Finishes the purge. These two surfaces were left out of the earlier commit
because the memory store is auto-generated, but both are read back into every
future session, so the name was still reaching the agent.
58 sites across 16 files: hostname, both LAN addresses, the SSID. Special-cased
the compound names a blind token swap breaks ('dank@the workstation/dankclaw',
'on the workstation/Dankclaw/Dankserver') and reworded the four that came out
ungrammatical. Facts kept: the hardware inventory of the box, the noctalia
architecture history, and today's own purge lessons all survive intact — only
the identifiers are gone.
Non-memory: the tracked .pre-pull-backup snapshot of a managed skill, whose
live counterpart was already clean.
description:Runbook for importing Dankware env-layer configs into the command-center repo and deploying them via stow on new fleet machines; decisions agreed with Frank 2026-09-09 (pre-execution — verify execution state before assuming done).
description:Runbook for importing client env-layer configs into the command-center repo and deploying them via stow on new fleet machines; decisions agreed with Frank 2026-09-09 (pre-execution — verify execution state before assuming done).
---
---
Design agreed with Frank for importing Dankware application/environment configs into the `~/.omp` command-center repo and deploying them on fresh machines. **Status: plan agreed, not yet executed** — check repo state (does `dotfiles/` or `scripts/deploy.sh` exist?) before re-deriving; if already implemented, this becomes the operating model, not a to-do.
Design agreed with Frank for importing client application/environment configs into the `~/.omp` command-center repo and deploying them on fresh machines. **Status: plan agreed, not yet executed** — check repo state (does `dotfiles/` or `scripts/deploy.sh` exist?) before re-deriving; if already implemented, this becomes the operating model, not a to-do.
## Agreed decisions (each was an explicit user choice)
## Agreed decisions (each was an explicit user choice)
1.**Target**: unknown/TBD fresh machine → design stays host-agnostic; deploy derives overrides from `$(hostname)` at run time. Nothing hardcodes a target.
1.**Target**: unknown/TBD fresh machine → design stays host-agnostic; deploy derives overrides from `$(hostname)` at run time. Nothing hardcodes a target.
3.**Layout**: stow-style package dirs `dotfiles/<pkg>/<home-relative path>` inside the repo; repo clones to `~/.omp` on every host that runs the harness (full command-center clone, not env-layer only).
3.**Layout**: stow-style package dirs `dotfiles/<pkg>/<home-relative path>` inside the repo; repo clones to `~/.omp` on every host that runs the harness (full command-center clone, not env-layer only).
4.**Deploy mechanism**: symlink farm — `stow -t ~` from `~/.omp/dotfiles`; relative symlinks within `$HOME` work because repo root lives inside `$HOME`.
4.**Deploy mechanism**: symlink farm — `stow -t ~` from `~/.omp/dotfiles`; relative symlinks within `$HOME` work because repo root lives inside `$HOME`.
5.**Dogfood order**: convert Dankware itself (real dirs → symlinks via adopt) BEFORE any target import; iterate where the compositor reload is observable.
5.**Dogfood order**: convert the workstation itself (real dirs → symlinks via adopt) BEFORE any target import; iterate where the compositor reload is observable.
6.**Secrets policy**: never commit secret values. Split rc files into committed logic + untracked sourced-if-exists stub (`~/.config/environment.d/secrets.env` or `.zshrc.local`); real values provisioned out-of-band only on machines that need them. Pre-commit secret scanner stays as safety net, not gate of first resort.
6.**Secrets policy**: never commit secret values. Split rc files into committed logic + untracked sourced-if-exists stub (`~/.config/environment.d/secrets.env` or `.zshrc.local`); real values provisioned out-of-band only on machines that need them. Pre-commit secret scanner stays as safety net, not gate of first resort.
7.**Divergence**: `zz-<hostname>` override package stowed LAST with `--override`; only per-host diffs live there. (Stow descends into conflicting symlinked dirs — per-file overrides only work via this last-stowed `--override` pattern.)
7.**Divergence**: `zz-<hostname>` override package stowed LAST with `--override`; only per-host diffs live there. (Stow descends into conflicting symlinked dirs — per-file overrides only work via this last-stowed `--override` pattern.)
8.**Deploy trigger**: explicit `scripts/deploy.sh` invocation, human- or agent-invoked. NO auto-deploy on pull — pull and deploy are separate decisions; `--check` = drift report mode.
8.**Deploy trigger**: explicit `scripts/deploy.sh` invocation, human- or agent-invoked. NO auto-deploy on pull — pull and deploy are separate decisions; `--check` = drift report mode.
@@ -22,7 +22,7 @@ Design agreed with Frank for importing Dankware application/environment configs
-`core.hooksPath` is a per-clone local setting — does NOT survive clone; bootstrap/deploy must set it on every fresh clone or the secret-gate pre-commit hook silently doesn't run there.
-`core.hooksPath` is a per-clone local setting — does NOT survive clone; bootstrap/deploy must set it on every fresh clone or the secret-gate pre-commit hook silently doesn't run there.
- dconf/gsettings are databases, not files: import as `dconf dump /` text into repo; deploy runs `dconf load` instead of symlinking.
- dconf/gsettings are databases, not files: import as `dconf dump /` text into repo; deploy runs `dconf load` instead of symlinking.
- p10k installer and toolchain hooks append to rc files at install time — expect conflicts on `.zshrc`-class files at first adopt; quarantine-and-replace handles it, but diff live-vs-repo before adopting so pre-existing drift is seen, not silently adopted.
- p10k installer and toolchain hooks append to rc files at install time — expect conflicts on `.zshrc`-class files at first adopt; quarantine-and-replace handles it, but diff live-vs-repo before adopting so pre-existing drift is seen, not silently adopted.
- Known secret at import time: `GITEA_TOKEN` export lived in `.zshrc` (line ~22) on Dankware — quarantine BEFORE importing rc files; consider token rotation at cutover since it had been exposed in plaintext rc + remote URL.
- Known secret at import time: `GITEA_TOKEN` export lived in `.zshrc` (line ~22) on the workstation — quarantine BEFORE importing rc files; consider token rotation at cutover since it had been exposed in plaintext rc + remote URL.
## Deploy script spec agreed
## Deploy script spec agreed
`scripts/deploy.sh`: sets hooksPath → stows all base packages → stows `zz-$(hostname)` with `--override` → first run uses `--adopt` for pre-existing identical files → divergent files quarantined-and-reported; `--check` mode emits drift report (modified tracked files vs deployed symlinks) without mutating.
`scripts/deploy.sh`: sets hooksPath → stows all base packages → stows `zz-$(hostname)` with `--override` → first run uses `--adopt` for pre-existing identical files → divergent files quarantined-and-reported; `--check` mode emits drift report (modified tracked files vs deployed symlinks) without mutating.
- Measuring WHICH card is the bottleneck in a llama.cpp layer-split LLM server on a fleet host (frank-medium, Dankserver 192.168.0.235, 2x R9700 asymmetric PCIe: ROCm0=x16 ~31.5GB/s, ROCm1=x1 ~1.97GB/s) — two traps that produced wrong numbers on 2026-09-18: (1) RUN THE TRANSIENT MEASUREMENT SERVER ON AN ISOLATED PORT, not the production port. If it takes over the prod port (7770), live fleet traffic from other agents hits it and interleaves with your timed request — I measured 35:30 at 2.51 t/s on :7770 vs the true 31.5 t/s; on isolated :7790 (prod stopped to free VRAM, fleet hits the down 7770) it read 31.52 t/s. Override LISTEN_PORT in a temp env copy (sed), point curl at the new port. (2) Use AVERAGE rocm-smi --showuse over a LONG request (2k tokens, ~60s), sampled every 0.5s — PEAK util lies for a layer split because both cards are touched every token and both hit ~100% at peak. Average is the bottleneck indicator. FINDING (durable, specific to frank-medium): the x1 card (ROCm1) is the STRUCTURAL bottleneck at 98% avg util in every config — 35:30/36:29, drafter on ROCm1/ROCm0 all gave util0(x16)=41-43% / util1(x1)=98%, decode ~31.5 t/s in all four. It's COMPUTE-bound, not PCIe-bound (only ~5.2 MB activation crosses the x1 link per 512-token ubatch). The per-step cost is the verify phase (both cards run the full target) + the LOGITS, which are pinned to the card holding the last layers; the MTP drafter is a small nextn head so moving it (SPEC_DRAFT_DEVICE=ROCm0) buys nothing. 35:30 is the optimal split (balanced VRAM 1.27/1.85 GiB free); 36:29 buys 2 util points on the x16 card and erodes its VRAM (fit free 0.45 GiB) for no throughput gain. No llama-server flag moves the lm_head off the last-layer card, so the x1 bottleneck is architectural, not a mis-tune. Recorded in scripts/frank-medium.env (committed 6c6464a).
- placeholder4
- placeholder3
- placeholder2
- placeholder
- frank-medium (Dankserver 192.168.0.235, llama.cpp layer-split 2x R9700 gfx1201) split-balance findings, 2026-09-17: (1) The LAYER SPLIT is NOT a decode-throughput lever at the 393216/MTP shape — 34:31, 35:30 and 36:29 all land in the same ~32 tok/s band (36:29 measured 31.76; 35:30 ~32.4 per env); total work, not imbalance, is the limit. (2) The x1 card (ROCm1, 0b:00.0, Gen4 x1 via B650 chipset ~1.97 GB/s) runs HOT (util ~91-97%) at 35:30 while the x16 card (ROCm0, 03:00.0, Gen4 x16 ~31.5 GB/s) idles (~45%) — because the MTP drafter + output layers + heavier compute buffers are STRUCTURALLY pinned to the card holding the LAST layers, NOT because the PCIe link is the bottleneck (only ~5.2 MB of boundary activation crosses per 512-token ubatch). (3) 36:29 makes the x16 card the utilized worker (util0=42%, util1=3%) at no throughput cost, but spends VRAM headroom (card0 free 0.45 GiB, under the 1.0 GiB floor; 2.22 GiB gap). Frank's standing preference: the card with the FASTEST PCIe lane (x16 = ROCm0) should be the more-utilized card. Deploying 36:29 requires re-fitting the seam test (tests/frank_medium_unit.bats) slack/floor, which currently rejects it.
- niri output config on blankpad (niri 26.04): transform values are COUNTER-CLOCKWISE, so "90" means the panel's top bezel must physically sit on the LEFT ("270" = top bezel right). Two traps when pinning outputs in ~/.config/niri/cfg/display.kdl (symlink into ~/.omp/dotfiles/zz-blankpad/): (1) an output "X" {} block RESETS every property you don't restate to its default — eDP-1 needed scale 1.25 repeated or it would silently drop to 1.0; (2) niri re-runs its auto-layout for any output without a pinned position, so pinning only the externals shoved eDP-1 to the far right (3936,0) — pin eDP-1 at x=0 too if you want the original left-to-right order. Apply with niri validate -c ~/.config/niri/config.kdl then niri msg action load-config-file; confirm with niri msg outputs (shows Logical size swapped to 1200x1920 and "Transform: 90° counter-clockwise"). Live one-off rotation without config: niri msg output NAME transform 90. _(context: Rotated both external PM1601 panels (DP-2, HDMI-A-1) to portrait on blankpad 2026-09-15; verified via niri msg outputs, committed 2baf0c0 to dank/command-center.)_
- niri output config on blankpad (niri 26.04): transform values are COUNTER-CLOCKWISE, so "90" means the panel's top bezel must physically sit on the LEFT ("270" = top bezel right). Two traps when pinning outputs in ~/.config/niri/cfg/display.kdl (symlink into ~/.omp/dotfiles/zz-blankpad/): (1) an output "X" {} block RESETS every property you don't restate to its default — eDP-1 needed scale 1.25 repeated or it would silently drop to 1.0; (2) niri re-runs its auto-layout for any output without a pinned position, so pinning only the externals shoved eDP-1 to the far right (3936,0) — pin eDP-1 at x=0 too if you want the original left-to-right order. Apply with niri validate -c ~/.config/niri/config.kdl then niri msg action load-config-file; confirm with niri msg outputs (shows Logical size swapped to 1200x1920 and "Transform: 90° counter-clockwise"). Live one-off rotation without config: niri msg output NAME transform 90. _(context: Rotated both external PM1601 panels (DP-2, HDMI-A-1) to portrait on blankpad 2026-09-15; verified via niri msg outputs, committed 2baf0c0 to dank/command-center.)_
- Frank's frank-large ceiling is now 200,000 context in ~/.omp/agent/models.yml (provider llamacpp (dankclaw), committed 60dc0bb to dank/command-center 2026-09-13), while halogen on 192.168.0.143:7770 still serves context=262144 — a deliberate under-serve, not drift. Convention when changing any contextWindow there: read the served value from curl baseUrl/health (it reports context, model, vision.enabled) and check KV-pool admission before/after — admission reserves prompt + max_tokens against the pool (401,408 positions since 2026-09-13), so contextWindow + maxTokens must stay under it or the request hangs the full 3600 s queue timeout before a 503. maxTokens stays pinned at 65536 because HALOGEN_MAX_TOKENS_CAP 400s anything larger. Also: OMP reads models.yml only at startup, so a running session keeps the old ceiling; tell the user new omp invocations pick up the change.
- Frank's frank-large ceiling is now 200,000 context in ~/.omp/agent/models.yml (provider llamacpp (dankclaw), committed 60dc0bb to dank/command-center 2026-09-13), while halogen on 192.168.0.143:7770 still serves context=262144 — a deliberate under-serve, not drift. Convention when changing any contextWindow there: read the served value from curl baseUrl/health (it reports context, model, vision.enabled) and check KV-pool admission before/after — admission reserves prompt + max_tokens against the pool (401,408 positions since 2026-09-13), so contextWindow + maxTokens must stay under it or the request hangs the full 3600 s queue timeout before a 503. maxTokens stays pinned at 65536 because HALOGEN_MAX_TOKENS_CAP 400s anything larger. Also: OMP reads models.yml only at startup, so a running session keeps the old ceiling; tell the user new omp invocations pick up the change.
- Frank's terminal font stack on Dankware: package ttf-terminus-nerd (Arch, group nerd-fonts), which provides the TERMINESS outline derivative — families "Terminess Nerd Font" / " Mono" / " Propo". The bitmap original terminus-font is NOT installed, so classic-Terminus crispness depends entirely on 1-bit hinted rendering. Crisp/bitmap look = freetype-load-flags = hinting,monochrome (monochrome = 1-bit, disables AA; the light flag stays in the resolved value but is inert once monochrome is on, per ghostty.5.md). Sizing rule that matters: pixel cell = pt × (96/72) × monitor_scale, and the crisp sweet spot is when that lands on Terminus' native bitmap heights (8x16, 9x18, 10x20, 11x22, 12x24). On Frank's eDP-1 (1920x1080 @ scale 1.25) font-size 12 → 20px → exactly the 10x20 grid; 14pt → 23.3px, fractional and blurry regardless of flags. Check scale with niri msg outputs. Also: ghostty +show-config prints only values differing from default, and Ghostty's default font-size IS 12 — so a 12pt setting vanishes from the diff output; confirm against +show-config --default before concluding the key was ignored.
- Frank's terminal font stack on the workstation: package ttf-terminus-nerd (Arch, group nerd-fonts), which provides the TERMINESS outline derivative — families "Terminess Nerd Font" / " Mono" / " Propo". The bitmap original terminus-font is NOT installed, so classic-Terminus crispness depends entirely on 1-bit hinted rendering. Crisp/bitmap look = freetype-load-flags = hinting,monochrome (monochrome = 1-bit, disables AA; the light flag stays in the resolved value but is inert once monochrome is on, per ghostty.5.md). Sizing rule that matters: pixel cell = pt × (96/72) × monitor_scale, and the crisp sweet spot is when that lands on Terminus' native bitmap heights (8x16, 9x18, 10x20, 11x22, 12x24). On Frank's eDP-1 (1920x1080 @ scale 1.25) font-size 12 → 20px → exactly the 10x20 grid; 14pt → 23.3px, fractional and blurry regardless of flags. Check scale with niri msg outputs. Also: ghostty +show-config prints only values differing from default, and Ghostty's default font-size IS 12 — so a 12pt setting vanishes from the diff output; confirm against +show-config --default before concluding the key was ignored.
- NEVER send SIGUSR1 to ghostty on Frank's workstations — it does not reload the config, it kills/respawns the terminal (observed 2026-09-13: PID churn, Frank said "stop killing the terminal"). The old recipe in agent/memories/--home-blank-.omp--/MEMORY.md and skills/dotfiles_git/scripts/reload_ghostty.sh ("pgrep ghostty | while read pid; do kill -USR1 $pid; done") is WRONG for ghostty 1.3.1. Correct way to apply a changed ghostty theme/config without losing shells: press the default keybind ctrl+shift+, (action reload_config) in each window — confirmed as the default in ghostty +show-config --default. For applying theme changes desktop-wide without touching terminals: noctalia msg templates-apply regenerates rendered theme files, and niri msg action load-config-file reloads the compositor (reload-config is NOT a valid niri subcommand). Custom noctalia palettes live at ~/.config/noctalia/palettes/Name.json and are selected with noctalia msg color-scheme-set custom "Name"; that source is immune to the community-palette catalog/md5 sync, unlike editing the cache in ~/.local/state/noctalia/community-palettes/. Also: noctalia msg screenshot-fullscreen saves to ~/Pictures (no need to scrape the PNG out of the clipboard with wl-paste).
- NEVER send SIGUSR1 to ghostty on Frank's workstations — it does not reload the config, it kills/respawns the terminal (observed 2026-09-13: PID churn, Frank said "stop killing the terminal"). The old recipe in agent/memories/--home-blank-.omp--/MEMORY.md and skills/dotfiles_git/scripts/reload_ghostty.sh ("pgrep ghostty | while read pid; do kill -USR1 $pid; done") is WRONG for ghostty 1.3.1. Correct way to apply a changed ghostty theme/config without losing shells: press the default keybind ctrl+shift+, (action reload_config) in each window — confirmed as the default in ghostty +show-config --default. For applying theme changes desktop-wide without touching terminals: noctalia msg templates-apply regenerates rendered theme files, and niri msg action load-config-file reloads the compositor (reload-config is NOT a valid niri subcommand). Custom noctalia palettes live at ~/.config/noctalia/palettes/Name.json and are selected with noctalia msg color-scheme-set custom "Name"; that source is immune to the community-palette catalog/md5 sync, unlike editing the cache in ~/.local/state/noctalia/community-palettes/. Also: noctalia msg screenshot-fullscreen saves to ~/Pictures (no need to scrape the PNG out of the clipboard with wl-paste).
- In OMP's ~/.omp/agent/models.yml there are NO per-model sampler keys. Sampling params (temperature, topP, topK, minP, presencePenalty, repetitionPenalty) are GLOBAL settings in config.yml, each defaulting to -1 = "do not send". The only per-model hook is compat.extraBody — a Recordstring, unknown of extra top-level fields merged into every request body. It is applied LAST in the openai-completions builder (applyOpenAIExtraBody runs after applyChatCompletionsCompatPolicy, via Object.assign), so extraBody wins over anything the compat policy or global settings set. Use wire names (top_p, top_k, min_p, presence_penalty, repetition_penalty), not OMP's camelCase setting names. Two traps that cost real debugging time: 1. preserve_thinking is NOT user-settable in models.yml. qwenPreserveThinking is auto-derived inside buildOpenAICompat from the AUTO-DETECTED thinkingFormat plus a local (loopback/RFC1918) baseUrl — and that derivation happens BEFORE applyCompatOverrides. So writing compat.thinkingFormat: qwen in models.yml does NOT turn on preserve_thinking. For a custom-named model (e.g. frank-large, which matches no Qwen id pattern) the only path is extraBody: { preserve_thinking: true }. 2. NEVER put chat_template_kwargs inside extraBody. Object.assign is shallow, so it replaces the whole chat_template_kwargs object and silently drops the reasoning_effort kwarg the compat policy already put there. Top-level preserve_thinking is honoured by halogen-flash-server (its own 400 message says the field "may also be sent as top-level fields"), so top-level is both sufficient and safe. Why it mattered: halogen-flash-server has no server-side sampler defaults — temperature absent or 0 is greedy decode. OMP sent nothing (global -1), so the endpoint was decoding greedy while appearing configured. _(context: Verified 2026-09-13 on the Dank fleet while pinning Qwen3.8-Flash-Next recommended sampling (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0) plus preserve_thinking onto frank-large. Ground truth read from the installed bundle: pi-ai/src/providers/openai-completions.ts (applyOpenAIExtraBody applied last), pi-ai/src/providers/openai-shared.ts (applyChatC)_
- In OMP's ~/.omp/agent/models.yml there are NO per-model sampler keys. Sampling params (temperature, topP, topK, minP, presencePenalty, repetitionPenalty) are GLOBAL settings in config.yml, each defaulting to -1 = "do not send". The only per-model hook is compat.extraBody — a Recordstring, unknown of extra top-level fields merged into every request body. It is applied LAST in the openai-completions builder (applyOpenAIExtraBody runs after applyChatCompletionsCompatPolicy, via Object.assign), so extraBody wins over anything the compat policy or global settings set. Use wire names (top_p, top_k, min_p, presence_penalty, repetition_penalty), not OMP's camelCase setting names. Two traps that cost real debugging time: 1. preserve_thinking is NOT user-settable in models.yml. qwenPreserveThinking is auto-derived inside buildOpenAICompat from the AUTO-DETECTED thinkingFormat plus a local (loopback/RFC1918) baseUrl — and that derivation happens BEFORE applyCompatOverrides. So writing compat.thinkingFormat: qwen in models.yml does NOT turn on preserve_thinking. For a custom-named model (e.g. frank-large, which matches no Qwen id pattern) the only path is extraBody: { preserve_thinking: true }. 2. NEVER put chat_template_kwargs inside extraBody. Object.assign is shallow, so it replaces the whole chat_template_kwargs object and silently drops the reasoning_effort kwarg the compat policy already put there. Top-level preserve_thinking is honoured by halogen-flash-server (its own 400 message says the field "may also be sent as top-level fields"), so top-level is both sufficient and safe. Why it mattered: halogen-flash-server has no server-side sampler defaults — temperature absent or 0 is greedy decode. OMP sent nothing (global -1), so the endpoint was decoding greedy while appearing configured. _(context: Verified 2026-09-13 on the Dank fleet while pinning Qwen3.8-Flash-Next recommended sampling (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0) plus preserve_thinking onto frank-large. Ground truth read from the installed bundle: pi-ai/src/providers/openai-completions.ts (applyOpenAIExtraBody applied last), pi-ai/src/providers/openai-shared.ts (applyChatC)_
- Frank's Gitea lives on Dankclaw at http://192.168.0.143:3000, single org/user dank. Repo inventory (verified 2026-09-12 via /api/v1/user/repos): agent (default master), command-center (main — this is the ~/.omp dotfiles/KB repo), frank-llm-bench (main), serverAppConfigs (main), vllmScripts (main). LESSON — never clone from a repo name Frank states verbally; resolve the exact full_name via the Gitea API first. He said "serverAppConfig"; the real repo is serverAppConfigs (plural). Cloning the guessed name would 404 with no useful signal. Working procedure: read the dank HTTP token from the existing origin URL in ~/.omp (git remote -v) — do NOT copy that token into memory, docs, or skills; reference its location only. Then GET /api/v1/user/repos and match on the real name/case/plurality before cloning. Local-vs-remote naming: the clone was placed at ~/Projects/serverAppConfig (singular, as Frank explicitly requested) while origin points at the plural repo. The path mismatch is intentional, not drift — pull/push work normally. Repo layout: dankclaw/, dankserver/, frankbook-pro/ (7 quadlet units: sonarr, radarr, lidarr, readarr, prowlarr, qbittorrent, gluetun), shared/ (homepage, alloy, loki, prometheus, searxng), tests/ (47-file E2E suite), serverAppConfigs-docs/ (115-file KB — read its index.md before touching the stack). _(context: Frank asked to "pull my serverAppConfig repository into ~/Projects/serverAppConfig"; Gitea API revealed the plural name. Credential handling: token reused from existing ~/.omp remote, deliberately not persisted here.)_
- Frank's Gitea lives on Dankclaw at http://192.168.0.143:3000, single org/user dank. Repo inventory (verified 2026-09-12 via /api/v1/user/repos): agent (default master), command-center (main — this is the ~/.omp dotfiles/KB repo), frank-llm-bench (main), serverAppConfigs (main), vllmScripts (main). LESSON — never clone from a repo name Frank states verbally; resolve the exact full_name via the Gitea API first. He said "serverAppConfig"; the real repo is serverAppConfigs (plural). Cloning the guessed name would 404 with no useful signal. Working procedure: read the dank HTTP token from the existing origin URL in ~/.omp (git remote -v) — do NOT copy that token into memory, docs, or skills; reference its location only. Then GET /api/v1/user/repos and match on the real name/case/plurality before cloning. Local-vs-remote naming: the clone was placed at ~/Projects/serverAppConfig (singular, as Frank explicitly requested) while origin points at the plural repo. The path mismatch is intentional, not drift — pull/push work normally. Repo layout: dankclaw/, dankserver/, frankbook-pro/ (7 quadlet units: sonarr, radarr, lidarr, readarr, prowlarr, qbittorrent, gluetun), shared/ (homepage, alloy, loki, prometheus, searxng), tests/ (47-file E2E suite), serverAppConfigs-docs/ (115-file KB — read its index.md before touching the stack). _(context: Frank asked to "pull my serverAppConfig repository into ~/Projects/serverAppConfig"; Gitea API revealed the plural name. Credential handling: token reused from existing ~/.omp remote, deliberately not persisted here.)_
- Ghostty 1.3.1 (and the 1.3.x line) has NO font-hinting and NO font-subpixel-render config key. The only font-smoothing key is freetype-load-flags, a comma-separated list: hinting, force-autohint, monochrome, autohint, light (each default-enabled except monochrome/force-autohint; prefix no- to disable). Ground truth: /usr/share/ghostty/doc/ghostty.5.md:598-628. To actually DISABLE anti-aliasing you need freetype-load-flags = hinting,monochrome (docs line 617: monochrome = 1-bit rendering, disables AA). Note hinting,light is a NO-OP — both are already Ghostty defaults. Verification workflow before editing any ghostty key: ghostty +validate-config (exit 0 = valid) plus grep the manpage for the key name; an unknown key silently does nothing useful and +validate-config is the authoritative gate. This matters because a poisoned memory claim about font-subpixel-render/font-hinting caused 3 wasted commits (4c84f02, df33310, 268dcbd) editing a nonexistent key on 2026-09-11. _(context: Verified 2026-09-12 against ghostty 1.3.1-arch2.2 manpage and ghostty +validate-config on Dankware. Corrects contradictory entries in agent/memories/--home-blank-.omp--/learned.md (line 2 asserts font-subpixel-render is the key; it is not).)_
- Ghostty 1.3.1 (and the 1.3.x line) has NO font-hinting and NO font-subpixel-render config key. The only font-smoothing key is freetype-load-flags, a comma-separated list: hinting, force-autohint, monochrome, autohint, light (each default-enabled except monochrome/force-autohint; prefix no- to disable). Ground truth: /usr/share/ghostty/doc/ghostty.5.md:598-628. To actually DISABLE anti-aliasing you need freetype-load-flags = hinting,monochrome (docs line 617: monochrome = 1-bit rendering, disables AA). Note hinting,light is a NO-OP — both are already Ghostty defaults. Verification workflow before editing any ghostty key: ghostty +validate-config (exit 0 = valid) plus grep the manpage for the key name; an unknown key silently does nothing useful and +validate-config is the authoritative gate. This matters because a poisoned memory claim about font-subpixel-render/font-hinting caused 3 wasted commits (4c84f02, df33310, 268dcbd) editing a nonexistent key on 2026-09-11. _(context: Verified 2026-09-12 against ghostty 1.3.1-arch2.2 manpage and ghostty +validate-config on the workstation. Corrects contradictory entries in agent/memories/--home-blank-.omp--/learned.md (line 2 asserts font-subpixel-render is the key; it is not).)_
- Ghostty's font antialiasing is controlled by font-hinting = no-hinting in the config file, not font-subpixel-render = off as the Noctalia reference skill states. When editing ghostty config, check /usr/share/ghostty/doc/ghostty.5.md or src/config/Config.zig — ghostty's --help confirms this. Invalid field causes font-subpixel-render: unknown field on restart. _(context: Discovered when user reported "font-subpixel-render: unknown field" after my initial edit to ghostty config)_
- Ghostty's font antialiasing is controlled by font-hinting = no-hinting in the config file, not font-subpixel-render = off as the Noctalia reference skill states. When editing ghostty config, check /usr/share/ghostty/doc/ghostty.5.md or src/config/Config.zig — ghostty's --help confirms this. Invalid field causes font-subpixel-render: unknown field on restart. _(context: Discovered when user reported "font-subpixel-render: unknown field" after my initial edit to ghostty config)_
- Noctalia is a Niri-based desktop environment at /usr/bin/noctalia. Config: ~/.config/noctalia/config.toml. Provides IPC via 'noctalia msg', launcher, lock screen, session menu, and 19 app templates (GTK, Ghostty, Alacritty, Hyprland, Sway, WezTerm, Helix, Btop, Starship, Qt, KDE, Cava, scroll, mango, umbriel, foot, emacs, kitty, alacritty). Templates under /usr/share/noctalia/assets/templates/. Color palette generator: 'noctalia theme [image]'. Text antialiasing: Ghostty uses 'font-subpixel-render' config key (set to 'off' to disable). Alacritty uses 'font.antialias = false' in alacritty.toml.
- Noctalia is a Niri-based desktop environment at /usr/bin/noctalia. Config: ~/.config/noctalia/config.toml. Provides IPC via 'noctalia msg', launcher, lock screen, session menu, and 19 app templates (GTK, Ghostty, Alacritty, Hyprland, Sway, WezTerm, Helix, Btop, Starship, Qt, KDE, Cava, scroll, mango, umbriel, foot, emacs, kitty, alacritty). Templates under /usr/share/noctalia/assets/templates/. Color palette generator: 'noctalia theme [image]'. Text antialiasing: Ghostty uses 'font-subpixel-render' config key (set to 'off' to disable). Alacritty uses 'font.antialias = false' in alacritty.toml.
- **dankserver (192.168.0.235):** ASUS B650-E with 2x AMD R9700 GPUs (gfx1201). Ubuntu 24.04. RCCL atomics require M.2 riser to move card #2 to CPU-attached M.2_2 slot.
- **RCCL Constraint**: Chipset-attached PCIe lanes fail RCCL atomics (broadcast/all-reduce/send-recv). Fix: Move card #2 to CPU-attached M.2_2 slot via PCIe x4 riser to restore atomic-op routing.
- **the workstation / blankpad:** Ryzen 5 PRO 5650U, RTX 4070 Max-Q. Passwordless SSH to fleet.
## LLM Model Portfolio
## LLM Model Fleet
- **frank-large (halogen)**: Qwen3.8-27B-ROCmFPX-Q8_0. Runs on Strix Halo (gfx1151).
- **frank-medium (dankserver :7770):**
-*KV Pool*: 524288 positions (~14.4 GiB). Admission is pool-bound, not slot-bound (caps at 3 simultaneous full-length requests).
-Model: Qwen3.8-27B (MXFP6 / Q8_K_XL).
-*Behavior*: Defaults to greedy. Stalls (full-CPU, no disk activity, no output) under co-tenancy with pinned trunk are documented behavior, not crashes.
-Architecture: Layer-split (35:30) across 2x R9700.
-*Constraints*: maxTokens <= 65536. First request after restart may be silent ~45-60s while first-touching KV pool.
-Drafter: DFlash2 (Q8_0).
- **frank-small**: Qwen3.8-27B GSQ-RCO IQ3_S. Runs on RTX 3090 (M.2 dock). Defaults to sampling (temp 1.0, top_k 20, top_p 0.95).
- **Podman**: Rootless deployment using Quadlet units for boot persistence. Note: `restart: unless-stopped` survives crashes, but not explicit stops (kernel power-off).
- **Architecture:** Quadlet-based podman units (boot-persistent).
-frank-large (halogen) healthcheck failure signature: the unit's healthcheck is halogen-healthcheck api with a 10 s timeout. During an engine PING-silence stall (log: "halogen: the engine has not answered PING for 45s"), /health HANGS rather than 503ing, so the healthcheck exits 125 (timeout) and podman shows the container (unhealthy) with a small FailingStreak. This is the documented transient stall — it self-recovers (streak resets to 0, /health 200) within ~1-2 min; do not restart over it. A completion request landing in the stall window takes ~60 s and then returns 200. A leftover failed transient *-132a6f9850c6fbd8.service (systemd-run podman healthcheck) may linger in the user manager — reset-failed it. _(context: 2026-09-17 11:28-11:30 PDT: frank-large 4x FailingStreak exit 125, self-recovered by 11:31; 65 s completion probe straddled the tail.)_
-Two lessons from a bulk identity purge (2026-09-28, a sold workstation across serverAppConfigs + dank/command-center): 1) NEVER delegate mechanical prose/file rewrites to a subagent without a full diff audit. Observed failure modes in one run: list markers (- , * ) stripped so bullets stop rendering, numbered lists left with duplicate 3. entries, paths "corrected" to directories that do not exist (system/workstation/), and whole sentences deleted rather than reworded. Subagent edits are cheap to revert (git checkout the touched paths, excluding any files that carried pre-existing WIP) and expensive to trust. Reliable substitute: an exact-string, expected-count-asserted substitution script over a file list (abort on any count mismatch, write nothing for the failing entry), then print every + line and read it. 2) Classification beats blanket search when purging an identity from a config+docs tree. Three buckets: (a) narrative mentions - reword to a class ("the workstation"); (b) LOAD-BEARING mentions - keep, e.g. install.sh resolved packages/$HOST.pkgs from $(hostname), so the manifest FILENAME was the lookup key and the --host case list had to keep the hostname; (c) evidence - keep but de-identify (syslog excerpts keep timestamps and lines, only the host field is generalized). Hardware model strings stayed: subject matter, not identity. Frank's preferences: distill per-host doc directories into general pages rather than renaming them into a graveyard, and exclude auto-learned memory (agent/memories/**) from purges. Also worth knowing: OMP's user persona is ~/.omp/agent/APPEND_SYSTEM.md (not config.yml — that only holds personality: default); ~/.omp IS the live config checkout for the dank/command-center repo, so a branch switch swaps live config under the running agent, and a small live-config fix is best committed on the current branch with only its own file staged. _(context: 2026-09-28: Frank sold the 'the workstation' workstation. Purge run across two repos (serverAppConfigs, dank/command-center at ~/.omp). Trigger for the lesson: I delegated a 17-file mechanical text scrub to a subagent and waited on it; it returned files with dropped list markers, a duplicated numbered item, an invented system/workstation/ path, and a deleted fact sentence. Reverted and redid it as a coun)_
-Fleet transport: blankpad has NO passwordless SSH to FrankBook-Pro — dank@192.168.0.112 rejects publickey (the 2026-09-12 key install only covered dankclaw/dankserver). Do not burn calls probing Mac SSH. Verify the *arr stack from the LAN instead: port probes at 192.168.0.112:{9696 prowlarr, 8989 sonarr, 7878 radarr, 8686 lidarr, 8787 readarr, 8080 qbittorrent} — 302 (login redirect) and 200 are both "up". Container-level inspection on the Mac needs a key installed on the Mac or a Mac-side session. _(context: 2026-09-17 fleet-wide container audit; ssh dank@192.168.0.112 → Permission denied (publickey), six-port LAN probe all answered.)_
-DeepSeek-V4.1-Flash vs the box it must run on — durable sizing facts (verified 2026-09-28 from the HF API, not model cards): PAYLOAD. deepseek-ai/DeepSeek-V4.1-Flash = **475.27 GiB** across 48 safetensors shards. Of that, **189.13 GiB is Engram** — exactly two 94.56 GiB shards holding layers.{1,14}.engram.embed.weight (196B params, 4-gram lookup, 384,006,168 / 384,016,682 rows). Routed-expert params = 40 layers x 384 experts x 3 x 5120 x 2304 = **543.6 B**, shipped as expert_dtype: fp4 = 269 GiB. Dense ~9.5 GiB. Lucebox's ROCMFP23.gguf = 377.51 GiB (179 GiB experts at 2.5/3.5 bpw + 189 GiB Engram + dense). **The Engram tables are never mapped** (2 table(s) embedded in the GGUF (not mapped)), so they do not evict the expert page cache. THE DECODE WALL ON A 63.7 GiB / 2x-R9700 BOX. A token selects 6x40 = 240 routed experts at ~11.96 MiB each. At the residency a hot stack can hold (37 of 384 per layer = 9.6%) that is **~2.4 GB of expert weights crossing PCIe per token**; PCIe 5.0 x8 does ~55 GB/s real, giving a **~23 tok/s absolute ceiling** — below a 35 tok/s goal before any attention/Engram/dense work. Resident fraction is the whole game. DS41.md's own reference host had 61 GB RAM; ours has 246 GiB, which is the only untested lever. CONTEXT — the three 2026-09-28 candidates: DeepSeek-V4.1-Flash 552B/16B-active (40 layers, 384 top-6, CED 20enc+20dec, CSA2, FP4 KV 890 B/tok, 1M ctx); MiMo-V2.6-Flash-MOPD 309B/15B (48 layers, 39 SWA/9 GA, 256 top-8, **no shared expert**, 5-layer MTP); GLM-5.3-Flash 320B/18B (KDA+DSA hybrid, mHC). Of these only V4.1 is 2.5x too big for this box; MiMo and GLM are smaller but have **no AMD engine at all** (vLLM/SGLang are NVIDIA-first; radiance-vLLM is welded to qwen4_exp). AFFINITY (Codeberg StillDeadcode) PORT SCOPE, from the real safetensors indices of both checkpoints. Its config parser is data-driven and its **expert machinery ports unchanged** — expert tensor naming is IDENTICAL (layers.N.ffn.experts.E.w{1,2,3}.{weight,scale}; 15,360 _(context: Frontier-MoE feasibility study on dankserver 2026-09-28: sizing arithmetic for DeepSeek-V4.1-Flash / MiMo-V2.6-Flash / GLM-5.3-Flash, the Affinity-to-V4.1 port scope from real checkpoint indices, and three stale knowledge-base claims corrected.)_
-Long-context speculative-decode tuning on llama.cpp: at 250k-token contexts, INCREASING draft depth (n-max 4) and busy-wait (poll 100) made decode WORSE (230–257 t/s) than depth 3 / poll 50 (257–302 t/s) — each longer verification round re-reads the full-context attention of the full-attention layers every round, which outweighs the extra accepted token. The productive prefill lever on a 24 GiB 3090 was ubatch 512-1024 (+5% pp at 254k, better tg too); ubatch 2048 is impossible because a spec draft context allocates its OWN pp compute buffer (~3.3 GiB at ub 2048 for a 12B model) against only ~2.2 GiB headroom AFTER the main context fits — the OOM signature is common_speculative_init_result: failed to create MTP context / graph_reserve: failed to allocate compute pp buffers AFTER the main model loaded fine. Also: idle VRAM is identical across logical batch sizes (4096 vs 8192) at the same ubatch — compute buffers reserve at max graph size, so a bigger logical -b is free. Verify with salted API probes (cache_prompt:false + uuid prefix) using the timings keys of modern builds: prompt_per_second / predicted_per_second / draft_n / draft_n_accepted. (frank-small, dankclaw 3090, 2026-09-17, deployed as 8192/1024, n-max 3, poll 50) _(context: Optimizing frank-small (Gemma4-12B QAT + MTP sidecar, 4x262144 slots, q8 KV, RTX 3090) for max prefill+decode at max context.)_
-Retiring a host from Frank's fleet means a name purge across four surfaces, and the doctrine is "generalize, never delete the lesson": (1) serverAppConfigs/ repo — grep -rniI -c name . (also sweep the machine's LAN IP, NM connection names and wifi SSID, which carry identity even after the hostname goes), retitle per-host doc sections to the CLASS of thing they describe ("## the workstation host DNS" → "## Multi-homed client DNS"), keep the operational content, and update any in-page [[#Anchor]] references the rename breaks; verify with ./tests/run.sh (346 bats seams incl. docs link integrity). (2) ~/.omp/AGENTS.md machine-facts table. (3) ~/.omp/agent/APPEND_SYSTEM.md — this IS the agent persona; the "### Your identity" paragraph names the fleet and the host, and new sessions only pick up edits. (4) ~/.omp/agent/managed-skills/*/SKILL.md and agent/memories/*/. A purge log entry in the work log must name the box neither, or it reprints what was just removed. Leave git history, browser history, and OMP logs/http-400-requests/* (past session payloads) alone unless explicitly asked; a history rewrite needs force-push + re-clones on both servers. TWO operational gotchas: ~/.omp is a LIVE deployment checkout (branch switches swap config under the running agent), so isolate a config fix by staging one path, not by branch-switching — switching back reverts the file you just fixed. And a dirty tree full of peer/agent WIP means git add path, never git add -A. _(context: 2026-09-28, Frank reported the dev workstation (the workstation) sold and asked for a full purge, then "fix the agent persona" after I flagged it. Persona location and the live-checkout hazard were the non-obvious finds; the rename-don't-delete rule is the reusable part for future host retirements.)_
-In the serverAppConfig checkout, concurrent peer sessions can stash YOUR uncommitted WIP and then DROP that stash (a peer did exactly this 2026-09-17 while deploying frank-small, dropping a stash that held the frank-medium env+seam changes). Dropped stash commits are dangling but reachable: git fsck --no-reflogs lists them, and the files recover cleanly with git checkout dangling-sha -- paths (restores content, no ref touched). Check git stash list AND git fsck for your files before re-doing work after a peer session has been active. Related standing rule for this repo: the deploy gate rejects a dirty tree, so a peer's uncommitted WIP blocks your deploy — stash their files (theirs, not yours), deploy, pop._(context: serverAppConfig 2026-09-17: peer's frank-small deploy stashed then dropped my frank-medium WIP; recovered via dangling stash commit cd0a554.)_
-Never carry line numbers across hashline edit calls: re-read the target range for a fresh tag before every subsequent edit, and never compute offsets from a botched apply's result. Multi-op edits resolve all PUT/CUT indices against the ORIGINAL numbering, and a post-failure diff view elides lines — so a second call built on those numbers duplicates text instead of replacing it. Same rule for the imperative edit tool: assert the whole replaced region in the anchor, re-read to confirm, and repair any duplicated lines immediately. (Context: a hostname purge on serverAppConfigs 2026-09-28 — a two-line edit left a 3-line remnant, and the follow-up PUT 29.=35/60.=63 then pasted 7 fresh lines beside 3 stale ones, duplicating the Proton-gate and deploy-flow paragraphs.)_(context: 2026-09-28 serverAppConfigs doc purge (5 files) — reproduced a line-number-drift failure in the hashline edit tool and repaired it.)_
-llama.cpp draft-mtp spec-decode cost law (calibrated 2026-09-17 on frank-medium, 2x R9700 layer split): the draft context shares the main model's weights (shares_model — no sidecar file, no second model copy), so the cost is one nextn layer's KV at 4096 B/position × the TARGET's ctx (the draft context is sized from the target, same law as any drafter) PLUS draft buffers that a formula-based fit model underestimates badly: at 393216 ctx the MTP total charge on the last-layer card measured 4.20 GiB vs the fit's ~1.7 GiB — 2.46 GiB hot, which moved the card's headroom from a predicted 3.54 to a real 1.08 GiB and forced the floor down to 1.0 GiB. Calibrate the fit constant to the measured rocm-smi load, not the formula. The MTP/DFlash2 trade on this host: MTP frees ~1.1 GiB on the drafter card (no 1.92 GiB sidecar + buffers) at the cost of ~11 tok/s (32.4 vs 43.5 measured) — that freed space is what made 2x196k contexts fit. _(context: frank-medium 2026-09-17: switched DFlash2 - MTP drafter to fit a 393216 pool (2x196608); ROCm1 loaded 30.78 GiB vs fit 28.32; seam constant recalibrated to 2,901,231,913 B.)_
-R9V radiance-vLLM (dankserver frank-primary): PLE mmap→HIP registration (54.4 GB) can exceed the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured 740 s, right after a bench teardown). Signature: worker log "TimeoutError: PLE offload worker did not become ready" ~16 min into load, then the PLE worker finishes registration AFTER the engine already tore down (BrokenPipeError on the late worker). WORST part: serve-frank-primary.sh exits 0 on docker wait, so systemd logs "Deactivated successfully" with NO error in the journal — a silent prod failure that reads like a clean stop. Fixed in production 2026-09-28: R9V_PLE_READY_TIMEOUT=1800 in scripts/frank-primary.env, forwarded by fork launch.sh 2de0357 (local-only). Any future R9V engine that waits on a registration/registration-like step: check for a hidden readiness timeout env in vendor/vllm/vllm/envs.py before debugging "clean" deactivations. _(context: frank-primary 2026-09-28: bench matrix E1/E3 init failures + a failed 09:46 production boot all traced to the 600 s PLE readiness timeout)_
-llama.cpp --fit with an assistant-style MTP sidecar (Gemma4: mtp-gemma-4-12B-it.gguf, arch gemma4-assistant) prints at boot llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set + failed to measure the memory of the extra model, fitting without it — this is the fit probe, self-described normal in the log, and the draft loads against the real context seconds later. Do NOT restart over it. Also: Gemma4 SWA + q8_0 KV made --fit take the full 262144 window in ~12 GiB of a 24 GiB RTX 3090 (idle 11,996 MiB with 7.4 GiB of Q4_K_M payloads), and Gemma4's chat template accepts every reasoning_effort level (all seven probed 2026-09-17: 200), unlike Qwen3.8's xhigh/medium/low gate. q8 KV on CUDA required rebuilding build-cuda with -DGGML_CUDA_FA_ALL_QUANTS=ON. Deployed as frank-small on dankclaw :7771 with --spec-type draft-mtp,ngram-mod (acceptance 0.38-0.46, mean len ~2.2). _(context: frank-small Gemma4-12B QAT cutover on dankclaw 2026-09-17 (commits 4b7de59, 87caca4; runbook serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md))_
-R9V radiance-vllm-qwen4exp (dankserver frank-primary, Q8_0) — verified 2026-09-28: (1) PLE registration (54.4 GB mmap - HIP) can exceed the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT (default, vendor/vllm/vllm/envs.py) under page-cache churn — measured 740 s right after a bench teardown. Signature: container dies ~19 min in, unit logs "Deactivated successfully" with ZERO error in journal (serve-frank-primary.sh exits 0 on docker wait), endpoint just never serves. Prod fix: R9V_PLE_READY_TIMEOUT=1800 in scripts/frank-primary.env, forwarded by launch.sh fork commit 2de0357 (local-only). (2) Bench matrix laws for the q8 shape: prefill chunk 4096 is the ceiling (16384 boots but 0.93 GiB VRAM free - requests hang/OOM; 8192 works: PP8K +25.9% but decode -8.5%); expert cache 80 slots is the POLICY max (doctor: "cache prepare backend requires capacities (48, 64, 80) with synchronous single-fill LRU" — async unsupported, 96/112 also fail expert-budget [108,198] rank maxima); MTP depth 6 is 3.7x SLOWER than depth 4 (454-489 vs 128-159 ms/step, tokens/step barely grows); unqualified shapes fail MID-REQUEST (health stays 200, stream dies). (3) Prefix caching (R9V_ENABLE_PREFIX_CACHING=1): zero VRAM cost (hits reuse freed KV blocks), cold path unchanged, reuse hits 32k 44.2-3.6-4.3 s (9x), 131k 439-6.9 s (63x); hit pattern rounds 1-2 MISS, round 3+ hit (identical-prompt and conversation-growth). Recommended for prod, pending Frank's go-ahead (q8 state opts out with =0; launcher default is ON). (4) deploy.sh refuses to run from non-main branches ("deploy.sh must run from the main branch") — ff main to the working branch and push before deploying. _(context: frank-primary bench matrix + 9h downtime incident + PLE timeout boot failure, dankserver 2026-09-28; repo commit f187165, runbook serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md "Prefill / decode bench matrix" section)_
-User preference (Frank, 2026-09-17): never do multi-step arithmetic by hand in prose — "stop doing math without python, you're an LLM. your handling numbers is like expecting determinism out of a pachinko board." All VRAM fits, per-token costs, gap/slack computations and the like must go through the eval tool (Python) and quote the computed numbers. Hand arithmetic is acceptable only for trivial single-step checks. _(context: frank-medium VRAM fit for 36:29 @ 327680 — I had computed card balances in-head before the user corrected)_
-frank-primary Q8_0 cutover (2026-09-27, commits e5a06f5-era trio + docs): (1) The R9V launcher now carries an opt-in chat-template passthrough — R9V_CHAT_TEMPLATE_FILE mounts file:/opt/r9v/share/chat-template.jinja:ro and passes --chat-template (wins over the tokenizer dir's metadata/chat_template.jinja); R9V_CHAT_TEMPLATE_KWARGS maps to --default-chat-template-kwargs. Value with braces in the env file gets the single-quote law (bare double quotes are stripped by bash assignment parsing). (2) Behavioral template test: request reasoning_effort "high" — the embedded template 500s ("Unexpected reasoning effort"), the sharp template folds high→xhigh → 200. (3) Probe gotcha: bare docker on dankserver is NOT the shim (PATH-dependent) — docker logs/inspect silently return nothing for podman-rootless containers; use podman logs (as dank) or export PATH with .podman-shim first. A green health probe plus "no such object" from docker inspect is the shim-missing signature, NOT the false-green stray law. (4) vLLM MTP acceptance lives in APIServer logs: SpecDecoding metrics: Mean acceptance length: N (metrics.py:120, ~every 10 s); measured 2.88–4.50 live. (5) systemd unmask returned success but LEFT the runtime mask symlink (/run/systemd/system/unit - /dev/null) in place on the relinked unit — verify with ls -la /run/systemd/system/unit after unmask, remove by mv if still present. (6) vLLM engine logs do NOT reach docker/podman logs when the launcher runs the engine detached inside the container; APIServer lines do — grep those for spec metrics. _(context: frank-primary q8 cutover on dankserver 2026-09-27; launcher commit 30d4a32 (fork local), repo commit + docs pushed)_
-llama.cpp mainline (fb27a525, the frank-medium build) does NOT autoload mmproj*.gguf from the model directory: a server started without an explicit --mmproj flag comes up healthy and text-only, answers image requests with 500 "image input is not supported - hint: if this is unexpected, you may need to provide the mmproj", and the load log contains no mmproj line at all — so /health and the journal both look fine. Always pass --mmproj file explicitly (the frank-small pattern), and verify vision behaviorally (send an actual image) after any deploy that touches the projector, because health checks cannot see it. Cost of the trap: one full extra deploy cycle on 2026-09-17 (frank-medium 36:29/327680 cutover). _(context: frank-medium 2026-09-17: NO_MMPROJ=0 + mmproj-BF16.gguf beside the model, first deploy healthy, vision 500'd; fixed with explicit --mmproj in serve-frank-medium.sh)_
-Why Q8_0 decoded ~11 tok/s vs uncensored IQ4_XS 59–96 tok/s on radiance-vllm dual-R9700 (verified 2026-09-27, not guessed): decode on this runtime is expert-bandwidth-bound; every token activates the same top-8×48-block expert set, so per-token expert BYTES decide speed. (1) Q8_0 doubles bytes/expert (1.0625 vs ~0.5625 B/param; 92.39→175.30 GiB payload). (2) Frozen-config budgets then HALVE, because the tiered residency budget is bytes-driven over fixed ~32 GiB cards: R9V_MAX_EFFECTIVE_EXPERTS_PER_RANK 222,428→108,198, R9V_TIERED_EXPERT_CACHE_SLOTS 160→80, R9V_MIN_FREE_VRAM_GIB_BY_RANK 1.5→3.0, PLE table 28.8 GB (residency ssd, 16 GiB pinned reserve)→54.4 GB (residency pinned). (3) Live uncensored residency that buys 59–96 tok/s: rank0 hot=4.36/cold_UVA=36.03/dynamic_cache=10.91 GiB/155 slots; rank1 hot=16.18/cold_UVA=4.24 GiB (~31 GiB expert working set in VRAM). Q8_0 cannot afford that shape. (4) The 2× bytes became 5–9× slowdown via the residency cliff: below the hot-set size the LRU hit rate collapses non-linearly and misses stream from pinned host RAM over PCIe (same law as the x4-pair experiment: residency fraction is the whole game). (5) Probe caveat: the 11.0 tok/s figure was ~the 3rd request of the container's life (80-slot LRU cold, reasoning tokens + TTFT included, single 256-tok shot) — a floor, not a steady-state number; measuring true steady state needs a bench window: stop prod, relaunch from /home/dank/r9v-state/q8xl/START-COMMAND.txt, warm the cache, run the cutover battery harness. Diagnostic template for any future quant comparison on this fork: diff the two state-dirs' setup.json config blobs (the R9V_* keys above are the ones that matter) and grep 'Tiered GGUF experts ready' from each runtime's logs for hot/cold_UVA/dynamic_cache per rank. Evidence: fork docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md, vault log.md 928fd81. _(context: Frank asked "why is decode so much slower?" after the Q8_0 qualification window on dankserver (2026-09-27). Answered from frozen-config diffs (r9v-state/q8xl vs uncensored setup.json) + live 'Tiered GGUF experts ready' log lines; no bench run needed to establish the mechanism.)_
-Measuring batch geometry and VRAM balance on a ROCm llama.cpp host — three durable rules (Dankserver 2026-09-16, frank-medium, 2x R9700, scripts/bench-batch-ubatch.sh): 1. SETTLE GATE, or your infeasibility verdict is an artifact. Loading a config, reading rocm-smi, killing it and immediately loading the next one fails with error loading model: unable to allocate ROCm0 buffer — amdgpu does not return ~26 GiB the moment the process dies, and a 180 s settle is NOT enough (600 s was still not enough in one run; the VRAM came back after the transient unit's cgroup was torn down). That failure is indistinguishable from "this configuration does not fit". Gate each load on rocm-smi showing both cards near-empty (awk over --showmeminfo vram --csv, wait until used 1 GiB per card), and run each config in its own transient unit so systemd's cgroup teardown does the killing. 2. A "plateau" claim must be tested from BOTH sides. Sampling only 512→2048 showed a shallow basin (1135-1170 tok/s) that read as "the knob does not matter"; extending downward found the actual PEAK at ubatch 512 (1193) with real losses on both sides — 256 → 1046 and 384 → 1068 (SMs not saturated), 2048 → 1077 (10 % off). The deployed 1536 was thus both 2.4 % slower than the optimum AND 3.71 GiB more expensive. 3. THE SPLIT IS A FUNCTION OF THE BATCH, because the compute buffers are not symmetric. Shrinking ubatch 1536 → 512 freed 1.30 GiB on the plain card but 2.41 GiB on the card holding the output layers and the drafter's graphs — so the balance point moved a layer (36:29 → 35:30 on a 65-layer hybrid) and the reachable minimum gap became one DeltaNet layer (0.46 GiB), not zero. Change the batch and you must re-measure the split. Method notes that made the result trustworthy: llama-bench records n_ubatch per test in its -o json output, so "is the knob even applied" is answerable from the artifact; and the VRAM columns scale monotonically with ubatch, an independent confirmation. The falsification test for _(context: Dankserver 2026-09-16: swept llama.cpp --batch-size x --ubatch-size on frank-medium (2x R9700 layer split, Qwen3.8-27B UD-Q8_K_XL + DFlash2) and applied the result. Operator had said "that sounds like something's wrong" about a flat prefill curve.)_
-Dankserver frank-primary experiment-window restore posture (proven 2026-09-27): (1) systemctl mask REFUSES a unit whose /etc/systemd/system file is the deployer's relink symlink into the repo checkout ("File already exists and is a symlink") — use systemctl mask --runtime unit instead: it shadows via /run/systemd/system (verified /run/systemd/system/frank-primary.service → /dev/null), leaves the relink untouched, and SELF-CLEARS at reboot, which is exactly the wanted window semantics (reboot returns the repo-configured llama.cpp). (2) Restore a frozen launch.sh-based vLLM deployment by dumping its state-dir setup.json config to an env file with export K=V lines, overriding the deltas (R9V_HOST_PORT, R9V_CONTAINER_NAME, R9V_HOST_BIND=0.0.0.0 — the base profile default is loopback-only, which breaks models.yml's LAN target), then R9V_PROFILE=base profile.env R9V_CONFIG_FILE=env file bash scripts/launch.sh — launch.sh's doctor (PASS=24 FAIL=0) then binds exactly like the incumbent. (3) vLLM containers hang on SIGTERM (60 s timeout → SIGKILL needed, same law as llama.cpp's SIGTERM cleanup hang). (4) launch.sh refuses to run when a stale exited container holds the name — docker rm it first. Evidence: health 200 in ~60 s (warm page cache), LAN-path completion, qualification record radiance-vllm-qwen4exp docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md (commit 50a79ec), vault log.md 928fd81. _(context: Q8_0 bring-up qualification window on dankserver (2026-09-27): q8 served + battery on :7773, then uncensored UD-IQ4_XS restored to the frank-primary :7770 slot; frank-primary.service masked for the window.)_
-In serverAppConfig, other sessions commit to the same checkout while you work, so line-anchored edits rot within minutes. Twice on 2026-09-16 an edit landed in the wrong section for the same reason: (1) AGENTS.md was edited against a read taken many turns earlier, and the peer session's commits had shifted the file — the frank-large text ended up split between the Model Config section and the service section, needing a repair; (2) an append at N used an N taken from wc -l, but the file had no trailing newline so its last content line was N+1 — the new section was inserted *before* the paragraph's final line, orphaning it. Discipline that prevents this: re-read the target region (read tool) immediately before editing, take the tag and line numbers from that read, prefer text-anchor insertions (PUT N: where N is a line you just saw) over remembered numbers, and immediately after an edit whose result looks even slightly off, re-read the region and check the surrounding structure (headings, list continuity, closing braces) rather than trusting the edit's summary. git diff --stat plus a targeted sed -n on the changed range is the cheap verification; for multi-hunk edits in a large markdown file, verify by reading the file, not by reasoning about the diff. Also: wc -l undercounts files that lack a final newline — never use it as the append anchor. _(context: Dankclaw/serverAppConfig 2026-09-16: while documenting the x4 link and Q4_K_S results, a peer session was committing frank-medium work to the same checkout; a stale-anchor AGENTS.md edit split a section, and a wc -l-based append split a paragraph in the x4 note. Both were caught by re-reading the files and repaired before commit.)_
-radiance-vllm-qwen4exp Q8_0 bring-up on dankserver (2026-09-27, fork commits 363d606+2fe6dbb, container r9v-qwen38-q8, port 7773): (1) The plugin's fused-HC uniformity check assumes the PUBLISHED IQ4 layout — injection shard (blk.N.hc_attn_inject → attn_hyper_connection.block_inject_weight) is F32 while down shard is Q8_0. Unsloth's Q8_0 conversion quantizes BOTH shards → ValueError "Detected some but not all shards of ...input_mix_weight_down_block_inject are quantized" at WorkerProc init (~112 s). Fix: dev-overlay qwen4_exp.py (env R9V_DEV_QWEN4_EXP_ADAPTER_PY, mount block in launch.sh) extends dequantize_qwen4_exp_packed_hc_down to dequantize a quantized inject shard in-stream + marks the module unquantized; numerics identical to file conversion, F32 path unchanged. (2) runtime.json env.always force-injects R9V_FULL_MUTABLE_CACHE=1, whose baseline check demands the QUALIFIED 160-slot uncensored geometry; q8 profile runs 80 slots → launch.sh gained a passthrough (profile-set R9V_FULL_MUTABLE_CACHE appended AFTER runtime-overlay envs, docker last-wins); q8 profile sets 0 → plain tiered path. (3) GOTCHAS: setup_profile.py start uses the FROZEN setup.json config (+os.environ), NOT profile.env — profile.env edits are inert at start time; pass R9V_PROFILE=q8 profile.env explicitly (launch.sh line 6 sources it). launch.sh sources the DEFAULT (dual-r9700) profile when R9V_PROFILE is unset. A stale exited container makes launch.sh exit 1 "Container already exists" — docker rm -f first. Without the podman-shim PATH, real docker answers "No such image". Tolerance note: Q8_0 roundtrip error ~0.03/step (gguf.dequantize of quantize'd (N,K) returns (N,K); quantize returns (N,K+20)). Profile: dual-r9700-mtp4-q8, PLE pinned 54.4 GB (preflight 108.8 required vs 140.9 available GiB), model load 21.84 GiB/158 s per worker, compile then serve ~16 min total cold. Verified: /health 200, /v1/models qwen3.8-flash-next, completions return content + reasoning_content (17×23=391). _(context: Qwen3.8-Flash-Next Q8_0 GGUF launch failure root-cause session on dank@192.168.0.235, fork repo /home/dank/GitPackages/radiance-vllm-qwen4exp, state /home/dank/r9v-state/q8xl/)_
-When a serving engine keeps some weights in host RAM, the decisive question is whether it *streams* them across PCIe or *computes* them there — the link decides the first and is irrelevant to the second. Dankclaw's 3090 is on Gen4 **x4** (measured H2D 6.72 GB/s, D2H 6.60 GB/s = 85 % of the 7.877 GB/s spec; idle-downshifts to 2.5 GT/s x4 and trains up under load). For an A3B MoE whose active experts are ~1.70 GiB/token (Q4_K_M, 75 GiB expert mass), streaming them costs 272 ms/token ⇒ **3.7 tok/s** (4.6 with a fifth resident), while computing the same bytes in host RAM at 256 GB/s costs 7.1 ms — the link is **31× slower than the memory behind it**. Measured alternatives on the same host: 20.46 tok/s for llama.cpp with experts computed on the CPU (-ot exps=CPU; the link then carries activations only), 25 for halogen, 26.5 for mainline on the iGPU. So "is the link fast enough?" is answerable in one line of arithmetic: per-token bytes the engine must transfer ÷ measured rate = seconds/token, and compare that against the compute-in-place path. Two measurement subtleties from the probe: cudaMemcpyAsync on one stream was *no* faster than sync, and pageable transfers ran within 1 % of pinned — i.e. the copies were link-limited rather than staging-limited, so don't assume the usual "pageable costs 2-3x" rule. Also, hardware ceilings must be read from the *platform port* (LnkCap of the upstream bridge) and not just the card: the 3090 is x16-capable, the Strix Halo GPP port under it is x8-capable, so the x4 lives in the adapter or cable and the realistic upgrade is x8 (2×, ≈7-9 tok/s) — still behind every alternative. And FreeToken specifically (expert LRU in host RAM = exactly the streaming shape) loads only modelopt block-FP8 HF safetensors for qwen3_5_moe, so a Q4_K_M GGUF is unloadable there before bandwidth is even the question. _(context: Dankclaw 2026-09-16: user asked whether the x4 link could support running the 111 GiB Q4_K_M Flash-Next model with FreeToken on the 3090 alone; built pcie-link-test.cu, stopped both services, measured, and derived the ~3.7 tok/s streaming ceiling vs 20.46 tok/s for llama.cpp's compute-in-RAM path.)_
-Verified model-path facts for the dankserver 2xR9700 fleet (2026-09-27, primary-source checked — supersedes any scout claims): (1) DeepSeek V4.1-Flash (released 2026-09-10): 552B backbone, 8B active prefill / 16B active decode, CED (Causal Encoder-Decoder) arch, 1M ctx, MIT, multimodal. NOT the April preview (DeepSeek-V4-Flash 284B/13B, ggml-org/llama.cpp#22319 — early vLLM/SGLang successes were Hopper/Blackwell only). BEST path = Lucebox (github.com/Luce-Org/lucebox): gfx1201 is a first-class tested platform (ROCm 7.2), payload Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF live on HF (24.9k downloads), heterogeneous R9700+Strix Halo measured 86 tok/s decode / 788 tok/s prefill @2K, and the fleet already ran this stack (frank-primary-lucebox-ds4 runbook) until the 2026-09-26 Qwen cutover — retired for the swap, not for failure. llama.cpp mainline CED support: unverified as of late Sept 2026; vLLM/R9V fork: qwen4exp-only, not a path. (2) GLM-5.3-Flash (zai-org): 320B/18B active, KDA+DSA hybrid attention + mHC, 1M ctx (default — hard-hung a 128 GB box once, always bind -c), native FP8 328 GB, MIT. llama.cpp arch glm5next lives in unslothai/llama.cpp branch glm5next/upstream = ggml-org PR #27754, NOT merged to mainline as of 2026-08-30 (check PR state before any build). Measured: 9.31 tok/s decode on Strix Halo UD-IQ1_S 93.1 GB, RADV Vulkan, no MTP (MTP landed d07e71ede 2026-08-30, unbenched); -fa off is MANDATORY (garbage output otherwise). Quants: unsloth ladder IQ1_S 93.1 GB (70.9% top-1) to UD-Q4_K_XL 199.7 GB (92.2%); GSQ-RCO 3.5-bit 137.07 GB (MMLU-Pro -1.40 pp vs Q8_0 ref 61.95%). On discrete R9700s GLM's ~7.9 GiB/token expert volume rides the exps=CPU law to ~4-5 tok/s [ESTIMATE] — #2 for dankserver, but a strong frank-large (Strix Halo) candidate. (3) Process law: LLM research-scout drafts carried invented citations twice in a row (fake repos, wrong PRs, impossible versions like "vLLM v0.21" in a 0.26 era); before any report ships, verify every load-bearing URL di _(context: Session 2026-09-27: research scout (R9700ModelResearch) returned two consecutive drafts with fabricated/incorrect sources ("MiaAI-Lab", "freenode", wrong PR, vLLM v0.21 claim); direct primary-source verification (GitHub issue 22319, HF model cards, julianmb/halo-glm53flash receipts, pfeifferj GSQ-RCO card) superseded it. HTML report delivered at /tmp/r9700-ds41-glm53-report.html (volatile path — f)_
-FreeToken v0.1.2 is an "edge-native MoE serving engine that keeps EXPERT WEIGHTS IN HOST RAM with an LRU GPU cache" — so unlike llama.cpp's -ot exps=CPU (which COMPUTES experts in host RAM and moves only activations over the bus), FreeToken STREAMS expert weights across PCIe every token. That makes it the one design where link width is the critical path, and its own Dankserver measurements prove the mechanism: Qwen3.6-35B-A3B-FP8 on a 3090 gave 56.7 tok/s at ~89% expert residency but 25.2 tok/s at 5% residency, and the runbook itself notes the bottleneck "flips to PCIe expert streaming" at low residency — 25.2 tok/s × ~2.4 GB/token of misses ≈ 60 GB/s, i.e. an x16 Gen4 link saturated. Two consequences for Dankclaw, where the 3090 sits behind Strix Halo GPP bridge 00:03.1 (LnkCap 16GT/s **x8**, LnkSta x4; the platform's other ports are x1/x4/x16-for-iGPU, so there is no x16 slot to move it to): (1) x4 = 7.877 GB/s and x8 = 15.75 GB/s cap that design at roughly 4.6–10 tok/s for a 111 GiB A3B model, because ~1.7 GiB of active expert bytes per token cannot cross in time; (2) it cannot even load this file — FreeToken's loader matrix is modelopt block-FP8 HF safetensors for qwen3_5_moe only (compressed-tensors FP8 fails; GGUF is not supported at all). The measured ceiling for "Q4_K_M experts-in-RAM on just the 3090" is therefore the CPU-compute path at 20.46 tok/s decode / 60–81 t/s prefill, and the host's fastest single-model arrangement remains the iGPU (25.8 tok/s Q4_K_M, 26.5 Q4_K_S at 64k depth). Also durable: FreeToken builds for CUDA only (its _pinned_tensor kernel links cudart), and torch.cuda.is_available() can report False on a box whose kernels are fine. Separately, the model-payload rules that bit me on 2026-09-16: mradermacher ships large quants as .partNofM fragments (a .gguf filter hides Q4_K_S entirely); curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body (three 153-byte "shards" with identical sha256 and a cheerful DOWNLOAD_CODE=done in 20 _(context: Dankclaw 2026-09-16/17: user asked whether the 3090's PCIe Gen4 x4 link is fast enough to run the 111 GiB Flash-Next Q4_K_M under FreeToken on just the 3090. Answered from the FreeToken runbook's own mechanism and measurements, the observed Dankserver residency-vs-throughput table, lspci -vv topology of the Strix Halo GPP bridge (x8 cap, x4 trained) and the earlier measured x4 bandwidth; the same )_
-R9V qwen4exp Q8_0+MTP4 profile bring-up (2026-09-27, dankserver): all artifacts staged and validated, bring-up blocked ONLY on GPU headroom (production uncensored holds 29.9/30.8 GiB per R9700; rank1 has 1.59 GiB free vs 3 GiB requested — a second instance cannot coexist; qualification needs a stop window for frank-primary). Staged state: model dir /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V (6 sha256-verified shards, 188.2 GB); PLE derived /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/r9v-data/per_layer_token_embd.iq4_nl.bin = 54,400,261,120 B sha256 91743e33b6de2272b50a6269a6fe692f56244946fdecf19ca1eba0ba640b303f; placement manifest validated (hot 28/198, rank0 static 4.56 GB / rank1 17.37 GB vs proven budgets 4.68/17.41 GB, MAX_EFFECTIVE=108,198); profile.env + bootstrap.json wrapper written; setup command ready: ./r9v setup qwen38-q8-mtp4 --state-dir /home/dank/r9v-state/q8xl --model-dir /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V --image sha256:2dac17a215fb5b0e3461e4c3e36a2981eec8ac3d6021e73183d247e819740c03 --local-image --reuse-from /home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS --accept-model-license. Bring-up bugs fixed along the way (durable): (1) GGUF expert tensor names split into 4 parts ("blk.0.ffn_up_exps.weight" - parts[0]="blk", layer=parts[1]) — the 3-part assumption crashed with int(''); (2) expert_budget.expert_memory(data, slots, cache_ranks, asynchronous=False, *, catalog=None, runtime=None) — runtime/catalog are KEYWORD-ONLY; (3) bootstrap.json wrapper canonical schema = exactly {schema, id, model_package, hardware, artifact_path, artifact_sha256, scope} — hardware "dual-r9700-32gb-asymmetric", scope "Unranked routing bootstrap only; does not certify fit or headroom", and wrapper id MUST equal the id declared in profile.json's placement descriptor or setup rejects with "does not match profile placement"; (4) nohup X & inside an ssh command races ssh teardown and can kill the script before it writes a byte — run remote long scri _(context: Frank ordered "modify the q4k xl profile for q8 with mtp" + "ensure ple always stays in ram, not disk" for the R9V fork on dankserver; pipeline authored and run to completion while production serves uncensored on :7770; scratch bring-up awaits a GPU window.)_
- Dankserver's frank-medium (2x Radeon AI PRO R9700, gfx1201/Navi 48) runs **ROCm/HIP, never Vulkan**: scripts/build-llama-rdna4.sh builds with -DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF -DGGML_HIP_RCCL=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1201 against a private bundled ROCm 7.14 tree (lib/rocblas, lib/hipblaslt, libamdhip64.so.7). Two log-reading traps that follow: (1) a HIP build still logs ggml_cuda_init: found 3 ROCm devices because upstream implements the HIP backend INSIDE ggml-cuda (so the CUDA name persists) — the giveaways are device names coming back as ROCm0/ROCm1 and ldd showing libamdhip64/librocblas/libhsa-runtime64; (2) ldd | grep -ci vulkan = 0 is the one-line proof of no Vulkan. Vulkan exists only on Dankclaw's separate build-dual tree (CUDA+Vulkan) used for the x4 3090+iGPU experiment; it has nothing to do with Dankserver. RCCL is compiled out by design because card #2 cannot dispatch an RCCL kernel (PCIe atomics), which is why the two cards are joined by llama.cpp -sm layer, not tensor/pipeline parallelism. _(context: 2026-09-16, Dankserver: user asked "are we using rocm or vulkan?" while a llama-bench batch x ubatch sweep was running on frank-medium. Verified from scripts/build-llama-rdna4.sh:85-88, llama-bench --version (0.4.1-dev, fb27a525d), and ldd on the binary.)_
- R9700 (gfx1201) model research — VERIFIED facts (2026-09-27, all URL-checked directly, NOT from the scout whose citations were partly fabricated): (1) DeepSeek V4.1-Flash released 2026-09-10: 552B backbone params, 8B active prefill / 16B active decode, Causal Encoder-Decoder (CED) arch, 1M ctx, multimodal, MIT (HF deepseek-ai/DeepSeek-V4.1-Flash model card). The April V4-Flash preview was 284B/13B (issue ggml-org/llama.cpp#22319, opened 2026-04-24; preview weights are FP4+FP8 mixed precision, community GGUF conversion initially produced garbage, dequant+requant IQ4 ~153 GB). (2) Lucebox = first-class R9700 path: Luce-Org/lucebox (2,883 stars) lists RDNA4 gfx1201 + ROCm 7.2 as a tested platform; heterogeneous "R9700 + Strix Halo" DeepSeek V4 measured 86 tok/s decode / 788 tok/s prefill@2K; Qwen3.8-27B UD-IQ4_XS + DFlash2 on one R9700 = 208.1 tok/s; HF org carries DeepSeek-V4.1-Flash-ROCMFP23-GGUF (the exact family frank-primary served 2026-09-26 per serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md) and DeepSeek-V4-Flash-0731-ROCmFP3. (3) GLM-5.3-Flash (zai-org): 320B MoE / 18B active, hybrid KDA+DSA attention + mHC, 1M ctx default, native FP8 checkpoint 328.3 GB. llama.cpp arch = glm5next, support lives in unslothai/llama.cpp branch glm5next/upstream = ggml-org PR #27754, NOT merged into mainline as of 2026-08-30; quant-side PR #27773 cited by the GSQ-RCO card. MTP/NextN landed upstream d07e71ede 2026-08-30 (glm5next.cpp:846 implements DECODER_MTP; earlier stub asserted at glm5next.cpp:690). -fa off is MANDATORY (flash-attn breaks MLA correctness → garbage); default 1,048,576 ctx must be explicitly bounded (hard-hung a 128 GB Strix Halo once). Only measured receipt: Strix Halo gfx1151 RADV Vulkan, UD-IQ1_S 93.1 GB → 9.31 tok/s decode / 19.7 prefill, 16 threads, no MTP (julianmb/halo-glm53flash, 2026-08-28). Quant ladder: unsloth UD-IQ1_S 93.1 GB (70.9% top-1) → UD-Q4_K_XL 199.7 GB (92.2%); GSQ-RCO (pfeifferj, IST-DASLab methods; taurusduan = mirror): 3.5-bit 13 _(context: Research turn 2026-09-27: scout output was partly fabricated (invented sources, wrong vLLM version); direct URL verification produced the reliable evidence set below. The HTML report on R9700 paths for DeepSeek 4.1 + GLM 5.3 Flash is still pending (bg_31 pipeline check in flight).)_
-In the serverAppConfig homelab repo, more than one agent session can be committing at once (I shared it with a peer working on frank-medium tonight). That breaks line-anchored edits: I edited AGENTS.md using a tag from a read several turns earlier, the peer's commits had shifted every line below my anchor, and my two hunks landed in the wrong sections — the first split the "Model Config (harness)" section from the "frank-large" heading, the second inserted a paragraph mid-sentence so the halogen description read "halogen-flash-server 0.5.8 as a rootless Quadlet again since 2026-09-16 [paragraph] (dankclaw/frank-large/quadlet/...), serving ...". The tell was in the edit response itself: neighbouring lines that had no business being adjacent. Discipline that prevents it: re-read the exact target region immediately before editing (or grep the target text for a fresh line/anchor) rather than reusing a tag from earlier in the session, and always glance at the returned context lines to confirm they are the ones you expect; repair a mis-landed edit with a range rewrite (CUT the stray lines, PUT the whole section back in order) rather than incremental patching. Second, durable purge practice learned the same night: before deleting a "model cache" on a serving host, (a) inventory what is actually there, (b) find which dirs *live* units mount (grep Volume=/MODEL_DIR in the units) and protect them by name, (c) verify each service payload is re-fetchable upstream before deleting — query the Hub API anonymously, because a gated repo returns 403 even with the host's token, (d) record byte-size fingerprints before deletion so the re-fetch can be compared afterwards (that is how I proved halogen's overlays had been stale in place: 2,477,677,120 → 2,572,466,560 and 2,383,306,048 → 2,478,095,488 while the checkpoint returned byte-identical), and (e) expect the bind-mount trap when clearing a container's model dir: the directory must exist but be empty, or podman fails with Error: sta _(context: Dankclaw/serverAppConfig 2026-09-16 night: operator ordered the model cache cleared with one-time rm permission; I purged ~600 GB (service payloads included) and re-fetched, which revealed the halogen sidecars had been stale in place. The same session's AGENTS.md doc update mis-landed because a peer session's commits had shifted the file between my read and my write.)_
-frank-primary on dankserver (192.168.0.235:7770) is now the R9V radiance-vLLM fork (supersedes the 2026-09-26 llama.cpp Q8_0 entry): drwolfen/radiance-vllm-qwen4exp, docker-pinned image bundle r9v-qwen38-flash-next-wmma@sha256:2dac17a2 (loaded from a containerd/OCI docker-save — podman keys images by config digest but keeps the manifest digest in RepoDigests), serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + blockfp8 MTP4 depth 4 + F16 mmproj, TP=2 tiered experts (UVA pinned host copy 43 GiB rank-0 + 16 GiB PLE pinned reserve; PLE residency ssd-mmap; budgets 146.25/78.75 GiB by device), CED on, MAX_MODEL_LEN=196608, tool parser qwen3_coder, reasoning channel reasoning. Production shape: repo-tracked ROOT unit scripts/frank-primary.service (User=dank, LimitMEMLOCK=infinity — but measured fact: the full tiered load reached healthy under a PLAIN dank session, so the pin path does not hard-require it on this host) + serve-frank-primary.sh which layers the on-host 104-key setup state (/home/dank/r9v-state/uncensored/setup.json — the tuned shape lives ON-HOST by design, repo env carries only bind/port/alias/container overrides) under scripts/frank-primary.env, execs the fork's OWN scripts/launch.sh through the docker→podman shim, then docker waits — launch.sh --detach alone would make a crashed engine a green unit. Docker→podman shim laws (in .podman-shim/docker, PATH-prepend, logs every call): (1) containerd-store ID law (inspect sha256:X answers X from RepoDigests; bare sha256:X run refs rewritten to repo@sha256:X); (2) LogConfig law (podman inspect answers LogConfig.Config:null where docker returns the log-opts map — the R9V doctor CRASHES on None.get; shim records run --log-opt pairs and normalizes inspect output); (3) docker info law (SecurityOptions/ServerVersion 27.5.1/DockerRootDir). R9V tooling gotchas: ./r9v start reads state from --state-dir (or default ~/.local/state/r9v/profiles/id) — passing --state-dir to setup but not to star_(context: frank-primary R9V radiance cutover on dankserver 2026-09-27, commits 095128b/7562e41/ca6ba6d; scratch-start debugging (state-dir mismatch, LogConfig crash, flag-consumer bug) all fixed in-shim or by rerun; production deploy green with MainPID check; e2e 32/33 with sleeping FrankBook the only failure.)_
-You can characterize a live LLM endpoint's real performance WITHOUT stopping or benchmarking it, by mining the serving runtime's own per-request telemetry from journald: llama.cpp (mainline, verbosity 3) logs slot print_timing per task with prompt eval time, eval time, total time, graphs reused, draft acceptance = A (n accepted / n generated), mean len = L, plus newer n_gen = N, tg = X t/s, tg_3s = Y t/s; halogen-flash-server logs one line per request: serve_api: mtp N tok in Xs = Y t/s | R rounds, commit Z/round | prompt P (C cached), prefill Ss | detok Uus/tok. Root unit - sudo journalctl -u unit; rootless quadlet - journalctl --user -u unit (never sudo). This is the policy-compliant substitute on hosts whose rules forbid running a bench beside a serving GPU, and it measures the REAL workload rather than a synthetic one. To attribute PCIe cost, run nvidia-smi dmon -s t -c N -o T (columns rxpci/txpci in MB/s) in the background while issuing ONE ordinary request; measured on dankclaw frank-small mid-generation: peak ~230 MB/s tx against a 63 Gb/s (Gen4 x4) link = ~3% utilization, zero at idle - i.e. a 4-lane link costs a fully-VRAM-resident model nothing, because decode traffic is only the sampled logits stream. THE SSH TRAP: if you background the curl and your remote command exits, ssh closes the session and kills the client, which CANCELS the in-flight generation - the journal shows W srv stop: cancel task, id_task = N and NO print_timing line at all, so you get 200-looking HTTP with no telemetry. Always end the remote command with wait so the session outlives the request. Also: per-request numbers are content-dependent (measured MTP draft acceptance 0.46-0.76 on the same build, mean accepted length 2.39-3.27), so a live sample is not comparable to a deploy-time baseline without a masked llama-bench run. _(context: 2026-09-16, dankclaw: user asked for "the performance of the model" after a PCIe x4 link discussion. Measured frank-small (llama.cpp + MTP on the RTX 3090) and frank-large (halogen on the iGPU) from production journal telemetry because AGENTS.md forbids benchmarking beside a live service, and proved PCIe x4 is not a factor with nvidia-smi dmon during one real 200-token generation.)_
-frank-primary (dankserver :7770, Qwen3.8-Flash-Next Q8_0 MoE): measured 2026-09-26 with AMD IBS on the live server that per-BLOCK expert read volume is FLAT (0.93–1.08× mean across all 40 CPU-resident blocks) — MoE top-8 routing fires every block every token, so "most active layers" do not exist at -ot granularity; ANY 8 GPU-resident blocks are equivalent and the only lever is COUNT (already at the 5/3 VRAM ceiling). Real skew is per-EXPERT (top 10% of 48×512 (block,expert) cells carry 40.5% of samples) — below tensor-override reach. Instrument: scripts/frank-primary-expert-activity.py (commit f41e038) — perf IBS op sampling + /proc/pid/maps attribution + GGUF header parsing. Critical implementation law: derive GGUF tensor extents from CONSECUTIVE header offsets, never from a quant type-size table — a wrong per-element size silently mis-buckets each tensor's tail into junk (first run lost 59% of expert samples to a 'gap' bucket with zero warning). Runbook: serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md "Which blocks, measured". _(context: Frank asked whether a tensor-override regex could target the most active layers of frank-primary's experts; measured live instead of guessing. Runbook updated + script committed f41e038; no config change (block choice proven immaterial).)_
-Running llama-bench on the Dank fleet (verified 2026-09-16 on dankserver, frank-medium's UD-Q8_K_XL + 2x R9700 layer split) — four facts that decide whether a sweep measures what it claims: 1. llama-bench has NO -c/--ctx-size. Source line: cparams.n_ctx = n_prompt + n_gen + n_depth. The context is a CONSEQUENCE of the test shape, so llama-bench can never test whether a config allocates beside the deployment's real KV (frank-medium's 16 GiB at 262144 over 2 slots). That check must be done by starting the real unit. Same reason there is no --parallel and no --spec-*: llama-bench measures single-sequence, drafterless numbers — its tg is the autoregressive baseline (~15.9 tok/s at 64k depth here) against the deployed 43.8 tok/s with DFlash2+ngram. -d N is a depth prefix that puts N tokens of KV in place before the measured run and is CACHED across reps via a state save/restore (llama_state_seq_*), so the fill is not re-paid per rep and is not part of the measurement. 2. Separator trap: in llama-bench, / separates devices WITHIN one configuration and , separates configurations to sweep; -ts copies this. -ts 36,29 silently becomes one value (36) and puts everything on one card, benching a different machine. llama-server takes the comma form for the same pair. Also useful: list-valued args accept ranges — -ub 512-2048+256. 3. A test that cannot run calls exit(1) IMMEDIATELY (failed depth/prompt/gen run → llama_free + exit(1)), so one infeasible ubatch in a big cross-product invocation takes every later cell with it and loses the results. Drive one invocation per cell, continue on failure, and resume by skipping cells that already have a non-empty output file. 4. memlock: the deploy units set LimitMEMLOCK=infinity because the ROCm backend pins staging buffers; an SSH shell on these hosts gets soft=hard=8 MiB (ulimit -l), so a bare llama-bench run can fail the load for a reason that has nothing to do with the model. Run it inside a transient unit: sudo systemd-run --unit=n - _(context: Dankserver 2026-09-16: operator asked for a batch x ubatch sweep at 64k context on frank-medium (full 4x7 cross-product, -p 65536 -n 128 -d 65536, -r 3 --no-warmup, no spec decoding). Harness written, dry-run tested with a stub llama-bench, deployed via git, run as a transient systemd unit.)_
-llama.cpp b11030 --override-tensor laws (proven live on frank-primary, dankserver, 2026-09-26): (1) The loader applies the FIRST matching pattern per tensor — a broad pattern like exps=CPU placed before block-specific GPU patterns SILENTLY swallows them (zero warnings in the log; buffers stay byte-identical to no-override). Specific patterns must precede the broad sweep. (2) Repeated --override-tensor flags log DEPRECATED ("only last value will be used" — misleading; actually all apply with first-match-wins); the canonical form is ONE flag with the full comma-joined list. (3) GPU-target overrides ARE legal under a layer split when the tensor sits on the card that holds that block's layers (both ROCm devices are in the main graph's backend list) — the x4-pair "=CPU only" wall applies to draft-graph borrows and non-primary devices, not main-graph co-located blocks. (4) Granularity: the whole per-block expert tensor (512 experts share one tensor, 2.42-2.49 GiB at Q8_0) is the atomic placement unit; there is no per-expert routing knob. (5) Env-file pitfall: the value with parentheses MUST be single-quoted in the env file — source parses values, unquoted ( is a bash syntax error. (6) Probe procedure: FRANK_PRIMARY_ENV_FILE=/tmp/probe.env + scratch PORT, warm loads ~10-30 s; verify placement via load_tensors: ROCmN model buffer size lines (exact arithmetic: base + N×2550 MiB) and headroom via rocm-smi bytes; total card = 34,208,743,424 B (31.86 GiB, not 32). ssh+zsh layers mangle sed backslash escapes and treat bare === as =cmd expansion — use heredocs. _(context: frank-primary 4-slot/V-q8/block-fill reshape on dankserver 2026-09-26, commits de85654/d317d6b/d67768e)_
-Three traps hit in one model-payload fetch on the Dank fleet (2026-09-16), all worth knowing before the next swap. (1) mradermacher's GGUF repos ship the LARGE quants as .partNofM fragments, not .gguf: mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF holds Q4_K_M as part1of3/2of3/3of3 (37.00+37.00+36.97 = 110.97 GiB — which is exactly how the deployed single-file payload was assembled) and Q4_K_S the same way (35.00+35.00+34.09 = 104.09 GiB), while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are single files. Listing the Hub tree and filtering for paths ending in .gguf therefore makes Q4_K_S look like it does not exist — I asserted that to the user twice before they pushed back. Filter on the quant name, not the extension. (2) A fetch script must use curl -f/--fail-with-body, compare byte counts to the Hub's declared size, and verify the LFS sha256; otherwise it reports success on an access error. curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body, so a gated repo produced three 153-byte "shards" with identical sha256 values and a cheerful DOWNLOAD_DONE in 20 seconds for 104 GiB — caught only because the time and the duplicate hashes were absurd. (3) A token passed as curl -H "Authorization: Bearer …" is visible in /proc/pid/cmdline, so pgrep -a curl while a download runs prints the secret into your transcript (it happened); use curl -K mode-600 file or an HF_TOKEN environment variable, and never pgrep -a a command carrying an auth header. Related: pgrep -f in a wait loop matches the loop's own command line when the pattern appears in it, so a "wait for the download to finish" loop can spin its full iteration count without ever observing completion. Gated mirrors: orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF carries the same Q4_K_S ladder but returns 403 even with the host's HF token until that account accepts the gate — mradermacher's copy of the same quant is the ungated source. _(context: Dankclaw 2026-09-16: user asked to download Q4_K_S of mradermacher/Qwen3.8-Flash-Next-Uncensored-i1 for a smaller-quant test; the fetch went through a wrong "the repo has no Q4_K_S" claim, a 403-on-gated-repo false success, and a token leak via pgrep -a, before landing a verified payload (111,769,915,680 bytes, parts hash-matched against the Hub's LFS oids).)_
-llama.cpp --override-tensor (b11030 unsloth build, verified 2026-09-26 on frank-primary): the loader applies the FIRST matching pattern per tensor — a broad exps=CPU listed before block-specific GPU patterns silently swallows them (all experts stayed CPU, zero warnings anywhere in the log; only the per-card "model buffer size" load-log lines revealed it). Order: specific patterns first, broad sweeps last. Repeated --override-tensor flags also log DEPRECATED (its "only last value will be used" text is misleading — matching is first-match-wins across the joined list); pass ONE comma-joined flag. Placement verification is buffer-size arithmetic, never absence-of-errors. Related measured fact: --cache-type-v q8_0 does NOT apply to the shared-MTP drafter's KV (stays f16 — draft pool = f16 law at the target pool size). Fleet workflow corollary: on the zsh-login hosts, write regex-bearing remote env files via heredoc cat file (sed piped through ssh+zsh layers eats backslash escapes), never use bare === as an echo marker (zsh =cmd expansion aborts the command), and quote globs inside remote grep patterns. _(context: frank-primary 4-slot/V-q8/hot-block reshape on dankserver 2026-09-26; commits de85654 (shape) + e330d56 (runbook law); the broad-first probe silently no-op'd the GPU block placement)_
-Two durable traps from rebalancing frank-medium's tensor split (2026-09-16, Dankserver; revert DSpark-DFlash2): 1. A RECORDED ARTIFACT SIZE IS A CLAIM, NOT EVIDENCE — verify against the Hub's tree API. The fetch script declared the DSpark drafter as 1,359,284,737 bytes while the Hub's authoritative size was 1,455,376,576 (96 MB short, the LFS object's own size), so --verify-only failed on a payload that was in fact byte-correct (its sha256 DID match the LFS oid, because my recorded hash was taken from the host file, not the Hub — a self-consistent loop that proves nothing). The check that settles it in one call: curl -sL "https://huggingface.co/api/models/repo/tree/main?recursive=1" then read each entry's size and lfs.oid (a 128-char sha256, pointerSize:135 for the pointer file). Compare BOTH against stat -c%s and sha256sum on the host. Note the bare /api/models/repo endpoint does NOT list files; the ?blobs=true variant returned null siblings on these repos, and piping the tree into jq failed with "cannot use null as iterable" when the response was partial — parse it in python from a saved file instead. 2. WHEN A REPO CARRIES A FITTED VRAM MODEL (a test seam), FITTING AGAINST IT REQUIRES MATCHING ITS ARITHMETIC UNIT CONVENTION EXACTLY, or the constants you derive are meaningless. frank_medium_unit.bats expresses compute buffers as 1442 * 1024 * ubatch — i.e. the coefficient is ~1.4 MB per ubatch UNIT, not 1442 bytes — while my first fit (in python) used 1442 * ubatch. The result was a fitted card-0 constant of -54,541,757 B (a negative "constant", the tell) and a seam that failed the floor assertion by ~2 GiB: 1442*1024*1536 vs 1442*1536 is a 2.1 GiB difference. Read the assertion expression literally before fitting to it, and treat a negative constant as evidence of a unit error rather than something to leave in. Also worth keeping: bash -n file.bats is NOT a syntax check — bats' @test "..." { } form is not valid bash, so it fails on a valid file (line 32 in that f _(context: Dankserver 2026-09-16: reverting frank-medium's spec drafter from magnitudedev DSpark to z-lab DFlash2 (DSpark's window-less draft KV spanned the full 262144 ctx = 5.0 GiB) and re-measuring the tensor split. The revert freed ~4.3 GiB on ROCm1 and moved the balance from 40:25 to 36:29. Repo: /home/blank/Projects/serverAppConfig, files scripts/frank-medium.env, tests/frank_medium_unit.bats, scripts/)_
-Dankserver fleet delta (2026-09-26 cutover, supersedes the frank-medium entry): frank-medium (vLLM/radiance/Swift-Qwen3.8-27B-PARO-MXFP6) is RETIRED — the active Dankserver endpoint is now **frank-primary**: unsloth llama.cpp b11030 prebuilt (root systemd unit, enabled+active), unsloth/Qwen3.8-Flash-Next-GGUF Q8_0 (175.3 GiB, 6 split parts, mmap — never --no-mmap) with shared-Q8_0 MTP drafter (depth 2) + ngram-mod, BF16 mmproj, 262,144 ctx × 1 slot, port 7770 @ 192.168.0.235, alias frank-primary, -ot exps=CPU,per_layer_token_embd.weight=CPU (experts+PLE computed in host RAM), -ts 1,1 across both R9700s, **SPEC_DRAFT_DEVICE=ROCm1** (last-layers card — ROCm0 crash-loops with the ggml-backend.cpp:941 borrowed output.weight abort). Measured: 11.8 t/s decode, 0.86 tool-task draft acceptance, ~7 s warm load, 9.4 GiB/card (29%/36% of 32 GiB), 29,696 B/token ctx-scaled KV law. KV is f16 (no --cache-type flags). Rollback: lucebox quadlet + 378 GiB payload retained on host (~/.AGENTTRASH/20260926-llamacpp-cutover/). Runbook: serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md. All three endpoints now export /metrics (frank-large was silently 501 since its own deploy; fixed 82ff9dd). Repo HEAD de4d227. _(context: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 cutover on dankserver 2026-09-26 (commits 8500600, a9b94a7, 5d632bb, 82ff9dd, de4d227))_
- llama.cpp multi-GPU layer-split sizing, learned on Dankserver (2x R9700, 262144 ctx targeted) 2026-09-16: 1. A SPECULATIVE DRAFTER'S KV IS SIZED FROM THE TARGET, NOT FROM ITS OWN FILE. common/speculative.cpp builds the draft context with cparams.n_ctx = llama_n_ctx(ctx_tgt) and there is NO flag to cap it independently (the --spec-draft-* surface has no ctx). So a drafter that declares no sliding window caches the WHOLE target context on whatever device --spec-draft-device names: DSpark (5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB/position) costs 5.0 GiB at 262144, where DFlash2's dflash.attention.sliding_window = 2048 had held it to ~40 MiB. Read sliding_window (and block_size, n_ctx_train) out of the drafter's GGUF header BEFORE sizing. Signature of getting it wrong: the model LOADS, /health answers, and the FIRST draft decode dies with ROCm error: out of memory inside common_speculative_impl_draft_dflash::process -- an allocation failure after a successful load is a SIZING failure, not a broken drafter, and a crash-looping unit makes it look like the latter. 2. A BIGGER --ubatch-size IS ASYMMETRIC ACROSS A LAYER SPLIT. Measured 2x R9700 at 262144: ubatch 512 - 1536 cost the output/drafter card ~3.3 GiB against ~1.7 GiB on the other (that card holds the output layers AND the drafter's graphs). So a split balanced at one batch is UNBALANCED at another -- 38:27 was balanced at 2048/512 and left the draft-carrying card at 0.08 GiB free after the batch tripled. One -ts step is worth ~1.5 GiB at that size; the balance point must be found by measurement (38:27 - 3.78/0.08; 41:24 - 1.47/2.38; 40:25 - 1.88/1.88). Corollary: the balanced split is also what MAXIMISES ctx for a given per-card headroom floor, because the headroom has to come out of CTX_SIZE. Also measured: batch 8192/ubatch 2048 does NOT fit at 262144 (its 2020 MiB pp buffer fails to allocate on the busier card), while 4096/1024 fits but leaves only 1.11 GiB. 3. A GPU PAGE FAULT IS NOT AN OOM. One split_(context: Dankclaw/Dankserver 2026-09-16: swapping frank-medium's drafter from z-lab DFlash2 to magnitudedev DSpark, then re-specifying batch/parallel/ctx per operator instruction. Repo: serverAppConfig, commits 7e11d24..718a283.)_
- llama.cpp b11030 (unsloth prebuilt family): (1) /metrics is DISABLED by default — without --metrics it answers 501, and Prometheus marks the target down while /health looks fine; both dankclaw frank-large and the new frank-primary had this silently (fixed 2026-09-26, commit 82ff9dd). (2) A spec-draft deployment that borrows target tensors (shared MTP head) must place the drafter on the card holding the target's LAST layers (-ts 1,1 → output.weight lands on ROCm1): with tensor overrides set, n_copies==1 and a draft context elsewhere aborts at load with ggml-backend.cpp:941 "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run the operation (NONE)" — crash-looped 4× at ~12 s until SPEC_DRAFT_DEVICE=ROCm1. (3) llama-server's SIGTERM cleanup can HANG after spec-decode service (logs "cleaning up before exit", never exits) — a timeout 300 probe survived its own kill signaland later blocked a deploy's port gate; kill -9 is the reliable stop for stray instances. (4) A deployer whose health check is a bare curl against the port cannot tell WHO answers — a stray process holding the port produces a FALSE GREEN deploy (seen 2026-09-26 when the deploy's wait_for_health was satisfied by a leftover probe while the real unit was stopped); after any aborted deploy, confirm the responder is the unit's own MainPID before trusting the health gate._(context: frank-primary llama.cpp cutover on dankserver 2026-09-26; commits 5d632bb (draft device), 82ff9dd (metrics + -lv 4), de4d227 (KV law))_
-Dank-fleet VRAM accounting, learned while answering "what's the VRAM breakdown?" (2026-09-16). (1) THE TWO DEVICES ON DANKCLAW ARE NOT THE SAME KIND OF MEMORY. The RTX 3090 is discrete: nvidia-smi --query-gpu=memory.total,memory.used --format=csv,noheader (24,576 MiB total; 22,582–22,592 MiB used by the single llama-server PID serving frank-small, confirmable with nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader). The gfx1151 iGPU is UMA: /sys/class/drm/card1/device/mem_info_gtt_total = 122,402,963,456 (114 GiB) and mem_info_gtt_used = ~36.2 GB (33.7 GiB) is the real "VRAM", while mem_info_vram_total = 2,147,483,648 (a 2 GiB carve-out window) with mem_info_vram_used ~208 MB. So iGPU VRAM IS host RAM, and a phrase like "halogen holds 95.4 GiB" is misleading: halogen's ledger shows 68 GiB of weights PINNED AS HOST PAGES (mlock/page cache, not GTT) plus ~33.7 GiB inside the device budget (11.0 GiB KV pool over 400,384 positions + working memory/scratch). Its own measurement lines are the best source: podman logs frank-large | grep -aiE "weights pinned|kv pool|left for everything|overlay". (2) THE REAL PER-BUFFER BREAKDOWN IS PRINTED BY llama.cpp AT INFO — llama-model.cpp:1827 ("%12s model buffer size = %8.2f MiB", per backend), llama-context.cpp:692 (compute buffer size), plus the KV self-size line — BUT NO ARTIFACT IN THIS FLEET CONTAINS IT: llama-bench suppresses the loader's INFO (its logs show only the device banner and the results table), and sudo journalctl -u frank-small | grep -c "model buffer size" = 0 across the unit's whole history. To actually get it, launch the model once through llama-cli or llama-server (not llama-bench) and keep the first ~200 lines; add -v/--verbose if the level filters it. Otherwise compose the breakdown by arithmetic: read the GGUF tensor table for group sizes (on Flash-Next: experts 75.0 GiB / 48 layers = 1.5625 GiB per layer, per_layer_token_embd 32.8 GiB, everything else 3.2 GiB / 48 = 0.067 GiB p _(context: Dankclaw 2026-09-16: after the 3090+iGPU split experiment, reconstructing what the VRAM looked like per device (measured totals from nvidia-smi + sysfs, halogen's ledger lines, GGUF tensor arithmetic) because no log carried llama.cpp's per-buffer accounting; also pinned down llama-bench's derived context lengths (n_ctx = n_prompt + n_gen + n_depth) for the runs whose parameters were being reviewed)_
-When debugging a crash-looping LLM service by running the serve script manually with a scratch env file (e.g. FRANK_PRIMARY_ENV_FILE=/tmp/probe.env), the probe binds the service port and can satisfy a subsequent deploy's health gate: deploy.sh exited green with "frank-primary healthy after ~104s" while systemctl showed inactive — the health curl hit my probe process, not the unit. Rule: before redeploying after a manual probe, kill the probe (pgrep llama-server on the host) and let the deployer's own unit answer; treat a green deploy that ran concurrently with a manual probe as unverified. Related: a wait_port_free failure on a lone LAST-ACK tail (dead client socket, e.g. 192.168.0.60 not ACKing FIN) is not a held listener — it often clears on its own in ~90 s; sudo ss -K sport = :port errors with RTNETLINK Invalid argument when only the tail remains, but by the next run the port is free — just rerun the deploy rather than editing the gate. _(context: frank-primary deploy 2026-09-26: LAST-ACK tail aborted deploy run 1; probe-A held :7770 while deploy run 2 went falsely green; unit was actually stopped)_
-Working convention on the Dank fleet for answering "what was running when it broke": experiment artifacts must survive on disk because /tmp does NOT (/tmp is cleared by a reboot — after the 2026-09-16 Dankclaw incident the x4*.sh experiment scripts were gone while their logs in /home/dank/models/logs/ survived). So launch experiment scripts from a persistent path and log to /home/dank/models/logs/tag.log. Reconstruction then works from three independent evidence sources: (1) log file mtimes (ls -l --time-style=+%m-%d_%H:%M:%S) minus each run's own real duration from time = that run's start, which orders/nests the runs; (2) set -x lines inside those logs are the authoritative record of the exact command line and parameters — but a log can be truncated mid-write if it is cleaned while a script still holds it, so a log's mtime is not proof the run *finished*; (3) the user journal gives the first-failure instant for any service (journalctl --user --since … | grep -m2 'invalid internal status'), and on this fleet container healthchecks shell out to the podman CLI, which makes them a sensitive early indicator that podman itself has broken. The answer to "what was running" is whichever invocation's [start, end] window contains that first-failure instant; with several overlapping resource-heavy invocations, check whether they overlapped at all (mine did not prove it: the sampler output was in /tmp and gone). Related trap when reading such logs through an agent tool: long command lines get display-truncated, so wrap them (sed 's/\(.\{130\}\)/\1\n /g') rather than concluding the parameters were short, and beware off-by-one log names (x4b.log vs x4b-experiment.log) silently skipping files in a loop. _(context: Dankclaw 2026-09-16, answering "what were the test parameters before the crash?" about the rootless-podman pause-process incident: reconstructed from log mtimes + per-run real durations + the first invalid internal status at 18:58:30, which pinned the in-flight run (launched 18:54:31) — after /tmp/x4*.sh had been wiped by the repair reboot.)_
-On a llama.cpp layer-split (-ts) deployment that carries --override-tensor flags AND a borrowed-head speculative drafter (draft-mtp shared head, e.g. Qwen3.8-Flash-Next's shared MTP GGUF), the drafter MUST sit on the card where the main model's output head landed — the last-layers card under a balanced -ts. Reason: -ot sets has_tensor_overrides(), which pins n_copies==1 in the draft's backend scheduler; a borrowed output.weight pre-allocated in card B's buffer aborts the draft graph with ggml-backend.cpp:941 "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run the operation (NONE)" when the draft context is on card A. Signature: SIGABRT ~12 s into load (right after "loading draft model"), crash-looping every restart. Verified live 2026-09-26 on frank-primary (dankserver 2x R9700, -ts 1,1, -ot exps=CPU,per_layer_token_embd.weight=CPU): SPEC_DRAFT_DEVICE=ROCm0 abort-looped 4x, ROCm1 loaded and served. This extends the fleet's x4-pair law (overrides ⇒ n_copies==1) into spec-decode territory, and matches the frank-medium layersplit precedent (SPEC_DRAFT_DEVICE=ROCm1) — check the repo's retired-era env files for precedent before pinning a device. _(context: frank-primary cutover 2026-09-26: deploy attempt 1 crash-looped; fixed by 5d632bb "pin the drafter to ROCm1")_
-Correction to my earlier note on the rootless-podman pause-process death (Dankclaw, 2026-09-16): the cause is NOT established. I had written that memory pressure from a GPU experiment that committed 116 of the host's 121 GiB killed it, but journalctl -k --since for that window records NO OOM kill and the kernel says nothing else about the pause process — so the mechanism was a guess presented as a finding. The operational content that IS verified stands: every podman command fails with invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process; /run/user/uid/libpod/tmp/alive holds a stale UUID token no process owns; container starts then fail because the rootless netns holding podman's bridges is gone (failed to bind udp listener on 10.89.x.1:53: Cannot assign requested address); podman system migrate panics in 5.7.0 (nil deref in teardownNetworkBackend); moving the stale token aside restores the CLI but not the bridges; a reboot repairs everything and all boot-persistent quadlets return on their own. The generalizable discipline: before publishing a cause, grep the evidence that would prove it (for a killed process: journalctl -k | grep -i oom), and if it is absent, publish "trigger not established" rather than a plausible story — an incident narrative that turns out to be invented is worse than an open question, because the next agent plans against it. _(context: Dankclaw 2026-09-16: writing up the pause-process incident in AGENTS.md, the x4-pair-experiment note and log.md, then checking journalctl -k and finding no OOM kill for the window — which falsified the cause I had already written down in three documents.)_
-GGUF pre-deploy verification (Dank fleet, frank-primary 2026-09-26): (1) A GGUF's KV metadata + tensor-info table sit at the file START, so a partially-downloaded split part already exposes its tensor names — scripts/dump-gguf.py (repo-tracked, dependency-free) can validate -ot override patterns mid-fetch, before the payload completes. This confirmed exps=CPU matches all of blk.N.ffn_{up,gate,down}_exps.weight on Qwen3.8-Flash-Next. (2) Parser gotchas: GGUF v3 string/array lengths are u64; KV value-type enum is 8=string, 9=array; nested array element types include enum 5 (i32) — handle unknown array types with a size-table skip ({0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:0,9:0,10:8,11:8,12:8} bytes/elem), don't raise. (3) The unsloth Qwen3.8-Flash-Next embedded tokenizer.chat_template (dumped from part 1's KV) natively folds high→xhigh before the xhigh/medium/low gate (only minimal/max raise) AND natively consumes preserve_thinking — so frank-primary needs NO external --chat-template-file; the models.yml reasoningEffortMap (minimal→low, max→xhigh) exists only for the two OMP-only levels. Pre-reading the embedded template from the GGUF before deploy prevents an entire deploy-cycle loss on a 500-ing template rejection. _(context: frank-primary cutover 2026-09-26: verified -ot tensor patterns and chat-template effort gate from the partially-fetched payload before deployment)_
-On a rootless-podman fleet host, the rootless "pause" process is load-bearing for ALL container management, and its death is not recoverable in place. Symptoms seen on Dankclaw 2026-09-16: every podman command (even podman ps, podman system df) fails with invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process; the alive file in /run/user/uid/libpod/tmp/ holds a stale UUID token (not a pid) that no process owns; every container start then fails with netavark: error while applying dns entries: aardvark-dns failed to start / failed to bind udp listener on 10.89.x.1:53: Cannot assign requested address because the rootless netns that held podman's bridge interfaces died with the pause process; and every Restart=always quadlet loops forever (28 of them here). Podman's own named remedy, podman system migrate, PANICS in 5.7.0 (panic: runtime error: invalid memory address or nil pointer dereference in libpod.(*Runtime).teardownNetworkBackend while tearing down a stopped container). Moving the stale alive/alive.lck aside makes podman respond again but the missing bridge interfaces remain, so containers still fail. The reliable repair is a REBOOT: /run/user clears, the pause process and rootless netns are recreated, and every boot-persistent quadlet returns by itself (verify loginctl show-user u -p Linger → Linger=yes and [Install] WantedBy=default.target present in each unit BEFORE rebooting). Plausible trigger in that incident: memory pressure from a GPU experiment that committed 116 of the host's 121 GiB — the pause process is ordinary OOM-killable host state. Also from the same recovery: a container cannot reach a Network=host service at the host's LAN IP, because netavark DNATs only PUBLISHED ports — from inside another container 192.168.0.143:published answers but 192.168.0.143:host-bind and the gateway 10.89.1.1 are refused; use host.containers.internal (169.254.1.2), which fixed a 502 o _(context: Dankclaw 2026-09-16 evening: after the x4-pair GPU experiment (which committed 116 of 121 GiB) the rootless podman pause process was gone; all 28 rootless quadlets were down/looping and the Caddy front door with them. Diagnosed the pause token + missing rootless netns, found podman system migrate panicking, repaired by reboot, then fixed a hermes vhost 502 caused by the netavark published-port-onl)_
-frank-medium (dankserver :7770) now serves hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, W4A8, 25.16 GiB) on image localhost/radiance-vllm:0.13.0-e060895 (PR #51) — the repo quadlet (dankserver/frank-medium/quadlet/frank-medium.container) is the source of truth; the AGENTS.md prose (MXFP4 / radiance-vllm 0.9.3) is STALE. Boot-crash lesson (2026-09-26): vLLM's startup check requires free VRAM = gpu_memory_utilization x 31.86 GiB per card, and free-at-check includes the worker's own HIP context plus the amdgpu per-boot baseline (~690 MiB total), which drifts tens of MiB between boots — so GPU_UTIL=0.98 (threshold 31.22 GiB) deterministically crash-looped a boot that measured 31.21 GiB free (ValueError "Free memory on device cuda:N ... less than desired GPU memory utilization"), identical numbers on every restart. Fixed by GPU_UTIL=0.97 (threshold 31.06 GiB, ~160 MiB headroom, cost ~5k tokens on the auto-sized ~486k-token KV pool, commit c5b55e1). If frank-medium crash-loops after a reboot with that ValueError, lower GPU_UTIL further rather than hunting for a GPU tenant — the GPUs are otherwise idle.
-llama.cpp's llama-bench separates a device/tensor-split LIST with / and SWEEP values with , (-dev dev0/dev1/..., -ts ts0/ts1/..). So -dev CUDA0,Vulkan1 is TWO single-device configurations, never a two-device run — and -ts 0.85,0.15 is two sweep values, each becoming [0.85,0,0,...] / [0.15,0,0,...], i.e. device 0 takes every layer. Write -dev A/B -ts x/y. On 2026-09-16 this artifact produced a completely wrong fleet conclusion on Dankclaw ("CUDA + Vulkan cannot serve one model"): the "abort" was a single-device run with -ot exps=Vulkan1 pointing at a device absent from the scheduler's list, and an "OOM at 25% split" was really all 111 GiB on the 24 GiB card. Tell-tale: llama-bench's own dev column prints the devices actually used per row — read it. Corollary that generalizes: before concluding anything from an abort or a resource failure, prove the harness parsed your intent (echo the resolved config, or check the tool's own reported parameters), and for placement-dependent numbers get an independent witness (sample the other device's memory with nvidia-smi while the run is loaded) rather than trusting the label. The real blocker in that episode was different and simpler: per-tensor cross-device placement is killed by --override-tensor itself — has_tensor_overrides() is a required term of the scheduler's pipeline_parallel flag (llama-context.cpp:433, the only read site), which sets n_copies in ggml_backend_sched_new; with n_copies == 1 a pre-allocated weight on a non-primary device has no backend and ggml-backend.cpp:941 aborts with the leaf's own op name (NONE). Two-device LAYER splits (no -ot) work fine, including cross-vendor CUDA+Vulkan. _(context: Dankclaw 2026-09-16: x4-pair experiment (3090 + Strix Halo iGPU under mainline llama.cpp, dual CUDA+Vulkan build 37b53fd); pure-Vulkan and cross-vendor layer splits then measured at 27.55 / 27.23 tok/s @64k depth vs 25.77 iGPU-only.)_
-serverAppConfigs e2e gate: the admin crawl test (tests/e2e/specs/11-admin.spec.ts:24) walks every admin open link, including Prowlarr at 192.168.0.112:8989 (the FrankBook-Pro Mac mini). When the mini is asleep (ARP STALE, 100% ping loss; it's a Mac mini with NO remote WoL/power-on path) that test fails with net::ERR_CONNECTION_TIMED_OUT — a pre-existing environmental failure, not a regression. For changes that don't touch the mini, deploy with --skip-e2e (justified) and note the dependency. The user crawl (12-crawl.spec.ts) is also slow (~1.2m) and can pass on retry. Every non-Mac-mini e2e test passes while the mini is down.
-Authoritative per-flag documentation for a llama.cpp build = its own common/arg.cpp: every flag is defined as add_opt(common_arg({"-x", "--flag"}, "TYPE", "multi-line help", parser)) with the help text inline (modern master, e.g. fb27a525 on Dankserver, keeps ALL arg definitions in thisone file; older trees split them across common/common.cpp + tools). Two traps: (1) llama-server --help on this master does NOT print help and exit — with no --model it silently starts a router-mode server on 127.0.0.1:8080 and stays there, so grep the source instead of probing the binary; (2) --help-style patterns won't match — grep for the literal quoted flag string, e.g. grep -n -A9 -e '"--split-mode"' common/arg.cpp. Recipe: ssh host 'P=$(find buildroot -maxdepth 5 -name arg.cpp -path "*common*" | head -1); cat "$P"' /tmp/arg.cpp (the checkout path can move under you mid-session — resolve it in the same call), then extract per flag locally. _(context: frank-medium exhaustive settings review, 2026-09-16: extracted all 44 live flags' help text from common/arg.cpp in the fb27a525 tree on Dankserver after --help started a server instead of printing.)_
-Fleet model input modalities: models.yml (/home/blank/.omp/agent/models.yml) is the authoritative source, NOT the AGENTS.md prose (partly stale — it claims "no audio anywhere in the fleet"). Ground truth: frank-large = text only (no vision projector); frank-medium = text+image (Gemma4 vision path, verified on held-out images); frank-small = text+image+audio. Audio nuance: frank-small's models.yml entry declares only input: [text, image] because OMP 18.0.6's schema rejects 'audio' (a failed schemacheck drops the WHOLE models file), but the endpoint genuinely accepts input_audio parts via the Gemma4 projector's audio encoder (verified 2026-09-17 deploy — a WAV transcribes verbatim). No fleet model does video. _(context: serverAppConfig 2026-09-26: adding per-model input modalities to the homepage LLM cards)_
-On the Dankclaw/Dankware fleet (and llama.cpp generally), a single model CANNOT be served by two different *device* backends — e.g. trunk+KV on the 3090 via CUDA and MoE experts on the Strix Halo iGPU via Vulkan — and this is a hard structural limit, not a tuning problem. Two failure modes, both measured 2026-09-16 on mainline llama.cpp 37b53fd: (1) per-tensor -ot pattern=device across device backends aborts in the scheduler with ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) — a weight in one device buffer cannot be consumed by an op the scheduler placed on another device; (2) per-layer cross-backend splits (-sm layer -ts, or layer-scoped -ot) fail at model load with only the generic "failed to load model". The CPU is the ONLY side that works as half of a heterogeneous split, because the CPU backend is the universal fallback and host buffers can be copied — which is exactly why -ot exps=CPU works. So on a one-NVIDIA-plus-one-AMD host, the 3090 can be the trunk OR the expert host but never the complement of an iGPU-resident peer; every working multi-device llama.cpp deployment in this fleet is either single-vendor (frank-medium's two R9700s via -sm layer) or device+CPU. The surprising measurement that came out of the same run, and which supersedes the old "mainline llama.cpp is slow on gfx1151" assumption: a CUDA+Vulkan dual-backend build with EVERYTHING on the iGPU (-dev Vulkan1 -ngl 999 -ot per_layer_token_embd=CPU) did 32.64 tok/s at ~0 depth and 25.77 tok/s at 64k depth, 456.7 tok/s fresh prefill, with the iGPU sampled at 99-100% busy and the 3090 idle at 0%. That MATCHES halogen's deployed ~25 tok/s (the ik-era "mainline = 1.01 tok/s" figure was the CPU+3090 hybrid, never the iGPU). It is nonetheless not deployable on Dankclaw: it commits 116 of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB PLE page cache) against halogen's 95.4 GiB held with 14.7 GiB spare besi _(context: Dankclaw 2026-09-16: dual-backend llama.cpp experiment testing whether the RTX 3090 (CUDA) plus the Strix Halo iGPU (Vulkan, RADV) could serve one 177B MoE (Flash-Next) together over the 3090's Gen4 x4 link. Result: not expressible; the iGPU-only configuration matches halogen but eats the host.)_
-serverAppConfig homepage (shared/homepage/): the three LLM status cards use a structured details object in services.json (fields role, spec, inputs[], use) instead of the plain description string the other ~32 cards use. app.js descHTML(svc) renders details as tiered divs — .desc-role (bold, --head pale green #aaffcc), .desc-spec (--dim), .desc-inputs (--input #00ff9d spring green, 'Inputs:' prefix, modalities joined with ' · '), .desc-use (--dim, 'Use:' prefix) — and falls back to esc(svc.description) when details is absent. Tiers are deliberately staggered light-dark-light-dark (pale role / dim spec / bright inputs / dim use). The /docs model guide (docs.html #models) mirrors it with Model/Capacity/Inputs/Best-for/Not-for bullets and was the source of a stale-info bug (it had drifted to retired model names/sizes — verify against models.yml/AGENTS.md before trusting it). Deploy: ./scripts/deploy.sh dankclaw --service homepage --skip-e2e (ships to /srv/homepage; e2e skipped because admin crawl test 11-admin.spec.ts:24 times out on the asleep Mac mini 192.168.0.112, which has no remote wake). Verify with bats tests/homepage_*.bats tests/docs_completeness.bats + a local browser render (python3 -m http.server + a scratch status.json, since file:// fetch is CORS-blocked; move the scratch status.json to .AGENTTRASH, never rm). _(context: serverAppConfig 2026-09-26: reworked the LLM model cards (structured role/spec/inputs/use, neutral roles, per-model input modalities, staggered green tones))_
-Deciding whether to split one model across two heterogeneous accelerators on a single host — the arithmetic that settles it, from the 2026-09-16 Dankclaw design pass (RTX 3090 + Strix Halo gfx1151 iGPU, RTX 3090 on a PCIe Gen4 **x4** M.2 dock): 1. LINK FACTS (measured): the 3090's LnkCap is 16GT/s x16 but LnkSta is x4, and it **idle-downshifts to Gen1 (2.5GT/s) x4 ≈ 1 GB/s**, reporting Speed 16GT/s, Width x4 only under load (7.88 GB/s). So the link is a *latency* resource, not a bandwidth one: a hidden state is ~10 KiB (Qwen3.8-27B embedding_length 5120 x 2 B), i.e. **1.3 µs per hop**, and 2 hops/layer × 64 layers ≈ 1 MiB/token ≈ 128 µs — ~0.3 % of a 40 ms token. Byte count is never the problem; **hop COUNT is** (ik_llama.cpp measured graph splits = 100/token with the GPU at 5 % SM). Design rule: partition by **where the weights live**, never by where activations flow; weights stay resident and only activations cross. 2. WHAT ACTUALLY PAYS: at batch-1 decode the KV cache is an **O(context) read every token** (Flash-Next: 28 KiB/position measured, so 5.5 GiB/token at 200k ≈ 21 ms of a 40 ms token at the LPDDR's shared ~256 GB/s vs 5.9 ms at the 3090's 936 GB/s). Expert weights are read once per token and are a rounding error beside it. So the highest-leverage move is **KV + its attention compute on the fast-VRAM device**, and KV must be co-located with its attention (separating them would push 5.5 GiB/token over a 7.88 GB/s link = ~700 ms/token, fatal). 3. WHAT DOESN'T: prefill is compute/amortized-weight bound, so moving weights to faster memory buys nothing and a device stage costs throughput. Calibration measured on this box, same model: moving 12.5 % of the expert mass to the GPU (-ncmoe 42) gave **decode +12 % (4.03 → 4.54 tok/s) but prefill 192 → 32 tok/s**. My bandwidth model predicts +10 % for that case, so it is usable for decode projections — and prefill is the casualty to budget for. 4. HARDWARE ASYMMETRY: on a UMA APU the iGPU is **not an independent memo _(context: Dankclaw 2026-09-16: after re-splitting frank-large (halogen on the iGPU) and frank-small (llama.cpp on the 3090), Frank asked what a *useful* 3090 + Strix Halo combination would be over the 3090's x4 link, then to elaborate the one-process split. Verified with live probes: lspci LnkSta/LnkCap idle vs loaded, sysfs VRAM/GTT windows, GGUF tokenizer identity between Qwen3.8-27B and Flash-Next, frank)_
-frank-small (Dankclaw :7771, Gemma4-12B QAT) DOES accept audio input (input_audio WAV transcribes verbatim, verified at the 2026-09-17 deploy) via the Gemma4 projector's audio encoder — even though the harness models.yml only declares input: [text, image] for it. The reason is a harness schema limit, not a model limit: OMP 18.0.6's models.yml schema only accepts input: "text"|"image", and a failed schema check drops the WHOLE models file, so audio is deliberately omitted from the list while the endpoint still accepts audio parts. Consequence: the AGENTS.md prose "There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11" is STALE — frank-small is the fleet's only audio-capable endpoint. When routing an audio task or fact-checking the docs, trust the models.yml comment + the Gemma4 projector over the AGENTS.md one-liner. (frank-large = text only, no vision projector; frank-medium = text+image via the Gemma4 merge path; none do video.)
-Speculative decoding is only worth deploying when you check the ARITHMETIC against the incumbent, not when it "sounds free". Durable method (verified on Dankclaw 2026-09-16): round cost = K·d_s + ~1.1·t_s (d_s = drafter per-token step, t_s = target per-token step); tokens/round A = (1−p^(K+1))/(1−p); rate = A/round; compare against the *incumbent's* rate (a target with its own MTP/EAGLE head is already speculating). Two structural facts that decide most cases: (1) a PLAIN autoregressive drafter spends K sequential forwards per round, so it must be 10–50× faster than the target to pay — a 27B drafting for a 177B MoE (which is sparse and therefore fast) fails: measured 12 ms/drafter-token vs 67 ms/target-step gives break-even p ≈ 0.70 at K=3; (2) trained multi-token heads (MTP/EAGLE/DFlash) draft K candidates in ~one pass, which is why they are the drafter of record. Acceptance reality: the fleet's TRAINED z-lab DFlash2 drafter accepted only ~41% at depth 7 on the matched 27B target — a bar external/cross-finetune pairs won't clear. Also durable: cross-device speculation is cheap (token ids once per round, not activations) and the drafter's VRAM on a discrete card is an ADDITIVE pool, unlike an APU iGPU whose GTT window is a mapping of host LPDDR (no extra capacity or bandwidth). Cross-model pairs require a byte-identical tokenizer — cheap to verify by hashing the GGUF tokenizer.ggml.tokens list (Dankclaw's Qwen3.8-27B and Qwen3.8-Flash-Next GGUFs matched exactly: vocab 248320, merges 247587, pre=qwen35, same sha256 over the first 2000 tokens), but tokenizer identity alone does not make a drafter cheap or accepted. _(context: Dankclaw 2026-09-16: user asked whether a 27B on the 3090 could draft for Flash-Next on the iGPU, then whether a DFlash drafter could be hooked to halogen. Verified halogen's drafter surface is closed (serve_api.py DRAFTERS/DRAFTER_CAP, live 400 for dflash2), llama.cpp exposes --spec-draft-device/-devd, and computed break-even tables from measured step times (frank-small 11.96 ms/token; halogen ~6)_
-Fleet model input modalities (authoritative source: /home/blank/.omp/agent/models.yml input: field): frank-large = text only (no vision projector); frank-medium = text + image (Gemma4 vision path, verified on held-out images); frank-small = text + image + audio. IMPORTANT correction: frank-small's audio is NOT in models.yml's input list because OMP 18.0.6's schema only accepts text|image (a failed schema check drops the WHOLE models file), but the Gemma4-12B-QAT projector's audio encoder DOES accept input_audio parts — verified at the 2026-09-17 deploy (an input_audio WAV transcribed verbatim). So the AGENTS.md line "There is no audio input anywhere in the fleet" is STALE/wrong for frank-small: audio left with the FLM NPU endpoint for the other models, but the Gemma4 swap brought it back on frank-small specifically. _(context: serverAppConfigs 2026-09-26: adding per-model input modalities to the homepage LLM cards; cross-checked models.yml vs the stale AGENTS.md audio claim)_
-llama.cpp speculative decoding, durable rules learned 2026-09-16 on Dankserver (frank-medium, Qwen3.8-27B UD-Q8_K_XL + zLayer split): (1) CHAINING IS GATED BY ctx_dft — common/speculative.cpp adds the draftless ngram family unconditionally but gates draft-mtp/draft-dflash/draft-dspark/draft-eagle3 on a draft CONTEXT existing, so a chain may hold one context-backed drafter plus any number of ngram types (draft-dflash,ngram-mod works), never two context-backed ones (draft-mtp,draft-dflash dies at load: "context type MTP requested but model doesn't contain MTP layers" — MTP builds against whatever model the draft context holds). Fixed internal order: ngram types first, then drafters, so CLI order is irrelevant. (2) MTP's head lives in the TARGET GGUF (blk.n.nextn.*) and only loads when MTP IS the draft model; with draft-dflash those tensors show as "unused tensor blk.64.nextn..." which is correct behaviour, NOT a truncated download. (3) ngram-mod prints NO startup banner in this build — a working chain looks unengaged in the journal; identify it behaviourally (draft_n0 only where text repeats) or via the per-task "draft acceptance = ... mean len = ..." line. It costs no VRAM (64 MiB host RAM). (4) Its window decides which traffic it can see: the fleet's 48/64 (n_min/n_max) wins because shorter windows propose on ordinary prose fragments, and rejected proposals still cost verification (43.9 - 36.0 tok/s); measured chain result: 2.6x the drafter alone on prompt-echo traffic, byte-identical to it on prose. (5) Measuring this is shape-driven and easy to get wrong: a "repeat verbatim" prompt with thinking ON puts the copy inside reasoning_content, so content is empty, cross-config identity checks compare '' to '', and you "verify" an echo you never looked at; disable thinking per request (chat_template_kwargs.enable_thinking=false) for echo/structured shapes. (6) Any harness that stops a production unit must restore it in a finally — mine crashed mid-run (a botched edit del_(context: serverAppConfig 2026-09-16: frank-medium moved from vllm-radiance replicas to mainline llama.cpp (layer split across 2x R9700, DFlash2 drafter) and then to a chained draft-dflash,ngram-mod. Findings live in serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md and AGENTS.md; harness is scripts/bench-spec-ab.py.)_
-serverAppConfigs homepage LLM cards: the three LLM status cards (frank-large/medium/small) in shared/homepage/services.json carry a details object {role, spec, inputs, use} rendered by app.js descHTML() as a 4-tier hierarchy — bold pale-green .desc-role / bright-green .desc-spec / bright-green .desc-inputs ("Inputs: a · b") / dim .desc-use ("Use: …") — falling back to the plain description string for all other cards. The desc-* CSS classes live in style.css and each needs a bare-mode (html[data-bare=1]) reset. Deploy with ./scripts/deploy.sh dankclaw --service homepage (the deploy_homepage handler copies the 6 assets to /srv/homepage, regenerates deployed.js, restarts the status timer). The /docs model guide is the #models section of shared/homepage/docs.html and must be kept in sync with the cards — it once drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8 ~31 GiB) and had to be corrected to Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8 19.4 GiB._(context: serverAppConfigs 2026-09-26: reworked the homepage LLM model cards into structured role/spec/inputs/use lines)_
-Speculative-decoding interfaces on the Dank fleet (verified 2026-09-16): HALOGEN (frank-large, iGPU) has a CLOSED drafter set — DRAFTERS = {"serial": 0, "mtp": 1, "dflash2": 2} in /halogen/tools/serve_api.py, wire values in the front-end→engine GEN protocol; DRAFTER_CAP gates mtp/dflash2; an unknown name is a deliberate 400 ("drafter 'x' unknown; use serial | mtp | dflash2"), and the entrypoint states there is no separate draft checkpoint for this model, so no external drafter can attach. Selection: per-request drafter field (omitted → engine applies the HALOGEN_DRAFTER env default), plus the image's own bench drafters max_tokens effort reps / sweep -d serial,mtp modes. Our deployment already runs the mtp drafter: engine verbose lines read mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round. LLaMA.CPP (frank-small, 3090) is the opposite: --spec-type takes a COMMA-SEPARATED LIST (none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,...), so drafters COMPOSE rather than replace, and there is a full external-draft surface (-md/--model-draft, -ngld/--gpu-layers-draft, -otd/--override-tensor-draft, --spec-draft-n-max, -ctkd/-ctvd, draft-cpu-moe/ncmoe). The RULE that generalizes: speculative decoding only exists inside the process that owns the target's KV, because verification needs a forkable/roll-back-able cache; hence "use device A as a drafter for device B" is only expressible as one llama.cpp process with both backends (needs a CUDA+Vulkan build — ours has GGML_VULKAN=OFF, GGML_RPC=OFF), and never as an HTTP-level arrangement. Also: the fast device belongs on the TARGET (the expensive stage), not the drafter. _(context: Answering "is it possible to use the 3090 as a drafter?" for Dankclaw's two endpoints (2026-09-16). Read from the halogen image's serve_api.py + entrypoint.sh and this llama.cpp build's --help.)_
-serverAppConfigs homepage (shared/homepage/): the LLM status cards use a structured details object in services.json — {role, spec, use} — rendered by app.js descHTML() as a bold pale-green role line, a bright-green spec line, and a dim "Use:" line (.desc-role/.desc-spec/.desc-use in style.css, plus a bare-mode reset). descHTML() falls back to the plain description string for any card lacking details (the other ~32 services). So: structured card = add a details object; plain card = add description. The e2e (10-homepage/11-admin) asserts only on .panel .name and a.open, NOT .desc content, and homepage_config.bats' schema test checks only required fields (name/category/section/probe) — so details is a safe optional extra field. Gotcha: the /docs model guide (docs.html) is a SEPARATE hand-maintained copy of the model facts and had drifted to retired hardware (it listed frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k and frank-medium as FP8 ~31 GiB, both stale); re-verify it against AGENTS.md ground truth whenever the models change. _(context: serverAppConfigs 2026-09-25: reworked the three LLM homepage cards from a single dense description string to a structured role/spec/use shape (details object + app.js descHTML + new CSS), neutralized the role labels, and fixed the drifted /docs model guide.)_
-Dankclaw's Strix Halo (Ryzen AI Max+ 395, Radeon 8060S/gfx1151) does NOT have a second memory pool for the iGPU — it has two *windows* onto the one 121.2 GiB LPDDR5X pool (8000 MT/s x 256-bit = 256 GB/s, verified via dmidecode; the 3090's 24 GB GDDR6X is 936 GB/s and genuinely additive). Probe: cat /sys/class/drm/card1/device/mem_info_vram_total = 2.0 GiB (BIOS UMA carve-out, genuinely private), .../mem_info_gtt_total = 114.0 GiB (GTT aperture mapping SYSTEM pages on demand); rocm-smi --showmeminfo all shows the same plus GTT Used, which tracks the engine's device-side footprint. Decisive proof that GPU memory here IS host RAM: halogen's own fit check is written 121.2 - 67.7 resident weights - reserve = device budget, and its ledger's "68.0 GiB locked in RAM + 11.0 KV pool + 16.5 working" shows up as GTT Used while the CPU counts the same pages. Consequences for anyone proposing a split: (1) capacity is not additive — moving layers "onto the iGPU" maps the same LPDDR pages through GTT, so a 117 GB model (84 GB experts + 33 GB PLE) still needs 117 GB; only the 2 GiB carve-out is private; (2) bandwidth is not additive — both engines share the 256 GB/s; (3) fencing more via amdgpu.gttsize (parm verified: "Size of the GTT userspace domain in megabytes") or the BIOS frame-buffer size subtracts from CPU-visible RAM, so it partitions rather than adds. This is why the split that paid on this box was device-vs-host (3090 GDDR6X for attention/dense/KV, LPDDR for experts via -ot exps=CPU), why iGPU-as-a-pipeline-stage buys nothing, and why "second memory pool" should be stated as "second compute engine over the same bus" — the iGPU's real value is as a whole-model runtime (halogen's ~4x prefill over llama.cpp on this SKU), not as a peer in a multi-device split. _(context: Answering "why is the iGPU not a second memory pool?" and "why was pipeline parallelism not attempted?" for Dankclaw (2026-09-16). Verified live on the host plus dmidecode/modinfo; the substantive correction was that the iGPU has addressable windows (2 GiB carve-out + 114 GiB GTT of a 121.2 GiB pool) but no independent capacity or bandwidth.)_
-Jellyfin 12.1 removed the legacy POST /Users/Auth/Password route (and its LegacyEmbyRouteRewriteMiddleware). Probing that dead route returns a SILENT 401 with ZERO plugin/UserManager log lines even at Debug level — it reads like an LDAP/auth failure but is just a dead route. The modern password-auth endpoint is POST /Users/AuthenticateByName, which REQUIRES the header Authorization: MediaBrowser Client="...", DeviceId="...", Device="...", Version="..." (no header → 401). The LDAP-Auth plugin runs on this modern path. So a "Jellyfin LDAP 401 after upgrade" that shows no plugin log is almost always a test hitting the removed legacy route, not a brokenintegration — verify with the modern endpoint + MediaBrowser header before touching plugin config. (Dankserver Jellyfin 12.1.0, 2026-09-25; alex authenticated HTTP 200 via AuthenticateByName with full plugin flow: service bind → search → ldapUid → user bind → token.) _(context: Dankserver Jellyfin 12.1.0 LDAP-Auth 401 diagnosis — the 401 was a test artifact from probing the removed legacy /Users/Auth/Password route; the web UI's modern /Users/AuthenticateByName endpoint works fine.)_
-frank-medium on dankserver (2x Radeon AI PRO R9700, gfx1201) is now one mainline-llama.cpp process, layer-split across both cards, and the reason is worth remembering: **vLLM cannot use both cards for one model here in ANY of its parallelisms** — TP needs an allreduce and PP's stage handoff is GroupCoordinator.send_tensor_dict, which sends metadata over the gloo CPU group but the hidden-state TENSORS over the RCCL device group (vllm/distributed/parallel_state.py, the comment "run on the device backend where is_completed() is reliable"). That is the same p2p primitive this host cannot dispatch, because ROCr refuses a kernel that declares a hidden hostcall buffer on a device without PCIe atomics (gate at rocclr/device/rocm/rocvirtual.cpp, case KernelParameterDescriptor::HiddenHostcallBuffer), and card #2 sits behind the B650 chipset (AtomicOpsCtl: ReqEn-, bridges Routing-). So "use PP to dodge the atomics problem" is exactly backwards. **llama.cpp's --split-mode layer is the path that works**: layers are distributed and the boundary activation is a per-backend copy, once per forward pass — no collective is ever launched. Measured here: 1991 t/s prefill / 18.8 tok/s decode layer-split vs 272 / 26.3 tensor (butterfly fallback). The ~$30 M.2→x4 riser into the free CPU-attached M2_2 is still the only way to get TENSOR parallelism (≈+40% decode), not the way to "use both cards" — that battle is already won in software. Three operational traps from the same cutover, all measured: 1. **rocBLAS/hipBLASLt bundle depth.** Bundling a ROCm runtime beside a llama.cpp binary with cp -r SRC DST/ where DST does not exist creates DST as a copy of SRC and flattens one level; the engine then loads 29 GiB of weights, answers /health, and dies on the FIRST request with rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1201. Device code must live at $ORIGIN/../lib/rocblas/library and .../hipblaslt/library/arch. 2. **An env fi _(context: 2026-09-16 frank-medium cutover on dankserver: vLLM (radiance MXFP4 replicas, then an FP8 + PP attempt) → mainline llama.cpp fb27a525 layer split + DFlash2, 262144 ctx/1 slot. Produced scripts/build-llama-rdna4.sh, scripts/download-frank-medium-gguf.sh, scripts/frank-medium.{env,service}, scripts/serve-frank-medium.sh, scripts/bench-spec-ab.py, tests/frank_medium_unit.bats, serverAppConfigs-docs/L)_
-Deploying a quadlet image change on the Dank fleet (dankclaw 192.168.0.143 / dankserver 192.168.0.235): pulling the repo does NOT update the running container, because the installed unit in ~/.config/containers/systemd/name.container is GENERATED state, not the repo file. A plain systemctl --user restart name restarts from the STALE installed unit (old image) — I hit this live (image IDs didn't change after a bare restart). To deploy a pinned-image bump: commit+push → git pull --ff-only on host → cp host/svc/quadlet/name.container ~/.config/containers/systemd/ → systemctl --user daemon-reload → systemctl --user restart name → verify podman inspect name --format {{.ImageID}} matches the freshly-pulled image. For local-build images (caddy/alloy/loki/comfyui, tagged :latest) no unit reinstall is needed (same tag), BUT deploy-remote.sh's build_compose_image gates on a cfhash label (sha256 of the Containerfile) and SKIPS the rebuild when the Containerfile is unchanged — so a base-image refresh needs a manual podman build --no-cache -t image -f containerfile . first. Gotcha: the hermes gateway container is named systemd-hermes, not hermes (no ContainerName= in its quadlet), so podman inspect hermes fails — use podman ps --filter name=hermes. anubis is pinned at v1.27.0 (latest stable; only pre-releases beyond). _(context: serverAppConfigs 2026-09-25: fleet-wide container update (pinned bumps + local-build rebuilds) across dankclaw + dankserver)_
-Dankclaw (GMKtec EVO-X2, Strix Halo) has a hard 4-lane budget for BOTH M.2 2280 PCIe 4.0 x4 slots, and the RTX 3090 lives in one of them on an M.2-PCIe riser: kernel boot logs "pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)". The GPU's root port 00:03.1 advertises LnkCap Width x8 (APU GPP bridge IP max) but only 4 lanesare wired at the socket, so the link trains x4; port 00:02.5 (NVMe) is x4 too. Therefore swapping the SSD and the GPU between the two M.2 slots cannot widen the GPU link - both are x4, and the GPU is already behind the wider-capable bridge. THE TRAP: lspci shows the idle NVIDIA GPU as "LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded)" because the NVIDIA driver downshifts the link to Gen1 for power when the GPU is idle - this is NOT a fault. Prove the real link by sampling NVML under load: nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv reads 4,4 while a generation request is in flight (and 1,4 at idle). Corroborating evidence: LnkCtl2 Target Link Speed: 16GT/s on both ends, and nvidia-smi hostmax/gpumax = 4. Also: this box's SMBIOS type-9 slot table is junk (it labels the RTL8125 Ethernet port and the SD reader as "M.2 Socket 3" slots and lists a phantom free slot at bus address 0200:00:03.2, a PCI domain that does not exist) - trust live lspci, not dmidecode -t slot. There is no OCuLink and no add-in slot on the EVO-X2; two M.2 2280 slots is the whole expansion story. _(context: 2026-09-16, user asked whether swapping the dankclaw SSD and the RTX 3090 between M.2 slots would let the GPU run at full speed. Verified live over ssh with lspci -vv, journalctl -k boot log, dmidecode -t slot, and NVML sampling during a real frank-small generation.)_
-Updating a container image on the Dank fleet (dankclaw/dankserver, rootless quadlet) — a naive git pull + systemctl --user restart silently keeps the OLD image, for two distinct reasons: (1) QUADLET INSTALLED-UNIT STALENESS: the repo quadlet file (e.g. dankserver/plex/quadlet/plex.container) is the source of truth, but the INSTALLED unit at ~/.config/containers/systemd/name.container is generated state that git pull does NOT touch. Bumping Image= in the repo, pulling, and restarting restarts the container from the STALE installed unit (old image). Fix: cp repo/host/svc/quadlet/svc.container ~/.config/containers/systemd/ + systemctl --user daemon-reload + systemctl --user restart svc. Verify by comparing podman inspect c --format {{.ImageID}} against the freshly-pulled image ID — if they differ, the unit was stale. (2) LOCAL-BUILD CFHASH GATE: the deploy functions (deploy_caddy, build_compose_image in scripts/deploy-remote.sh) rebuild only when the Containerfile's sha256 (the cfhash image label) changed. A base-image bump (caddy:2, grafana/loki:latest, grafana/alloy:latest, the rocm/vllm comfyui base) leaves the Containerfile byte-identical, so the gate SKIPS the rebuild and the old base stays. To force a refresh: podman build --no-cache -t image -f containerfile. from the repo root (caddy=franksdank/caddy-porkbun:latest, alloy/loki=franksdank/{alloy,loki}:latest — loki needs --format docker for its HEALTHCHECK, comfyui=comfyui-rocm:gfx1151). hermes (image localhost/hermes-agent:latest, Pull=never): podman pull docker.io/nousresearch/hermes-agent:latest + podman tag ... localhost/hermes-agent:latest + restart the trio (hermes, hermes-dashboard, hermes-webui). Gotcha: the hermes GATEWAY container is named systemd-hermes (not hermes), so podman inspect hermes fails. anubis is pinned at v1.27.0 in deploy-remote.sh (latest stable; GHCR tag-list API is capped at 1000 unsorted tags — use the GitHub releases API to find the latest). Excluded from "update to latest": frank- _(context: serverAppConfigs 2026-09-25 fleet-wide container update: pinned bumps (authentik 2026.8.3, tautulli ls245, calibre-web ls402, plex 1.43.4) + local rebuilds (caddy, alloy, loki, comfyui-rocm, hermes x3). Hit the installed-unit staleness trap on dankserver (restarted with old image IDs until the units were re-copied + daemon-reloaded).)_
- Dankclaw (GMKtec EVO-X2, NucBox_EVO-X2, Strix Halo APU) has exactly TWO M.2 2280 slots, both PCIe 4.0 x4 on the SoC root complex (no chipset), SATA M.2 unsupported, 8TB max per slot. Both are OCCUPIED: slot 1 = 2TB AirDisk NVMe (MAXIO MAP1608 DRAM-less) at full Gen4 x4 (16GT/s), holding the root FS and all model weights; slot 2 = RTX 3090 (frank-small) in a dock/riser, and that dock negotiates only **Gen1 x4** (2.5GT/s, downgraded from the card's x16 Gen4 LnkCap) — the dock/riser is the likely cap, ~$30 Gen4 riser class of fix. Consequence: unlike dankserver (free CPU-attached M.2_2 = R9700 riser target), dankclaw has NO free M.2 slot, so there is no multi-GPU-via-M.2 path on this box — the only other GPU is the integrated gfx1151. PCI addresses: NVMe c3:00.0, 3090 c4:00.0. _(context: 2026-09-16, live ssh dankclaw: lsblk/lspci/dmidecode; user asked "tell me about the m.2 slots on dankclaw".)_
- Dankclaw guard-chained vhost 400 diagnosis (Anubis → Authentik embedded outpost, e.g. marreta.franksdank.cloud): a 400 where the APP's own logs show NO user request (only periodic health probes) is the embedded outpost's handle_auth_callback returning 400 with the warn "auth callback without a valid session cookie" (src/outpost/proxy/application/handlers/mod.rs). The outpost sets a session cookie on the external host during auth_start; the callback must carry it back. If the browser (observed: Firefox) doesn't send it — private/incognito window, cookie blocking (ETP Strict / extension), or stale site data — the callback 400s and the app never sees the request. The cookie name is 'authentik_proxy_' + first 8 hex of SHA256(provider.client_id) (compute: hashlib.sha256(client_id).hexdigest()[:8]); it is SameSite=Lax, Secure, HttpOnly, host-only when provider.cookie_domain is empty — and the callback is same-site (auth.→vhost., same registrable domain) so Lax allows it, meaning the server is fine. Confirm server health by running the single-vhost e2e spec (clean Chromium drives the full chain; a pass means the cause is browser-side). Two server-side gotchas that ALSO produce the same 400: provider.access_token_validity unset/None → session_max_age()=0 → zero-max-age cookie the browser discards; and cookie_domain set to a domain that doesn't match the callback host. ProxyProvider is at authentik.providers.proxy.models; on this fleet the marreta provider is pk 13 named 'marreta-proxy' (NOT 'marreta'). The outpost runs inside the authentik-server container (logs via journalctl --user -u authentik-server), reached as remote = the Anubis container IP. _(context: serverAppConfigs 2026-09-25: user reported "400 from marreta"; traced to the embedded outpost's handle_auth_callback missing-session-cookie 400, verified server healthy via e2e spec 16-marreta (3/3 pass), root cause = Firefox not sending the authentik_proxy_6536c831 session cookie on the callback.)_
-On a host where the second GPU cannot dispatch RCCL (no PCIe atomic routing on its bridge chain), vLLM cannot use both cards for one model in ANY of its parallelisms: tensor parallelism needs a per-layer allreduce, and **pipeline parallelism needs the same class of primitive** — vLLM's PP stage handoff sends metadata over the gloo CPU group but the hidden-state tensors over the *device* backend (vllm/distributed/parallel_state.py, isend_tensor_dict/recv_tensor_dict; the comment at the tensor-send path says these "run on the device backend where is_completed() is reliable"). So "use PP to dodge the atomics problem" is exactly backwards. The working software path for two cards on one model is **llama.cpp --split-mode layer**, which launches no collective at all: layers are distributed in order and only the boundary activation crosses PCIe, once per forward pass. Also note the kyuz0/amd-r9700-vllm-toolboxes images are stale for this purpose (Docker Hub latest = 2026-06-13, dev = 2026-08-12 vLLM 0.27.1 with dflash/ but no dflash2/), and that switching to such an image changes nothing about the atomics gate because the gate lives in ROCclr (rocclr/device/rocm/rocvirtual.cpp, case KernelParameterDescriptor::HiddenHostcallBuffer → !pcie_atomics_ → hipErrorIllegalState) and the shipped RCCL build is the same one that already failed. Measured on dankserver 2026-09-16: layer split 1991 t/s prefill / 18.8 tok/s decode vs tensor 272 / 26.3 (butterfly fallback). _(context: serverAppConfig 2026-09-16: frank-medium cutover from a vllm-radiance MXFP4 replica pair to mainline llama.cpp layer-split across two R9700s (the user asked first for vLLM + PP, then approved the llama.cpp pivot).)_
-In authentik (2026.8.0, Dankclaw), self-serve password change is a PROMPT FIELD, not a stage. The UserWriteStage writes whatever the preceding PromptStage collected: for key password it calls user.set_password(value) (stages/user_write/stage.py:116). So the fix is a password field on the default-user-settings prompt (field_key='password', type='password', required=False) — NOT a PasswordStage. A PasswordStage is an AUTHENTICATION stage: its validate_password runs authenticate() against the backends to check the CURRENT password; it never sets a new one. (A PasswordStage was added then removed for exactly this reason on 2026-09-25.) Two gotchas: (1) a blank password field must be guarded — set_password('') leaves the account with no password, so bind an ExpressionPolicy to the prompt via PromptStageValidationPolicy that denies when prompt_data['password']=='' (allow when set/absent); (2) a prompt field is a Prompt row linked through PromptStageField(prompt_stage, prompt) — the order comes from Prompt.order, and there is no PromptField model. Verify the write path by driving UserWriteStageView.update_user on a throwaway user (it only modifies in-memory; call user.save() after, as dispatch() does). The self-serve flow /if/flow/default-user-settings-flow/ is open to ANY authenticated user (its authorization policy only gates email/name/username changes, not password); the admin Directory→Users screen is separate and admin-only.
-frank-medium (dankserver, 2x R9700 gfx1201) is now ONE mainline-llama.cpp process layer-split across both cards — and the decisive insight is that **vLLM's pipeline parallelism is not an escape from the PCIe-atomics wall**: PP sends its hidden-state handoff on the *device* backend (RCCL send/recv, vllm/distributed/parallel_state.py ~1117-1119), the same primitive card #2 cannot dispatch (AtomicOpsCtl: ReqEn- behind Routing- chipset bridges). llama.cpp's --split-mode layer needs no collective at all (layers distributed, boundary activation is a per-backend copy), so it is the one software path that puts both cards on one model: 1991 t/s prefill layer-split vs 272 tensor (butterfly fallback), and it delivered 1.0-1.4k tok/s prefill at 4k-140k tokens with 43.8 tok/s decode. Switching to the kyuz0 R9700 toolbox image changes nothing about this — its librccl is byte-identical in version to the radiance image's (RCCL 2.30.4 compiled with ROCm 7.14.0.0-9999-2b22ab01), and the toolbox's published tags are stale for DFlash2 (latest=2026-06-13, dev=2026-08-12 = vLLM 0.27.1, has spec_decode/dflash/ but no dflash2/); DFlash2 landed in llama.cpp via PR #27342 and is in main at fb27a525. Three durable traps from this deployment: (1) **Bundling a ROCm runtime next to a llama.cpp binary**: cp -r SRC DST/ with a missing DST creates DST as a copy of SRC and drops one directory level — rocBLAS/hipBLASLt look for device code at a fixed depth (../lib/rocblas/library, ../lib/hipblaslt/library/arch), so the flattened copy loads the model, answers /health, and dies on the FIRST request with rocBLAS error: Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201. Declare the destination then copy contents. (2) **DFlash2 in llama.cpp**: --spec-type draft-dflash + --spec-draft-model DFlash2 gguf; the init line block_size=8, n_extract=5, sample_from_anchor=true is the proof it took the DFlash2 path, n_max must be = block_size-1 = 7, and with draft-dflash the target _(context: 2026-09-16, dankserver frank-medium cutover: user asked for kyuz0 R9700 toolbox vLLM + d0xin FP8 + z-lab DFlash2 + pipeline parallelism; investigation showed PP needs the same RCCL p2p that the atomics wall blocks and that a 29.8 GB FP8 checkpoint cannot fit one 31.86 GiB card, so the user pivoted to llama.cpp + ajgazin UD-Q8_K_XL + the DFlash2 GGUF drafter + layer split as the sole LLM runtime. R)_
-authentik 2026.x default-user-settings-flow (the self-serve account page, https://auth.domain/if/flow/default-user-settings-flow/) ships with ONLY a PromptStage + a UserWriteStage (name/email/locale) — NO PasswordStage, so users cannot change their password. To add it: create a PasswordStage (backend authentik.core.auth.InbuiltBackend, the "standard password" backend) and bind it to the flow. Two ORM gotchas (verified on dankclaw authentik-server 2026.8.0): (1) Flow.stages.add(stage) FAILS with "Can't bulk create a multi-table inherited model" because the through model FlowStageBinding is multi-table-inherited — instead create the row directly: FlowStageBinding.objects.create(target=flow, stage=ps, order=N) where N is max(existing orders)+1; (2) PasswordStage.objects.create(name=...) alone hits a NOT NULL violation on the backends ArrayField — you must pass backends=[...]. Valid backend values come from authentik.stages.password.__init__: BACKEND_INBUILT="authentik.core.auth.InbuiltBackend", BACKEND_LDAP, BACKEND_APP_PASSWORD (TokenBackend), BACKEND_KERBEROS. Flow execution order is the FlowStageBinding.order field, NOT the M2M insertion order (flow.stages.all() returns a different order — don't trust it). To verify a stage is in the executed plan WITHOUT a browser: FlowPlanner(flow).plan(request) (request.user set) → plan.bindings lists the stages in order; FlowPlanner.__init__ takes only the flow, user/request go to .plan(request). Note: a headless browser will NOT render authentik flow stages (the Lit app renders via XHR after the shell, leaving only "Powered by authentik") — verify via FlowPlanner, not the DOM.
-Dankclaw/Dank fleet serving conventions established 2026-09-16 (serverAppConfig), all verified live: 1. PORT RELEASE RACE (deployer convention). Starting an LLM replacement while the predecessor still holds the published port fails: pasta (rootless podman's userspace net stack) binds the published port at container CREATE, and the ik/LLM server logged "Deactivated successfully" ~1s before pasta still reported Address already in use (unit exit 126); only Restart=on-failure recovered it. Repo fix: wait_port_free port in scripts/deploy-remote.sh polls ss -Htan "sport = :$port" (listener + accepted + TIME_WAIT) and fails loudly with the socket table. Ordering matters: stop this service's OWN quadlet BEFORE waiting, or a redeploy deadlocks against itself. 2. THINKING-LEVEL TRAP. A model's own chat template can reject the harness's reasoning level: Qwen3.8's template accepts only xhigh|medium|low for reasoning_effort and raises — HTTP 500 naming Jinja, not the model. Subagents run at high, so every subagent on that endpoint died until compat.reasoningEffortMap folded minimal/low→low, medium→medium, high/max→xhigh in models.yml. Halogen accepts the wider set (verified minimal/low/medium/high/xhigh all 200), so per-provider entries legitimately differ. Corollary: probe each reasoning level against a new endpoint before routing roles at it. 3. TOOLS DEPEND ON THE RUNTIME, NOT THE MODEL. The ik_llama.cpp runtime rejected every request carrying tools (500 tools param requires --jinja flag), silently breaking every tool-using subagent on that endpoint; mainline llama.cpp defaults --jinja on, and halogen accepts tools and returns tool_calls. Always test a tool-call round trip before pointing task/smol roles at an endpoint. 4. TWO ENDPOINTS, ONE HOST = two providers in .omp/agent/models.yml (one provider per HOST:PORT; the file cannot express same-host-different-port), plus a separate modelRoles entry per role. The harness config lives with the harness on the dev box (/home/blank _(context: Dankclaw re-split 2026-09-16: halogen restored as frank-large (pool 400384 = 2x200192, 2 slots, MAX_TOK 24576 for the co-tenant) and frank-small created (mainline llama.cpp on the RTX 3090, Qwen3.8-27B GSQ-RCO IQ3_S -mtp + BF16 mmproj, port 7771). Failures: pasta bind race on cutover, subagent tools 500 from ik, reasoning_effort high rejected by the Qwen3.8 template.)_
-Running a single Playwright spec via the fleet e2e runner requires the file path to come BEFORE --project: tests/e2e/run.sh specs/18-openwebui-sso.spec.ts --project standard. The reverse order (--project standard specs/...) fails with Project(s) "specs/..." not found because playwright parses the positional file path as a project name. Spec 18 is in the "standard" project (testMatch /(10|12|13|16|17|18)-.*\.spec\.ts/), which depends on the "setup" project (00-auth.setup.ts) that logs in and writes .auth/qa-e2e.json + .auth/qa-e2e-admin.json. The runner needs tests/e2e/.env.local (git-ignored) present and auto-installs chromium. This lets you verify one SSO/guard spec against the live fleet without running the whole (slow, Mac-mini-dependent) suite. _(context: serverAppConfigs 2026-09-24: ran 18-openwebui-sso.spec.ts in isolation toverify the SSO-only flip against the live fleet)_
-Dankclaw now serves TWO LLM endpoints and the split produced three durable traps. (1) SHAPE: frank-large = halogen-flash-server 0.5.8 rootless Quadlet on :7770 (gfx1151 iGPU), retuned 2026-09-16 to HALOGEN_CTX=200192 + KV_POOL_POSITIONS=400384 (pool == 2 × ctx, so two 200k conversations are exactly resident) + KV_SLOTS=2 + MAX_TOK=24576 + HOST_RESERVE_GIB=20, VISION_TOWER unset; frank-small = mainline llama.cpp 37b53fd root systemd unit on :7771 (RTX 3090), 131072 ctx / 1 slot / f16 KV, alias frank-small. Measured: halogen budget 121−67.7−20=33.5 vs need 27.2 (no clamp), 95.4 GiB held, 14.7 GiB left, co-tenant stall NOTE prints again with 4,350→2,199 compaction stalls; frank-small 22,362 MiB of 24,576. (2) PORT RACE: rootless podman publishes through pasta, which binds the published port at container CREATE — a just-stopped predecessor still holds it (exit 126, "Address already in use"), so the deployer must stop the service's own unit, then poll ss -Htan 'sport = :PORT' empty (listener + accepted + TIME_WAIT) before installing/starting, and must stop its own quadlet first or a redeploy deadlocks against the gate. (3) TEMPLATE/TOOL TRAPS that break subagents invisibly: ik_llama.cpp serving frank-large returned 500 tools param requires --jinja flag for every request carrying tools (every tool-using agent on that endpoint was broken and it read like a model fault; mainline llama.cpp 0.4.1 defaults --jinja on, halogen accepts tools); and Qwen3.8's own GGUF template accepts ONLY xhigh|medium|low for reasoning_effort and raises a 500 otherwise — harness subagents run at high, so models.yml needs compat.reasoningEffortMap {minimal:low, low:low, medium:medium, high:xhigh, max:xhigh} for any endpoint whose template is that strict (halogen accepts all five, so the two provider entries legitimately differ). One provider per HOST:PORT in models.yml, and one role per endpoint: smol/tiny/task/vision → frank-small, slow/plan/designer/advisor → frank-large. _(context: 2026-09-16: re-split Dankclaw's inference (halogen restored as frank-large; new llama.cpp frank-small on the 3090), live cutover + docs; three defects found by driving the fleet rather than reading config.)_
-Flipping OpenWebUI (dankclaw, ai.franksdank.cloud) to SSO-only requires changing BOTH the quadlet env AND the per-key config DB, because the config DB (webui.db config table) wins over env vars. Concretely: (1) add Environment=ENABLE_LOGIN_FORM=false and Environment=ENABLE_PASSWORD_AUTH=false to dankclaw/open-webui/quadlet/open-webui.container; (2) set ui.enable_login_form to false in the config table on the live data dir (/home/dank/Containers/open-webui/data/webui.db). If you only set the env, the DB override ui.enable_login_form=true silently keeps the password form showing. ui.enable_signup was already false. Verified 2026-09-24: the login view at /auth renders zero inputs and only the "Continue with authentik" button, and the OIDC round-trip lands signed in. To restore the password form: set ui.enable_login_form back to true (Admin Panel or DB) and drop the two env vars. Note the dankclaw host has NO sqlite3 CLI — use python3 -c "import sqlite3; ...". _(context: serverAppConfigs 2026-09-24: OpenWebUI SSO-only flip (commit 33cdeca) + config DB override flip)_
-dankserver's two R9700s cannot cooperate through RCCL, and the escape is a ~$30 piece of hardware. Causes, all verified: (1) ROCclr gates the hostcall buffer RCCL collective kernels need on a *queried* hardware property — Device::checkAtomicSupport() in rocdevice.cpp:181 reads LinkAttribute::kLinkAtomicSupport from the CPU fine-grain pool and only then sets info_.pcie_atomics_ = true; there is no else branch and no env/driver/sysfs override anywhere near the gate (grepped ROCclr for getenv/HSA_*/ROCR_*), and the value is consumed at rocdevice create time so setpci writes are useless. (2) The board's second x16 slot is chipset-attached (AtomicOpsCap: Routing- on both bridges, endpoint ReqEn-). ASUS TUF GAMING B650-E WIFI lane wiring from ASUS's own techspec: PCIEX16_1 = CPU x16; PCIEX16_2 = B650 chipset, x16 slot in **x1 mode**; M.2_1 = CPU PCIe 5.0 x4 (boot NVMe, occupied); **M.2_2 = CPU PCIe 4.0 x4 and free**; M.2_3 = chipset x4. No x8/x8 bifurcation on this board. So an M.2→PCIe x4 riser into M.2_2 is the cheapest route to two CPU-attached cards and therefore to real TP=2; a CPU x8/x8 board (X670E/X870E class) is the better-bandwidth alternative. Bandwidth is not the blocker: a 27B TP=2 allreduce at batch 4 moves ~10 MB per token-step, ~1.4 ms at Gen4 x4. (3) Empirically, RCCL launches NOTHING involving the chipset card: rank0 (CPU card) completes broadcast/all_reduce/barrier while rank1 fails, and dist.send/recv — the primitive pipeline parallelism would use — fails too, so PP=2 is dead as well, not just TP=2. (4) vLLM's one collective-free path (IPC custom allreduce, custom_ar=True compiled in the radiance image) is gated off for ROCm RDNA4 purely by a hardcoded CUDA-sm-keyed table CUSTOM_ALL_REDUCE_MAX_SIZES = {9.0, 10.0, 10.3, 10.7} while RocmPlatform.get_device_capability() returns DeviceCapability(major=12, minor=0) → '12.0' not in table → False; patching that table is NOT sufficient, because broadcast/barrier/all_gather on the TP init path have no non-RCCL _(context: 2026-09-16 dual-R9700 TP=2 research on dankserver (ASUS TUF GAMING B650-E WIFI, Ryzen AM5, 2x Radeon AI PRO R9700). Produced the ranked options and the source-level proofs.)_
-OpenWebUI (dankclaw, ai.franksdank.cloud, quadlet on bridge 10.89.3.0/24, published 3005) can do authentik OIDC SSO per the official integration docs, with these fleet-specific adaptations: (1) OpenWebUI's container cannot reach authentik-server:9000 by name (isolated bridge), but CAN reach https://auth.franksdank.cloud — Caddy publishes 0.0.0.0:443 and netavark DNATs published ports from other containers — so OPENID_PROVIDER_URL uses the public URL; (2) authentik runs 2026.8.0 (≥2026.5), so the provider needs the new typed Strict Authorization redirect URI (https://ai.franksdank.cloud/oauth/oidc/callback), not the old plain list; (3) OAUTH_CLIENT_SECRET must not enter the repo quadlet — use EnvironmentFile= pointing at a git-ignored host env file (Caddy-env pattern); (4) OpenWebUI persists config in /app/backend/data fill-missing-only, so on the existing data dir the OAuth settings must be set via Admin Panel → Settings, env vars only seed a fresh data dir; (5) rollout: enable SSO first, verify, then flip ENABLE_LOGIN_FORM/ENABLE_PASSWORD_AUTH=false — flipping first locks you out of the admin panel if the wiring is wrong. _(context: serverAppConfigs 2026-09-24: answering whether the goauthentik.io OpenWebUI integration docs apply to the fleet setup)_
-NEVER run RCCL/NCCL collective experiments on a live GPU host that is serving production. On 2026-09-16 I ran a 2-rank RCCL probe (torch.distributed, backend=nccl, both R9700s) on dankserver — the production LLM host — *while frank-medium was serving*, reasoning that the probe needed only ~200 MB of free VRAM and that a failure would just error out. It hit the known PCIe-atomics wall ('rank1 broadcast/all_reduce/barrier/send-recv FAIL', 'rank0 OK'), the process hung until my 200-300 s timeout killed it, and within ~15-20 minutes the whole host was hard down: no ARP reply (ip neigh FAILED from a peer), 100% ICMP loss from two subnets, every port closed, and it did not come back over 25+ minutes — i.e. powered off or kernel-wedged, needing a physical power cycle (consumer board, no BMC). A hung AQL dispatch / failed hostcall on the atomics-less card is a plausible cause of a GPU-driver wedge that takes the box down, and this host has a documented precedent (the 2026-08 pinned-host-buffer incident which 'starved co-tenants into unreachability until a power cycle'). The correct procedure for any multi-GPU collective experiment is the benchmarking policy: sudo systemctl stop frank-medium && sudo systemctl mask frank-medium first (mask, never disable), then experiment, then unmask and verify /health — and ideally experiment on a host that is not serving users at all. Two secondary lessons from the same hour: (1) when a host goes unreachable, ip neigh show ip from a peer distinguishes 'powered off / NIC dead' (FAILED) from 'kernel alive but wedged' — ARP usually still answers on a merely wedged box; (2) validate the *naming* of your own probes before reporting an outage: I briefly believed Gitea was down on dankclaw because systemctl --user is-active gitea returned 'inactive', but the unit is gitea-server and the container was 'Up 9 hours (healthy)' — a check against a non-existent unit name produced a false outage report. _(context: dankserver hard-down for 25+ min on 2026-09-16 during the dual-R9700 TP=2 research pass, ~20 min after running a bare 2-rank RCCL collective/P2P probe on its GPUs while frank-medium was live.)_
-When a subagent is told to write its report to local://name.md, the file lands in the GLOBAL dir /home/blank/.omp/local/name.md — NOT in the session-local dir (/home/blank/.omp/agent/sessions/session/local/). In the 2026-09-22 serverAppConfigs session, three subagent reports written to local:// all resolved to the global dir; a first read attempt at the session-local path failed and needed a find /home/blank/.omp -name file to locate them. Durable rule: read subagent local:// outputs from /home/blank/.omp/local/ directly; if not found there, find under /home/blank/.omp. _(context: serverAppConfigs session 2026-09-22: subagent outputs (appdocs-ai, research-job-search, research-popular-ai-apps) all landed in the global dir, not the session-local one)_
-In serverAppConfig, the highest-yield overnight work (4 of 4 passes found something real) came from one audit family: **surfaces whose signals were already wrong before I arrived**. Concrete recurring defects and their fixes: (1) Homepage status cards can be red *by construction* — the checker runs on Dankclaw, so it reaches only that host's loopback, its container names, and *published* LAN addresses; mariadb/valkey/Postgres probed container-internal DB ports and could never be up. Rule now in Infrastructure/Homepage.md: a card must be probeable from the monitor's host or be wip (Gluetun was the pre-existing example). (2) Prometheus jobs comfyui, services, gitea scrape HTML or a 404 /metrics — a scrape cannot express "is this HTTP service answering?"; remedy is the blackbox exporter (/probe?target=&module=), which also gives guarded vhosts an uptime signal since Prometheus cannot solve an Anubis challenge. (3) **Retiring a file makes every assertion that greps it pass vacuously** — with dankclaw/compose.yaml gone, four bats tests (observability_quadlet, comfyui_rocm, caddyfile×2, authentik dead var) passed by absence (a missing path makes grep exit 2 and ! succeed). Fix pattern: assert the invariant directly ("no dankclaw compose file competes with the quadlet units") *and* keep a conditional live check for a re-added entry — then prove teeth by recreating the file and watching the tests go red. (4) A linter must be validated against the resolver it models before acting on its output: my vault-link checker produced 45 false positives (Obsidian wikilinks are case-insensitive, [[folder/Note]] resolves by path suffix not stem, %20 in markdown links is URL-encoding) before reaching the 2 real breaks. It now lives at scripts/check-vault-links.py with a docs_completeness seam. (5) Your own change invalidates prose and unit comments: check the repo's rule files first — AGENTS.md claimed "ComfyUI runs CPU-only" (false since the GPU unit was restored) and "observability RUN _(context: serverAppConfig overnight session 2026-09-16: found a permanently-red homepage board, three unprobeable Prometheus jobs, four hollow tests after retiring dankclaw/compose.yaml, three dangling vault links, and a flatly wrong AGENTS.md lesson about ComfyUI. Suite 315 green, e2e 32.)_
-ComfyUI v0.37.0 (released 2026-09-21, tag v0.37.0) ships NATIVE Qwen-Image 2.1 support (CORE-423, PR #16400 — comfy/ldm/qwen_image21/model.py, model detection + supported-models registration). The Dankclaw ComfyUI image (comfyui-rocm:gfx1151, built from dankclaw/Containerfile.comfyui) currently bakes in CUSTOM GGUF patches (apply_qwen21_patch.py: VAE 4D→5D expansion, in_channels shape[0] read, QwenImage21 arch registration) for the Qwen-Image 2.1 GGUF setup. On the next image rebuild, check whether the native 0.37.0 path makes those patches redundant BEFORE re-applying them. Also: the repo now lives in the Comfy-Org org (comfyanonymous/ComfyUI redirects). _(context: serverAppConfigs vault fact-check 2026-09-22, AppDocsAI upstream research)_
-In serverAppConfig, grep-style bats seams and doc claims silently rot whenever a file they reference is retired — the audit that keeps finding real defects is "which signal here cannot be correct?" Three concrete traps, all hit on 2026-09-16: (1) VACUOUS ASSERTIONS — after dankclaw/compose.yaml was retired, four tests that ended in ! grep -qE '^\s+svc:' "$REPO_ROOT/dankclaw/compose.yaml" passed because grep exits non-zero on a MISSING FILE, so the negation succeeded. Fix pattern: assert the real invariant ([ ! -f "$cf" ]) in ONE place plus keep a live conditional check ([ ! -f "$cf" ] || ! grep ...) for a re-added entry, and TEETH-TEST by recreating the file and watching the tests go red. (2) PERMANENTLY-RED PROBES: homepage cards and Prometheus jobs probing container-internal or HTML endpoints can never report up (mariadb 192.168.0.235:3306, valkey/postgres 127.0.0.1:6379/:5432 unpublished; comfyui/services scrape / = HTML; gitea /metrics 404). Rule: a monitor signal must be probeable from the monitor's host or be marked wip/none — an always-red signal trains the operator to ignore the surface. (3) STALE CLAIMS AFTER YOUR OWN CHANGE: retiring a file or adding a feature invalidates prose — AGENTS.md was telling agents "ComfyUI runs CPU-only, no ROCm wheels for gfx1151" after ComfyUI had been restored with GPU access (an agent trusting it would re-break it), and it still said observability was "RUNNING BUT UNCONFIGURED" plus named halogen as the active frank-large runbook. Grep the repo for every reference to anything you retire/change, not just code. When auditing a docs vault, validate the LINTER against the resolver it models before acting: a naive Obsidian link checker reported 45 breaks of which only 3 were real (case-insensitivity, path-suffix resolution for [[folder/Note]], and %20 URL-decoding in markdown links were the three false-positive classes). scripts/check-vault-links.py (docstring records all three) plus the no unresolved links in the docs vault seam _(context: Overnight 2026-09-16 passes on serverAppConfig: homepage board audit (3 impossible cards), vault link integrity + checker, four hollow bats assertions re-armed, AGENTS.md/doc stale-claim sweep after retiring dankclaw/compose.yaml and adding alert rules.)_
-frank-medium image rebuilds on dankserver: build.sh invokes docker build (rootful docker), but the quadlet runs under ROOTLESS podman — a freshly built image is NOT visible to podman. Bridge it: docker save img:tag | podman load (lands as docker.io/name:tag), then podman tag name:tag localhost/name:tag because rootless quadlets cannot resolve registry-prefixed refs. The 2026-09-22 MXFP6 image was built this way as ggz14/vllm-radiance-mxfp4:0.13.0-e060895 → localhost/radiance-vllm:0.13.0-e060895. The /patches clone must be at the same commit as the image (e060895) so the entrypoint re-applies the matching patches.
-Fleet convention: a monitoring/status signal must be probeable from the vantage point of whatever produces it, or it must be marked intentional (wip) — an always-red signal teaches the operator to ignore the surface it lives on. Instances found on 2026-09-16: (1) the homepage status checker runs on Dankclaw, so it can reach that host's loopback, its container names, and published LAN addresses — but NOT container-internal ports of another host; three cards (mariadb probing 192.168.0.235:3306, valkey 127.0.0.1:6379, Postgres 127.0.0.1:5432) were red by construction while the databases were healthy, and publishing a DB port on the LAN to make a badge green is not a trade this fleet makes, so they became wip with the reason in their description (Gluetun/WireGuard-UDP was the pre-existing example of the same call); (2) four Prometheus jobs read down and three always will — comfyui and the two services probes scrape HTML pages (Prometheus refuses to parse HTML, error 'unsupported Content-Type text/html'), and gitea's /metrics is a 404 — none is an outage, and the tempting fix (deleting the job) is refused because an absent scrape and a healthy one look identical; the documented remedy is a blackbox exporter in probe mode, and Gitea's metrics enabled or the job repointed; (3) a bats test failed on a clean tree for many sessions ('comfyui Containerfile ...') because its selector did find | head -1 and read the wrong file — a permanently red test trains everyone to ignore red, so it was fixed at the root rather than excused. Diagnostic pattern that works: for every signal that is not green, ask what produces it and from where; if the producer cannot reach the target, it is a config defect, not an outage — classify before touching. When fixing, prefer honest state (wip / documented) over widening exposure or deleting the signal. _(context: Dankclaw/Dankserver 2026-09-16 night: audited the homepage board (31 cards, 3 impossible-red DB cards), the Prometheus target list (7/11 up, 3 impossible), and the bats suite (one chronic red test); fixed each and documented the rule in the Homepage and Observability pages.)_
-Diagnosing frank-medium (radiance) "restart loop" false alarms: the journal's repeating lines every ~10 s — [radiance.gemm]/[radiance.paroquant]/[radiance.mxfp4]/[radiance.paroquant_mxfp6] registered plus a NOOP file already applied line — are NOT a crash loop. They are the entrypoint's ~20 sequential python3 patch_*.py invocations, each re-importing sitecustomize (which prints the radiance banners) and printing its own NOOP line in FORWARD order, then exec vllm. Proof it's not looping: the container reaches Application startup complete and podman inspect shows Started=t Restarts=0 with a stable Started timestamp. A real crash loop is the RuntimeError: Engine core initialization failed / ValueError: Free memory … less than desired GPU memory utilization traceback (a real prior failure: the 0.99 GPU_UTIL crash). Distinguishing: grep for Traceback|ValueError|Engine core initialization failed (real failure) vs the registration/NOOP cadence (normal slow patch phase, ~3 min).
-Migrating a compose-only container to a boot-persistent Quadlet on this fleet has three mechanical traps that bit on 2026-09-16, all invisible until something later breaks — a skill for the general procedure already exists (compose-to-quadlet-cutover), so these are the additions to remember. (1) THE CONTAINER, NOT THE SERVICE, IS WHAT FAILS: restart: unless-stopped (or Restart=always) restarts the *existing* container with its *old* config; when the compose service later gains GPU/devices/volumes, the running container keeps the stale config forever. ComfyUI died 9,146 times this way — created before devices:/group_add: were added, so it started with no /dev/kfd, no /dev/dri, no keep-groups and failed on "No HIP GPUs are available" while the YAML on disk looked perfect. Diagnosis: podman inspect c --format '{{.Config.CreateCommand}}' (the argv actually used) and {{.Created}} — NOT .HostConfig.Devices, which reported [] for a container that demonstrably had the devices (verified: torch saw the GPU, --device /dev/kfd was in the create command). Fix: remove the stale container so the unit's own can take the name. (2) A HAND-WRITTEN UNIT COLLIDES BY NAME with the one the generator produces (~/.config/systemd/user/comfyui.service shelling out to podman compose up -d vs the generated comfyui.service); retire it — stop, disable, move to ~/.AGENTTRASH — and never let both exist. (3) ONE OWNER PER CONTAINER NAME: delete the compose service, and rewire the deploy path in the same commit — planner allowlist (DANKCLAW_SERVICES/DANKSERVER_SERVICES) + a path rule for target/svc/**, the dispatcher's service_is_allowed entry, a handler that installs/restarts/health-waits, and install_quadlet pointed at "$TARGET/observability/quadlet"-style per-target trees rather than a hardcoded host. When a target ends up with zero compose services, retire the file rather than leaving a services: key with nothing under it (the compose-validity seam asserts ≥1 service). (4) SEAMS THAT READ COMPOSE _(context: Dankclaw/Dankserver 2026-09-16: migrated Dankserver node-exporter and Dankclaw ComfyUI from compose to Quadlet (ComfyUI had 9,146 restarts from a stale device-less container), retired Dankclaw's now-empty compose.yaml, and found two services whose test seams read the stale compose while the unit diverged.)_
-HF download verification gotcha: the resolve endpoint's X-Linked-ETag differs between authenticated (Bearer token) and unauthenticated requests for the SAME LFS file. The actual downloaded file's sha256 matches the UNAUTHENTICATED etag. So scripts/hf-manifest.py (which does Bearer-token HEAD) and hf-fetch.sh's sha256 check produce a systematic false-positive "mismatch" for this repo (e.g. hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6). This is HF API behavior, not a script bug or a stale manifest. To verify a download, compare local sha256 against an UNAUTHENTICATED curl -sI https://huggingface.co/repo/resolve/main/file X-Linked-ETag (strip CRLF with tr -d '\r'). The 2026-09-22 MXFP6 download was verified correct this way (PASS=19 FAIL=0).
-Before tightening a container's bind address, enumerate EVERY consumer of it — a bind is a contract with something, not just an exposure choice. On 2026-09-16 I set HERMES_WEBUI_HOST=127.0.0.1 on dankclaw's hermes-webui (reasoning: the compose service it replaced published loopback, and the repo policy favours internal-only), and broke hermes.franksdank.cloud with a 502: that unit is HOST-NETWORKED (it shares the loopback with the chrome-devtools CDP daemon), so Caddy cannot reach it by container name and fronts the guarded vhost with reverse_proxy 192.168.0.143:8787 — the host's LAN address, recorded in dankclaw/Caddyfile with the same explanation. With network_mode host there is no Podman port forwarder, so the app's bind IS the only path in: the wildcard serves both consumers (Caddy on the LAN address, the SSH tunnel on the host's localhost), and loopback serves only the latter. The doc's own gotcha already said this was "an exposure change to decide deliberately, not a silent fix" — read the service's doc before changing its bind. Two process lessons from the same near-miss: (1) the e2e suite stayed 100% green through it because playwright.config.ts matches specs BY NUMBER PREFIX, not glob, so a new spec file that is not registered in a project silently never runs — register the prefix and let tests/e2e_scaffold.bats fail if any spec matches no project; (2) for every guarded vhost, at least one gate should assert that the vhost reaches its app through the guard chain (status 500 is the load-bearing check: a dead upstream is a 502 from Caddy), especially for services whose upstream is a host address rather than a container name. _(context: Dankclaw 2026-09-16: tightened hermes-webui's bind to loopback, broke the hermes.franksdank.cloud vhost (502) with the whole e2e suite green; reverted, added specs/17-hermes-vhost.spec.ts, registered it, and added a guard that every spec matches a playwright project.)_
-frank-medium (dankserver, dual R9700 TP=2): vLLM's startup memory check (gpu_worker.py request_memory) runs BEFORE weight load and requires gpu_memory_utilization × total free-at-device-init. On the 32 GB R9700 thereported total is 31.86 GiB but only 31.23 GiB is free at device-init — the 0.63 GiB gap is the ROCm/HIP context + NCCL + radiance's custom all-reduce buffers (84 MB fine-grained AR + 44 MB AR_QUANT). So the max that starts is 0.98 (0.98×31.86=31.22 31.23); 0.99 (31.54) fails with "ValueError: Free memory on device cuda:0 (31.23/31.86 GiB) on startup is less than desired GPU memory utilization" and crash-loops. 0.98 is the ceiling AND the prior production value. The repo fit-gate test does NOT model this device-init baseline (it only checks GPU_UTIL×total vs weights+KV), so it wrongly passed 0.99. If startup failures recur on reboot the lever is a further drop (0.975).
-On this fleet, "which container will not come back after a reboot?" has a one-line answer: every running container should carry a PODMAN_SYSTEMD_UNIT label; anything without one is compose-only or bare and will die silently at the next boot/shutdown. Audit with for c in $(podman ps --format '{{.Names}}'); do printf '%s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"; done and compare against the host's ~/.config/containers/systemd/*.container and the repo's host/svc/quadlet/. THREE separate instances of this failure mode were found on 2026-09-16 alone: (1) Dankserver's node-exporter — compose-only, Exited (143) for 37 h, no unit, Prometheus's node-dankserver job scraping nothing, and the compose form bound the wildcard address; fixed by adding dankserver/observability/quadlet/node-exporter.container (loopback + LAN 9100, the deliberate LAN listener posture). (2) ComfyUI on Dankclaw — its container was created BEFORE the GPU declarations were added to the compose service and restart: unless-stopped then kept restarting that stale, device-less container for 9,146 restarts (RuntimeError: No HIP GPUs are available); the service was fine, the container was old. Fixed by a Quadlet unit + deleting the compose service. (3) hermes/hermes-dashboard/hermes-webui were ALREADY quadlet-owned (labels proved it) while dankclaw/compose.yaml still defined them — a second owner for the same container names, which is how a stray podman compose up creates a duplicate that fights the unit. Generalisations worth keeping: **editing a compose service does not replace a running container** (restart policy will keep the old config alive forever — recreate explicitly); **podman inspect --format '{{.HostConfig.Devices}}' is not a reliable device-passthrough check** (it read devices=0 on a container that demonstrably had --device /dev/kfd — use {{.Config.CreateCommand}} for the real argv, and confirm from inside with e.g. podman exec c python3 -c "impo _(context: 2026-09-16 night session on the serverAppConfig fleet (Dankclaw + Dankserver): restored Dankserver's node-exporter (37 h dark) and ComfyUI (9,146 restarts, 7 h down) as boot-persistent Quadlets, audited every running container's systemd owner, and found that all three hermes services were already units while compose.yaml still duplicated them. Verified fixes live: node_* targets up, ComfyUI health)_
-The integration of ComfyUI as the image generation engine in OpenWebUI is fully verified. The workflow is set, the engine is active, and the 'red cube' prompt successfully generated the expected visual output.
-The recurring outage class on this fleet is a container whose RUNNING config has drifted from its DEFINITION, kept alive by a restart policy: compose restart: unless-stopped (or a hand-written systemd unit calling podman compose up -d) restarts the stale container forever instead of recreating it from the edited definition, so a service can sit broken for days looking merely "unhealthy". Found twice on 2026-09-16: Dankserver's node-exporter had been Exited (143) for 37h with no unit at all (compose-only, dead since a reboot), and ComfyUI had restarted 9,146 times because its container was created 2026-09-15 14:12 BEFORE the /dev/kfd //dev/dri/keep-groups declarations were added to the compose service — so torch reported "No HIP GPUs are available" on every start while the *definition* was correct. Diagnostic trap that nearly produced a wrong root cause: podman inspect --format '{{.HostConfig.Devices}}' reported EMPTY for a container that demonstrably had --device flags; the trustworthy view is --format '{{.Config.CreateCommand}}' (the actual argv). Also: find who started a container via the PODMAN_SYSTEMD_UNIT label — a podman-compose@… label means compose owns it, regardless of what unit files exist. The durable fix in this repo is the convention already written down: one owner per container name (delete the compose service), a repo-tracked Quadlet under host/svc/quadlet/ installed by the deployer, and retirement of any legacy compose-era unit file into ~/.AGENTTRASH (a leftover ~/.config/systemd/user/svc.service COLLIDES by name with the unit the Quadlet generator produces). Remediation ordering that worked: build image - install unit - stop/disable+move the legacy unit - podman rm -f the stale container - daemon-reload - restart unit - health-wait - then verify the real capability INSIDE the container (e.g. torch.cuda.is_available()), not just that it is Up. _(context: serverAppConfigs, 2026-09-16 night: restored Dankserver node-exporter (37h dark, compose-only) and ComfyUI (9,146 restarts, stale device-less container) by migrating both to repo-tracked Quadlet units; also documented that three Prometheus jobs (comfyui + two services probes scraping HTML, gitea /metrics 404) can never be green.)_
-OpenWebUI successfully generates images using the ComfyUI engine and the Qwen-Image 2.1 workflow. The 'red cube' prompt produced a 1024x1024 image with a red center, confirming the end-to-end integration is complete.
- Two durable facts from the 2026-09-16 night (Dankclaw/Dankserver fleet). (1) TRAP: podman inspect c --format "{{len .HostConfig.Devices}}" reports 0 and .HostConfig.GroupAdd reports [] even for a container that WAS started with --device /dev/kfd --device /dev/dri --group-add keep-groups. I nearly misdiagnosed ComfyUI as "no GPU devices configured" from that field. The ground truth is the actual argv: podman inspect c --format "{{.Config.CreateCommand}}" (or podman exec c python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" for the behavioural truth — which printed True | Radeon 8060S Graphics once fixed). (2) ROOT-CAUSE PATTERN for "a service crash-loops after a reboot while its config looks right": a container created by an EARLIER revision of the compose service (before devices:/group_add: were added) is kept alive by restart: unless-stopped; podman compose up -d does not recreate it, so it restarts forever with the OLD config (ComfyUI: RuntimeError: No HIP GPUs are available, 9,146 restarts, ~7h down). Decisive check: compare the container's creation timestamp against when the config gained the missing lines, and read .Config.CreateCommand for the flags it actually has. Fix on this fleet is not "recreate the compose container" but migrating the service to a repo-tracked Quadlet (the repo's own rule: every container is a boot-persistent, self-healing unit with repo provenance). Result: ComfyUI now runs from dankclaw/comfyui/quadlet/comfyui.container — image comfyui-rocm:gfx1151, /dev/kfd + /dev/dri, GroupAdd=keep-groups, four comfyui-data volumes, loopback-only publish 127.0.0.1:8188 (unguarded service = internal-only; homepage card probes loopback), Restart=always, WantedBy=default.target, healthcheck via a mounted file (the podman 5.x v1 generator splits HealthCmd on spaces, so inline python3 -c "..." is a quoting trap). Unit is generated, healthy, restarts=0, and curl 127.0.0.1:8188 → 200. Also fixed the same night: Dankser _(context: Dankclaw ComfyUI outage found while auditing Prometheus targets during the 2026-09-16 frank-medium night shift: 9,146 restarts, stale device-less compose container, migrated to a Quadlet; verified hip available True. Dankserver node-exporter restored the same way.)_
- The 4 missing models for ComfyUI have been downloaded to the correct directories in the dankclaw host. This completes the model download task for the Qwen-Image 2.1 and Qwen3VL models.
-Fleet state after the 2026-09-16 night session on serverAppConfigs (frank-medium + observability work): (1) DANKSERVER NODE-EXPORTER RESTORED — it had run only from dankserver/compose.yaml, died Exited (143) at a reboot and left Prometheus' node-dankserver job scraping nothing for ~37 h; it is now dankserver/observability/quadlet/node-exporter.container (generated, Restart=always, binds exactly 127.0.0.1:9100 + 192.168.0.235:9100, no wildcard — verified with ss -ltn), the compose service is deleted (one owner per container name), and deploy_observability now installs from "$TARGET/observability/quadlet" instead of a hardcoded Dankclaw path. Dankclaw keeps five units there, Dankserver one. (2) PENDING FOR FRANK — ComfyUI on Dankclaw is crash-looping: 9,146 restarts, exit 0, log says RuntimeError: No HIP GPUs are available, unit shows active (exited) while the container respawns; deliberately not touched because the fix depends on which posture Frank wants (its documented posture is CPU-only --cpu, which contradicts that error). (3) THREE PROMETHEUS TARGETS ARE CONFIGURATION DEFECTS, NOT OUTAGES: gitea returns 404 on /metrics (service itself fine), and the two services targets (SearXNG :8081, OpenWebUI :3005) answer 200 with Content-Type text/html, which Prometheus 3 refuses to parse — "blackbox-style probes" without a blackbox exporter are inert; the observability doc's own blackbox-exporter note is the fix. So "7 of 11 targets up" is the expected reading, and an alert on up==0 across all jobs would be permanent noise (the LLM rules are scoped to frank-* and are unaffected). (4) Also this session: tests are GREEN (313, ALL TESTS PASSED) — the long-standing "comfyui test 73 fails on a clean tree, ignore it" belief is now false and must not be repeated; the homepage card for Dankserver :7770 was mislabelled frank-small (retired) and is now frank-medium; Grafana has a provisioned Prometheus datasource + an LLM fleet dashboard; Prometheus evaluates 5 LLM alert rules but _(context: serverAppConfigs, 2026-09-16 pre-dawn session: frank-medium (2x R9700, vllm-radiance, two TP=1 replicas behind an adaptive-routing proxy) plus an observability audit that found the node-exporter gap, the mislabelled homepage card, a red test that had been excused for sessions, and a crash-looping ComfyUI.)_
-Testing the /api/v1/generate endpoint with the 'comfyui' model to verify if it's the correct endpoint for producing images in the current OpenWebUI version.
-Durable fact for this repo (serverAppConfigs): the measurement harnesses for the CURRENT LLM serving stack now live in git — do not rewrite them in /tmp. Four files, added 2026-09-16 with their traps documented in each header: scripts/bench-vllm-corpus.py (builds a code-shaped corpus from the repo's own sources; synthetic --dataset-name random prompts misreport speculative draft acceptance badly — measured 2.9-3.6 tokens/update on random against ~5.8 on code, so tuning SPEC depth on random data picks the wrong value), scripts/bench-vllm-serve.sh (concurrency sweep via vllm bench serve from inside the serving image; it exists mainly to encode two traps — the client needs --tokenizer /models/... with /models mounted, because with only --model frank-medium it tries to resolve that alias on HuggingFace and dies before sending a request, producing a run that looks like a benchmark and reports nothing; and --ignore-eos output is not comparable with runs that lack it), scripts/bench-turn-latency.py (per-turn TTFT for an agent-loop shape — the instrument that decided the deployed prefix-affinity routing policy, and the fastest way to demonstrate cold-vs-warm prefill: measured 12.5 s cold vs 1.4 s warm on the same ~50k-token prompt), and scripts/bench-llm-soak.py (sustained multi-family load; this is what found the two-card capacity wall). Prompts in every harness carry a salt as their leading bytes, because that changes both the router's identity window and vLLM's prefix-cache key — without it a run silently inherits a warm cache from the previous run and reports fantasy numbers. Also durable: scripts/*.py are now compiled by the scripts_syntax seam (previously only scripts/*.sh were checked), and the whole suite is green (311 tests) with no known-failure exceptions to explain away. Older llama.cpp-era harnesses (bench-decode*, bench-spec-decode.py, bench-threads-batch*, generate-prompts.py) are still in scripts/ and target a different runtime's endpoint — check which stack _(context: Night session 2026-09-16 on Dankserver frank-medium (vllm-radiance MXFP4, two R9700 replicas behind an adaptive-routing proxy): after building and using throwaway measurement scripts for hours, the useful four were committed to scripts/ and smoke-tested against the live fleet, and the script-syntax seam gained its missing python half.)_
-Confirmed that setting 'image_generation.engine' to 'comfyui' and 'image_generation.model' to 'comfyui' correctly wires the two. The workflow is the most important part, defined as a JSON string in the SQLite config.
-CORRECTION to the long-standing belief recorded in this project's memory that "./tests/run.sh exits 1 on a clean tree because of a pre-existing comfyui test-73 failure, and that is expected, so don't treat a red suite as blocking." That is now FALSE as of 2026-09-16: the test was fixed properly and the suite is GREEN (310 tests, ALL TESTS PASSED). Root cause of the old failure was a selector bug in tests/comfyui_rocm.bats — find dankclaw -name 'Containerfile*' | head -1 returned whichever Containerfile the directory listing happened to put first (dankclaw/alloy/Containerfile), so the assertion read a file that never claimed to be ComfyUI's and failed; the same selector would have passed on a wrong file. The fix resolves the Containerfile the way the build does — build.context joined with build.dockerfile, read from dankclaw/compose.yaml — so the test checks the file the deploy actually builds. The generalisable lesson: in this repo a test that guesses a path with find ... | head -1 is a bug waiting to be believed as "a known failure"; read the contract (compose, unit file, config) instead of the filesystem. Also: a permanently red test trains every operator and every future agent to ignore red — treat "known failure" as a defect with an owner, not as a baseline. A dead duplicate was retired alongside it (dankclaw/comfyui/Containerfile was referenced by nothing — compose builds Containerfile.comfyui), staged in .AGENTTRASH/ awaiting Frank's purge decision. _(context: serverAppConfig repo, 2026-09-16 night session: fixed tests/comfyui_rocm.bats test "comfyui Containerfile is based on the gfx1151 torch image" after it had failed across many sessions; suite went from 285 ok + 1 not ok to 310 ok, ALL TESTS PASSED.)_
-Final verification of OpenWebUI image generation with ComfyUI as the engine using the 'red cube' workflow. The endpoint is confirmed to be responsive once the 'image_generation.model' is explicitly set to 'comfyui'.
-Dankserver frank-medium as of 2026-09-16 is NOT a single tensor-parallel process: it is two single-GPU TP=1 replicas of the vllm-radiance MXFP4 stack (frank-medium-a on R9700 #1 :7771, frank-medium-b on #2 :7772) behind a prefix-affinity proxy (frank-medium-proxy :7770, routing.py policy + affinity-proxy.py server), each at 114688 context / 4 sequences / CHUNK=8192 / KV_MEM=0, with the DFlash2 drafter at SPEC=7. TP=2 is impossible on this host for a hardware reason: RCCL's collective kernels carry a hidden hostcall-buffer parameter and ROCr refuses to dispatch them on a device without PCIe atomics, and the second R9700 sits behind chipset bridges 05:00.0/06:09.0 advertising AtomicOpsCap: Routing- (read-only) — so no software enables atomics there while the first card, on CPU root port 00:01.1, has them. The practical capacity ceiling per card is 122,636 KV tokens (the engine's own ledger), i.e. about two 30k-token conversations; three concurrent 50-80k contexts thrash the prefix cache (turns alternate ~1.4 s warm / 22-45 s cold) and neither load-balanced placement nor per-card admission control fixes it, because a running sequence and its cached prefix draw on the same pool. CHUNK is not a lever: available KV is 5.44 GiB at 8192 versus 4.22 at 16384 and 4.16 at 4096 (which refuses to boot), so 8192 is a sweet spot. Long-prefix caching itself is healthy — the same ~50k-token prompt sent twice to an idle card is 22.3 s then 1.4 s — and the engine's lifetime "Prefix cache hit rate" line is an average over all traffic ever scraped, not a verdict on any request. The fleet's Prometheus scrapes both LLM endpoints (frank-medium and frank-large) and five rules watch them; nothing is notified because there is no Alertmanager, so alerts only appear on Prometheus' /alerts and the dashboard's ALERTS panel. _(context: Night-long session on Dankserver frank-medium: TP=2 ruled out by PCIe atomic routing, deployment reshaped as two TP=1 replicas + adaptive-routing proxy, capacity measured, four optimization hypotheses tested (routing and failover shipped; placement, admission, CHUNK variants rejected with numbers), then Prometheus scraping + Grafana provisioning + five alert rules added.)_
-The ComfyUI integration is verified by successfully setting the workflow and engine in the OpenWebUI config. The next step is to confirm the visual output via a real request to the /api/v1/generate (or similar) endpoint to ensure the 'Method Not Allowed' error was resolved.
-On the Dankclaw fleet, Grafana is configured as CODE, not by clicking: shared/grafana/provisioning/{datasources,dashboards} is installed to /opt/grafana/provisioning by copy_grafana_provisioning() in scripts/deploy-remote.sh (the grafana quadlet already mounts that path read-only), and scripts/deploy-plan.py maps any shared/grafana/** path to the grafana service, so ./scripts/deploy.sh dankclaw --service grafana is the whole apply step. VERIFICATION TRAP (cost me a false negative): a *provisioned* dashboard does NOT appear in the legacy dashboard table of grafana.db on the current Grafana build — dashboards live in unified storage behind the dashboard.grafana.app API group — so querying that table returns empty and looks like the dashboard failed to load. The datasource DOES appear in the data_source table, which makes the asymmetry easy to misread. Correct verification, in order: podman logs --since 10m grafana | grep -i "inserting datasource\|provision dashboards" (the provisioning paths log), then the API — curl -u admin:admin http://127.0.0.1:3002/api/search?type=dash-db for dashboards and /api/datasources/proxy/uid/uid/api/v1/query?query=... to prove the datasource actually reaches Prometheus. Grafana on this host publishes only on loopback :3002 (the off-host path is the guarded vhost), and its admin login is still the default admin:admin — worth changing, and worth remembering because every metric/dashboard assertion needs it. One more habit that paid off twice in this repo: before writing any dashboard panel or config comment, grep the metric names out of the live /metrics endpoint (curl -s host:port/metrics | awk '{print $1}' | sort -u | grep word); a comment naming vllm:prefix_cache_hit_rate shipped once and that series does not exist (the real pair is prefix_cache_hits_total / prefix_cache_queries_total). _(context: Dankclaw 2026-09-16: provisioned the Prometheus datasource plus an "LLM fleet — capacity and reuse" dashboard from git (shared/grafana/provisioning/*), after discovering Grafana had zero datasources so the newly scraped vLLM series were only reachable through the Prometheus API. Verified via the Grafana API after the legacy dashboard table gave a false empty.)_
-Setting 'image_generation.model' to 'comfyui' ensures that even if 'image_generation.engine' is 'comfyui', the model picker and internal logic have an explicit model identifier to refer to, preventing potential ambiguity in the SQLite config.
-frank-medium on Dankserver (192.168.0.235:7770) is no longer one tensor-parallel process: since 2026-09-16 it is TWO single-GPU vLLM replicas (frank-medium-a on :7771, frank-medium-b on :7772) behind a prefix-affinity proxy on :7770, because TP=2 cannot launch a collective on this host (RCCL kernels need PCIe atomics; the second R9700 sits behind chipset bridges advertising AtomicOpsCap: Routing-). Each replica is TP=1 at 114688 context / 4 sequences / CHUNK=8192 / SPEC=7 with vLLM-profiled KV. Concrete consequences a future session must not re-derive: the harness entry in /home/blank/.omp/agent/models.yml is contextWindow 114688 / maxTokens 32768 and omp --model frank-medium -p "..." works end to end; per-card KV is 122,636 tokens, so two concurrent large contexts are comfortable and three thrash; the deploy handler installs a/b/proxy and moves any stale TP=2 unit into ~/.AGENTTRASH; Prometheus now scrapes both LLM endpoints (frank-medium vLLM, frank-large llama.cpp) on their LAN addresses; and the trio is proven boot-persistent by an actual unattended reboot (active 59 s after boot, healthy ~3.5 min, no manual step). Full evidence lives in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md — read it before assuming TP=2 or the retired ROCmFPX deployment. _(context: frank-medium deployment and fleet wiring, 2026-09-15/16: TP=2 → two TP=1 replicas + adaptive proxy, harness/models.yml update, Prometheus scrape jobs, reboot verification.)_
-Successfully integrated ComfyUI as the image generation engine for OpenWebUI. The workflow for a "red cube on a white background" was successfully loaded via the SQLite config table. The engine was set to 'comfyui' and the base URL was set to 'http://host.containers.internal:8188'.
-Benchmarking an LLM serving deployment honestly (used repeatedly on frank-medium 2026-09-16, applies to any future swap incl. frank-large): (1) vllm bench serve needs an explicit --tokenizer local path with /models mounted — passing only --model alias makes the client resolve the alias on HuggingFace and die before sending a request, a green-looking run that produces no numbers; (2) synthetic --dataset-name random under-reports speculative decoding badly (2.9-3.6 tok/update measured vs 5.8 on code) so tuning SPEC depth on it picks the wrong default for a code workload — build the corpus from the repo's own sources instead (JSONL of {"prompt": instruction + file body}); (3) vLLM's Prefix cache hit rate: log line is a LIFETIME average over every prompt that replica ever saw, not a per-request verdict — to learn whether long-prefix reuse works, send the same prompt twice to an idle card and compare TTFTs (measured: 22.3 s cold → 1.4 s warm on a 50k-token prefix); (4) A/B arms must use different leading bytes (a per-arm salt at the front of every prompt), otherwise the second arm silently reuses the first arm's warm cache; (5) repeat each arm at least twice — I saw 5-10% run-to-run variance, and a single measurement was enough to make me nearly ship the wrong SPEC depth; (6) when a proxy sits in front of several replicas, measure the workload shape the fleet actually runs (sequential conversation turns) separately from batch benchmarks: a cold batch of distinct prompts favors spreading across cards, sequential turns favor pinning to one, and the two conclusions pull in opposite directions. _(context: frank-medium (2x R9700, two TP=1 vLLM replicas behind a proxy) optimization pass, 2026-09-16: corpus design from repo sources, salted per-arm corpora, repeated arms, sequential-turn client vs batch benchmarks, engine counter semantics.)_
-OpenWebUI's /api/images/generate returned 'Method Not Allowed', suggesting the endpoint might be /api/v1/generate or requires a different structure. The 'image_generation.model' was empty, which might also affect the resolution.
-frank-medium on Dankserver (2x R9700 gfx1201, since 2026-09-16) serves as TWO single-GPU TP=1 replicas behind a proxy on :7770 (adaptive routing + failover), because TP=2 is impossible on this AM5 box: RCCL collective kernels need PCIe atomics and card #2's chipset path advertises AtomicOpsCap: Routing- (read-only). Per-replica shape: MAXLEN=114688, MAXSEQS=4, CHUNK=8192, KV_MEM=0 (passed through literally — blanking it makes vLLM size KV from GPU_UTIL alone and OOM), SPEC=7. Engine ledger facts: **GPU KV cache size: 122,636 tokens per card** (5.44 GiB), 1.07x concurrency at full context. Measured and REJECTED tonight, do not retry without new evidence: (1) load-balanced family placement — no improvement (heavy 25.4→46.7 s median); (2) large-request admission control, one 16k-token request per card — no clear win (heavy 25.4→48.2 s, mediums improved on small samples); (3) CHUNK=4096 or 16384 to free KV — available KV is NON-monotonic in chunk size (4.16 GiB at 4096 where the engine refuses to boot, 5.44 at 8192, 4.22 at 16384), so 8192 is a sweet spot; (4) SPEC=5 for batch throughput — 7 stays (46.2 vs 41.5 tok/s single-stream). The wall is capacity arithmetic: live sequences and cached prefixes share one pool, so a card holding two 50k-token families must carry 2x50k cached + 50k live = 150k against 122,636 — co-tenants evict each other regardless of placement or serialization. Caching itself is healthy (a 50k-token prefix: 22.3 s cold then 1.4 s warm, with a stable or varying tail). Trap: the engine's "Prefix cache hit rate: N%" log line is a LIFETIME average over every prompt that replica has seen, not a per-request verdict. Verified elsewhere tonight: the trio survives an unattended reboot (units generated, Linger=yes, active 59 s after boot, replicas healthy ~3.5 min in), and the harness path works (omp --model frank-medium -p --no-tools → correct reply in ~8-10 s) with models.yml contextWindow 114688 / maxTokens 32768. _(context: Dankserver frank-medium, 2026-09-16 overnight session: two-replica MXFP4 deployment, routing/failover shipped, four optimization hypotheses measured (two shipped, two rejected), capacity ceiling quantified, reboot-tested.)_
-The 4 missing models for ComfyUI were: qwen_image_2.1_bf16.safetensors, qwen_image_2.1_int8_convrot.safetensors, qwen3vl_8b_bf16.safetensors, and qwen3vl_8b_w4a8.safetensors. These were identified as the missing pieces for the Qwen-Image 2.1 and Qwen3VL models on Dankclaw.
-On a single-GPU vLLM replica, a request's LIVE sequence and its cached prefix draw on the same KV pool — so placement/routing policy cannot rescue an overloaded working set. Measured on 2x R9700 radiance MXFP4 (122,636 KV tokens per card, 1.07x concurrency at 114k context): one 80k-token family plus two 50k-token families, each turn re-sending its document, thrash the cache no matter how families are placed — counter placement: heavy median 25.4 s (23% cached), mediums 34.7/45.2 s (14/15% cached); load-balanced placement (new families to the card carrying less context, continuations still pinned): heavy 46.7 s (20% cached), mediums 23.4/34.3 s (0% cached). No improvement, change reverted. Why: a card running two 50k-token generations has already spent its 122,636 tokens on live sequences before any cache entry is considered, so placement only decides WHICH card is over capacity. Lever for more concurrent large contexts = fewer in flight (context compaction) or another card, never a routing tweak. Two measurement disciplines that made this decision possible, both worth repeating: (1) design the A/B so the two policies actually differ — with tiny "light" families nothing is under pressure and both arms look identical, so size the families to straddle the card's KV ceiling (e.g. 80k + 50k + 50k against 122k); (2) before trusting an arm, unit-check that the mechanism separates as intended (here: heavy-card0, both mediums-card1, confirmed by calling the routing function directly) — otherwise "no improvement" may just mean the feature never fired. Same discipline caught two other hypotheses in the same session: CHUNK=4096 also reduced available KV (4.16 GiB) rather than freeing it, so 8192 is a sweet spot, not a compromise. _(context: 2026-09-16, Dankserver frank-medium: A/B of load-balanced vs counter placement for two radiance MXFP4 replicas behind an adaptive-routing proxy; hypothesis rejected on measurement and reverted, documented in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md under "Placement does not rescue an overloaded working set".)_
-When 'image_generation.engine' is set to 'comfyui', the 'image_generation.model' key can be left empty (""), but 'image_generation.comfyui.workflow' must be a JSON string containing the full workflow dictionary. In the current setup, 'image_generation.comfyui.base_url' should point to 'http://host.containers.internal:8188'.
-frank-medium (Dankserver, 2x R9700) runs as TWO single-GPU vLLM-radiance replicas behind an adaptive proxy — TP=2 is impossible on this AM5 box (RCCL needs PCIe atomics; card 2's chipset path has AtomicOpsCap Routing-, read-only). Durable numbers measured 2026-09-16: each replica holds 5.44 GiB KV = 122,636 tokens at KV_MEM=0/MAXLEN=114688/CHUNK=8192/MAXSEQS=4/GPU_UTIL=0.98 (1.07x concurrency at full context). The arena/KV budget is NOT monotonic in CHUNK: available KV is 5.44 GiB at 8192 but 4.22 GiB at 16384 and 4.16 GiB at 4096 (where the engine refuses to boot: "5.0 GiB KV cache is needed"), so 8192 is a sweet spot and CHUNK is NOT a lever for freeing KV — more concurrent large contexts need another card. A 25-min soak of three concurrent large-context agent loops (each turn re-sending a ~30k-token document) gave 155/155 requests answered, zero errors/failovers, but only 25% cache-hit turns (median 1190 ms) against 75% re-prefills (median 13.9 s): the ceiling shows up as latency, not failure. Boot persistence is PROVEN by rebooting dankserver unattended: units active 59 s after boot, both replicas healthy ~3.5 min after boot, harness (omp --model frank-medium -p) working, no manual step — though SSH took ~9 min to return while systemd reports a 2m14s boot. The stack's /v1/completions returns raw continuations (no chat template), so api: openai-completions entries work only because the harness builds its own prompt; verify harness paths with omp --model alias -p --no-tools rather than assuming. OPERATIONAL TRAP hit twice: shipping files straight into the host checkout (/home/dank/serverAppConfigs) makes the next ./scripts/deploy.sh fail with "Your local changes ... would be overwritten by merge" — restore that path (git checkout -- path, or move the hand-copied file aside) before deploying, or the deploy silently stops at the pull. _(context: Dankserver frank-medium 2026-09-16 (overnight session): two TP=1 radiance MXFP4 replicas behind an adaptive-routing proxy on :7770; bench/soak/reboot verification; repo commits up to 99ed1ac.)_
-ComfyUI Qwen-Image 2.1 (qwen_image21) setup gotchas, verified 2026-09-21: (1) The VAE MUST be qwen_image_2.1_vae_bf16.safetensors (z_dim=64, 5D video VAE with time_conv). The Comfy-Org qwen_image_vae.safetensors (z_dim=16) is INCOMPATIBLE — the model's QwenImage21 latent format has latent_channels=64, so a z_dim=16 VAE causes "expected input to have 16 channels, but got 64" in VAEDecode. Both are video VAEs (5D convs); the 4D→5D expansion patch in comfy/sd.py is needed for both. (2) The model detection in_channels must come from img_in.weight.shape[0] (=64, the latent input dim), NOT shape[1] (=4096, the hidden dim). The GGUF stores the weight transposed relative to PyTorch's [out_features, in_features] convention — the model definition has in_channels=64, inner_dim=4096, so the GGUF tensor [64, 4096] is [in, out]. (3) Three patches are required for an unknown GGUF arch: general.architecture set in the GGUF file, the arch string in IMG_ARCH_LIST (loader.py line 12), and a detection block in model_detection.py. All three are baked into the image via apply_qwen21_patch.py run by comfyui-start.sh. _(context: ComfyUI Qwen-Image 2.1 Q8 GGUF setup on Dankclaw, 2026-09-21. After fixing the VAE 4D→5D patch and the in_channels shape, the remaining failure was using the wrong VAE file (Comfy-Org z_dim=16 vs repo z_dim=64).)_
-Two counterintuitive, measured facts about vLLM KV capacity on the frank-medium radiance stack (Dankserver, 2x R9700, TP=1 replicas, 27B MXFP4, 114688 ctx) — both cost hours if re-derived from reasoning: 1. Available KV is NOT monotonic in the prefill chunk size. Measured per card: CHUNK=4096 - 4.16 GiB available KV and the engine REFUSES TO BOOT ("To serve at least one request with the model's max seq len (114688), 5.0 GiB KV cache is needed, which is larger than the available KV cache memory (4.16 GiB)"); CHUNK=8192 - 5.44 GiB (the sweet spot, and the deployed value); CHUNK=16384 - 4.22 GiB. So shrinking the chunk to "free memory for KV" is wrong in both directions. Get the real capacity from the engine's own boot ledger, never from arithmetic: podman logs container | grep -E "GPU KV cache size|Maximum concurrency|Available KV cache memory" - here "GPU KV cache size: 122,636 tokens" and "Maximum concurrency for 114,688 tokens per request: 1.07x". 2. 122,636 KV tokens per card is about TWO 30k-token conversations with their cached prefixes. A 25-minute soak with three concurrent agent loops (each turn re-sending a ~30k-token document, growing history) came back perfectly stable — 155 requests, 155 answers, 0 failures, 0 proxy failovers — but only 25% of turns hit the prefix cache; the other 75% paid a 13.9 s median re-prefill because a third family on the same card evicts. Capacity exhaustion on this fleet shows up as latency, not errors. Related: the boot warning "Add 3 padding layers, may waste at most 60.00% KV cache memory" is real — the mamba/attention hybrid wastes KV, so the usable budget is well under the nominal pool. Also: the harness path is verifiable directly with omp --model frank-medium -p --no-tools --no-session "prompt" (9.8 s round trip here) — that exercises models.yml provider/baseUrl/alias end to end, which reading YAML cannot. Note the provider uses api: openai-completions, i.e. /v1/completions, where this stack returns a RAW continuation (no _(context: Dankserver frank-medium 2026-09-16, after the two-replica + adaptive-routing deployment: soak of three concurrent agent loops, CHUNK sweep (4096/8192/16384), engine boot-ledger capacity read, and a real omp harness call.)_
-frank-large (dankclaw :7770, Strix Halo gfx1151 iGPU) was cut 2026-09-21 to stock llama.cpp (unsloth b11030 prebuilt, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.4 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) + q8_0 KV + 262144 ctx / 1 slot + text-only + Qwen-Sharp v22.4.0 template. Verified live: health ok, answer "Paris" + reasoning_content, prompt 177 tokens (template wired, NOT the 73,600-token peg-native fallback), DFlash2 draft acceptance 0.85 (short easy prompt) / 0.519 (mean len 3.08, longer varied "explain a socket" prompt), warm decode ~17 t/s, GTT 57 GiB of 114 GiB. Durable facts: (1) unsloth b11030 does NOT support fp8 KV — allowed types are f32,f16,bf16,q8_0,q4_0,q4_1,iq4_nl,q5_0,q5_1; q8_0 is the highest-precision 8-bit option (~8 GiB at 262k). (2) --ngl (double-dash) is INVALID in b11030 — use --n-gpu-layers or -ngl. (3) curl -C - resume corrupted a 31.5 GiB HF download (213 MB oversized); fresh re-download (no resume) is the safe fix. (4) The 8-bit UD-Q8_K_XL decodes ~17 t/s vs the prior 4-bit Kairic Edge IU4 at ~22 t/s — the ~23% larger weight size (31.4 vs 26 GiB) explains the gap, it is not a regression. (5) DFlash2 draft acceptance is text-variety-dependent: 0.85 on a trivial one-word answer, 0.519 on a varied explanatory prompt — do not read a single-request acceptance as the steady-state. _(context: serverAppConfigs frank-large swap to stock llama.cpp UD-Q8_K_XL + DFlash2, 2026-09-21, commits 7689092 + 600812d)_
-A reverse proxy in front of N LLM replicas that trusts a periodically-refreshed health table will hand clients 5xx errors during every restart window: with a 10 s health TTL, requests routed to a card that just went down fail before the table notices — and a deploy restarts all replicas. Fix: on connection failure (nothing written to the client yet), demote that replica immediately and retry the request on another live replica; log the failover, because a silent one hides a downed card. Safe for stateless OpenAI-compatible endpoints (chat/completions carries no session state) but NEVER retry after streaming has begun. Verify behaviourally: stop one replica and fire a burst of requests in the same second — all must return 200 with exactly one failover line in the proxy log. Measured on frank-medium (2x R9700, adaptive routing): 6/6 requests 200, one failover log line, before the change they returned 502. Also observed: a stopped radiance replica takes ~150 s to come back with a warm compile cache, so the deploy health wait must budget for it. _(context: Dankserver frank-medium 2026-09-16: closed the 502-during-restart window in the two-replica proxy (routing.py + affinity-proxy.py), verified by stopping card 1 and firing a burst; deployed green with the e2e gate 30/30.)_
-ComfyUI Qwen-Image 2.1 GGUF setup (dankclaw, 2026-09-21): Three non-obvious issues required patches baked into the image via a startup wrapper (apply_qwen21_patch.py). (1) VAE/model latent channel mismatch: the model's QwenImage21 latent format has latent_channels=64, but the Comfy-Org VAE (qwen_image_vae.safetensors) is a video VAE with z_dim=16 — the repo's VAE (qwen_image_2.1_vae_bf16.safetensors) has z_dim=64 and is the correct one. Both are video VAEs (5D convs, time_conv); only the repo's matches the model's latent channel count. (2) in_channels shape: GGUF stores linear layer weights as [in_features, out_features], not [out_features, in_features] like PyTorch — model detection must read shape[0], not shape[1]. (3) 4D→5D VAE expansion: the model produces 4D image latents but the video VAE expects 5D; fix is elif self.latent_dim == 3 and samples_in.ndim == 4: samples_in = samples_in.unsqueeze(2) in comfy/sd.py decode(). Also: Q8_0 quantized 1D tensors have raw data shape ≠ logical tensor shape; dequantize all 1D tensors unconditionally in the GGUF node loader.
-Benchmarking a speculative-decoding LLM endpoint on this fleet taught a discipline that changed three conclusions in one night: (1) Each configuration under test needs its OWN salted corpus — prefix caching persists inside a running engine, so reusing a document across configs silently measures a warm cache; put a unique token sequence at the very start of every prompt so both the router hash and vLLM's cache key change. (2) Synthetic random prompts under-report speculative decoding badly: the same model/server measured 2.9-3.6 tok/update on --dataset-name random and 3.86 tok/update on a corpus built from the repo's own source files, so a code/RAG/agent workload must be measured with a code-shaped corpus (vllm bench --dataset-name custom with JSONL {"prompt": ...}). (3) Run each setting at least twice: the same config measured 118.7 and 127.0 tok/s (conc 4) and 136.1 / 129.4 (the other config) — ~5-7% run-to-run variance, so a single-run difference under ~10% is not a decision, and the two settings' ranges overlapped while the ordering held. (4) Conversation workloads need a sequential-turn client measuring per-turn TTFT, not a batch benchmark: a batch of distinct prompts parallelises prefill across replicas and even makes round-robin look better, while the agent-loop shape (same prefix, growing history) is what exposes cache behaviour. Separately, a sizing fact for the radiance/vLLM stack on a 32 GiB card: the prefill arena scales with --max-num-batched-tokens and shares one device budget with the KV cache, so CHUNK=16384 leaves 4.22 GiB of KV where 114688 context needs 5.21 GiB and the engine refuses to start — CHUNK and MAXLEN are a matched pair, not independent knobs. _(context: Dankserver frank-medium 2026-09-16/17: tuning two single-GPU radiance MXFP4 replicas (SPEC depth 7 vs 5, CHUNK 8192 vs 16384) and their adaptive-routing proxy; every number in the runbook came from these corpora and repeats.)_
-llama.cpp's --chat-template-kwargs alone does NOT load a custom template — you must also pass --chat-template-file (or --chat-template) for the binary to use it. Without --chat-template-file, the binary falls back to the built-in template (logged as "Chat format: peg-native"), which can produce a wildly different prompt size than the custom template. On the Kairic Edge build (ciru-ai/ROCmFPX, gfx1151), the built-in peg-native template produced a ~73,600-token prompt for a ~250-token request; the Qwen-Sharp v22.4.0 template (wired via --chat-template-file) produces a 62-175-token prompt for the same request. The "Chat format: peg-native" log line is the format-type label, NOT the template indicator — the custom template is active even when that line appears. The serve script must pass --chat-template-file explicitly in its ARGS array; setting LLAMA_ARG_CHAT_TEMPLATE_FILE as an env var before invoking the script is insufficient because the script constructs the binary command line explicitly and does not forward that env var.
-On the vllm-radiance MXFP4 stack (frank-medium on Dankserver's R9700s), two KV-sizing traps cost deploy cycles and are counterintuitive: (1) KV_MEM=0 must be passed THROUGH to the engine, not blanked. The entrypoint expands ${KV_MEM:+--kv-cache-memory "$KV_MEM"}; the image's patched KV sizing reads an explicit 0 as "profile this shape". If the variable is blanked instead (so the flag is omitted), vLLM sizes the KV cache from --gpu-memory-utilization alone, overshoots, and the allocation dies with torch.OutOfMemoryError right after the weights load — measured: "Tried to allocate 2.16 GiB. GPU has 116 MiB free, 29.34 GiB allocated by PyTorch". Symptom shape: the quadlet crash-loops every ~2 min ("Engine core initialization failed", status=1) while the same config runs fine when started by hand with the older entrypoint — compare entrypoint versions when a hand-run works and the unit does not. Do not "fix" the entrypoint by making the expansion empty. (2) CHUNK (--max-num-batched-tokens) is coupled to the context: the prefill ARENA grows with the chunk size and shares one device budget with the KV cache. On a single 32 GiB card at MAXLEN=114688, CHUNK=16384 leaves 4.22 GiB of KV against the 5.21 GiB needed and the engine refuses to start ("To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)"). Raising CHUNK therefore means cutting MAXLEN. Per-card single-card prefill at CHUNK=8192 measured ~2.5k tok/s (~15k-token prompt, 5992 ms TTFT), so the trade was not worth taking. Also: 262144 context needs 9.61 GiB of KV and 131072 needs 5.58 GiB, which is why one card tops out at 114688 in this shape. _(context: Dankserver frank-medium, 2026-09-16: two TP=1 radiance MXFP4 replicas, 114688 context each. These two traps are documented in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md; capturing them in memory because both look like engine bugs rather than sizing rules.)_
-In serverAppConfigs, adding a new docs runbook (e.g. serverAppConfigs-docs/LLM/foo.md) requires linking it in TWO indexes or the "docs index links every docs page" bats test (tests/docs_completeness.bats:24) fails: (1) the section index (serverAppConfigs-docs/LLM/index.md), and (2) the TOP-LEVEL serverAppConfigs-docs/index.md — the test greps ONLY the top-level index for [[basename]]. Adding the link to just the LLM index is a false sense of completion; the test still fails. When a runtime is swapped same-day (old ACTIVE → new ACTIVE), also mark the old runbook RETIRED in both indexes and add it to the AGENTS.md retired-runbook table so the chain stays documented. _(context: serverAppConfigs, 2026-09-21: created frank-large-kairic-edge-qwen38-27b.md runbook, added it to the LLM index, but tests/run.sh failed on "docs index links every docs page" because the top-level index.md also needed the link.)_
-When fronting multiple single-GPU vLLM replicas with a proxy, plain round-robin destroys prefix-cache hits: an agent loop re-sends the same system prompt + file context every turn, so alternating replicas makes every other turn pay a cold prefill. Measured on 2x R9700 (90k-token contexts): round-robin warm-turn TTFT 2676 ms vs pinned 1096 ms. But pinning everything is also wrong — a cold batch then loses a card's prefill parallelism (16 requests: 18.6 s spread vs 30.1 s pinned). The right rule is workload-shaped: pin *continuations* (requests carrying 1 message = an agent loop resending context) to the replica that served the prefix, and spread everything else (one-shot questions, batch members, parallel subagents) by counter. Measured result of that hybrid on the same host: 1095 ms warm turns AND 19.97 s burst. Two traps found by measuring, not reasoning: (1) hashing a fresh prefix to choose a card sends every member of a same-prefix burst to the same card (29.8 s) — spread must come from a counter, only continuations may be pinned; (2) the conversation identity is the first N KiB of the prompt (4096 by default) — if the prefix is shorter than that window, turns hash differently and never pin. Also keep the routing policy in its own stdlib-only module: a proxy that imports aiohttp at module scope cannot be unit-tested in an environment without it, and the first version of the routing code shipped a bug (stale index - HTTP 500 on every continuation) that only a policy-level test catches cheaply. _(context: Dankserver frank-medium 2026-09-16: two TP=1 radiance MXFP4 replicas behind a proxy on :7770 (TP=2 impossible — PCIe atomics). Implemented routing.py + affinity-proxy.py, tests/frank_medium_proxy.bats, deployed via deploy.sh with the e2e gate.)_
-frank-large (dankclaw :7770, Strix Halo iGPU gfx1151) was swapped 2026-09-21 from the unsloth llama.cpp prebuilt (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE, 87 GiB) to the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B DENSE, IU4 W4A4). Key facts: (1) DFlash is NOT available for this model — the Kairic Edge runtime's spec-type constants are DRAFT_EAGLE, DRAFT_MTP, DRAFT_SIMPLE, NGRAM_* (no DRAFT_DFLASH); the spec method is native MTP depth 4 + n-gram drafting (24/64/64). (2) The fast-greedy path rejects sampling, penalties, probabilities, grammar-constrained tool calls, logit bias, LoRA and a reasoning budget; the harness needs all three, so COMPATIBILITY_MODE=1 (9.7% slower per the build guide). (3) The payload is 4 files (15.5 GiB GGUF + 7.99 GiB FFN + 1.88 GiB GDN + 0.70 GiB GDN-Output .pfs sidecars), sha256-verified. (4) The runtime is built from source (scripts/build-kairic-edge-gfx1151.sh), TheRock 7.15.0a20260718 toolchain, CK fdf4bb7f + gfx1151-iu4 patch. (5) Reasoning is enabled (auto + deepseek format, thoughts land in reasoning_content). (6) The Qwen3.8 template gates reasoning_effort to {xhigh, medium, low}; the harness folds levels via models.yml reasoningEffortMap. (7) The 27B dense model fits the ~114 GiB GTT aperture easily (vs the 125B MoE's 87 GiB). Rollback: the unsloth prebuilt is still on disk at /home/dank/GitPackages/unsloth-llama-b11030/ and the model at /home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/.
-Multi-GPU RCCL collectives on AMD require PCIe atomics on EVERY participating GPU, and on AMD desktop platforms the second x16 card usually sits behind chipset bridges that advertise AtomicOpsCap: Routing- (read-only). RCCL's collective kernels carry a hidden hostcall-buffer parameter; ROCr refuses to give them one when the device lacks atomics (rocclr/device/rocm/rocvirtual.cpp:3963, gated on pcie_atomics_ set once at driver probe from the CPU-agent link info), and the launch dies as hipErrorIllegalState ("the operation cannot be performed in the present state") with NO mention of atomics. Diagnostics: AMD_LOG_LEVEL=3 prints the truth (Pcie atomics not enabled, hostcall not supported → AQL dispatch failed! → hipLaunchKernel: Returned hipErrorIllegalState); lspci -vv -s gpu on the endpoint AND every upstream bridge must show AtomicOpsCap: Routing+; the endpoint's Device Control 2 (setpci -s gpu CAP_EXP+28.w) must have bit 6 set. Setting that bit by hand does NOT help — the driver consumes it at probe time. Traps: a single-rank (world_size=1) collective succeeds on either card because NCCL short-circuits, so one-card tests prove nothing; the failing kernel version was ruled out across 7.0.0-28/-30/-31; a from-source RCCL rebuild reproduces the identical failure, so the library is never the bug. Workaround when only one card is atomic-capable: two single-GPU TP=1 replicas behind a round-robin proxy (vLLM's OpenAI API is stateless per request). Related KV trap found the same night: passing --kv-cache-memory 0 explicitly means "profile this shape" (radiance-patched vLLM); BLANKING the variable so the flag is omitted makes vLLM size KV from --gpu-memory-utilization alone and the allocation OOMs right after weights load._(context: Dankserver frank-medium (2x R9700, AM5) 2026-09-16: TP=2 radiance MXFP4 could not launch any collective; root cause traced to PCIe atomic routing on the second card's chipset path; deployment changed to two TP=1 replicas + round-robin proxy, deployed green with benches (conc1 42.8, conc4 147.6, conc8 230.8 tok/s; TPOT 23.6-28.8 ms; 32k prefill ~2.8k tok/s per card).)_
-vllm-radiance 0.9.3 (stilldeadcode/vllm-radiance + ggz14/radiance-vllm-mxfp4) is RDNA-4 (gfx1201) end to end — NOT portable to Strix Halo (gfx1151 / RDNA 3.5) as published. Confirmed live (frank-medium image torch arch list = ['gfx1201']; no radiance image on the Strix Halo host — the only gfx1151 image there is the stock ComfyUI base, a different vLLM 0.16.0) and by primary source (Dockerfile ARG GFX_ARCH=gfx1201; README "Compiled for gfx1201 only, won't run on other GPUs. Two GPUs (TP=2) is the only configuration tested"). THE HARD BLOCKER is the MXFP4 W4A8 GEMM: it is built on __builtin_amdgcn_wmma_f32_16x16x16_fp8_fp8_w32_gfx12 (fp8 WMMA), which clang gates on gfx12-insts (RDNA 4); there is NO fp8 WMMA for gfx11-insts (RDNA 3.5). The kernel's own comment: "The only way to the 2x is v_wmma_f32_16x16x16_fp8_fp8 by hand." So recompiling it for gfx1151 won't work — you'd dequantize weights to f16/int8 and use the f16/int8 WMMA, losing the exact fp8 speedup that makes MXFP4 fast. Component status for gfx1151: (a) image gfx1201-only; (b) aiter unified-attention Triton path DOES support gfx1151 (explicit tuning: waves_per_eu=8 if arch=="gfx1151", "tuned for the Qwen3-Omni"; gfx1151 is in RDNA_ARCHS) but fp8 ATTENTION is excluded (FP8_ARCHS={gfx942,gfx950,gfx1200,gfx1201} omits gfx1151) → bf16 attention; (c) GDN fused kernels use bf16/int8 WMMA (have gfx11 forms) → port with a source change; (d) R4D one-shot allreduce is a multi-GPU collective, moot at TP=1 (single Strix Halo). To port: gfx1151 image rebuild (build is parameterized by GFX_ARCH, possible in principle, never published) + patch aiter's gate (radiance enables it for on_gfx12x() only) + bf16 KV/attn + rebuild GDN/R4D kernels with gfx11 WMMA + replace the W4A8 GEMM (lose the 2x). "Work well" is separate from "run at all": even ported, the W4A8 GEMM + attention are bandwidth/compute-bound and Strix Halo's LPDDR5X is far slower than the R9700's HBM3 (RDNA 3.5 is a gen behind RDNA 4); Strix Halo's advantage is m_(context: serverAppConfigs research 2026-09-21: "does radiance-vllm optimizations also work well for strix halo, or just r9700s?" — live probes (torch arch list, aiter JIT, image inventory) + librarian primary-source research (StillDeadcode/vllm-radiance Dockerfile/README, ROCm/aiter v0.1.17, StillDeadcode/libr4d, ggz14/radiance-vllm-mxfp4, clang BuiltinsAMDGPU.def).)_
-serverAppConfig deploy.sh pre-flight gotcha (2026-09-16): it requires a WHOLE-TREE clean git status --porcelain (untracked included) AND the main checkout on branch main — a linked git worktree fails the "must run from the main branch" check, so worktrees cannot bypass a dirty tree. When another agent's uncommitted files block a needed deploy, do NOT commit their WIP; use a targeted stash cycle: git stash push -m "deploy-window: park peer WIP" -- peer files && ./scripts/deploy.sh target ... && git stash pop. The stash survives a conflicting pop (kept on failure), the peer's edits are restored byte-identical, and the window is seconds. Check git status --short afterward to confirm the peer's files are back. _(context: 2026-09-15 frank-large deploy: deploy.sh failed twice (dirty tree from peer files; worktree rejected by main-branch check), then succeeded via targeted stash cycle; peer WIP verified restored (git status identical, stash dropped cleanly).)_
-OpenWebUI (dankclaw) LLM wiring — three non-obvious facts, verified 2026-09-21: (1) The model picker is built from the model TABLE in webui.db ONLY (Models.search_models, filtered on base_model_id != None), NOT live-merged with connection-discovered models — a fresh data volume shows whatever rows are seeded, and connection discovery alone will not populate the picker. (2) llama.cpp serves /v1/models in Ollama shape ({"models":[{"name":...}]}); OpenWebUI's extract_data() only reads the OpenAI {"data":[{"id":...}]} shape, so connection discovery SILENTLY misses llama.cpp endpoints (frank-large :7770, frank-small :7771) → 500/404 'Model not found'. Fix: set model_ids in each connection's api_config (when non-empty, OpenWebUI builds the model list directly and never fetches /v1/models); also set base_model_id = the model name on each model row so the picker filter passes. vLLM (frank-medium) serves the OpenAI shape and would discover fine. (3) The open-webui container sits on a netavark BRIDGE; it can hairpin to a DIFFERENT host's LAN IP (dankserver 192.168.0.235) but NOT to its own host's non-published ports via the host LAN IP (192.168.0.143:7770/7771 → connection refused). host.containers.internal (169.254.1.2) resolves on the bridge and reaches both on-host llama.cpp services — this CORRECTS the stale OpenWebUI.md gotcha that claimed host.containers.internal is 'slirp4netns-only and does not resolve on a bridge' (verified end-to-end: all 3 models returned completions). The model table + connection configs are DB-only (not git-managed); a fresh volume loses them. Hermes surfaces: web UI client = hermes.franksdank.cloud (:8787), agent management dashboard = hermesadmin.franksdank.cloud (:9119); Pi-hole (replaced the retired dnsmasq quadlet 2026-08-24) admin = pihole.franksdank.cloud (:8083, 403 basicauth counts as up).
-CORRECTION (2026-09-16): the "e2e 11-admin.spec fails on 192.168.0.112:8989 from blankpad — environmental, expect 29/30" lesson is STALE. On 2026-09-16 tests/e2e/run.sh passed 30/30 from blankpad, including all three 11-adminspecs, with no known intervention. Do not pre-declare the admin-SSO specs environmentally doomed or treat 29/30 as the expected baseline — run the suite and read the actual result. The old diagnosis (blankpad TCP-path to the arr ports blocked while peers succeeded) no longer reproduces; if an admin spec fails again, re-diagnose fresh rather than citing the old blocker. _(context: 2026-09-16 frank-large ik cutover wrap-up: tests/e2e/run.sh from blankpad returned 30/30 passed including specs/11-admin.spec.ts against 192.168.0.112 raw ports; repo commits 795b0b3/ad7a9c4.)_
-serverAppConfig homepage (www.franksdank.cloud, /admin/) — how to verify a card/link change is actually live: (1) Panels are rendered CLIENT-SIDE by shared/homepage/app.js from services.json + status.json; the raw HTML (fetched via curl) is just the shell, so grepping the fetched HTML for a link's href gives a false negative. Verify via the DEPLOYED /srv/homepage/services.json on the serving host (Dankclaw) and/or the e2e admin-link test (tests/e2e/specs/11-admin.spec.ts "admin open links reach their apps"), not by grepping HTML. (2) scripts/homepage-status.py probe() treats ANY HTTP response — including 403 — as "up" (it only returns "down" on a connection/URLError). So a basicauth-gated service (Pi-hole admin at 192.168.0.143:8083 → 403) probes up fine; no need to find a 200 path. (3) A probe of 127.0.0.1:X is relative to the HOST RUNNING THE CHECKER. Running homepage-status.py on the dev box (the workstation) gives a false "down" for host-local probes like hermes-webui 127.0.0.1:8787; scp the script+config to the serving host (Dankclaw) and run it there. (4) Pi-hole FTL (a dnsmasq fork) retired the standalone dnsmasq quadlet on 2026-08-24 — the homepage had carried the stale "dnsmasq" card (probe-only, no link) until it was replaced with a linked Pi-hole card (2026-09-21, commit af36595). Hermes has two surfaces: the web UI client (hermes.franksdank.cloud :8787) and a separate admin dashboard (hermesadmin.franksdank.cloud :9119).
- frank-large ik_llama.cpp performance campaign (2026-09-16): decode ~3.8-4.3 t/s is gated by a SERIAL per-token critical path — ~100 CPU↔GPU backend splits/token with the 3090 at 5% SM utilization (launch/sync-bound across the PCIe x4 dock); thread count is irrelevant (T1≈T16); prefill ~192 t/s sits at the CPU AVX512-VNNI compute ceiling for CPU-resident experts. The 30 t/s halogen bar is unreachable on this model+hardware: ik's CUDA-graph capture aborts on qwen4exp's per-layer ggml_repeat_4d hc-broadcast ops (capture-illegal op), mainline llama.cpp 37b53fd runs 1.0 t/s (CPU MoE kernels 4x slower than ik's iqk), the GGUF has no MTP head (mradermacher stripped it), and 84 GiB of experts cannot fit 24 GiB VRAM. Re-evaluate when mainline's hc-ops + graph handling mature (ggml-org PR #28901). DANGERS learned: (1) systemd Restart=always respawns a pkill'd llama-server ~4s later — manual experiments MUST sudo systemctl stop the unit first or measurements get phantom instances; (2) ik --defer-experts does NOT keep experts off the GPU under -ngl 999 (only PLE defers) → cudaMalloc OOM crash at 79.7 GiB; (3) ik -ot/-ncmoe overrides disable mmap for the whole load (create_tensors returns false); (4) -ncmoe N without GGML_CUDA_NO_PINNED_WEIGHTS=1 page-locks ~108 GiB (the co-tenant killer); (5) never run -j16 builds concurrently with decode measurements. _(context: 2026-09-15/16 overnight frank-large decode campaign on dankclaw: goal was halogen's ~30 t/s decode; measured evidence chain (perf pause-spin analysis, T1=T16 sweep, nvidia-smi 5% SM during decode, graph splits=100, mainline comparison 1.0 t/s, GGUF MTP-head check) recorded in serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md, commit 795b0b3.)_
- frank-medium (dankserver :7770, vllm-radiance TP=2, 2x R9700) was cut on 2026-09-21 to the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, quark/W4A8, 19.37 GiB) with UNQUANTIZED bf16 KV (--kv-cache-dtype auto), replacing the brief FP8/fp8-KV shape (orcarouter/...FP8, 09-20→09-21). Three durable facts: (1) CAUSAL: the endpoint hardcoded --kv-cache-dtype fp8 on the COMMON exec path (outside the IS_MXFP4 quant branch) in every commit since the MXFP4 era, so BOTH FP8 and MXFP4 ran fp8 KV identically — fp8 KV therefore CANNOT explain the FP8-vs-MXFP4 throughput gap (92.4 vs 100.0 t/s); that gap was weight precision. fp8 KV stacked on MXFP4's already-W4A8 weights compounds attention-path precision loss → a QUALITY (not throughput) hypothesis. KV_DTYPE is now a shared quadlet tunable (entrypoint default fp8); quadlet sets auto. (2) VLM: the MXFP4 checkpoint DOES serve vision despite config.json language_model_only=true — the Qwen3_5ForConditionalGeneration arch carries the vision tower through the Gemma4 merge path and it is NOT MXFP4-quantized, so vLLM serves image input (verified: solid red 48x48 PNG named "Red"; vLLM would 400 if unsupported). The old quadlet "TEXT ONLY...400 here" comment was wrong; harness models.yml keeps frank-medium input:[text,image]. RULE: don't trust language_model_only=true as evidence vLLM rejects images — probe with a distinctive-color image. (3) POOL: f16 KV = exactly 2x the bytes of fp8 KV per token → pool halves. MXFP4+fp8 was 943,581 tokens; MXFP4+bf16 auto-sized to 486,464 (Maximum concurrency for 262,144: 1). MXFP4's freed weights (19.4 vs FP8's 30.9 GiB) keep the full native 262,144 window fitting (MAXLEN stays 262144); FP8+bf16 would only reach ~192,000 so it needs MAXLEN=192000. Rollback to FP8: SNAP + SPEC + KV_DTYPE=fp8 (+ MAXLEN=192000 only if KV stays bf16). _(context: serverAppConfigs frank-medium MXFP4 + unquantized bf16 KV deploy 2026-09-21, commits a45f396 + d8dbd4f; user hypothesis that fp8 KV was hurting MXFP4 quality.)_
-serverAppConfig repo convention (established 2026-09-16, commit f227786): AGENTS.md is evergreen rules + key lessons only. Fleet-swap history, retired-runtime detail, and per-flag sizing rationale do NOT go back into AGENTS.md — they live in serverAppConfigs-docs/LLM/ runbooks, and AGENTS.md points at them (Runbook Index table). The "Key Lessons Learned" section holds only durable, cross-runtime operational facts (ROCm layer-split, HIP env collisions, mmap/PLE, swap ordering, quadlet/podman gotchas). When a runtime swap or retirement happens: update the ACTIVE topology table and the Runbook Index row, add any new durable lesson, and write the detail into the runbook — do not grow AGENTS.md history sections. Known pre-existing test failure: tests/run.sh not ok 73 (comfyui Containerfile selector bug, test-side) is unrelated to doc changes and does not block commits. _(context: AGENTS.md rewrite session 2026-09-16; Frank approved "lessons + pointer table" for retired runtimes and "condense to operational-critical" for halogen detail via grilling. Commit f227786 (734→412 lines).)_
-frank-medium (dankserver :7770, vllm-radiance) KV-cache dtype is a SINGLE shared flag on the common exec line of radiance-entrypoint.sh — OUTSIDE the IS_MXFP4 quantization branch — so it applies to BOTH the FP8 and MXFP4 checkpoints, not per-path. It was --kv-cache-dtype fp8 for both (confirmed in commits 6b647c1 MXFP4-era and d389d09 FP8-swap). Per-card VRAM budget (31.9 GiB R9700 @ 0.98 util): ~15.29 GiB FP8 weights + ~1.7 GiB HIP graphs leaves ~8.61 GiB for KV = 389,624 tokens at fp8, ~194,800 at f16/bf16 (exactly 2x bytes). So unquantized (auto/bf16) KV CANNOT hold the native 262,144 window — the max window drops to ~192k (set MAXLEN=192000). KV_MEM=0 auto-sizes the pool to fill remaining VRAM; the pool halves on fp8-f16. The fit-budget gate is tests/frank_medium_unit.bats "the KV pool holds a full window" (asserts the maxlen+KV_DTYPE shape when KV_MEM=0). If KV_DTYPE is a shared quadlet tunable, the MXFP4 rollback inherits it — harmless there (MXFP4 weights are smaller, 19.4 GB, so f16 KV + 192k fits with headroom), but no longer byte-for-byte the old MXFP4 shape (fp8 KV + 262144). _(context: frank-medium KV-quantization work 2026-09-21: user asked to turn off KV quant (fp8-auto/bf16); discovered the dtype flag is shared across both quant paths and that f16 KV halves the auto-sized pool below the 262k window.)_
-blankpad skill directories (2026-09-16): the pi-era user skills at ~/.pi/agent/skills and ~/.agents/skills were wiped; omp discovers user skills at ~/.omp/agent/skills (global) and .omp/skills (project), both of which now exist again. Restored 9 skills from source repos: 8 from github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff — code-review and handoff were mislabeled "Local" in the old omp-docs/pi/skills.md; they are from mattpocock) + find-skills from github.com/vercel-labs/skills. Copied as plain dirs (skill/name/SKILL.md, spec-compliant frontmatter) — the npx skills CLI was bypassed because it auto-detects the wrong agent (claude) and doesn't know omp. UNRECOVERABLE until dankware (192.168.0.226, offline, no SSH alias) is back online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base. New skills are discovered on the NEXT session, not the current one. _(context: blankpad skill restoration session 2026-09-16: user noticed /skill:grilling unresolved. Traced pi-era skills (~/.pi/agent/skills + ~/.agents/skills, installed via vercel-labs skills CLI) as wiped; recovered 9 of 13 from source repos on GitHub into ~/.omp/agent/skills.)_
-vLLM logs a non-default args: {...} line at boot (api_utils.py) — the AUTHORITATIVE set of every arg that differs from vLLM's internal defaults. Everything NOT in that dict is at vLLM's stock default. This is the right way to answer "what is vllm endpoint running with / what's still default" (better than diffing the cmdline against remembered defaults). Parsing gotchas: (1) it's Python repr (single quotes), not JSON → ast.literal_eval, not json.loads; (2) it contains enum reprs like DynamicShapesType.BACKED: 'backed' that break literal_eval → re.sub(r"[^]*", "'ENUM'", s) first; (3) the line is long and display-truncates at ~768 chars → save to a file and parse it, don't eyeball. Live argv: podman exec c cat /proc/1/cmdline | tr '\0' ' ' (tr runs host-side; a /proc/... redirect inside podman exec reads the host /proc and returns empty). Runtime-adjusted "defaults" (auto-sized KV pool, mamba block size 16→1664, max_num_scheduled_tokens, CUDA-graph capture) come from the boot log, not the CLI. Rootless quadlet: podman logs = current instance only; journalctl --user -u c spans recreations. _(context: Tabulating frank-medium (dankserver :7770, vllm-radiance 0.9.3 / vLLM v0.27.1) runtime args; the non-default args line gave the authoritative 24-override list; the enum repr and 768-char truncation both broke naive parsing.)_
-The AGENTS.md fleet-topology snapshot embedded in session context can lag the repo: on 2026-09-15 the serverAppConfig tree at HEAD (commits d53962f/4183e31) described a NEWER layout than the context file — frank-medium (Dankserver 2x Radeon AI PRO R9700, gfx1201) replacing frank-small, and frank-large moved to ik_llama.cpp serving orcarouter/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M on Dankclaw's RTX 3090. Before reasoning about the LLM fleet from the AGENTS.md context (service names, models, runtimes, ports), verify against git log --oneline + the tracked serve/env files at HEAD; the context file is a snapshot of a past date. Related trap from the same session: the working tree carried UNCOMMITTED radiance-era edits (scripts/deploy-plan.py, scripts/deploy-remote.sh, tests/deployment_scoping.bats) plus a DELETED test (tests/frank_large_quadlet.bats — deletion must go through .AGENTTRASH per policy, never rm); reconcile git status before any deploy step. _(context: 2026-09-15 frank-medium (dankserver 2x R9700, vllm-radiance MXFP4) cutover session: interrupted while reconciling working-tree state before host-side trash staging. The system-prompt AGENTS.md describes the 2026-09-11/13 fleet; commits d53962f/6085a99 ("serve the new R9700s, retire frank-small") plus scripts/frank-large.env (2026-09-15 header, ik_llama.cpp, dankclaw RTX 3090) show the fleet was re)_
-OpenWebUI (container open-webui on dankclaw, DB /app/backend/data/webui.db): the model picker (/api/models/list → Models.search_models) reads the model table ONLY — an enabled connection does NOT auto-appear in the picker; a model must be a row in the model table. LLM connections live in the config table as position-indexed parallel arrays openai.api_base_urls / openai.api_configs (+ openai.api_keys, openai.enable); Ollama in ollama.base_urls/ollama.enable. RAG: rag.embedding_engine == "" (empty string) means a local SentenceTransformer loaded from rag.embedding_model. To run backend python: podman exec -w /app/backend open-webui python — working dir MUST be /app/backend, the package is open_webui (underscore, not openwebui), and you MUST set os.environ["WEBUI_SECRET_KEY"] BEFORE importing any open_webui module (import-time hard requirement). Model rows are pydantic ModelModel objects (attribute access m.id/m.base_model_id, not .get()). llama.cpp ignores the model field in chat requests, so a stale/renamed model name still routes to the loaded model.
-FLEET STATE UPDATE (2026-09-15): frank-small (FreeToken/CUDA, RTX 3090) on dankserver is RETIRED. The 3090 was removed and replaced with two AMD Radeon AI PRO R9700 (Navi 48, gfx1201, 32,624 MiB each; no NVIDIA device remains; host iGPU gfx1036 hidden from ROCm via HIP_VISIBLE_DEVICES=0,1). The :7770 dankserver endpoint is now frank-medium: charlie12345/ROCmFPX llama.cpp (c49ebdb, built gfx1200;gfx1201 with the fork's self-contained ROCm 7.14 TheRock toolchain, RPATH-bundled — host /opt/rocm-6.3.3 was purged and never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (26.28 GiB Q8_0 + MTP Q4_0 draft pinned ROCm1) with layer split across both cards. Deploy path: ./scripts/deploy.sh dankserver --service frank-medium (unit is a root systemd unit symlinked to the repo file; first deploy retires the old frank-small unit). Clients already cut over: blankpad ~/.omp/agent/{models.yml,config.yml} and Hermes live config on dankclaw (provider key stays 'dankserver-small' as label-only). Any memory/doc still describing dankserver as FreeToken/CUDA frank-small is stale. Trash staged awaiting purge approval: repo scripts/.AGENTTRASH/{frank-small.env,frank-small.service,serve-frank-small.sh}, dankserver ~/.AGENTTRASH/{20260915-frank-small-retire,20260915-rocm633-cleanup}, dankclaw hermes models_cache.json-20260915. _(context: 2026-09-15 GPU swap on dankserver (RTX 3090 out, 2x Radeon AI PRO R9700 gfx1201 in); frank-medium deployed via repo git flow, commits 6085a99/d53962f/d4024a9/db29c4d; runbook serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md; supersedes the fleet-topology lines in older memory (frank-small FreeToken/CUDA on dankserver).)_
-SearXNG (Dankclaw) agent access + limiter, verified in the live :latest image 2026-09-21: (1) Agents/OpenWebUI do NOT go through the Anubis→Authentik guard chain — that chain sits only on the public vhost search.franksdank.cloud and is browser-only (a plain HTTP client gets Anubis's "not a bot" PoW HTML, not JSON). Agents dial the internal host-published port http://192.168.0.143:8081 (gluetun's published 8081:8080; SearXNG shares gluetun's netns), which bypasses Caddy entirely. SearXNG's own SEARXNG_BASE_URL is that internal port. (2) The JSON rate cap is API_MAX=4 req/IP/hour for any non-HTML format — a BARE PYTHON CONSTANT at searx/botdetection/ip_limit.py:82, no config knob in limiter.toml or settings.yml. (3) pass_ip is checked FIRST in searx/limiter.py and short-circuits the whole botdetection chain (return None before user-agent/header/ip_limit), so any pass_ip source is fully exempt from the cap. pass_ip covers 127.0.0.1, 192.168.0.0/24 (LAN), 10.88/10.89.0.0/16 (all rootless podman bridges) — so every on-host agent and LAN client is exempt (verified: 8-req burst from inside open-webui = 8x200, no 429). To exempt a new source, widen pass_ip (config-native); patching API_MAX or server.limiter:false are the only other (fragile/nuclear) options. (4) OpenWebUI web search is wired to SearXNG via per-key SQLite config (webui.db, web.search.*): engine=searxng, searxng_query_url=http://192.168.0.143:8081/search, enable=true. Quadlet env vars (ENABLE_WEB_SEARCH, WEB_SEARCH_ENGINE, SEARXNG_QUERY_URL, SEARXNG_LANGUAGE, WEB_SEARCH_RESULT_COUNT) SEED that table on first start (fill-missing only); afterwards the DB is authoritative and Admin-UI edits persist there. Now git-managed in dankclaw/open-webui/quadlet/open-webui.container + documented in OpenWebUI.md. _(context: Answered "can agents access SearXNG behind Authentik / can we disable the limiter cap" and wired OpenWebUI web search, serverAppConfig repo, commit d63acd4)_
-Dankserver frank-medium (2x Radeon AI PRO R9700, gfx1201) ops facts, measured 2026-09-15: (1) llama-bench multi-GPU gotcha — '-dev ROCm0,ROCm1' runs two SEPARATE single-card benches; a real 2-GPU bench needs '-dev ROCm0/ROCm1'. (2) Parallelism on this RCCL-less ROCmFPX build: layer split (1991 t/s prefill, ~17.6-18.8 t/s tg) beats tensor split (~272 t/s prefill, 7x worse) because tensor mode falls back to the meta-backend butterfly allreduce; with MTP serving, layer also wins decode (35.7-46.2 vs 28.7 tok/s). Draft head must be pinned to ROCm1 — unpinned draft under tensor mode aborts KV build (GGML_ASSERT(buffer)). (3) First-request-after-restart on the engine can go silent ~45-60s (health engine_unresponsive, 503s) then self-recovers — do not restart over it. (4) Request-level speculative:{n_max:0} does NOT disable drafting. (5) Hermes provider KEY 'dankserver-small' is label-only and deliberately NOT renamed (DB references) even though its model id is now frank-medium; repo example dankclaw/hermes/config.yaml.example synced in db29c4d. (6) Trash still staged awaiting purge approval: repo scripts/.AGENTTRASH/{frank-small.env,frank-small.service,serve-frank-small.sh}, dankserver ~/.AGENTTRASH/{20260915-frank-small-retire,20260915-rocm633-cleanup}, dankclaw hermes models_cache.json-20260915. _(context: 2026-09-15 frank-medium bring-up on Dankserver (2x Radeon AI PRO R9700, charlie12345/ROCmFPX llama.cpp, kingjones Q8_0_ROCMFPX-AGENT); runbook at serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md, commits 6085a99/d53962f/d4024a9/db29c4d.)_
-Agents/services (OpenWebUI backend, LLM agents) reach SearXNG via the INTERNAL port http://192.168.0.143:8081 — gluetun's published 8081:8080, SearXNG shares gluetun's netns — NOT the guarded public vhost search.franksdank.cloud (a plain HTTP client hitting that gets Anubis's "not a bot" PoW HTML, not JSON). SearXNG's SEARXNG_BASE_URL is already that internal port and formats includes json. Limiter (shared/searxng/limiter.toml): the JSON cap is a HARDCODED 4 requests/IP/hour in Python; the ONLY bypass is a pass_ip source (127.0.0.1, ::1, 192.168.0.0/24, 10.88.0.0/16, 10.89.0.0/16). Rootless podman strips the real client IP on published-port hits, so the source is the bridge IP (10.88/10.89) → pass_ip → UNRESTRICTED. Verified live from inside the open-webui container: 8-request JSON burst all 200 (no 429), egress IP seen by SearXNG = 10.89.3.2. So on-host/LAN agents are rate-cap-exempt; to wire OpenWebUI web search just point it at http://192.168.0.143:8081. For an OFF-HOST agent that needs a DNS name: add an unguarded search-api vhost straight to :8081 (exempt-by-construction, same pattern as Gitea/OpenWebUI), optionally bearer/IP-gated — NOT an Authentik service token, since the guard chain is proxy-provider inline flow with no forward_auth/token gate.
-frank-large (halogen-flash-server, Dankclaw) pool history and post-restart behavior, measured 2026-09-13: pool trimmed 524288 → 401408 (deployed, repo commits 91406af + 27e6335) to restore co-tenant headroom. Measured boot ledger at 401408/MAX_TOK=32768/reserve 15: 68.0 weights + 11.0 KV + 21.0 working = 100.0 GiB held, host left 9.1 GiB / 1396 contiguous 2 MiB blocks (vs 7.1 / 310 at 524288), engine printed need ~31.5 GiB vs modelled ~33.4, and the co-tenant stall NOTE no longer prints (threshold is somewhere below 9.1 GiB / 1396 blocks). TWO new facts: (1) On the FIRST request after any (re)start the engine can go silent ~45-60 s — /health flips to engine_unresponsive "no PONG within 30s" and the front-end 503s, then self-recovers. Do not restart the unit over it; the quadlet HealthStartPeriod=300s covers it. First completion attempt timed out during this window; a retry 20 s later returned instantly. (2) Host-left gain (+2.0 GiB) came in below the modelled +3.4 because the engine holds freed KV space as working slack (working 21.0 at 401k vs 20.6-ish at 524k with arena). The omp models.yml on blankpad (/home/blank/.omp/agent/models.yml) had frank-large maxTokens: 262144 — above halogen's 65536 cap and a 503-queue hazard after the pool cut; fixed to 65536, and the stale flm (dankclaw-npu) provider + its modelRoles (vision/tiny/task) were removed per the AGENTS.md route map (vision→frank-large, tiny→frank-small, task→frank-large:medium). _(context: frank-large KV pool trim to 401408 on Dankclaw, 2026-09-13, followed by OMP models.yml/config.yml cleanup on blankpad.)_
-When trimming a CPU-side memory cap on a llama.cpp LLM host (e.g. --cache-ram), verify the effect with Pss_Anon from /proc/PID/smaps_rollup, NOT RSS. Pss_Anon is the process's proportional share of anonymous memory — where the cache actually lives — so its delta matches the expected reduction. Measured on frank-small (dankclaw 3090) cutting --cache-ram 8192→2048: Pss_Anon dropped 8.97→3.03 GiB (~5.9 GiB ≈ the 6 GiB delta), but RSS dropped only ~4.3 GiB (9.75→5.48). RSS is misleading because it counts shared + file-backed pages and a CUDA context re-maps file-backed pages on restart, so RSS can drop LESS than the memory actually freed (here Pss_Anon dropped MORE than RSS). Cross-check with host free -h available as the real-world benefit (6.1→13 GiB). Related facts: --cache-ram (env LLAMA_ARG_CACHE_RAM) build default is 8192 MiB; -1 = no limit, 0 = disable. Gemma4-12B is NOT a DeltaNet model, so the repo's "12.8 GiB CACHE_RAM floor" rule (which protects Gated-DeltaNet state) does not apply to it. _(context: frank-small --cache-ram 8192→2048 on dankclaw, 2026-09-21. Verify via Pss_Anon delta (matched the 6 GiB cap reduction) while RSS under-reported the drop; host available doubled.)_
-Dankclaw host swap (as of 2026-09-12): 64 GiB file at /swap.img, ext4 on /dev/nvme0n1p2, fallocated hole-free, driven by /etc/fstab line 12 (/swap.img none swap sw 0 0) which generates the swap.img.swap systemd unit — fstab is the only control point, there is no hand-written unit and no repo file, so deploy.sh cannot see swap state. vm.swappiness=60 (distro default, UNTESTED against this workload), zswap disabled. Raised from 8 GiB; 748 GiB still free on /. The old 8.1 GiB file is staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img awaiting Frank's approval to purge. Important framing: swap on this host protects the ~28 co-tenant containers, NOT frank-large — its ~68 GiB of pinned weights are not relieved by swap, and paging the KV pool (which IS anonymous) would be worse than the OOM kill it prevents, so if the box thrashes the knob is lower swappiness or a cgroup cap, not more swap. _(context: Raised swap on Dankclaw 2026-09-12 after the frank-large reserve-15 config left only ~7.1 GiB for co-tenants. Documented in serverAppConfigs-docs/Dankclaw/Swap Configuration.md and AGENTS.md memory-model section.)_
-Homepage stale-cache trap (serverAppConfig fleet, 2026-09-21): deploys overwrite /srv/homepage in place and Caddy file_server sends no revalidation headers, so a user's browser keeps serving pre-deploy CSS/JS/HTML — the new sectioned board rendered VERTICAL for Frank while a fresh profile proved the live page was correct. Fix (commit 8ea6486): the @home handler on the post-auth :8017 listener in dankclaw/Caddyfile sends header Cache-Control "no-cache" (ETag round-trip, cheap 304); pinned in tests/homepage_caddy.bats. RULE: forany user-facing static surface deployed in place, verify revalidation headers exist or users see stale assets. Live-page verification trick that worked well: reuse the e2e Playwright storage state (tests/e2e/.auth/qa-e2e.json, user; qa-e2e-admin.json, admin) — map its cookies and page.setCookie(...) in a headless browser, then navigate to the guarded vhost (bypasses the Anubis+Authentik chain); measure layout with getBoundingClientRect and read response headers with fetch(url, {cache:'no-store'}).
-In the serverAppConfig repo, ./tests/run.sh exits 1 on a **clean tree** because of one pre-existing failure: tests/comfyui_rocm.bats test 73 "comfyui Containerfile is based on the gfx1151 torch image". It is a test-side selector bug, not a product regression — its find ... | head -1 picks dankclaw/alloy/Containerfile instead of the ComfyUI one, so the assertion reads the wrong file. Verify before blaming your own change: a good run reports 285 ok plus this single not ok 73. Confirmed present at HEAD across several sessions (2026-09-12 frank-large work), independent of frank-large/Caddy/quadlet edits. Do NOT treat a green deploy as blocked by it, and do NOT "fix" it by weakening the assertion — the correct fix is to target the actual ComfyUI Containerfile path. Note deploy.sh gates on its own per-service health plus the Playwright e2e suite, not on tests/run.sh, so this failure does not block deploys. _(context: Every ./tests/run.sh run during the 2026-09-12 frank-large KV-pool/reserve work returned exit=1 with exactly not ok 73 comfyui Containerfile is based on the gfx1151 torch image while 285 tests passed, including on commits where nothing ComfyUI-related changed.)_
-Frank's homepage (www.franksdank.cloud) layout and copy preferences, set 2026-09-21: (1) Sections render as horizontal ROWS of panels (auto-fill grid, as many per row as the viewport allows) with sections stacking vertically — "render the sections horizontally" means panels-in-a-row, NOT side-by-side section columns (that was the wrong read and got reverted). (2) User page section order: Web Apps, Media Library Apps, AI — the AI section sits at the BOTTOM. Admin page order: Media Library Apps, Observability, AI, Infrastructure. (3) AI model card + /docs#models descriptions are MODEL-ONLY: hosted model, size, context/parallel capacity, brief what-it's-for. "My users don't care about the infrastructure" — no runtimes, GPUs, tensor parallelism, or draft heads in user-facing copy. Sizes: frank-large 87.3 GiB UD-IQ4_XS, frank-medium ~31 GiB FP8, frank-small 7.4 GiB Q4_K_M. Pinned in tests/homepage_pages.bats (row layout + AI-at-bottom order greps).
-Frank resolves contested tuning calls by **running the experiment**, not by reading the argument. On 2026-09-12 he overruled documentation-based pushback twice in a row with "lets just try it". Working with this, rather than against it: when he pushes back on a knob, state the prediction numerically up front, then run it and report measured vs predicted — he engages with that and it twice produced a better model than the docs (the arena linearity, and the vision-tower term). Corollaries: (1) get the rollback staged and named BEFORE the risky deploy, because the experiment may take production down — the 65536 trial OOMed frank-large for ~2 minutes; (2) use deploy.sh --skip-e2e for risky single-service experiments so the 4-minute e2e gate is not sitting on top of an outage; (3) after a failure, git revert sha + redeploy restores in ~10s, because the weights are already page-cached. Also learned the hard way: when a config experiment ends in a state that quietly reverses the user's original request (32768 + KV_POOL_FIT clamp meant the pool stayed 262144, i.e. the doubling never happened), say so plainly — do not let a green health check imply the goal was met. Separately, on log forensics: journalctl --user -u frank-large.service did NOT reach back to the 2026-09-11 boot even when asked with --since 2026-09-11 (rootless user journal retention is short). To compare against a container's ORIGINAL startup ledger, capture podman logs BEFORE the first deploy of the session — that is the only reliable copy of pre-change boot state. _(context: frank-large KV-pool doubling work on 2026-09-12: Frank overruled doc-based pushback twice ("let's just try it") — once raising HALOGEN_MAX_TOK to 65536 (OOMed the production service for ~2 min) and once back to 32768, which quietly un-did the doubling he had originally asked for. Also: journalctl --user retention for frank-large.service did not cover the 2026-09-11 boot even when queried with --si)_
-frank-large (dankclaw, Strix Halo APU, unsloth llama.cpp b11030 gfx1151, Qwen3.8-Flash-Next UD-IQ4_XS 87 GiB 3-way split, -c 200000 -np 1 -ngl 999) — where the memory goes, measured 2026-09-21. REAL iGPU memory = /sys/class/drm/card*/device/mem_info_gtt_used (~70 GiB of a 113 GiB GTT aperture); rocm-smi VRAM is MISLEADING on this APU (reports the 2 GiB carveout, ~172 MiB used). Process RSS (~28-30 GiB, mostly anon) UNDERCOUNTS total LPDDR because GTT allocations are separate from the process page tables; system free showed ~115/121 GiB used. Of the 87 GiB model: ~70 GiB (MoE experts + attention + MTP) is RESIDENT in GTT (MoE must keep all experts in memory); ~17 GiB (the PLE n-gram embedding tables, blk.N.ple_*.weight) is STREAMED from NVMe on demand via --lazy-mode auto (the DEFAULT — the serve script never overrides it). --lazy-mode and -ngl 999 COMPOSE, not conflict: -ngl picks the compute device, lazy-mode controls data residency for per-layer embedding tensors 4 GiB. Proof of streaming = disk I/O during inference: watch /proc/PID/io read_bytes — a NEW prompt raises it ~1-15 MiB, a REPEAT prompt raises it 0. KV cache is only ~4.6 GiB at 200k ctx because Qwen3.8-Flash-Next is a hybrid SSM model: block_count=48, full_attention_interval=4 (only 12 full-attention layers, 36 Gated DeltaNet with fixed per-slot state) + aggressive GQA (head_count=24, head_count_kv=2, key/value_length=256, f16 KV = 2 heads × 512 × 2 B × 12 layers × 200192 tok). So the memory wall is the MoE experts, NOT the KV: shrinking -c saves little, -ot exps=CPU is the real footprint lever. To read GGUF arch values: llama-gguf part1.gguf r lists KEYS only (no values) — parse the metadata with a Python GGUF reader (value types 4=u32,5=i32,6=f32,8=str,9=array,10=u64,11=i64; part 1 has n_tensors=0, it is all metadata). _(context: frank-large cutover follow-up Q&A on dankclaw: user asked whether the PLE/engrams stream from disk, whether lazy-mode conflicts with -ngl 999, and how large the KV cache is. Answered by measuring mem_info_gtt_used, free -h, /proc/PID/smaps_rollup, /proc/PID/io across new-vs-repeat prompts, and parsing the GGUF metadata.)_
-CORRECTION to the earlier "e2e 11-admin.spec.ts failing on 192.168.0.112:8989 means the Mac is asleep — wake it and re-run" lesson. That diagnosis is incomplete and can send you in circles. On 2026-09-12 the admin SSO spec failed on page.goto http://192.168.0.112:8989/ with ERR_CONNECTION_TIMED_OUT, and waking the Mac did NOT fix it: the Mac was verifiably awake (ping 33-54ms), all six socat forwarders were bound to 192.168.0.112, macOS Application Firewall reported State = 0 (disabled), netstat showed 8989 listening, and from **dankclaw** the ports answered 8989→200, 7878→302, 9696→302. Yet from **blankpad** (the dev box where Playwright actually runs) the same ports returned curl rc=28 / 000, while ICMP ping to .112 succeeded. So the blocker is a blankpad↔192.168.0.112 path problem — ICMP allowed, TCP to the arr ports not — not sleep and not the Mac's own firewall. Distinguish the two causes by probing from the box the browser runs on, not from a fleet peer: curl -s -o /dev/null -w '%{http_code}' --max-time 7 http://192.168.0.112:8989/ on blankpad vs ssh dankclaw 'curl ... 192.168.0.112:8989'. Peer 200 + local timeout = path/firewall issue, and no amount of waking or re-running will make that spec pass from this box. Consequence: the mandated e2e gate currently cannot go fully green from blankpad; 29/30 pass, 11-admin fails on Sonarr's raw IP. It is environmental and unrelated to what is being deployed — verify the deployed service directly (ledger, /health, real completion) rather than treating the red gate as a regression, and say which gate was skipped and why. _(context: 2026-09-12 frank-large KV-pool work: e2e admin SSO spec failed on Sonarr raw IP while the Mac was provably awake and dankclaw reached it fine; blankpad could not open the TCP ports at all.)_
-frank-large (dankclaw, Strix Halo APU iGPU, unsloth-llama b11030 serving Qwen3.8-Flash-Next UD-IQ4_XS): the PLE n-gram embedding table ("engrams") is LAZY-STREAMED from NVMe by default, not resident in LPDDR. Mechanism: llama.cpp --lazy-mode defaults to auto = on for per-layer-embedding tensors 4 GiB; the serve script (scripts/serve-frank-large.sh) does NOT override it, so it runs at that default. The PLE is exactly that shape: blk.N.ple_{key,value,norm_*,conv1d}.weight across 48 blocks, each head a ~20M-row table (qwen4exp.ple.head_vocab_sizes = [20000003, ...]). Consequences: model RSS ~28 GiB (not the 87 GiB file), Pss_File ~204 MiB, fast load (~34s), fits in shared LPDDR; cost is a little NVMe I/O per NEW n-gram, and the page cache keeps repeated/similar content at 0 disk cost. To make it fully resident: --lazy-mode off (pushes whole table into LPDDR, slower load, no gain here). DIAGNOSTIC METHOD (reusable, to prove lazy tensor streaming on any llama.cpp endpoint): run a throwaway server on a scratch port with the prod flags, then measure /proc/PID/io read_bytes (actual disk I/O, NOT rchar) before/after a completion. Signature = content-dependent disk reads: a NEW prompt reads ~1-15 MiB from disk (PLE rows for new n-grams), a REPEATED prompt reads 0 MiB (rows now cached). Cross-check with /proc/PID/smaps_rollup: Pss_File tiny vs large model = table not resident. Always kill the throwaway and sudo systemctl start frank-large (root unit) after; verify :7770 health + a completion. _(context: frank-large PLE lazy-streaming investigation, 2026-09-21, serverAppConfigs. User asked "is it set to stream the engrams from disk instead of memory?")_
-halogen-flash-server (frank-large, Dankclaw) device-RAM sizing is ONE shared budget, and it is now measured rather than estimated: device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB (121 - 67.7 - 24.0 = 29.5 GiB here, plus a 1.5 GiB vision margin). The KV pool costs ~28 KiB/position (524,288 = 14.4 GiB) and the PREFILL ARENA IS LINEAR IN HALOGEN_MAX_TOK at 3.80 GiB base + 8.40 GiB per 16,384 tokens — verified against three restarts to 0.1 GiB. So the ceiling is max_tok = (budget - margin - pool - 3.8)/8.4 * 16384; with the doubled pool that is max_tok = 19,115, making the deployed 16384 the CEILING, not a conservative setting. Corner check confirms: max_tok 32768 leaves 7.4 GiB = ~269k positions, exactly the old 262144 config. TWO DIFFERENT FAILURE MODES when you violate this: (a) under the ceiling, HALOGEN_KV_POOL_FIT silently LOWERS the pool and deploy.sh still reports healthy in 12s — you think you doubled it, you did not; (b) OVER the ceiling the arena alone outgrows the budget and the server WILL NOT BOOT at all — measured at max_tok=65536: "WARNING even 262144 positions ... needs ~44.5 GiB", then "dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB (out of memory)". Mode (b) is the dangerous one because systemctl stays active, the container reads Up (starting), and deploy.sh waits out the full LLM_WAIT instead of failing fast — an outage that looks like a slow deploy. Always grep podman logs frank-large for kv pool: and confirm the reserved position count equals the requested one; /health 200 proves nothing about pool size.
-"Change not visible on the live homepage" diagnosis (serverAppConfig fleet): deploys overwrite /srv/homepage in place, so a user's browser can keep serving pre-deploy CSS/JS/HTML from cache — new HTML (e.g. class="sections") running against stale CSS silently falls back to the old layout, and the user reports "it's still vertical" while the server is correct. Procedure: (1) grep the deployed file on the host (server-side truth); (2) prove it in a FRESH headless profile by reusing the e2e SSO storageState — tests/e2e/.auth/qa-e2e.json (user) / qa-e2e-admin.json (admin) are regenerated by every e2e run; load them with page.setCookie(...cookies) (map expires0 only) then navigate to https://www.franksdank.cloud/ — no need to drive the Anubis/Authentik login UI. Measure getBoundingClientRect/computed styles, screenshot. Permanent fix landed 2026-09-21 (commit 8ea6486): the @home handler on the post-auth :8017 Caddy listener sends Cache-Control: no-cache (ETag revalidation, cheap 304), so any deploy is visible on the next navigation; tests/homepage_caddy.bats pins the header. Also learned the same session: "render the sections horizontally" from Frank meant panels in horizontal ROWS per section (sections stack vertically), NOT sections side by side — ambiguous layout asks get a screenshot before shipping. _(context: Homepage sectioning session 2026-09-21: user reported the horizontal section layout was "still vertical"; live file was correct, browser cache was the cause; verified via fresh-profile drive with e2e storageState cookies; added no-cache header to the Caddyfile @home handler.)_
-The e2e suite's 11-admin.spec.ts navigates to the *arr services at their raw LAN IPs (192.168.0.112:8989 Sonarr, 7878 Radarr, 9696 Prowlarr), and Playwright runs from the DEV BOX (blankpad), not from Dankclaw. There are therefore TWO independent ways that spec fails, and they need different fixes: (1) The known one: FrankBook-Pro asleep - arr ports unreachable from everywhere. Wake it, re-run. (2) NEW, found 2026-09-12: blankpad cannot reach those forwarded ports AT ALL, even with the Mac fully awake. Measured: from blankpad, ping 192.168.0.112 succeeds (33-54 ms, 0% loss) but TCP to 8989/7878/9696 all return curl rc=28 (connection timeout). From dankclaw the same ports return 200/302/302. On the Mac itself: socat forwarders ARE running and bound to bind=192.168.0.112, curl 127.0.0.1:8989 - 200, and socketfilterfw --getglobalstate reports "Firewall is disabled (State = 0)". So the Mac is not the blocker — something on the blankpad-FrankBook-Pro path drops TCP to those ports while passing ICMP. Diagnostic order that separates them, before assuming "the Mac is asleep": a. From the dev box: curl -s -o /dev/null -w '%{http_code}' --max-time 7 http://192.168.0.112:8989/ AND ping -c2 192.168.0.112. b. From dankclaw: same curl. If dankclaw gets 200 but the dev box times out while ping works, it is failure (2), not sleep — re-running the spec will never go green from the dev box. c. On the Mac (via ssh dankclaw "ssh dank@192.168.0.112 ..."): pgrep -fl socat, curl 127.0.0.1:8989, firewall state. Implication: a red 11-admin.spec.ts is NOT automatically evidence of a fleet outage, and it is NOT caused by whatever was just deployed. Confirm which failure mode by comparing the dev-box probe against the dankclaw probe before attributing the failure to your change. _(context: 2026-09-12, after doubling frank-large's KV pool on Dankclaw. The e2e gate's 11-admin.spec.ts failed on http://192.168.0.112:8989/ (ERR_CONNECTION_TIMED_OUT) on both the initial run and retry. Waking the Mac did NOT fix it, which is why this qualifies the older sleep lesson.)_
-Serving a concatenated split GGUF is a trap: concatenating the three -0000N-of-00003.gguf parts yields a byte-correct single file, but part 1's header still carries ggml.file_count=3, so llama.cpp's loader "advertises itself as a split" and dies with llama_model_load: error loading model: invalid split file name (the unit crash-loops, restart counter climbs, yet an older orphan llama-server may still hold the port and answer /health — so the endpoint looks up while the unit is down). The fix (peer commit d6f9b85): point -m at PART 1 and let llama.cpp load the split natively (parts 2-3 auto-discovered in the same directory) — do not concatenate for serving. This bit frank-large on 2026-09-21 after a re-download. Diagnosis shortcut: ls -la the payload dir and compare the concatenated file's byte size to the sum of the parts (a mismatch like exactly 1e9 bytes short = truncated concat), and read the full journalctl -u svc line for the "invalid split file name" path.
-halogen-flash-server (frank-large on Dankclaw): raising HALOGEN_KV_POOL_POSITIONS can SILENTLY NOT TAKE EFFECT, while the deploy still reports healthy. HALOGEN_KV_POOL_FIT=1 (default) measures the device budget at startup and lowers the pool itself if the configured value will not fit — and it never raises. The budget is ONE pool covering the KV AND the prefill arena together: host RAM − resident weights − HALOGEN_HOST_RESERVE_GIB. Measured on Dankclaw 2026-09-12: 121 GiB − 67.7 weights − 24.0 reserve = 29.5 GiB device budget. Pool 524288 with HALOGEN_MAX_TOK=32768 needs ~35.0 GiB (+1.5 margin) → refused, silently lowered to 262144, deploy reported "healthy after 12s". Fix: HALOGEN_MAX_TOK=16384 halves the arena (~20.6 → ~12.3 GiB working mem), bringing the 524288 need to ~26.6 GiB, which fits with ~2.9 GiB margin at ~8-9% prefill cost. Net result was CHEAPER than the old config: 94.6 GiB held / 13.8 GiB host-left vs 96.3 / 12.1. RULE: after ANY halogen restart, read podman logs frank-large | grep 'kv pool:' and confirm the pool it reserved matches what the unit asked for — a green /health and a green deploy.sh say nothing about pool size. The engine prints "LOWERING THE POOL TO N" in the refusal; that line is the tell.
-When a user reports a homepage/static-site change "isn't showing" after a green deploy, suspect browser cache BEFORE doubting the deploy: deploys overwrite /srv/homepage in place, and with no revalidation header the browser keeps serving the pre-deploy CSS/JS/HTML (new HTML ran against old CSS and silently fell back to the old layout). The fix (commit 8ea6486): the @home handler on Caddy's post-auth :8017 listener sends Cache-Control: no-cache (ETag round-trip, cheap 304), pinned by tests/homepage_caddy.bats. To PROVE what the live page actually renders — independent of the user's cached tab — drive it in a FRESH headless profile using the e2e SSO cookies: load tests/e2e/.auth/qa-e2e.json (user) or qa-e2e-admin.json (admin) via page.setCookie(...cookies), goto https://www.franksdank.cloud/, waitForFunction on '#grid .panel', then read getBoundingClientRect of '#grid .section' (equal y = horizontal, distinct x = columns) or getComputedStyle. The e2e .auth files are regenerated by every deploy's e2e gate, so they're current. The user still needs one manual hard-refresh of their already-open tab (stale CSS is in memory until reload).
-frankbook-pro (FrankBook-Pro.local, macOS 192.168.0.112) is reachable from this dev box ONLY via the dankclaw jump host: ssh dankclaw "ssh dank@192.168.0.112 'bash -l -s'" script.sh (fleet inter-host key works for dank@112; frank@/admin@ are denied; NO passwordless sudo on the Mac — sudo -n fails). The *arr stack (sonarr 8989, radarr 7878, readarr 8686, lidarr 8787, prowlarr 9696, qbittorrent, gluetun) runs inside podman-machine-default (applehv); on the macOS host, socat forwarders bind 192.168.0.112:port → 127.0.0.1:port (gvproxy → VM). macOS Application Firewall is DISABLED and /etc/pf.conf is stock. THE FLAKE: when the Mac sleeps, the arr ports go unreachable from the LAN (curl connect 0.000s refused / ERR_CONNECTION_TIMED_OUT in Playwright), which fails the e2e 11-admin.spec.ts SSO test. Waking the Mac (SSH/ping) restores all ports to 200/302 within seconds. So an admin-SSO e2e failure on a *arr link is usually "the Mac is asleep" — wake it and re-run before treating it as areal outage. Diagnostic order: (1) ping 112 (high latency ~150ms = asleep/waking), (2) from the Mac host curl 127.0.0.1:8989 and curl 192.168.0.112:8989 (both 200 = chain healthy, block is external/sleep), (3) podman machine list + podman ps -a (Up = services fine). _(context: Diagnosing the e2e admin SSO test failing on Sonarr raw-IP (112:8989) during the 2026-09-12 marreta work — turned out to be frankbook-pro sleeping, not a real outage.)_
-Serving a llama.cpp model that ships as split parts (HF caps uploads at 50 GB, so any 50 GB GGUF is multi-part): do NOT concatenate the parts into a single file to serve. Part 1's GGUF header carries ggml.file_count=N, so a byte-correct concatenation still advertises itself as an N-way split, and the loader rejects a non-split filename with "invalid split file name" (fails in ~50 ms, before reading any weights — reads like a corrupt file but the bytes are fine). Fix: point -m at part 1 (name-00001-of-NNNNN.gguf) and let llama.cpp load the split natively; parts 2..N are auto-discovered in the same directory. The split parts ARE the canonical payload. Caught 2026-09-21 on frank-large (unsloth Qwen3.8-Flash-Next UD-IQ4_XS, 3-way split, 87.3 GiB) — the assembled file crash-looped the systemd unit ~110 times before the diagnosis. Generalizes to every multi-part GGUF. _(context: frank-large cutover to unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on dankclaw, 2026-09-21; serverAppConfigs repo, runbook serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md)_
-authentik embedded-outpost proxy providers: a ProxyProvider is INVISIBLE to the Rust embedded outpost unless it is linked to an authentik Application. The outpost's config API (ProxyOutpostConfigViewSet in authentik/providers/proxy/api.py) filters ProxyProvider.objects.filter(application__isnull=False). So a provider created via ProxyProvider.objects.create(...) with no Application never reaches the outpost, and requests to its external_host fall through to raw flow serving on the app's own host — where the IdP session cookie (scoped to auth.domain) is invisible, so the user is stuck on a login wall (e.g. /if/flow/default-authentication-flow/ or /flows/-/default/authentication/). FIX: create Application(name=..., slug=..., provider=proxy_provider, launch_url=...) and restart authentik-server + the outpost. Also required for the same provider to work: invalidation_flow must be a real UUID (null breaks the Rust outpost's serde with "invalid type: null, expected a formatted UUID string" and takes down ALL providers on that outpost), and external_host must exactly match the Host header (trailing-slash mismatch = no match). Diagnose by: (1) ak shell → list providers with application/invalidation_flow; (2) journalctl --user -u authentik-outpost.service | grep -iE "error|invalid" for the serde error; (3) compare a working provider's full field set against the broken one. _(context: Fixed marreta.franksdank.cloud login-wall on Dankclaw 2026-09-12: the marreta-proxy ProxyProvider was created without a linked Application, so the embedded outpost never loaded it. Root cause chain: null invalidation_flow (serde crash) - missing flows - missing Application link.)_
-serverAppConfig multi-agent repo, two more peer-collision facts (2026-09-21): (1) A peer's git add can sweep YOUR in-flight uncommitted files into THEIR commit (my homepage layout changes landed inside the peer's frank-large fix commit d6f9b85) — after any surprise "nothing to commit, working tree clean", check git log --oneline -5 + git log -S your-string before assuming your work vanished (stash was the first suspect from an earlier lesson; a peer commit is the other). (2) A service-scoped deploy (deploy.sh host --service X) leaves other changed services PENDING on the remote: a later local --dry-run prints "changed services: (none; the remote will check pending service state)" even though /srv assets are stale — run the real deploy and the remote --pending runner dispatches the leftover services. Verify staleness by grepping /srv/svc assets on the host, not by the local plan. _(context: Horizontal homepage section deploy, 2026-09-21: peer committed frank-large fix sweeping my 4 in-flight files; peer's scoped frank-large deploy left homepage pending; my full deploy dispatched it (e2e 32/32).)_
-NEVER send SIGHUP to PID 1 inside a podman container to reload a service. In podman containers PID 1 is the container supervisor/init, not the app — SIGHUP to it TERMINATES the whole container. On 2026-09-12 I tried podman exec pihole kill -HUP 1 to reload dnsmasq addn-hosts; it killed the pihole container and took LAN DNS down for ~15s until the quadlet unit's Restart=always (plus an explicit systemctl --user restart pihole.service) brought it back. The app process (pihole-FTL) was PID 54, not 1. Correct ways to reload a service in a podman container: (a) systemctl --user restart svc.service (the quadlet unit — clean, intended), or (b) signal the ACTUAL service PID found via podman exec c ps -e -o pid,comm, never PID 1. For Pi-hole specifically, the repo flow is deploy.sh dankclaw --service pihole which does the addn-hosts cp + unit restart. _(context: 2026-09-12, closing the marreta addn-hosts gap on the live Pi-hole. pihole reload is not a valid subcommand (printed usage, exit 0), so the || kill -HUP 1 fallback fired and killed the container.)_
-llama.cpp: concatenating a split GGUF's parts into a single .gguf file does NOT produce a loadable single-file model — part 1's header carries ggml.file_count=3, so the byte-correct concatenation still advertises itself as a split and the loader fails with "invalid split file name: path" (it demands a -NNNNN-of-NNNNN filename). The fix (frank-large, 2026-09-21, commit d6f9b85): point -m at part 1 of the split; llama.cpp auto-discovers parts 2-3 in the same directory and loads the split natively. serve-frank-large.sh and the deploy-remote.sh LLM_LARGE_MODEL gate both follow this. Don't re-attempt the concat-and-rename shape. _(context: frank-large crash-loop diagnosis on dankclaw: unit failed with "invalid split file name" on the 93.68 GB concatenated GGUF; peer fixed by native split load; endpoint active/healthy after.)_
-On the Dankware/Dankclaw homelab, a "connection timeout" to any *.franksdank.cloud service from a LAN client is usually a CLIENT split-DNS problem, not a server outage. The public DNS record for franksdank.cloud points at the WAN IP (71.227.227.172) which has NO 443 port-forward (filtered). Pi-hole (192.168.0.143) returns the correct LAN answer 192.168.0.143 via addn-hosts. If a client's systemd-resolved is using 1.1.1.1 (or any external resolver) as its current server, franksdank.cloud resolves to the WAN IP → TCP SYN to a filtered port → timeout (curl rc=28). LAN-only names (dankclaw/grafana/marreta) return NXDOMAIN instead. Diagnose: dig +short www.franksdank.cloud @192.168.0.143 (→143) vs @1.1.1.1 (→WAN); resolvectl status shows the current server. Fix on the client: a systemd-resolved drop-in routing the domain route-only to Pi-hole. Verify the server itself is fine by pinning the LAN IP: curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 https://www.franksdank.cloud/ → 200. _(context: 2026-09-12 homepage "links gone / not loading / connection timeout" on dankware. Server was fully healthy (Caddy @home 200, Anubis challenge valid, Authentik outpost healthy, files render). Root cause was dankware's resolved using 1.1.1.1. Fixed with /etc/systemd/resolved.conf.d/franksdank-lan.conf [Resolve] DNS=192.168.0.143 Domains=~franksdank.cloud ~franksdank.casa + restart systemd-resolved.)_
-Peer-agent coordination on blankpad: hub list can show ZERO peers even when another omp process is running in the same repo directory (separate broker instances are not visible to each other; ps -eo pid,args + /proc/pid/cwd is how you find the peer's session). When a live-fleet anomaly appears that belongs to a peer's in-flight deploy (e.g. frank-large crash-loop + orphan llama-server during a homepage deploy), do NOT fix it unilaterally — verify your own deploy only dispatched its planned services, gather the evidence (journal, process age, file sizes), and hand it to the user as a relayable handoff. Seen 2026-09-21: homepage deploy was clean while a peer's frank-large work left the unit crash-looping ("invalid split file name") with an orphan llama-server holding :7770; the assembled GGUF was exactly 1,000,000,000 bytes short of the parts sum (truncatedconcat). _(context: Homepage deploy vs peer frank-large deploy, 2026-09-21)_
-Dankware fleet — "service unreachable / connection timeout" triage for Anubis+Authentik+Caddy guard-chain vhosts. Two durable gotchas: (1) Rootless-podman published-port SNAT makes Anubis log x-real-ip: 10.89.1.22 (the podman gateway) for EVERY client — the real client IP is lost at the published-port layer, so never use Anubis x-real-ip for per-client identity/rate-limit reasoning. (2) The WAN IP 71.227.227.172 has NO 443 port-forward; any client resolving *.franksdank.cloud to the WAN IP (i.e. NOT using Pi-hole 192.168.0.143 as DNS, or off-LAN) gets a TCP connection timeout — the SYN hits the filtered public address. LAN clients must resolve via Pi-hole to 192.168.0.143. To PROVE a guard-chain service is server-side healthy (and thus the fault is the complaining client's DNS/network, not the server): test from a LAN peer — ssh dankserver then dig +short www.franksdank.cloud (expect 192.168.0.143), /dev/tcp/192.168.0.143/443 (expect OPEN), curl -sk https://www.franksdank.cloud/ (expect 200). Also check Pi-hole addn-hosts has the 192.168.0.143 host line and iptables -L INPUT policy is ACCEPT. If the LAN peer gets 200 but one device times out, the server is fine — fix that device's DNS (point at 192.168.0.143) or its LAN connectivity. Note: a curl to the bare IP without matching SNI returns code 35 (SSL name mismatch) — that is NOT a real failure; always use the hostname URL with --resolve or real DNS. _(context: 2026-09-12: homepage reported "links gone" then "connection timeout" from danklivingroom. Exhaustive server-side trace (Anubis challenge page valid, Authentik outpost healthy, Caddy @home/@data 200, files render with visible links, no CSP/redirect-loop, no fail2ban) proved the stack healthy; dankserver LAN peer got https 200 in 7ms. Root cause isolated to the complaining client resolving to the fi)_
-Homepage registry contract (2026-09-21, commit 384747a): every shared/homepage/services.json entry now carries a section field that MUST be in app.js's SECTION_ORDER for its category (user: AI, Web Apps, Media Library Apps; admin: Media Library Apps, Observability, AI, Infrastructure) — a section not in SECTION_ORDER is SILENTLY dropped from the page (pinned by tests/homepage_config.bats). wip is no longer a boolean: it is false | "open" | "closed", and homepage-status.py passes it through as state "wip-open"/"wip-closed" (badges "WIP: OPEN" amber / "WIP: CLOSED" dim). Panels render under section headings via shared renderPanel()/renderSection() components in app.js (both pages share one app.js, so that IS the shared-component mechanism). Adding a homepage service = one JSON entry with the right section; no deploy-script change (deploy copies a fixed file list). _(context: Homepage sectioning deploy, serverAppConfig repo, 2026-09-21)_
-On dankclaw's Anubis→Authentik→Caddy guard chain, the rootless-podman published-port SNAT masks the real client IP: Anubis logs x-real-ip: 10.89.1.22 (the Caddy container's podman-network address) for EVERY client — verified with both host-local curl and a browser from a different machine (blankpad). So Anubis cannot distinguish clients by IP; clearance is cookie-based, not IP-based. This is pre-existing fleet behavior, NOT caused by config edits, and does not break clearance. When a user reports a guarded page "broken" but the server is healthy, the break is client-side/session — you need the authenticated view (Authentik creds) or the exact symptom, not more server probing. _(context: 2026-09-12 homepage "links gone / not loading" investigation; server-side fully verified healthy, root cause not reached (awaiting user symptom).)_
-Fleet security-posture audit (franksdank.cloud), traps + 2026-09-21 baseline: (1) WAN port scans MUST run from gluetun's ProtonVPN egress on dankclaw (ssh dankclaw 'podman exec gluetun ...', verify external IP via api.ipify.org first) — LAN→WAN is hairpin NAT and lies. (2) TLS-port gotcha: plain-HTTP wget probes against TLS ports (443, Plex PMS 32400) are FALSE NEGATIVES — re-probe with wget -qO- https://IP/ (the gluetun container's wget may lack --secure; the https:// URL suffices). (3) Calibre on dankserver is a HOST GUI APP, not a container service — only Calibre-Web (:8083, books. vhost) is on the wire; the optional calibre-server (:8080) is not running. AGENTS.md's stack list ("Calibre, Calibre-Web") misleads; don't audit or chase Calibre as a service. If calibre-server is ever started: host Calibre 9.2.1 9.9.0 (content-server file-read fix) and use --enable-auth. (4) 2026-09-21 verified baseline: WAN has ONLY 80+443 open (Caddy edge); public names = www, search, ai, gitea, plex, books, jelly, auth (IdP); 9 vhosts serve the Anubis challenge (www, search, grafana, prometheus, loki, seerr, marreta, romm, files); all exempt vhosts behaviorally gated (OpenWebUI /api/v1/users/me 401, Gitea 401 + repos private, *arr API 401, qB 403, pihole basicauth 401); LLM endpoints (143:7770, 143:7771, 235:7770) accept requests with a BOGUS Bearer token (authentirely absent — LAN trust model); Grafana admin:admin confirmed live but loopback-only (127.0.0.1:3002); plex.franksdank.casa public record was STALE (73.59.75.210 = previous Comcast IP, dead on 80/443 — works on LAN only because Pi-hole shadows it); Plex vhost is WAN-broken (302 to LAN origin 192.168.0.235:32400); :8017 carries vestigial *arr handlers (exempt vhosts never enter the chain). A managed skill fleet-security-posture-audit exists from an earlier audit — enhance it with these traps rather than duplicating.
-TRANSPORT UPDATE (2026-09-12, supersedes the old "blankpad has no fleet SSH" lesson): blankpad (~/.ssh on user blank) NOW has passwordless SSH to the whole fleet. An ed25519 key was generated (id_ed25519, comment blankpad-deploy-20260912) and its pubkey installed on dank@192.168.0.143 (dankclaw) and dank@192.168.0.235 (dankserver). ~/.ssh/config defines Host aliases dankclaw and dankserver (User dank, IdentityFile ~/.ssh/id_ed25519, IdentitiesOnly yes). So ssh dankclaw / ssh dankserver and ./scripts/deploy.sh dankclaw ... work from this box now — no more "ask Frank for a transport." Passwordless sudo on the targets is separate (FRANK_SUDO_PW / NOPASSWD). MARRETA/LARAVEL APP_KEY GOTCHA: a Laravel APP_KEY must be the base64:-prefixed form — base64:$(openssl rand -base64 32). A bare base64 string (no prefix) is parsed as a raw key of the wrong length and every app route 500s with Unsupported cipher or incorrect key lengthat Encrypter.php:61, while the health route /up (which skips the encrypter) still returns 200 — so a green healthcheck masks a fully-broken app. Fix before first boot (no encrypted data persisted yet). _(context: Deployed marreta on dankclaw 2026-09-12; had to set up SSH keys from blankpad first, then fix a bare-base64 APP_KEY that 500'd all app routes.)_
-Fleet security posture as audited 2026-09-21 (all behavioral): the WAN (71.227.227.172) exposes ONLY ports 80+443 (Caddy edge); 8 public vhosts — www/search behind the Anubis→Authentik guard chain, ai/gitea/plex/books/jelly exempt with verified own auth (401s/login redirects), plus auth. (IdP, public by design). Every other vhost (grafana, prometheus, loki, seerr, marreta, romm, files, pihole, *arr, hermes, hermesadmin) is LAN-only by DNS absence. Known residuals: (1) all three LLM endpoints bind 0.0.0.0:7770/7771 with NO auth — LAN trust model, the first thing to gate if untrusted LAN devices appear; (2) Grafana still answers basic auth admin:admin on loopback 127.0.0.1:3002 (200 with creds, 401 without) — loopback-only and behind the chain via vhost; (3) plex.franksdank.casa is a STALE public record at 73.59.75.210 (the previous Comcast WAN IP — current is 71.227.227.172), so the plain-HTTP legacy name is dead from the WAN; (4) plex.franksdank.cloud is WAN-broken functionally because Plex Web 302-redirects to its LAN origin 192.168.0.112… correction: 192.168.0.235:32400, unreachable off-LAN; (5) the :8017 post-auth listener carries vestigial @prowlarr/@radarr/@lidarr/@readarr/@qbittorrent handlers that nothing routes to (exempt vhosts never enter the chain). Probe traps: plain-HTTP wget against TLS ports (443, 32400) is a false negative — re-probe with https://; busybox wget has no --secure flag and fails silently inside && chains; the authentik flow page is React so grep for 'password' finds nothing — a 200 on /if/flow/... is the proof. _(context: Security posture audit of the franksdank.cloud fleet, 2026-09-21, from blankpad)_
-On the Dankware/Dankclaw fleet, LLM services that are rootless Quadlet units (e.g. frank-large on Dankclaw, user dank) must be read WITHOUT sudo: sudo podman logs name fails with "no such container" because rootless storage is per-user. And because Quadlet runs --replace --rm, podman logs only covers the CURRENT container instance — history across recreations lives only in journalctl --user -u name.service. So: current instance → podman logs; anything before the last recreate → journalctl --user. Separately: the dev box /home/blank/Projects/serverAppConfig (user blank) has NO ssh key or host alias for the fleet hosts — ~/.ssh holds only known_hosts, and dank@192.168.0.143 / frank@192.168.0.143 both return Permission denied (publickey,password). Do not burn calls probing fleet SSH from this box; ask Frank for a working transport before claiming live verification, and label repo-grounded answers as not live-verified. _(context: Answering "how do I retrieve logs for frank-large?" (2026-09-12); grounded in dankclaw/frank-large/quadlet/frank-large.container and serverAppConfigs-docs/LLM/halogen-flash-server.md:191-219. Live verification attempt failed on missing SSH credentials.)_
-qBittorrent WebUI on FrankBook-Pro (http://192.168.0.112:8080, published on the gluetun unit): username admin; password = QBIT_WEBUI_PASSWORD in /home/core/Containers/qbittorrent/env on the Mac — a 20-char random string generated once at provisioning by the deploy handler (scripts/deploy-remote.sh VM_SEED: openssl rand -base64 18 | tr -d '/+=' | head -c 20), machine-local per the repo secret rules, with a PBKDF2-HMAC-SHA512/100k hash of the same value embedded in qBittorrent.conf. Access gap (verified 2026-09-21): blankpad (current dev box) has NO SSH alias or key for core@192.168.0.112 — the frankbook-pro alias lived on the retired the workstation dev box; neither jump host (dankclaw, dankserver) has the Mac's key; blankpad has no sshpass/expect. Bootstrap a key (ssh-[REDACTED] skill) or use the Mac's password before reading the env file. _(context: Frank asked for the qBittorrent WebUI credentials; repo + skill pointed to the machine-local env file, but no SSH path from blankpad existed (4 probes: direct, alias, both jump hosts))_
- deploy.sh's [3/3] Playwright e2e gate can outlast a 300 s bash timeout and get killed mid-suite, but the deploy itself is already complete: the service was restarted, health-checked, and its STATE revision recorded BEFORE the gate starts. When the gate is cut off, do NOT re-run deploy.sh (it would re-dispatch) — run ./tests/e2e/run.sh standalone to finish the gate; the repo's deploy-completeness rule (e2e green) is satisfied by that standalone run. Flaky tests (fail once, pass on retry) do not fail the suite. Seen 2026-09-21 on the frank-medium chat-template deploy: gate died at 14/32, standalone re-run gave 31 passed + 1 flaky (12-crawl). _(context: frank-medium Qwen-Sharp chat template deploy, 2026-09-21, serverAppConfig repo)_
- In the serverAppConfig multi-agent repo, when expected WIP vanishes from a clean tree (files missing, git status empty), the first place to look is git stash list — peer agents stash in-flight work with descriptive messages (e.g. "peer WIP: frank-medium qwen-sharp template (stashed for prometheus deploy)") to land their own commits. Verify contents with git stash show --include-untracked --stat stash@{N} (untracked files live in the stash's third parent, stash@{N}^3) and git stash pop on the clean tree. Seen 2026-09-21: the Qwen-Sharp template + quadlet flip had been stashed by a peer for a prometheus deploy and looked lost until the stash list was checked. _(context: frank-medium Qwen-Sharp chat template deploy, 2026-09-21, serverAppConfig repo)_
- On the frank-medium FP8 Qwen3.8-27B checkpoint (dankserver, vllm-radiance TP=2), the z-lab DFlash2 EXTERNAL drafter beats the model's own MTP self-speculation on DECODE (geomean 100.0 vs 85.9 t/s, burst-4 287.1 vs 179.7 t/s) despite a LOWER mean accept length (4.24/7 vs 5.66/8). The trap: accept length is a PROXY, not the metric. Decode throughput = (accept+1) / (draft time + verify time). The DFlash2 drafter is a small separate model that steps fast; the MTP head is the target's own layer applied autoregressively N times, so its draft is slower per step. The faster step more than makes up the lower accept length. MTP still wins on TTFT (0.06 vs 0.19 s) and prefill (1096.9 vs 923.0 t/s); the drafter costs 2.1 GiB VRAM (KV pool 384,238/1.47x vs MTP's 461,540/1.76x). RULE: when choosing between MTP self-speculation and an external drafter, measure decode throughput on the SAME target — never compare accept lengths across different targets/quantizations (that was the apples-to-oranges error: MTP-on-FP8 5.66/8 vs DFlash2-on-MXFP4 ~4.1). FP8 + DFlash2 (100.0 t/s) also beats MXFP4 + DFlash2 (92.4 t/s) — the higher-precision target verifies more of the drafter's proposals. DFlash2 is the deployed shape (SPEC_METHOD=dflash, SPEC=7). _(context: frank-medium DFlash2 vs MTP A/B test on the FP8 checkpoint, 2026-09-20. User: "so, you're lying to me then... test it" — the prior MTP-vs-DFlash2 comparison was across different targets/quantizations.)_
- In vLLM/radiance speculative decoding, mean accept length is a PROXY, not the metric — a small separate drafter (DFlash2) can beat the target's own autoregressive MTP head on decode throughput DESPITE a LOWER accept length, because the separate drafter steps faster than the target's own layer applied autoregressively K times. Throughput = (accept+1)/(draft time + verify time); the verify (one target forward pass) is shared, so the draft time is the differentiator. Measured on the FP8 Qwen3.8-27B target (frank-medium, dankserver, 2026-09-20): DFlash2 (4.24/7 accept, 100.0 t/s decode, 287.1 t/s burst-4) beat MTP self-speculation (5.66/8 accept, 85.9 t/s, 179.7 t/s); MTP wins on TTFT (0.06 vs 0.19 s) and prefill. The DFlash2 drafter is quantization-agnostic (proposes, target verifies) and works with any Qwen3.8-27B target; it costs 2.1 GiB VRAM (KV pool 1.47x vs MTP's 1.76x). FP8 + DFlash2 (100.0 t/s) also beats MXFP4 + DFlash2 (92.4 t/s) — a higher-precision target verifies more of the drafter's proposals. Rule: when A/B-ing spec-decode variants, measure decode t/s (the bench-conversation.py battery), never accept length alone; flip the quadlet SPEC_METHOD, deploy, read the boot-log SpecDecoding metrics, run the battery, keep the winner. _(context: frank-medium DFlash2 vs MTP A/B on the FP8 checkpoint, 2026-09-20; commit f998a2d. Prior turn I wrongly claimed MTP was the better choice based on accept length (5.66 vs 4.1) without testing DFlash2 on the FP8 target; the user called it out ("you're lying to me then... test it").)_
- frank-medium (dankserver :7770, vllm-radiance TP=2) model swaps are ONE quadlet edit + deploy, because the entrypoint (dankserver/frank-medium/radiance-entrypoint.sh) is QUANTIZATION-AWARE: it reads quantization_config.quant_method from the checkpoint's config.json and branches. fp8 - plain path (no MXFP4 patches, --quantization fp8, ROCM_AITER_UNIFIED_ATTN attention, MTP self-speculation SPEC_METHOD=mtp SPEC=8 — the model's own single-layer MTP head, no external drafter). mxfp4/quark/empty - patch path (R4D attention if R4D_ATTN=1, DFlash2 external drafter SPEC_METHOD=dflash SPEC=7, applies radiance-vllm-mxfp4 patches every start). A swap edits exactly 4 env vars in the quadlet: SNAP=/models/name, SPEC_METHOD, SPEC, KV_MEM (0=auto-size). The deploy handler (deploy-remote.sh deploy_llm_medium) reads SNAP/SPEC_METHOD from the quadlet and gates on the right assets (drafter only for dflash, libr4d only for mxfp4/quark). KV pool auto-sizing is quantization-dependent: bigger weights leave less VRAM (FP8 30.9GB - 461,540 tokens 1.76x; MXFP4 19.4GB - 943,581 tokens 3.59x on the same 2x R9700). One-command swap: scripts/swap-frank-medium.sh --repo o/r --target /home/dank/models/name --generate-manifest --apply --deploy. Full process: serverAppConfigs-docs/LLM/frank-medium-model-swap.md. _(context: frank-medium MXFP4-FP8 swap 2026-09-20; the entrypoint was made quantization-aware so future swaps are a quadlet edit, not an entrypoint rewrite)_
- Generating a HuggingFace fetch manifest (filename/size/sha256 per file) from the repo API: the GET https://huggingface.co/api/models/owner/repo response's siblings[] carries ONLY rfilename — no size, no lfs hash. So the size and LFS sha256 must come from a HEAD on the resolve endpoint https://huggingface.co/owner/repo/resolve/main/file: size = Content-Length header, LFS sha256 = X-Linked-ETag header (64-hex; non-LFS files like config.json/[REDACTED] have no 64-hex etag, so leave the hash blank and verify size only). Token via Authorization: Bearer header. This is what scripts/hf-manifest.py does (dankserver frank-medium model-swap toolchain, 2026-09-20). A naive siblings[].size reads 0 for everything. _(context: frank-medium MXFP4-FP8 swap streamlining; first hf-manifest.py version printed size 0 for all files because it read siblings[].size)_
- When probing a reasoning-on LLM endpoint for a capability (vision, tools), a small max_tokens budget (e.g. 32) can be fully consumed by the thinking block, leaving content null/empty — which reads as "the capability is broken" when it is not. Re-probe with headroom (200+) before concluding. Seen 2026-09-20 on frank-medium (vLLM): a 32-token vision probe returned content None; the same image at 200 tokens answered "The image is white."
- When running a Python heredoc over ssh host 'python3 - PY ... PY', the REMOTE shell (bash) expands $var inside the heredoc body before Python sees it — an unquoted PY delimiter with a $job in the code silently becomes replace("", ...) (inserts the replacement between every character) and produces misleading results (here: 21/21 "HTTP 400 parse error" from a query with frank-medium spliced into every letter of rate(...)). Use a quoted delimiter 'PY' (or escape the $) whenever the script body contains shell-expandable $ sequences. Diagnose by printing repr() of the first extracted string — the splicing is visible immediately.
- On dankclaw, the Grafana quadlet container is published to 127.0.0.1:3002, NOT 3000 — port 3000 is a different service that answers 404/"Not found." to every path (this masqueraded as "dashboard not provisioned" for a day; the file was in the container and the provider was active the whole time). Grafana 13.2.0 has no GF_SECURITY_ADMIN_PASSWORD set, so basic auth is admin:admin. Verify any dashboard via curl -u admin:admin http://127.0.0.1:3002/api/dashboards/uid/uid and /api/search?type=dash-db.
- Two prefix-cache semantics traps measured 2026-09-20 on the Dank fleet: (1) llama.cpp's llamacpp:prompt_tokens_total EXCLUDES cached tokens (its own HELP line: "Number of prompt tokens processed, excluding cached tokens"), while llamacpp:prompt_[REDACTED] counts reused tokens — so the reuse FRACTION is cached/(cached+prompt), and the naive cached/prompt ratio is unbounded (read 77 on a fully-warm frank-large, 2.98 on frank-small). (2) vLLM (radiance 0.9.3, Qwen3.8 hybrid GDN/mamba) prefix caching: metric names are vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total, both token-counted (NOT 'reused'); the attention block size is padded to the mamba page size (1,648 tokens here, boot log "Setting attention block size to 1648 tokens"), and reuse happens at that granularity — an identical 13,021-token prompt went 2.30s cold → 0.68s warm with 9,888 tokens (76%) reused, while an identical 2,521-token prompt (one full block) registered +0 hits. So shared prefixes under ~2 blocks (~3.3k tokens) re-prefill in full. The API returns prompt_tokens_details: null in this build, so the counters are the only instrument. Prefix caching in mamba 'align' mode is flagged experimental at boot ("Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental."). _(context: frank-medium radiance TP=2 cutover monitoring work, 2026-09-20, serverAppConfigs)_
- When parallel doc-writing agents edit an OKF/Obsidian vault (serverAppConfigs-docs), the recurring damage is STRUCTURAL, not factual, and no existing gate catches it: (1) prose prepended ABOVE the --- frontmatter opener (so the frontmatter silently stops being frontmatter), (2) whole body sections duplicated wholesale — an agent appended its rewrite instead of editing in place, leaving two ## Architecture/## Why TP=2 cannot run here blocks plus a stray second ---, (3) a literal model tool-call token ("|channel}tool_call||tool_callcall:read{i:) pasted into the body, (4) line-by-line bullets emitted with a leading - on EVERY wrapped line (- - , - 2026-09-20 ...) and stray + diff-artifact prefixes, (5) an agent line-range replacement that dropped a sentence tail, leaving a dangling fragment. check-vault-links.py reports 0 unresolved links the whole way through because links resolved while structure rotted. Remedy that worked: git checkout HEAD -- file to restore the destroyed file (move the agent version to .AGENTTRASH/date-topic/ first so nothing is lost), then re-apply the intended flips by hand; salvage genuinely unique sections with CUT N.=M @reg + PUT K @reg before deleting duplicate ranges. VERIFY AFTER EVERY AGENT DOC PASS: frontmatter --- is line 1 and appears exactly twice, headings are unique (grep -n '^## ' | sort -k2 | uniq -f1 -d), no +- /- - /literal token artifacts, and the link checker plus the docs-completeness seam both run. Also: agents keep writing after you think they are done — a completed job re-wrote a file I had already fixed, so re-check wc -l/structure after the last job settles. _(context: 2026-09-20 frank-medium radiance TP=2 cutover in serverAppConfigs: two doc agents (EvidenceAndLog, RunbookSwap) left these wounds across serverAppConfigs-docs/LLM/index.md (restored from HEAD twice), frank-medium-radiance-mxfp4.md (deduped 827-549 lines), frank-medium-llamacpp-dflash2.md, index.md, log.md.)_
- The 3090 on dankclaw is an EVGA FTW3 Ultra V2 (PCI 10de:3982); its RGB is driven by an iCX3 lighting MCU on the GPU's OWN nvidia i2c bus — i2c-1 (NVIDIA port 1), address 0x2D. The nvidia i2c driver exposes ports 1,5,6,7,8 as /dev/i2c-1..5 but only i2c-1 carries a live slave. Protocol (raw I2C_RDWR, NOT the SMBus ioctls): write reg 0xB2 04 C6 EB EA 15 (enable, required once first), then reg 0xC0 mode 09 m0 m1 m2 m3 .. .. .. .. sync where m0..m3 are the four zone modes and MODE_OFF=0x00 (OFF = 09 00 00 00 00 FF FF FF FF 00), then reg 0x90 04 9E EB 00 90 to commit to EEPROM. Read the mode back by writing reg 0xC0 and reading 16 bytes — bytes 1..4 are the zone modes (0x00=OFF), bytes 6..9 are per-zone LED counts. The EEPROM save persists across a full reboot with NO Linux-side daemon — verified 2026-09-19: after sudo reboot the mode read back 09 00 00 00 00 1c 04 05 3c 00 … (all four zones OFF). Works headless (no display engine / X needed). /dev/i2c-* are root:i2c 660, so run the tool with sudo (dank has passwordless sudo). _(context: dankclaw 3090 RGB turned off + EEPROM-saved, then reboot-verified 2026-09-19)_
Context: 1970s Moscow cafeteria (stolovaya). Protagonist: Pyotr Arkadyevich Volkov (chef). Problem: The 'kotleta' (cutlet) lacked architecture, falling apart when eaten with a fork. Solution: Enclosing it in a pampushka bun to make it a 'hand-consumed' burger. Bureaucracy: The invention moved from a chef's observation to a formal memo, through a 9-link bureaucratic chain, eventually becoming a GOST standard (Meat cutlet in a soft bread bun). Key Philosophy: 'The standard gives the product a body. The kitchen gives it a life.' The 'half-millimeter' tolerance is the space where the individual's touch meets the state's law.
## 01a0e144-473f-71a5-96f8-0316232910aa
updated_at: 1790503503
### Placement & Tensor Laws
- **Override Logic:** `--override-tensor` is first-match-wins. Specific patterns must precede broad sweeps (e.g., `exps=CPU` should be listed after specific blocks but before general ones).
- **PCIe Strategy:** `-ot exps=CPU` is the primary method to stop crossing the PCIe bus; it works only if `--no-mmap` is NOT used (as no-mmap forces tensors into anonymous memory, breaking the CPU-resident trick).
- **Atomic Units:** The unit of placement is the per-block expert tensor (512 experts = 1 tensor). There is no per-expert routing knob.
- **Expert Distribution:** Top 10% of experts carry 40.5% of samples. Hottest layers do not exist at -ot granularity because Top-8 routing fires on nearly every block.
### Spec Decoding & Drafter Laws
- **Composition:** Legal combinations include [one context-backed drafter (MTP/DFlash/DSpark/Eagle3)] + [any number of ngrams]. Two context-backed drafters cannot share a process.
- **KV Scaling:** Drafter `n_ctx` automatically scales to the target's `n_ctx`.
- **Drafter VRAM Traps:** DSpark (no sliding window) can consume ~5.0 GiB at 262k context, while DFlash2 (sliding window) consumes only ~40 MiB.
- **Device Pinning:** Pin the drafter to the card holding the main model's output head (e.g., ROCm1 under `-ts 1,1`) to avoid `SIGABRT` on pre-allocated buffer mismatches.
### Performance & Implementation
- **Ubatch Optimum:** For frank-medium, the peak is 512 (1193 tok/s). Below 512 is under-saturated; above 512 starts to see diminishing returns and higher VRAM costs.
- **GGUF Extraction:** Derive tensor extents from **consecutive header offsets** rather than quant type-size tables to avoid mis-bucketing tail elements into 'gap' buckets.
- **Fleet State:** Primary (192.168.0.235:7770), Large (192.168.0.143:7770), Small (192.168.0.143:7771).
## 01a0dfa8-7193-7235-980d-69e138218fcc
updated_at: 1790474002
Model: GLM-5.3-Flash-Uncensored-GGUF (orcarouter). Params: 321.3B MoE. Key Constraints: 1. Requires `NVIDIA_TF32_OVERRIDE=0`. 2. Must use `-fa off` for correct MLA output. 3. Use `-ub 128` to mitigate microbatch-dependent long-context collapse. 4. Use `-ot exps=CPU` for offloading experts as Q6_K (~264GB) is slightly larger than available RAM (246GB). Features: Supports draft MTP (`--spec-type draft-mtp`). Context: Previous `frank-medium` unit was in a crash loop on ParoQuant MXFP6, highlighting the need for stable deployment of this new large MoE model.
## 01a0df97-1e50-77d7-9084-92708bfd1152
updated_at: 1790458743
Connectivity: Repeated 'Unable to connect' errors (ID 135168) occurred on dankserver using frank-medium. Framework: Introduced /guided-goal, a structured workflow requiring a 5-point interview (Success criteria, Verification, Attempt cap, Boundaries, Stop conditions) before executing a persistent autonomous objective loop. Current Project: Deploy https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF on dankserver using the https://github.com/Luce-Org/lucebox runtime.
## 01a0d66d-f9e2-7681-a5e3-e77b896dcc3f
updated_at: 1790407605
Homepage CSS Update: Flipped the stagger pattern in shared/homepage/style.css. Role and Inputs now use --dim (#2f8f52, bold), while Spec and Use use --fg (#33ff66). Removed the --input variable. Deployment: Committed cd07a66, pushed, and deployed to /srv/homepage. OpenWebUI/Authentik: OpenWebUI is running as a quadlet in dankclaw. Authentik integration (OAuth2/OIDC) is confirmed as the primary method for passing login state, matching the current containerized architecture.
## 01a0c81c-56dc-7359-ac24-6da36332ad84
updated_at: 1790068527
## ComfyUI 0.37.0 Native Qwen-Image 2.1
- Released 2026-09-21, tag v0.37.0, CORE-423, PR #16400
- 8-bit is ~23% larger weight size than 4-bit Kairic Edge, explaining ~22 t/s vs higher throughput.
- DFlash2 draft acceptance is text-variety-dependent: 0.85 on trivial one-word answers, 0.519 on varied explanatory prompts. Do not read a single request as steady-state.
- Kairic Edge payload + runtime remain on disk as rollback target.
- Pending: purge 87 GiB assembled file + 31.7 GB corrupted model on dankclaw (awaiting approval).
- Model: just1moremodel/Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, Swift finetune)
- Template: Qwen-Sharp v22.5.0 (178 prompt tokens for a simple math question; 73,600 is the fallback sentinel)
- Drafter: DFlash2-FP8 (Qwen3.8-27B-based, not Swift-specific) — works fine, acceptance 85.7% easy / 58.3% varied
- max_model_len: 262144
- KV pool: 425,047 tokens (1.62x max_model_len), 14.42 GiB/card available
### GPU_UTIL ceiling on 32GB R9700
- 0.99 is INFEASIBLE: vLLM startup check runs BEFORE weight load. 0.99 x 31.86 GiB = 31.54 GiB required, but only 31.23 GiB free at device-init. The 0.63 GiB gap is consumed by ROCm/HIP context + NCCL + radiance custom all-reduce buffers.
- 0.98 is the ceiling: 0.98 x 31.86 = 31.22 < 31.23. Prior production value.
- Symptom of 0.99: ValueError at engine init → crash loop. The 'registered' lines in journal are NOT the loop — they're ~20 sequential patch_*.py invocations each printing NOOP sitecustomize lines.
### Weight sizes (affects KV pool)
- MXFP4: 19.37 GiB
- MXFP6: 25.16 GiB (5.79 GiB more → less KV headroom)
5. Verify: health 200, model name, spec decode config, completion (prompt token count confirms template), tools, drafter acceptance
### Commits
- e9a01cd: MXFP6 swap
- d5c8e40: GPU_UTIL 0.98 revert
- 24f7a77: measured boot shape in runbook
### Open item
- Drafter is Qwen3.8-27B-based, not Swift-matched. Acceptance healthy (58-86%) so it's a net win, but a Swift-specific drafter would be the next lever if needed.
- If storage may have published and repair cannot be proven durable: `SessionPersistenceIndeterminateError` fails closed with original + recovery errors.
**Directory**: `llamacpp/` at repo root (planned, not yet created).
**Phases**:
0. Recon: git status, disk space, identify forks on GitHub, read runbooks, verify SSH to dankclaw/dankserver.
1. Clone mainline llama.cpp into `llamacpp/`.
2. Trim AGENTS.md — remove rules that block private-fork freedom (e.g. 'don't add dependencies', 'keep PRs small', 'no breaking API changes').
3. Merge charlie12345/llama.cpp — source of ROCmFP4 quantization for RDNA4. Test on frank-large (dankclaw, gfx1151 iGPU, port 7770). Must stop halogen service first per benchmark policy.
4. Merge ik_llama.cpp — improvements for unique setup. Runbook: `frank-large-ik-uncensored.md`.
5. Create managed skill: merge-llamacpp-forks (merge latest from all parent forks + TDD, noting quirks).
- dankclaw (192.168.0.143): RTX 3090 CUDA + gfx1151 iGPU. frank-large service on port 7770.
- the workstation / blankpad (dev boxes): Ryzen 5 PRO 5650U, RTX 4070 Max-Q. Has passwordless SSH to fleet.
**Fork sources**:
- Mainline: ggml-org/llama.cpp
- charlie12345/llama.cpp: ROCmFP4 quantization (RDNA4-focused). Also referenced as 'ROCmFPX fork' in runbooks.
- ik_llama.cpp: used on frank-large 2026-09-15 to 09-16.
**Pitfalls encountered**:
- frank-medium on dankserver was unreachable (10 retries exhausted, ~15.6s each). Service down.
- frank-large context overflow: prompt 144085 tokens + max_tokens 57344 > context 200192. Need to reduce max_tokens when prompt is large, or increase context.
- TDD skill mandates seam confirmation with user before writing tests — one consolidated question needed.
- Deploy planner: new top-level directory `llamacpp/` should be a no-op (unknown paths don't map to services).
**Status**: Phase 0 (recon) was in progress when session was interrupted. No code written yet.
- **deploy-remote.sh** (target): takes explicit service list (never infers). State store at `~/.local/state/serverAppConfigs/deploy-state/<service>` with `revision=` and `status=`. Services marked `pending` before handler, `deployed` after success. Crash recovery: pending services retried next run. `--pending` (default) catches up services whose revision lags. Retired-service lockout protection (trash, not rm).
- Key invariants: master = deployed state (Q33), ff-only pull + exact revision match, service-scoped (unrelated services never restarted), deploy not complete until e2e gate passes
- E2e gate can take ~20 minutes (admin link-walk alone budgets 240s)
## User Request Context (not yet implemented)
- Wanted sections on homepage: AI, Web Apps, Media Library Apps
- Wanted sections on admin: Media Library Apps, Observability, AI, Infrastructure
- Wanted separate labels: 'Work in Progress: Open' vs 'Work in Progress: Closed'
- Noted outdated AI descriptions and missing tautulli link on admin page
## 01a0c510-1716-724c-b99b-b1ee4df08178
updated_at: 1790020691
## Calibre Content Server Deployment (Dankserver)
**Image**: `lscr.io/linuxserver/calibre:9.15.0` — this is a DESKTOP build (Selkies GUI), NOT headless. The dedicated headless images are dormant (calibre 6.10/2017, below CVE fix threshold). Must run `calibre-server` as the s6 main service via `Exec=` in the Quadlet unit.
**Port traps**:
- Internal port 8080 is the image's own nginx → `Address already in use` if you try to bind calibre-server there. Use internal 8090 → host 8084.
- Default auth mode is **Digest**, not Basic. Every Basic-auth login returns `400 Unsupported authentication method`. Fix: `--auth-mode=basic`.
**Bootstrap password trap**:
-`echo "password" | ...` stores `password\n` (trailing newline) → every login 401. Use `printf '%s'` on stdin.
**pasta port-forward mode**: On this host, pasta runs in port-forward mode. Only explicitly published ports are reachable from outside the container netns. The desktop GUI (if started) would be unreachable without an additional `PublishPort` line.
**Final shape**: Boot-persistent Quadlet (`Restart=always`), image pinned, userdb bootstrap idempotent in deploy handler, 3 new seam tests added.
## Calibre-Web 0.6.27 SSO Analysis
- NO generic OIDC support. Login types: local, LDAP, GitHub, Google only.
- No `authlib` installed, no OIDC config keys.
- Upstream feature request open; maintainer declining more login options.
- Grafana's RP-mode OIDC path is NOT available.
- Correct path: guard chain (Anubis + Authentik + Caddy) with zero app changes. Calibre-Web's own login remains as a second layer (no disable mode), same posture as RomM.
3. Anubis guard set + Caddy edge + provider list deployed together
## Project Context
- Repo: serverAppConfigs — single source of truth for Dankserver (192.168.0.235, Ubuntu 24.04) and Dankclaw (192.168.0.143, Ubuntu 26.04)
- Deployment: Podman/Quadlet, service-scoped git deploy via `scripts/deploy.sh`
- Test suite: 328 tests via `./tests/run.sh`, e2e gate 32/32 Playwright
- Docs vault: `serverAppConfigs-docs/` with link integrity checked by test suite
## 01a0c09e-b740-7547-8216-e31373fa0316
updated_at: 1789968641
## Why vllm-radiance TP=2 failed on Dankserver
- Hardware: 2x AMD Radeon AI PRO R9700 (gfx1201), chipset Gen4 x1 with AtomicOps Routing- (no PCIe atomics)
- RCCL collectives (allreduce, all_gather, broadcast, barrier) require PCIe atomics on upstream bridges — unavailable on this chipset
- PP (pipeline parallelism) also needs p2p send/recv using the same atomic primitive
- vLLM's layer-split mode isn't expressible in the needed form
- The radiance fork's RADIANCE_USE_R4D_AR (one-shot P2P push allreduce over gloo IPC handles) covers prefill+decode allreduce but does NOT intercept PyNcclCommunicator.__init__ warmup allreduce or pynccl_comm.all_gather (direct RCCL calls)
- NCCL symmetric memory: unavailable (shipped RCCL advertises NCCL 2.30.4 but exposes no ncclDevComm* API)
- Workaround that worked: two single-GPU replicas (frank-medium-a/-b) behind a round-robin proxy on port 7770
- Corroborated by Codeberg StillDeadcode/vllm-radiance issue #22 (different host, same image, kernel 7.0.0-30, same enqueue.cc failure)
- Retired 2026-09-16; replaced by mainline llama.cpp with DFlash2 drafter
## models.yml update (MTP → DFlash2 drafter swap)
- File: ~/.omp/agent/models.yml (NOT in git repo)
- Actual API settings unchanged: contextWindow=262144, maxTokens=32768, reasoning=true, input=[text,image]
- Stale comments fixed:
- Model: Qwen3.8-27B-Uncensored-FP8 (30.9 GB / 7 shards), accept 4.24/7, throughput 100.0 vs 85.9 t/s (A/B), 100.0 vs 45.7 t/s
- Pitfall: a schema parse error in models.yml drops the ENTIRE file silently (learned from the `audio` incident) — always verify 0 schema errors after edits
- Running sessions keep their process-start snapshot; changes take effect on next new session
## 01a0b5e1-46b7-7387-a64d-f6f69f79be10
updated_at: 1789847739
## Fleet Model Swap Workflow (llm-fleet-model-swap skill)
- Read HF model card first: runtime pin, hard constraints, recommended args
- Check host for existing model files before downloading: `ls /home/dank/models/`, `df -h /home`, `du -sh`
- Artifact integrity: LFS oid in HF API = sha256; compare size AND sha256sum of on-host file
- Deploy via git flow: commit → push → deploy.sh dankclaw --service frank-small → health check → e2e
- Pre-deployment live verification + port-holder health-check trap check
## vLLM MTP Speculative Decoding for Gemma-4-12B
- Main model: google/gemma-4-12B-it-qat-w4a16-ct (HuggingFace)
- llamacpp (dankserver) provider can have transient 'Unable to connect' errors; auto-retry recovers (seen 2 consecutive failures then recovery)
- frank-small was previously on FreeToken (2026-09-02 → 09-15), then llama.cpp master 37b53fd with HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M, CTX_SIZE=1048576, PARALLEL=4, batch 8192/1024, q8_0 KV
## 01a0b693-7972-7104-bb4b-2bbec7f42acc
updated_at: 1789773472
## frank-medium CACHE_RAM sizing law
**Constraint**: CACHE_RAM must exceed one full slot's prompt state size, which is `n_ctx_slot × 68 KiB` (64 KiB from 16 full-attention layers at 4 KV heads × 256 head_dim × f16 + 4 KiB MTP draft KV). At n_ctx_slot=196608: ~12.8 GiB. Must also stay under half host RAM (62.2 GiB → cap < 31 GiB). 16384 MiB satisfies both.
**Why it's load-bearing for hybrid DeltaNet models**: With `--ctx-checkpoints 0`, the recurrent cell (DeltaNet state) sits at the END of the prompt. If the RAM cache refuses the state snapshot, llama.cpp has no checkpoint to roll back to — it takes the 'forcing full prompt re-processing' path. The RAM cache is the ONLY thing that can carry a state across slots. A refused state is simply gone.
**Historical evidence**: 137 lines of `prompt state size 11xxx MiB exceeds cache size limit 8192.000 MiB, skipping` in 2 days (2026-09-16→18). Max observed: 11,388.99 MiB.
**Verification**: argv (`--cache-ram 16384`), code path, arithmetic, and historical refusal logs. The >8 GiB threshold was NOT exercised live (needs 123k+ token conversation ≈ 2-3 min prefill; AGENTS.md says not to run beside live service). Tell for future: refusal lines should only name states >16384 MiB; an eviction line naming an entry >8192 MiB would be direct proof.
## Test seam pattern (tests/frank_medium_unit.bats)
New assertion: CACHE_RAM must clear one full slot's state (~12.8 GiB) AND stay under half host RAM. Fails on 8192, passes on 16384. This is the canonical way to guard against regressions when n_ctx_slot or model architecture changes.
## Deploy housekeeping
Peer's uncommitted frank-small/vLLM WIP blocked deploy's dirty-tree gate. Pattern: `git stash` → deploy → `git stash pop`. Note: with peer WIP present, `./tests/run.sh` is red (7 frank-small/frank-large seams fail) — full suite isn't a clean signal until that work lands.
- frank-medium: port 7770, tensor-split 35:30, ctx 393216, parallel 2, flash-attn on, spec-type draft-mtp
- Unit file: symlink /etc/systemd/system/frank-medium.service → /home/dank/serverAppConfigs/scripts/frank-medium.service
- Login shell on dankserver is ZSH — avoid words starting with `=` in SSH commands
### Key Insight
Loopback scratch servers are invisible to endpoint monitors (they don't bind the prod port), so they strand the fleet silently instead of failing loudly. Two bench-protocol strandings occurred on the same day (2026-09-18): morning was a simple stop-without-restore; this one was worse because the orphan held GPUs.
### Suggested Guards (not yet implemented)
-`restore-frank-medium.sh` trap in bench harness (EXIT handler to self-restore)
- Fleet check that alerts when GPUs are busy while no prod unit is active
### Commit
- cfabe07 in serverAppConfigs-docs/log.md
## 01a0b30f-1e89-763e-b65d-48d87e6ee683
updated_at: 1789756946
## Incident: frank-medium left down after bench protocol (2026-09-18)
- **Symptom:** connection refused on 192.168.0.235:7770. Unit was `inactive` (enabled, unmasked, NRestarts=0) — not a crash.
- **Root cause:** AGENTS.md bench stop-and-mask protocol executed from PWD=/home/dank via agent SSH. Timeline: stop+mask 02:47:00, unmask+start 04:21:27, final stop 04:25:59. Step 3 of the protocol (unmask/start + health verify) never ran for the last round.
- **Recovery sequence (safe order):**
1. Confirm GPUs idle: both R9700s ~60 MiB, no llama-bench/llama-server procs.
2.`systemctl start frank-medium` → active, health 200 first probe.
- **Durable lesson:** The restore half of the bench protocol is not optional. A session that benches owes the fleet a health 200 before it ends. Recommended fix: make the bench harness self-restoring (trap EXIT → start + health-verify) so a dead session can't strand the endpoint.
- BusyBox wget in caddy container: NO -H flag, use `--header "Host: ..."`.
- 8016/8017 live in caddy container's netns; curl from host returns 000 (expected).
- Anubis container is distroless; podman exec sh fails.
- x-real-ip is masked to 10.89.1.22 (podman SNAT) for ALL clients; clearance is cookie-based.
- Homepage app.js does Promise.all([fetch services.json, fetch status.json]); on ANY fetch failure the grid stays empty and summary shows 'status unavailable: <err>'.
## 01a0b23a-7a27-7792-b8da-97d7421b99c3
updated_at: 1789756944
## Plex /mnt/entropy group permission fix on dankserver
### Problem
Plex container on dankserver cannot read files in /mnt/entropy. Host-side permissions on /mnt/entropy must NOT be changed (no chmod/chgrp/chown/setfacl on dir or contents).
2.`ssh dankserver 'stat -c "%A %U:%G %n" /mnt/entropy; getfacl -p /mnt/entropy; ls -ln /mnt/entropy | head'` — identify owning group of files
3.`ssh dankserver 'findmnt /mnt/entropy'` — determine mount type (NFS, mergerfs, local disk)
4.`ssh dankserver 'id dank'` — check if host user is already in the required group
5. Confirm container sees the tree: `podman exec plex ls /mnt/entropy | wc -l`
6. Probe a specific file: `podman exec plex head -c 16 <file>`
### Fix approach (container-side only)
- **Rootless quadlet**: Use `GroupAdd=keep-groups` in the quadlet unit. This passes the host user's supplementary groups into the container. Prerequisite: host user `dank` must be a member of the group that owns /mnt/entropy files. Adding dank to that group is a host change (allowed), not a /mnt/entropy change.
- **Rootful container**: Use `GroupAdd=<gid>` with the numeric GID of the owning group directly.
- **Alternative**: `--group-add keep-groups` in compose if still compose-based.
### Key fleet patterns
- Dankserver media stack (Plex, Jellyfin, RomM, Copyparty, Calibre-Web) uses compose.yaml in repo `dankserver/` dir, but boot persistence requires quadlet units (compose-to-quadlet cutover in progress).
- Rootless podman user-unit spawns carry NO supplementary groups by default — must explicitly use `GroupAdd=keep-groups` or enumerate GIDs.
- PMS may return HTTP 000 on /identity while port 32400 is listening (pre-bind before HTTP layer ready); especially after long downtime. Wait or retry.
### Session status
Interrupted before final fix applied. Next step: confirm the GID owning /mnt/entropy files, ensure `dank` is in that group (if not, `sudo usermod -aG <group> dank`), set `GroupAdd=keep-groups` in the plex quadlet unit, deploy, verify with `podman exec plex head -c 1 <file>`.
- CRITICAL: OMP snapshots model config at process start; existing sessions are permanently pinned to the old value until restarted (use `omp -r` to resume)
- Wire-verified via tcpdump on dankclaw: fresh one-shot omp sends max_completion_tokens: 57344
## Testing
- 326 seams passing (up from 321; a peer added a seerr test)
## 01a0ac72-2620-750f-82dd-79c84e00e98d
## 01a0ac72-2620-750f-82dd-79c84e00e98d
updated_at: 1789631572
updated_at: 1789631572
DANKCLAW HARDWARE: AMD Strix Halo APU (gfx1151), Radeon 8060S iGPU. Has an M.2 dock slot used to carry an RTX 3090 at PCIe x4 (link downgraded from x16). This is distinct from dankserver (ASUS TUF GAMING B650-E WIFI) which has a free CPU-attached M.2_2 slot for an R9700 riser (~$30 M.2→PCIe x4 riser).
DANKCLAW HARDWARE: AMD Strix Halo APU (gfx1151), Radeon 8060S iGPU. Has an M.2 dock slot used to carry an RTX 3090 at PCIe x4 (link downgraded from x16). This is distinct from dankserver (ASUS TUF GAMING B650-E WIFI) which has a free CPU-attached M.2_2 slot for an R9700 riser (~$30 M.2→PCIe x4 riser).
@@ -152,7 +694,7 @@ updated_at: 1789544414
-`npx skills` CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
-`npx skills` CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
- Newly installed skills surface on the NEXT session, not the current one.
- Newly installed skills surface on the NEXT session, not the current one.
Verified fleet health across dankclaw/dankserver/frankbook-pro. Deployed frank-medium 35:30 tensor split (VRAM headroom on bottleneck card tripled to 3.00 GiB; layer lever exhausted at 36:29). Diagnosed and confirmed fix for frank-large max_tokens 400 dead zone (maxTokens 57344 in models.yml closes the gap; old OMP sessions must be restarted to pick up config). 326 seams green.
Diagnosing Plex container read-permission failure on /mnt/entropy at dankserver. Constraint: cannot modify /mnt/entropy permissions; fix must be container-side via GroupAdd. Session interrupted before final fix was applied and verified.
frank-medium (192.168.0.235:7770) found stopped after an incomplete bench protocol run: the final `systemctl stop` at 04:25:59 was never followed by unmask+start+health-verify. Restored safely (GPU idle check → start → health 200 → /proc/MainPID/cmdline geometry verify → behavioral completion). Durable lesson: the restore half of the bench protocol is not optional; a session that benches owes the fleet a health 200 before it ends. Consider making the bench harness self-restoring via trap EXIT.
frank-medium outage on 2026-09-18 (second of the day) caused by an orphaned merge-build llama-server (PID 2519965, PPID 1, bound 127.0.0.1:17770) holding 15.5+17.1 GiB across both R9700s after a successful load test that was never cleaned up. Unit was in clean-stopped state (inactive/enabled/unmasked/NRestarts=0). Diagnosed via rocm-smi showing busy GPUs despite inactive unit; proved abandonment via frozen /proc/pid/stat, zero established conns, loopback-only bind, 3h stale log. Fixed by killing stray and restarting frank-medium (503→200 in ~20s). Logged as commit cfabe07.
Doubled CACHE_RAM 8192→16384 MiB for frank-medium (llama.cpp, Qwen3.8-27B hybrid DeltaNet on 2x R9700). The old cap refused real states (137 refusals in 2 days, up to 11.4 GiB) because a full slot state is ~12.8 GiB at 196608 ctx (68 KiB/token from 16 full-attn layers + MTP draft KV). With --ctx-checkpoints 0, a refused state means full re-prefill next turn — the RAM cache is the only cross-slot state carrier for hybrid DeltaNet models.
Two topics: (1) vllm-radiance TP=2 is impossible on Dankserver's 2x R9700 (gfx1201) because the chipset lacks PCIe atomics (Gen4 x1, AtomicOps Routing-), which RCCL collectives require; the workaround was two single-GPU replicas behind a round-robin proxy, later replaced by llama.cpp DFlash2. (2) Updated stale comments in ~/.omp/agent/models.yml after MTP→DFlash2 drafter swap: model is now Qwen3.8-27B-Uncensored-FP8 (30.9GB/7 shards), accept 4.24/7, KV pool shrank from 943,581 to 384,238 tokens (3.59→1.47 concurrent) due to DFlash2's 2.1GiB drafter cost.
Calibre Content Server deployed on Dankserver (192.168.0.235:8084) with four critical traps discovered: LS image 9.15.0 is desktop not headless, internal port 8080 is nginx so use 8090, default auth is Digest not Basic, and echo-piped passwords include trailing newline. Calibre-Web 0.6.27 confirmed to have no generic OIDC; guard chain is the only SSO path.
frank-medium (vLLM/radiance on dankserver) already supports --chat-template via $CHAT_TEMPLATE in radiance-entrypoint.sh line 253. The Qwen-Sharp chat template is archived at scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp. Applying it to frank-medium requires pointing CHAT_TEMPLATE to that file (or a copy in a stable location). The frank-large swap (Kairic Edge → new model + DFlash2 drafter) was completed in commits 7689092 + 600812d.
Explored homepage service architecture (services.json + app.js category-based rendering) and the three-script deployment pipeline (deploy.sh → deploy-plan.py → deploy-remote.sh) with path-aware service scoping, per-service state store for crash recovery, and a Playwright e2e gate against the live fleet.
Explored homepage architecture (shared/homepage/ with services.json, app.js, style.css) for section-based reorganization; discovered Open WebUI image generation API endpoint at /api/images/generate in the open-webui podman container on dankclaw.
OMP session persistence architecture: JSONL files at ~/.omp/agent/sessions/<project-slug>/<timestamp>_<id>.jsonl; blob externalization (strings>500K truncated, image data URLs always content-addressed as blob:sha256:<hash>, other base64 images at 1024 chars); storage backends (File/Memory/Indexed); discovery reads 4KiB prefix + 32KiB tail; HistoryStorage is separate SQLite at ~/.omp/agent/history.db for prompt recall.
Exhaustive fact-check and OKF compliance pass on serverAppConfigs-docs vault (136 pages). Key corrections: ComfyUI 0.37.0 has native Qwen-Image 2.1 (custom GGUF patches may be redundant on rebuild); OpenWebUI imagegen endpoint is POST /api/v1/images/generations; CVE-2025-7404 is Calibre-Web not Calibre; GTT is 114 GiB not 96; deployment is Quadlet/deploy-plan.py based. Three research briefs added (local TTS/voice cloning, AI job search tools, popular self-hosted AI apps Jun-Sep 2026) plus HTML report. All 336 tests pass, 0 unresolved wikilinks.
A 3000-word narrative depicting the invention of the 'Soviet burger'—a kotleta (meat patty) nestled in a pampushka bun. It explores the transition from a disorganized cafeteria meal to a standardized GOST product, highlighting the tension between bureaucratic perfection (the standard) and the variable 'life' of the kitchen.
Deploying GLM-5-Next (GLM-5.3-Flash) 321.3B MoE model on dankserver. Key features include vision tower support, MTP draft mode, and a specific requirement for `-fa off` and `-ub 128` to handle MLA and long-context collapse. Since the Q6_K weights (~264GB) exceed the 246GB RAM, it uses `mmap` with `-ot exps=CPU` to offload experts to host RAM.
Established foundational engineering laws for the Frank fleet: -ot (tensor overrides) follows first-match-wins logic; -ot exps=CPU reduces PCIe overhead; spec-decoding limits (one context-backed drafter + multiple ngrams); ubatch optimum is 512; and GGUF extents must be derived from header offsets to avoid 'gap' buckets.
The standard 'bench stop-and-mask' protocol. A session that benches owes the fleet a health 200 before it ends. Safe order: 1. Confirm GPUs idle. 2. systemctl start. 3. Health 200. 4. Verify geometry (/proc/MainPID/cmdline). 5. Behavioral proof (completion).
- 2026-09-14 dankware outage: upstream commit 49afdfe ("Noctalia theme migration") committed config.kdl with include "noctalia.kdl" but noctalia.kdl is a per-host GENERATED file, never in the repo; the same pull set ghostty theme = noctalia and alacritty import = themes/noctalia.toml, both also template-generated and absent here. Niri 26.04 does NOT exit on a config error — it logs "error parsing" and boots with DEFAULT config, producing a half-dead desktop (no keybinds/rules/autostart shell, default output placement). noctalia msg templates-apply answers "ok" but is a NO-OP on noctalia 5.0.1 (opt-in schema mismatch). Working fix (offline, no session restart needed): noctalia theme --theme-json ~/.config/noctalia/colors.json --render "builtin template:output" for each: templates/niri/niri.kdl → ~/.config/niri/noctalia.kdl (plain host-local file, NOT a repo symlink), templates/ghostty/ghostty → ~/.config/ghostty/themes/noctalia, templates/alacritty/alacritty.toml → ~/.config/alacritty/themes/noctalia.toml. Tokens resolve fully from the 16-key colors.json (primary_container etc. are derived). Verify: niri validate rc0; fresh login loads full config (journal "loaded config" + custom output placement). Also verified: kill -USR1 was NOT the cause of the session ending that night — journal showed a fresh niri-session login; don't accept blame for user-initiated logouts.
- 2026-09-14 the workstation outage: upstream commit 49afdfe ("Noctalia theme migration") committed config.kdl with include "noctalia.kdl" but noctalia.kdl is a per-host GENERATED file, never in the repo; the same pull set ghostty theme = noctalia and alacritty import = themes/noctalia.toml, both also template-generated and absent here. Niri 26.04 does NOT exit on a config error — it logs "error parsing" and boots with DEFAULT config, producing a half-dead desktop (no keybinds/rules/autostart shell, default output placement). noctalia msg templates-apply answers "ok" but is a NO-OP on noctalia 5.0.1 (opt-in schema mismatch). Working fix (offline, no session restart needed): noctalia theme --theme-json ~/.config/noctalia/colors.json --render "builtin template:output" for each: templates/niri/niri.kdl → ~/.config/niri/noctalia.kdl (plain host-local file, NOT a repo symlink), templates/ghostty/ghostty → ~/.config/ghostty/themes/noctalia, templates/alacritty/alacritty.toml → ~/.config/alacritty/themes/noctalia.toml. Tokens resolve fully from the 16-key colors.json (primary_container etc. are derived). Verify: niri validate rc0; fresh login loads full config (journal "loaded config" + custom output placement). Also verified: kill -USR1 was NOT the cause of the session ending that night — journal showed a fresh niri-session login; don't accept blame for user-initiated logouts.
- On this host's niri build, niri msg do reload-config is rejected ("unrecognized subcommand 'do'") — verified 2026-09-10 during the noctalia unification deploy. The working config-reload path is kill -USR1 $(pidof niri) (SIGUSR1 to the niri process); verified on both fleet hosts (dankware pid via /proc, blankpad over ssh). Future keybind/config deploys that need live reload must use the SIGUSR1 path, not niri msg do.
- On this host's niri build, niri msg do reload-config is rejected ("unrecognized subcommand 'do'") — verified 2026-09-10 during the noctalia unification deploy. The working config-reload path is kill -USR1 $(pidof niri) (SIGUSR1 to the niri process); verified on both fleet hosts (the workstation pid via /proc, blankpad over ssh). Future keybind/config deploys that need live reload must use the SIGUSR1 path, not niri msg do.
- Noctalia fleet architecture UNIFIED 2026-09-10: both dankware and blankpad now run bare noctalia (v5 native, package noctalia 5.0.1-1.1) as the desktop shell, launched via niri spawn-at-startup "noctalia" from repo autostart.kdl. The v4 Quickshell-hosted architecture (qs -c noctalia-shell, JSON settings.json) is retired on dankware; qs host process killed and replaced live by a transient systemd-run unit (noctalia-live) that hands off to autostart at next login — single-instance guard in the binary makes overlap safe. Live config layer on both hosts = TOML only: hand-edit ~/.config/noctalia/config.toml; runtime state ~/.local/state/noctalia/settings.toml loads last and overrides same keys (never hand-edit). Verified mechanics: noctalia msg resolves the running instance via D-Bus session bus, so it works from systemd user-unit context (systemd-run --user --pipe --wait) even though niri's own environ lacks WAYLAND_DISPLAY; spawn-at-startup children do receive a working display connection. Lock keybinds/swayidle unified to noctalia msg session lock; verified verb grammar includes panel-toggle {launcher,session,control-center,...}, volume-up/down/mute, mic-mute, media next/previous/playPause (graceful no-op rc 0 with no active player), brightness-up/down. Wallpapers: wallpaper-next/wallpaper-get confirm rotation draws from the configured [wallpaper] directory on both hosts; current wallpaper stays on the built-in default asset until first rotation after config-reload. Leftover dead packages on dankware (noctalia-shell 4.7.7, noctalia-qs fork) are inert — removal is optional cleanup, zero dependents post-flip.
- Noctalia fleet architecture UNIFIED 2026-09-10: both the workstation and blankpad now run bare noctalia (v5 native, package noctalia 5.0.1-1.1) as the desktop shell, launched via niri spawn-at-startup "noctalia" from repo autostart.kdl. The v4 Quickshell-hosted architecture (qs -c noctalia-shell, JSON settings.json) is retired on the workstation; qs host process killed and replaced live by a transient systemd-run unit (noctalia-live) that hands off to autostart at next login — single-instance guard in the binary makes overlap safe. Live config layer on both hosts = TOML only: hand-edit ~/.config/noctalia/config.toml; runtime state ~/.local/state/noctalia/settings.toml loads last and overrides same keys (never hand-edit). Verified mechanics: noctalia msg resolves the running instance via D-Bus session bus, so it works from systemd user-unit context (systemd-run --user --pipe --wait) even though niri's own environ lacks WAYLAND_DISPLAY; spawn-at-startup children do receive a working display connection. Lock keybinds/swayidle unified to noctalia msg session lock; verified verb grammar includes panel-toggle {launcher,session,control-center,...}, volume-up/down/mute, mic-mute, media next/previous/playPause (graceful no-op rc 0 with no active player), brightness-up/down. Wallpapers: wallpaper-next/wallpaper-get confirm rotation draws from the configured [wallpaper] directory on both hosts; current wallpaper stays on the built-in default asset until first rotation after config-reload. Leftover dead packages on the workstation (noctalia-shell 4.7.7, noctalia-qs fork) are inert — removal is optional cleanup, zero dependents post-flip.
- Divergence explained (as of 2026-09-10): dankware and blankpad both show noctalia 5.0.1-1.1 but run different live architectures because each host's autostart froze at its adoption-era architecture and package upgrades never rewrote autostart commands. Fossil record from pacman install dates: noctalia-shell 4.7.7-3 installed on dankware Jul 3 (v4 Quickshell-hosted architecture, JSON config; module files owned at /etc/xdg/quickshell/noctalia-shell/, launched via qs -c noctalia-shell from base niri package autostart.kdl); noctalia-qs 0.0.12-2 (Quickshell fork hosting the shell) installed Aug 24; v5-era noctalia package landed Sep 6 as a parallel package owning ONLY /usr/bin/noctalia + desktop file + README — CLI/client surface, no shell module. blankpad was provisioned fresh Sep 9 entirely from CachyOS repo packaging (noctalia 5.0.1-1.1 + vanilla quickshell 0.3.1-1.1 + cachyos-niri-noctalia 1.4.0-1; no v4-era packages exist in that repo set), so its zz-blankpad override autostart launches bare noctalia (native v5, TOML config). Consequence: settings.json is genuinely the live config on dankware (v4 code), while blankpad runs true v5 with TOML state at ~/.config/noctalia/config.toml + ~/.local/state/noctalia/settings.toml. Unification candidate when approved: flip both to noctalia as single entrypoint, but verify first that the v5 packaging on dankware ships the shell module (current -Ql evidence says CLI-only) — a unified flip likely requires installing the full v5 package set on dankware first. Diagnostic recipe for this class of question: query pacman install dates + package ownership (pacman -Qi/-Ql/-Qo, pgrep -af) to reconstruct which architecture each host actually runs rather than trusting version labels.
- Divergence explained (as of 2026-09-10): the workstation and blankpad both show noctalia 5.0.1-1.1 but run different live architectures because each host's autostart froze at its adoption-era architecture and package upgrades never rewrote autostart commands. Fossil record from pacman install dates: noctalia-shell 4.7.7-3 installed on the workstation Jul 3 (v4 Quickshell-hosted architecture, JSON config; module files owned at /etc/xdg/quickshell/noctalia-shell/, launched via qs -c noctalia-shell from base niri package autostart.kdl); noctalia-qs 0.0.12-2 (Quickshell fork hosting the shell) installed Aug 24; v5-era noctalia package landed Sep 6 as a parallel package owning ONLY /usr/bin/noctalia + desktop file + README — CLI/client surface, no shell module. blankpad was provisioned fresh Sep 9 entirely from CachyOS repo packaging (noctalia 5.0.1-1.1 + vanilla quickshell 0.3.1-1.1 + cachyos-niri-noctalia 1.4.0-1; no v4-era packages exist in that repo set), so its zz-blankpad override autostart launches bare noctalia (native v5, TOML config). Consequence: settings.json is genuinely the live config on the workstation (v4 code), while blankpad runs true v5 with TOML state at ~/.config/noctalia/config.toml + ~/.local/state/noctalia/settings.toml. Unification candidate when approved: flip both to noctalia as single entrypoint, but verify first that the v5 packaging on the workstation ships the shell module (current -Ql evidence says CLI-only) — a unified flip likely requires installing the full v5 package set on the workstation first. Diagnostic recipe for this class of question: query pacman install dates + package ownership (pacman -Qi/-Ql/-Qo, pgrep -af) to reconstruct which architecture each host actually runs rather than trusting version labels.
- Noctalia shell runtime split on the fleet (as of 2026-09): both dankware and blankpad install noctalia 5.0.1-1.1 (-dirty = git build), but launch differently — dankware hosts it via qs -c noctalia-shell (Quickshell host; reads JSON-era config ~/.config/noctalia/settings.json), blankpad runs the bare v5 binary (reads hand-edit ~/.config/noctalia/config.toml, runtime state written to ~/.local/state/noctalia/settings.toml which overrides declarative values last). Lock chain on both hosts: primary qs -c noctalia-shell ipc call lockScreen lock, fallback noctalia msg session lock|suspend|lock-and-suspend|logout|reboot|shutdown. Quirk: noctalia msg ... from a non-session context (ssh command exec) always answers "noctalia is not running" even when the shell runs — detection needs session-scoped state absent in sshd-spawned shells; DBUS_SESSION_BUS_ADDRESS override does NOT fix it. Correct remote procedure: write config files directly (they hot-reload or apply at next start); never treat the msg failure over ssh as evidence the shell is dead. Wallpaper dir default when unconfigured = XDG Pictures ($HOME/Pictures) via user-dirs.dirs; explicit [wallpaper] directory key overrides it — blankpad was seeded with directory = /home/blank/.local/share/noctalia/wallpapers (repo-tracked symlink target).
- Noctalia shell runtime split on the fleet (as of 2026-09): both the workstation and blankpad install noctalia 5.0.1-1.1 (-dirty = git build), but launch differently — the workstation hosts it via qs -c noctalia-shell (Quickshell host; reads JSON-era config ~/.config/noctalia/settings.json), blankpad runs the bare v5 binary (reads hand-edit ~/.config/noctalia/config.toml, runtime state written to ~/.local/state/noctalia/settings.toml which overrides declarative values last). Lock chain on both hosts: primary qs -c noctalia-shell ipc call lockScreen lock, fallback noctalia msg session lock|suspend|lock-and-suspend|logout|reboot|shutdown. Quirk: noctalia msg ... from a non-session context (ssh command exec) always answers "noctalia is not running" even when the shell runs — detection needs session-scoped state absent in sshd-spawned shells; DBUS_SESSION_BUS_ADDRESS override does NOT fix it. Correct remote procedure: write config files directly (they hot-reload or apply at next start); never treat the msg failure over ssh as evidence the shell is dead. Wallpaper dir default when unconfigured = XDG Pictures ($HOME/Pictures) via user-dirs.dirs; explicit [wallpaper] directory key overrides it — blankpad was seeded with directory = /home/blank/.local/share/noctalia/wallpapers (repo-tracked symlink target).
- blank@blankpad (CachyOS, Arch-like) deployment COMPLETED 2026-09-09 (supersedes earlier "pending action awaiting user go-ahead" note): full-replica rsync of /home/dank/.omp/ → /home/blank/.omp/ (.git included; push to Gitea NOT required for this transport) followed by deploy.sh sync on blankpad. SSH was initially blocked there by firewall default-drop (timeout, not refuse) and opened only after the user started sshd locally — expect that blocker pattern again for new fleet hosts. Host facts verified at deploy time: login shell fish; niri runs as niri --session reading the repo-symlinked config.kdl; installed binaries include noctalia CLI (its keybinds override package uses noctalia msg ..., NOT canonical qs -c noctalia-shell ipc call), alacritty, firefox, nautilus, fish, zsh, shelly-notifications; MISSING: qs/quickshell, ghostty, gnome-calculator — canonical keybinds relying on those are silent no-ops if deployed there. ~/.p10k.zsh on blankpad is an inert repo symlink until powerlevel10k-zsh is installed and sourced. zz-blankpad override package (display/keybinds/misc/rules/autostart/.zshrc) is tracked in the repo; deploy.sh's target derivation must strip the nested base-package segment for zz-* packages or overrides land on phantom paths like $HOME/niri/.config/...
- blank@blankpad (CachyOS, Arch-like) deployment COMPLETED 2026-09-09 (supersedes earlier "pending action awaiting user go-ahead" note): full-replica rsync of /home/dank/.omp/ → /home/blank/.omp/ (.git included; push to Gitea NOT required for this transport) followed by deploy.sh sync on blankpad. SSH was initially blocked there by firewall default-drop (timeout, not refuse) and opened only after the user started sshd locally — expect that blocker pattern again for new fleet hosts. Host facts verified at deploy time: login shell fish; niri runs as niri --session reading the repo-symlinked config.kdl; installed binaries include noctalia CLI (its keybinds override package uses noctalia msg ..., NOT canonical qs -c noctalia-shell ipc call), alacritty, firefox, nautilus, fish, zsh, shelly-notifications; MISSING: qs/quickshell, ghostty, gnome-calculator — canonical keybinds relying on those are silent no-ops if deployed there. ~/.p10k.zsh on blankpad is an inert repo symlink until powerlevel10k-zsh is installed and sourced. zz-blankpad override package (display/keybinds/misc/rules/autostart/.zshrc) is tracked in the repo; deploy.sh's target derivation must strip the nested base-package segment for zz-* packages or overrides land on phantom paths like $HOME/niri/.config/...
- blankpad is a fleet machine reached only as blank@blankpad.local (mDNS FQDN; bare hostname does not resolve from Dankware; 192.168.0.83). As of 2026-09-09 it runs the command-center deployment at /home/blank/.omp with host-specific divergence preserved as tracked override package dotfiles/zz-blankpad/: its own display.kdl output layout (not Dankware's DP/eDP rig), autostart/keybinds targeting the bare noctalia CLI instead of qs -c noctalia-shell ipc call, misc.kdl without renderD129, rules.kdl without the Noctalia floating-window rule, and blank's own minimal .zshrc. Its login shell is fish. Initial blocker (port 22 drop-not-refuse = sshd down or firewall default-drop) was fixed same day by user locally; if SSH drops again on blankpad, suspect firewall default-drop first, not auth.
- blankpad is a fleet machine reached only as blank@blankpad.local (mDNS FQDN; bare hostname does not resolve from the workstation; 192.168.0.83). As of 2026-09-09 it runs the command-center deployment at /home/blank/.omp with host-specific divergence preserved as tracked override package dotfiles/zz-blankpad/: its own display.kdl output layout (not the workstation's DP/eDP rig), autostart/keybinds targeting the bare noctalia CLI instead of qs -c noctalia-shell ipc call, misc.kdl without renderD129, rules.kdl without the Noctalia floating-window rule, and blank's own minimal .zshrc. Its login shell is fish. Initial blocker (port 22 drop-not-refuse = sshd down or firewall default-drop) was fixed same day by user locally; if SSH drops again on blankpad, suspect firewall default-drop first, not auth.
- blankpad (user blank) is fleet member at 192.168.0.83, reachable ONLY via mDNS FQDN blankpad.local — bare hostname does not resolve from Dankware. As of 2026-09-09 its SSH was unreachable from the LAN segment: ping fine (~1 ms, wired) but TCP 22 and alt ports (2222/8022/22222/10022) all TIMEOUT = firewall default-drop or sshd down; drop-not-refuse distinguishes this from sshd-listening-but-auth-failing. Pending action awaiting user go-ahead: deploy command-center to blank@blankpad via full-replica rsync of /home/dank/.omp/ (incl .git so core.hooksPath stays armed pre-push) → deploy.sh --check review → sync; transport deliberately chosen over clone-from-Gitea because main was 6 commits ahead of origin/main and push is reserved as a user-only trigger. Note: origin remote on Dankware embeds the Gitea token in plaintext URL — keep it off other machines' .git/config by using rsync as transport. _(context: User asked to set up the dotfiles/command-center deployment on blank@blankpad with a password credential; SSH entry point unreachable from this segment, so work was staged and handed back with exact unblock steps (systemctl enable --now sshd + firewalld internal-zone ssh service).)_
- blankpad (user blank) is fleet member at 192.168.0.83, reachable ONLY via mDNS FQDN blankpad.local — bare hostname does not resolve from the workstation. As of 2026-09-09 its SSH was unreachable from the LAN segment: ping fine (~1 ms, wired) but TCP 22 and alt ports (2222/8022/22222/10022) all TIMEOUT = firewall default-drop or sshd down; drop-not-refuse distinguishes this from sshd-listening-but-auth-failing. Pending action awaiting user go-ahead: deploy command-center to blank@blankpad via full-replica rsync of /home/dank/.omp/ (incl .git so core.hooksPath stays armed pre-push) → deploy.sh --check review → sync; transport deliberately chosen over clone-from-Gitea because main was 6 commits ahead of origin/main and push is reserved as a user-only trigger. Note: origin remote on the workstation embeds the Gitea token in plaintext URL — keep it off other machines' .git/config by using rsync as transport. _(context: User asked to set up the dotfiles/command-center deployment on blank@blankpad with a password credential; SSH entry point unreachable from this segment, so work was staged and handed back with exact unblock steps (systemctl enable --now sshd + firewalld internal-zone ssh service).)_
- The FLM (FastFlowLM) service on Dankclaw (gemma4-it:e4b, port 7771) only supports PNG and JPEG/JPG image formats. WebP images are rejected with "Unsupported image format, skipping this image." The data URL formats it expects are data:image/png;base64, data:image/jpeg;base64, and data:image/jpg;base64. To use WebP screenshots with the NPU, convert them to PNG first. Verified 2026-08-30: a WebP screenshot (1024x576, 71K) was rejected while the same content as PNG was correctly identified by the vision model. _(context: Dankclaw NPU (FLM, gemma4-it:e4b, port 7771) — image format support investigation)_
- The FLM (FastFlowLM) service on Dankclaw (gemma4-it:e4b, port 7771) only supports PNG and JPEG/JPG image formats. WebP images are rejected with "Unsupported image format, skipping this image." The data URL formats it expects are data:image/png;base64, data:image/jpeg;base64, and data:image/jpg;base64. To use WebP screenshots with the NPU, convert them to PNG first. Verified 2026-08-30: a WebP screenshot (1024x576, 71K) was rejected while the same content as PNG was correctly identified by the vision model. _(context: Dankclaw NPU (FLM, gemma4-it:e4b, port 7771) — image format support investigation)_
- When configuring dankclaw-npu (FLM NPU endpoint on port 7771) in models.json, the model entry must include a thinkingLevelMap field. The other two providers (dankserver, dankclaw) have it; if absent, schema validation fails. Pattern: {"thinkingLevelMap": {"max": "xhigh"}}. Also verify gemma4-it:e4b model ID is registered and .xclbin kernel symlink exists under /home/dank/.config/flm/xclbins/. _(context: Dankclaw NPU endpoint (dankclaw-npu provider, port 7771) had a schema error because gemma4-it:e4b was missing thinkingLevelMap. Fixed by adding it to both models.json and models.yml. The FLM NPU serves 34 models in its registry; gemma4-it:e4b is the currently wired model.)_
- When configuring dankclaw-npu (FLM NPU endpoint on port 7771) in models.json, the model entry must include a thinkingLevelMap field. The other two providers (dankserver, dankclaw) have it; if absent, schema validation fails. Pattern: {"thinkingLevelMap": {"max": "xhigh"}}. Also verify gemma4-it:e4b model ID is registered and .xclbin kernel symlink exists under /home/dank/.config/flm/xclbins/. _(context: Dankclaw NPU endpoint (dankclaw-npu provider, port 7771) had a schema error because gemma4-it:e4b was missing thinkingLevelMap. Fixed by adding it to both models.json and models.yml. The FLM NPU serves 34 models in its registry; gemma4-it:e4b is the currently wired model.)_
- Dankclaw NPU serves 34 FLM (FastFlowLM) models on port 7771. Only one is wired in models.json (gemma4-it:e4b) but all 34 are registered and available via per-request model routing without restart. Known behavior: single-lane NPU queues FIFO, per-request model switch costs ~7–11s wall for 4B-class reload. Common pitfall: some registered tags fail if .xclbin kernel pack symlink is missing under /home/dank/.config/flm/xclbins/ (fix: ln -s /opt/fastflowlm/share/flm/xclbins/PackName). Registry probe: curl http://192.168.0.143:7771/v1/models.
- Dankclaw NPU serves 34 FLM (FastFlowLM) models on port 7771. Only one is wired in models.json (gemma4-it:e4b) but all 34 are registered and available via per-request model routing without restart. Known behavior: single-lane NPU queues FIFO, per-request model switch costs ~7–11s wall for 4B-class reload. Common pitfall: some registered tags fail if .xclbin kernel pack symlink is missing under /home/dank/.config/flm/xclbins/ (fix: ln -s /opt/fastflowlm/share/flm/xclbins/PackName). Registry probe: curl http://192.168.0.143:7771/v1/models.
- Frank's cursor size is 36 (up from default 24) on Dankware running niri/Wayland. Applied via three channels: gsettings org.gnome.desktop.interface cursor-size=36 (primary), GTK settings.ini gtk-cursor-theme-size=36 (fallback), XCURSOR_SIZE=36 in ~/.profile (sessions). Cursor theme: capitaine-cursors. _(context: Dankware workstation — Frank requested a bigger mouse pointer; set to 36 across gsettings, GTK config, and environment)_
- Frank's cursor size is 36 (up from default 24) on the workstation running niri/Wayland. Applied via three channels: gsettings org.gnome.desktop.interface cursor-size=36 (primary), GTK settings.ini gtk-cursor-theme-size=36 (fallback), XCURSOR_SIZE=36 in ~/.profile (sessions). Cursor theme: capitaine-cursors. _(context: the workstation workstation — Frank requested a bigger mouse pointer; set to 36 across gsettings, GTK config, and environment)_
- Frank's default browser is Firefox (firefox.desktop), set via xdg-settings on Dankware (niri/Wayland). Previously was Chromium. Changes persist across reboots. _(context: Dankware workstation — Frank changed default from Chromium to Firefox via xdg-settings set default-web-browser firefox.desktop)_
- Frank's default browser is Firefox (firefox.desktop), set via xdg-settings on the workstation (niri/Wayland). Previously was Chromium. Changes persist across reboots. _(context: the workstation workstation — Frank changed default from Chromium to Firefox via xdg-settings set default-web-browser firefox.desktop)_
-Dankware (RTX 4070 Laptop, nvidia-open 610.57.04, kernel 7.2.0-1-cachyos, DynamicPowerManagement=2) suffers the known 610.x NVKMS DIFR-prefetch deadlock: nvidia-modeset/kthread_q livelocks in DifrPrefetchEventDeferredWork→nvWriteGpEntry holding the NVKMS API semaphore; any compositor (niri) atomic commit then wedges in D-state in ApplyModeSetConfig/nvkms_ioctl_from_kapi_try_pmlock. Symptoms: GUI frozen, no VT switch, SSH alive, nvidia-smi still answers (GPU idle P8, no Xid). Trigger: display-idle then flip/modeset. Unrecoverable in place — reboot via systemctl reboot --force --force. Candidate mitigations: NVreg_RegistryDwords="RMDisableDIFR=1" (one upstream report says ineffective) and DynamicPowerManagement=0. Bug class spans 595.71–610.57 upstream. _(context: 2026-08-30 GUI lockup diagnosis on dankware; user suspected nvidia modesetter, confirmed via hung-task stacks. Pending user decision on writing modprobe mitigation.)_
-the workstation (RTX 4070 Laptop, nvidia-open 610.57.04, kernel 7.2.0-1-cachyos, DynamicPowerManagement=2) suffers the known 610.x NVKMS DIFR-prefetch deadlock: nvidia-modeset/kthread_q livelocks in DifrPrefetchEventDeferredWork→nvWriteGpEntry holding the NVKMS API semaphore; any compositor (niri) atomic commit then wedges in D-state in ApplyModeSetConfig/nvkms_ioctl_from_kapi_try_pmlock. Symptoms: GUI frozen, no VT switch, SSH alive, nvidia-smi still answers (GPU idle P8, no Xid). Trigger: display-idle then flip/modeset. Unrecoverable in place — reboot via systemctl reboot --force --force. Candidate mitigations: NVreg_RegistryDwords="RMDisableDIFR=1" (one upstream report says ineffective) and DynamicPowerManagement=0. Bug class spans 595.71–610.57 upstream. _(context: 2026-08-30 GUI lockup diagnosis on the workstation; user suspected nvidia modesetter, confirmed via hung-task stacks. Pending user decision on writing modprobe mitigation.)_
- When downloading large model shards (40GB) to dankclaw via SSH: 1. NEVER run rm on partial downloads — always clean up BEFORE starting the first download 2. Ensure only ONE wget process per file — if you see duplicate wget PIDs killing them with kill -9 before restarting, or use fuser to verify exclusivity 3. Use wget -c (continue) to resume partial downloads — SSH connections often drop mid-download on large files (~300s timeout) 4. Download sequentially (not in parallel) when using background task agents that may spawn duplicate processes, or coordinate downloads through a single agent session 5. Verify each file's size against expected values after download completes
- When downloading large model shards (40GB) to dankclaw via SSH: 1. NEVER run rm on partial downloads — always clean up BEFORE starting the first download 2. Ensure only ONE wget process per file — if you see duplicate wget PIDs killing them with kill -9 before restarting, or use fuser to verify exclusivity 3. Use wget -c (continue) to resume partial downloads — SSH connections often drop mid-download on large files (~300s timeout) 4. Download sequentially (not in parallel) when using background task agents that may spawn duplicate processes, or coordinate downloads through a single agent session 5. Verify each file's size against expected values after download completes
- kingjones777's Qwen3.8-Flash-Next ROCmFP4 builds for Strix Halo (gfx1151) have three tiers: FAST (~88 GiB), STRIX_LEAN (~98.5 GiB, 3-shard), STRIX (~113.5 GiB). All are text-only — no vision tower, no mmproj, no fused MTP draft head. Uses TYPE_100/TYPE_101 ROCmFP4 tensor types (general.file_type=106), requires patched llama.cpp PR #27742 from ROCmFPX fork. STRIX_LEAN carries a 35.76 GiB PLE n-gram table streamed off SSD at runtime; ~63 GiB GTT for non-PLE weights, ~75 GiB at full 262K context. Uncensored variants exist from orcarouter but only for Qwen3.8-27B (not Flash-Next), and in non-ROCmFP4 format. _(context: User requested kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN-GGUF for frank-default-large on dankclaw. Search revealed the repo doesn't exist; only the base STRIX_LEAN tier is available, and it's text-only with no MTP.)_
- kingjones777's Qwen3.8-Flash-Next ROCmFP4 builds for Strix Halo (gfx1151) have three tiers: FAST (~88 GiB), STRIX_LEAN (~98.5 GiB, 3-shard), STRIX (~113.5 GiB). All are text-only — no vision tower, no mmproj, no fused MTP draft head. Uses TYPE_100/TYPE_101 ROCmFP4 tensor types (general.file_type=106), requires patched llama.cpp PR #27742 from ROCmFPX fork. STRIX_LEAN carries a 35.76 GiB PLE n-gram table streamed off SSD at runtime; ~63 GiB GTT for non-PLE weights, ~75 GiB at full 262K context. Uncensored variants exist from orcarouter but only for Qwen3.8-27B (not Flash-Next), and in non-ROCmFP4 format. _(context: User requested kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN-GGUF for frank-default-large on dankclaw. Search revealed the repo doesn't exist; only the base STRIX_LEAN tier is available, and it's text-only with no MTP.)_
- Model provider config is SINGLE-SOURCE: `~/.omp/agent/models.yml` (OMP is the primary harness). The pi-era dual-copy convention (`~/.pi/agent/models.json` as source of truth + generated YAML import) is retired — there is no second file to keep in sync. Validate edits with a YAML parse. Label strings are NOT cached in ~/.omp/agent/models.db (model_cache table), so no cache flush is needed after label edits — new labels apply on the next config read.
- Model provider config is SINGLE-SOURCE: `~/.omp/agent/models.yml` (OMP is the primary harness). The pi-era dual-copy convention (`~/.pi/agent/models.json` as source of truth + generated YAML import) is retired — there is no second file to keep in sync. Validate edits with a YAML parse. Label strings are NOT cached in ~/.omp/agent/models.db (model_cache table), so no cache flush is needed after label edits — new labels apply on the next config read.
Repository /home/dank/.omp: develop has stashes, main synced with origin. Uncommitted: modified AGENTS.md, config/models, deleted skills. Desktop unification complete: dankware & blankpad run bare noctalia v5 native via live runtime flip. Lock uses D-Bus msg without WAYLAND_DISPLAY pinning. Keybinds use verified v5 verbs. Config reload via kill -USR1. v5 uses TOML-only (legacy v4 JSONs inert). Commit 961de3b local on dankware; rsync deploy to blankpad verified.
Repository /home/dank/.omp: develop has stashes, main synced with origin. Uncommitted: modified AGENTS.md, config/models, deleted skills. Desktop unification complete: the workstation & blankpad run bare noctalia v5 native via live runtime flip. Lock uses D-Bus msg without WAYLAND_DISPLAY pinning. Keybinds use verified v5 verbs. Config reload via kill -USR1. v5 uses TOML-only (legacy v4 JSONs inert). Commit 961de3b local on the workstation; rsync deploy to blankpad verified.
- Dankclaw's charlie12345/ROCmFPX llama.cpp fork (build at ~/GitPackages/ROCmFPX/build-qwen38) shares the mainline --spec-type vocabulary plus draft-dspark; --spec-draft-n-max default is 16 there (mainline default 3). ngram-mod params on the fork: --spec-ngram-mod-n-min (default 48), --spec-ngram-mod-n-max (default 64), --spec-ngram-mod-n-match (default 24). Constraint in common/speculative.cpp: n_min is clamped to = n_max (std::max(0, std::min(n_min, n_max))) and n_max clamped to a server-side n_draft_max; NO constraint between n_match and n_min/n_max — n_match (lookup window) is independent of draft token count. Use these when building the ngram-mod pyramid sweep for frank-llm-bench. _(context: frank-llm-bench project; Dankclaw ROCmFPX fork at ~/GitPackages/ROCmFPX/build-qwen38)_
- Dankclaw's charlie12345/ROCmFPX llama.cpp fork (build at ~/GitPackages/ROCmFPX/build-qwen38) shares the mainline --spec-type vocabulary plus draft-dspark; --spec-draft-n-max default is 16 there (mainline default 3). ngram-mod params on the fork: --spec-ngram-mod-n-min (default 48), --spec-ngram-mod-n-max (default 64), --spec-ngram-mod-n-match (default 24). Constraint in common/speculative.cpp: n_min is clamped to = n_max (std::max(0, std::min(n_min, n_max))) and n_max clamped to a server-side n_draft_max; NO constraint between n_match and n_min/n_max — n_match (lookup window) is independent of draft token count. Use these when building the ngram-mod pyramid sweep for frank-llm-bench. _(context: frank-llm-bench project; Dankclaw ROCmFPX fork at ~/GitPackages/ROCmFPX/build-qwen38)_
- llama.cpp (mainline, verified 2026-08) spec-decoding flags: --spec-type takes a comma-separated list: none, draft-simple, draft-eagle3, draft-mtp, draft-dflash, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache. Multiple speculators stack via comma (e.g. "draft-mtp,ngram-mod"). n-max-spec maps to TWO different flags: --spec-draft-n-max N (default 3) controls draft-model token count; --spec-ngram-mod-n-max N controls ngram-mod token count. For stacked configs, set both to the same N. MTP capability is detected from GGUF metadata key {arch}.nextn_predict_layers (0 means model has embedded MTP head). --spec-type draft-mtp requires an MTP-capable GGUF; it does not add MTP at runtime. ROCmFPX fork (charlie12345) may use different flag names at its llama.cpp base point — verify fork's --help on Dankclaw before relying on these. _(context: Grilling session for frank-llm-bench; flags confirmed from local ~/GitPackages/llama.cpp/build/bin/llama-server --help.)_
- llama.cpp (mainline, verified 2026-08) spec-decoding flags: --spec-type takes a comma-separated list: none, draft-simple, draft-eagle3, draft-mtp, draft-dflash, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache. Multiple speculators stack via comma (e.g. "draft-mtp,ngram-mod"). n-max-spec maps to TWO different flags: --spec-draft-n-max N (default 3) controls draft-model token count; --spec-ngram-mod-n-max N controls ngram-mod token count. For stacked configs, set both to the same N. MTP capability is detected from GGUF metadata key {arch}.nextn_predict_layers (0 means model has embedded MTP head). --spec-type draft-mtp requires an MTP-capable GGUF; it does not add MTP at runtime. ROCmFPX fork (charlie12345) may use different flag names at its llama.cpp base point — verify fork's --help on Dankclaw before relying on these. _(context: Grilling session for frank-llm-bench; flags confirmed from local ~/GitPackages/llama.cpp/build/bin/llama-server --help.)_
- Frank's three machines: (1) Dankware — dev laptop, Intel Core Ultra 7 155H, Arc iGPU + RTX 4070 Max-Q Mobile (discrete, non-unified memory), 62GB RAM, 22 threads. NEVER runs inference / never a bench target. llama.cpp mainline built at ~/GitPackages/llama.cpp/build/bin/. (2) Dankserver — currently runs FreeToken (not llama.cpp). (3) Dankclaw — Strix Halo AMD APU, single unified GPU memory pool → hits the n-cpu-moe [REDACTED] (non-unified-memory check fails). Runs charlie12345/ROCmFPX llama.cpp fork. Service frank-default-large, port 7770, health http://192.168.0.143:7770/health. Model: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN, 56B MoE (512 experts, 10 used), ~98GB, Q4_0_ROCMFP4. Text-only — NO embedded MTP draft head (confirmed in serverAppConfigs scripts/frank-default-large.env). Gitea self-hosted on Dankclaw at 192.168.0.143:3000, owner dank, API token embedded in serverAppConfigs git remote URL. No tea CLI installed. _(context: Grilling session for frank-llm-bench; facts gathered from serverAppConfigs AGENTS.md, env files, and live probes.)_
- Frank's three machines: (1) the workstation — dev laptop, Intel Core Ultra 7 155H, Arc iGPU + RTX 4070 Max-Q Mobile (discrete, non-unified memory), 62GB RAM, 22 threads. NEVER runs inference / never a bench target. llama.cpp mainline built at ~/GitPackages/llama.cpp/build/bin/. (2) Dankserver — currently runs FreeToken (not llama.cpp). (3) Dankclaw — Strix Halo AMD APU, single unified GPU memory pool → hits the n-cpu-moe [REDACTED] (non-unified-memory check fails). Runs charlie12345/ROCmFPX llama.cpp fork. Service frank-default-large, port 7770, health http://192.168.0.143:7770/health. Model: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN, 56B MoE (512 experts, 10 used), ~98GB, Q4_0_ROCMFP4. Text-only — NO embedded MTP draft head (confirmed in serverAppConfigs scripts/frank-default-large.env). Gitea self-hosted on Dankclaw at 192.168.0.143:3000, owner dank, API token embedded in serverAppConfigs git remote URL. No tea CLI installed. _(context: Grilling session for frank-llm-bench; facts gathered from serverAppConfigs AGENTS.md, env files, and live probes.)_
- Frank prefers merge (not rebase) for resolving branch drift before PRs. He dislikes the historical revision that comes with excessive rebasing. When updating a feature branch against a moved main, merge main into the branch rather than rebasing the branch onto main. This applies to all git workflows on his repos unless he says otherwise. _(context: Grilling session for frank-llm-bench project init; Q2 drift rule.)_
- Frank prefers merge (not rebase) for resolving branch drift before PRs. He dislikes the historical revision that comes with excessive rebasing. When updating a feature branch against a moved main, merge main into the branch rather than rebasing the branch onto main. This applies to all git workflows on his repos unless he says otherwise. _(context: Grilling session for frank-llm-bench project init; Q2 drift rule.)_
- Primary repo: /home/dank/Projects/serverAppConfigs on Dankware/Dankclaw/Dankserver contexts. Current repository state always wins over this memory.
- Primary repo: /home/dank/Projects/serverAppConfigs on the workstation, Dankclaw, and Dankserver contexts. Current repository state always wins over this memory.
- Dankserver: 192.168.0.235. Media services and frank-default-small/media stack pieces.
- Dankserver: 192.168.0.235. Media services and frank-default-small/media stack pieces.
- FrankBook-Pro: 192.168.0.112 / FrankBook-Pro.local, user dank, macOS x86_64 with Homebrew, Podman 5.4.2, applehv/vfkit machine.
- FrankBook-Pro: 192.168.0.112 / FrankBook-Pro.local, user dank, macOS x86_64 with Homebrew, Podman 5.4.2, applehv/vfkit machine.
-Dankware: workstation/operation host.
-Workstation: the operation host.
- Passwordless SSH was verified among Dankclaw, Dankserver, FrankBook-Pro, and Hermes containers. Hermes containers use /root/.ssh/id_ed25519 for root/dank host-to-host checks.
- Passwordless SSH was verified among Dankclaw, Dankserver, FrankBook-Pro, and Hermes containers. Hermes containers use /root/.ssh/id_ed25519 for root/dank host-to-host checks.
- Authentik 2026.8 embedded outpost (franksdank.cloud fleet) caches claims per browser session in the authentik_providers_proxy_proxysession table (session_data JSON holds claims incl. groups; observed row TTLs 1–24 h). A session authorized BEFORE a user is added to a group keeps serving the old group list to Caddy header_regexp RBAC gates → persistent 403 despite correct membership, and the 403 generates ZERO authentik events (it never reaches authentik), so logs look silent. Re-login does not help (browser cookie re-maps to the same stale row). Fix (verified live 2026-09-01 for akadmin /admin 403): DELETE rows WHERE session_data-'claims'-'preferred_username' = 'user' in the authentik-postgres container — the django login session is untouched, so the next visit re-authorizes silently and mints fresh claims. Do NOT rotate AUTHENTIK_SECRET_KEY for this (invalidates every outpost session fleet-wide). Diagnosis SQL + silent-reauth verification documented in serverAppConfigs-docs/Infrastructure/Authentik.md (commit ea0bba3). _(context: Frank reported persistent 403 at www.franksdank.cloud/admin after being added to homepage-admins; recon found his live proxysession row (expiring 2026-09-02 22:06 UTC) carried ["authentik Admins"] only, minted before the group existed. Silent re-auth proven by deleting qa-e2e-admin's 13 rows and running the e2e admin spec with --no-deps against the stale storage state (2 passed, 2.9 s, no login pr)_
- Authentik 2026.8 embedded outpost (franksdank.cloud fleet) caches claims per browser session in the authentik_providers_proxy_proxysession table (session_data JSON holds claims incl. groups; observed row TTLs 1–24 h). A session authorized BEFORE a user is added to a group keeps serving the old group list to Caddy header_regexp RBAC gates → persistent 403 despite correct membership, and the 403 generates ZERO authentik events (it never reaches authentik), so logs look silent. Re-login does not help (browser cookie re-maps to the same stale row). Fix (verified live 2026-09-01 for akadmin /admin 403): DELETE rows WHERE session_data-'claims'-'preferred_username' = 'user' in the authentik-postgres container — the django login session is untouched, so the next visit re-authorizes silently and mints fresh claims. Do NOT rotate AUTHENTIK_SECRET_KEY for this (invalidates every outpost session fleet-wide). Diagnosis SQL + silent-reauth verification documented in serverAppConfigs-docs/Infrastructure/Authentik.md (commit ea0bba3). _(context: Frank reported persistent 403 at www.franksdank.cloud/admin after being added to homepage-admins; recon found his live proxysession row (expiring 2026-09-02 22:06 UTC) carried ["authentik Admins"] only, minted before the group existed. Silent re-auth proven by deleting qa-e2e-admin's 13 rows and running the e2e admin spec with --no-deps against the stale storage state (2 passed, 2.9 s, no login pr)_
- Frank wants periodic "what did we do" HTML change reports rendered in Firefox (asked 2026-09-01; first one at /home/dank/Reports/serverAppConfigs-24h-2026-09-01.html). Preferred standard: Keep a Changelog 1.1.0 categories (Added/Changed/Fixed/Removed/Security, omit empty) inside a post-change-review skeleton (summary → timeline → categories → code samples → verification → incidents → user actions). Reports are deliverables outside the repo at ~/Reports/ (never tracked; keeps tree clean). Frank reads the report to plan his code-review pass, so include a ready-made git show/git diff command list per theme. _(context: Built a 38 KB self-contained HTML report for serverAppConfigs covering 39 commits/24h; verified render via headless browser screenshot before handing over; opened via firefox --new-tab file:///....)_
- Frank wants periodic "what did we do" HTML change reports rendered in Firefox (asked 2026-09-01; first one at /home/dank/Reports/serverAppConfigs-24h-2026-09-01.html). Preferred standard: Keep a Changelog 1.1.0 categories (Added/Changed/Fixed/Removed/Security, omit empty) inside a post-change-review skeleton (summary → timeline → categories → code samples → verification → incidents → user actions). Reports are deliverables outside the repo at ~/Reports/ (never tracked; keeps tree clean). Frank reads the report to plan his code-review pass, so include a ready-made git show/git diff command list per theme. _(context: Built a 38 KB self-contained HTML report for serverAppConfigs covering 39 commits/24h; verified render via headless browser screenshot before handing over; opened via firefox --new-tab file:///....)_
- Playwright automation of authentik's Lit flow executor (fleet e2e, tests/e2e/lib/login.ts akLogin): a programmatic page.fill races the stage re-render — the Lit web component re-renders right after mount and wipes the input, so the form submits an EMPTY password ("This field is required") even though fill() succeeded. Hardened pattern: fill, VERIFY the value survived (expect the input's value, short retry), submit with Enter (native form path, no button re-render race), and retry the whole stage up to 3 attempts before failing. Two adjacent musts: every login runs in a FRESH browser context (a reused/half-finished context poisons authentik's pending_user stage and hangs the next login), and the browser needs a real Chrome UA (Anubis denies obvious headless UAs); budget 60s assertions for the first Anubis PoW per fresh context (subsequent guarded vhosts skip challenge thanks to COOKIE_DOMAIN solve-once). Conventions documented in serverAppConfigs-docs/Architecture/E2E Suite.md (commit b945dd0). _(context: frank-default e2e suite hardening, 2026-09-01: 10/11 spec flake root-caused to Lit re-render race + Anubis PoW budgets; suite now 17/17 green across setup/standard/admin/guest projects.)_
- Playwright automation of authentik's Lit flow executor (fleet e2e, tests/e2e/lib/login.ts akLogin): a programmatic page.fill races the stage re-render — the Lit web component re-renders right after mount and wipes the input, so the form submits an EMPTY password ("This field is required") even though fill() succeeded. Hardened pattern: fill, VERIFY the value survived (expect the input's value, short retry), submit with Enter (native form path, no button re-render race), and retry the whole stage up to 3 attempts before failing. Two adjacent musts: every login runs in a FRESH browser context (a reused/half-finished context poisons authentik's pending_user stage and hangs the next login), and the browser needs a real Chrome UA (Anubis denies obvious headless UAs); budget 60s assertions for the first Anubis PoW per fresh context (subsequent guarded vhosts skip challenge thanks to COOKIE_DOMAIN solve-once). Conventions documented in serverAppConfigs-docs/Architecture/E2E Suite.md (commit b945dd0). _(context: frank-default e2e suite hardening, 2026-09-01: 10/11 spec flake root-caused to Lit re-render race + Anubis PoW budgets; suite now 17/17 green across setup/standard/admin/guest projects.)_
- Fleet VPN truth (verified 2026-09-01): the gluetun VPN_SERVICE_PROVIDER=custom profile on Dankclaw (endpoint 149.102.254.77:51820) points at a **ProtonVPN Seattle WireGuard server**, NOT a self-run box. Evidence: RDAP via curl -sL https://rdap.org/ip/IP (works when whois is empty) — endpoint IP → CDNEXT-CG-SEA/Datacamp; Dankclaw's tunnel v6 2a07:b944::2:2 → CH-PROTONVPN-20160719; exit IP 149.22.88.19 → CDNEXT-SEA-CG. Full 65535-port scan fingerprint: no SSH anywhere, only silent TLS listeners on 443, 4443–4468, 5995, 7770, 8443 (Proton protocol-stealth port map). Consequences: (1) adding a "peer" is never server admin — it's generating a new WireGuard config in Frank's Proton dashboard; each config = one keypair, Proton has NO BYO pubkey (a locally generated keypair is useless; one was retired to dankclaw ~/.AGENTTRASH/20260901-frankbook-wg-keypair). (2) Two hosts sharing one Proton config make the server bounce the endpoint between them — reusing Dankclaw's config for FrankBook-Pro would kill the Dankclaw tunnel (SearXNG). (3) Mac gate: second dashboard config (e.g. named frankbook) → PrivateKey/Address paste into /home/core/Containers/gluetun/env; server pubkey utqNH4P+C0fA1wTgIUaIfhm1b7ai0iEGvXyiw4KC3Fs=, gluetun ignores the .conf DNS line. General technique: identify mystery infra with rdap.org RDAP + nmap port fingerprint before asking the user to administer it. _(context: FrankBook-Pro WG gate investigation, 2026-09-01. Probes: no SSH config aliases on Dankware/Dankclaw, port 22 timeout, full nmap scan, RDAP on 3 IPs. Docs rewritten: serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md "Mac provisioning" section; log.md entry. Repo commits 2d06be8, 3094ffa, f1171b9.)_
- Fleet VPN truth (verified 2026-09-01): the gluetun VPN_SERVICE_PROVIDER=custom profile on Dankclaw (endpoint 149.102.254.77:51820) points at a **ProtonVPN Seattle WireGuard server**, NOT a self-run box. Evidence: RDAP via curl -sL https://rdap.org/ip/IP (works when whois is empty) — endpoint IP → CDNEXT-CG-SEA/Datacamp; Dankclaw's tunnel v6 2a07:b944::2:2 → CH-PROTONVPN-20160719; exit IP 149.22.88.19 → CDNEXT-SEA-CG. Full 65535-port scan fingerprint: no SSH anywhere, only silent TLS listeners on 443, 4443–4468, 5995, 7770, 8443 (Proton protocol-stealth port map). Consequences: (1) adding a "peer" is never server admin — it's generating a new WireGuard config in Frank's Proton dashboard; each config = one keypair, Proton has NO BYO pubkey (a locally generated keypair is useless; one was retired to dankclaw ~/.AGENTTRASH/20260901-frankbook-wg-keypair). (2) Two hosts sharing one Proton config make the server bounce the endpoint between them — reusing Dankclaw's config for FrankBook-Pro would kill the Dankclaw tunnel (SearXNG). (3) Mac gate: second dashboard config (e.g. named frankbook) → PrivateKey/Address paste into /home/core/Containers/gluetun/env; server pubkey utqNH4P+C0fA1wTgIUaIfhm1b7ai0iEGvXyiw4KC3Fs=, gluetun ignores the .conf DNS line. General technique: identify mystery infra with rdap.org RDAP + nmap port fingerprint before asking the user to administer it. _(context: FrankBook-Pro WG gate investigation, 2026-09-01. Probes: no SSH config aliases on the workstation/Dankclaw, port 22 timeout, full nmap scan, RDAP on 3 IPs. Docs rewritten: serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md "Mac provisioning" section; log.md entry. Repo commits 2d06be8, 3094ffa, f1171b9.)_
- When a fleet credential "mystery" arises (auth works/doesn't contradict user memory): (1) assume session-archive images are corrupted — OCR of transcript frames mangles strings while preserving shape (Jellyfin admin password read as "ResFace_3241" instead of "<REDACTED-rotated-2026-09-10>"; the prefix was garbage, the digits were real; the user's own recall had the right prefix but flipped digits). (2) NEVER brute-force a live auth endpoint — pull the stored hash and verify candidates offline: Jellyfin stores it in jellyfin.db (Users table, Password column) as $PBKDF2-SHA512$iterations=210000$salt$hash where salt AND hash are HEX-ENCODED (32 B salt → 64 hex chars, 128-char hash). A parser that assumes base64 silently tests zero candidates — always print back the parsed lengths (saltbytes/hashbytes) and candidate count before concluding "no match". (3) A single matching hash is conclusive; a live POST adds nothing and just logs auth-failure warnings. _(context: Jellyfin 401 mystery on Dankserver, 2026-09-01. User's correction of the corrupted prefix + digit flip; offline hash check vs stored DB hash decided it. Files: serverAppConfigs-docs/Dankserver/Jellyfin.md, serverAppConfigs-docs/log.md (both corrected).)_
- When a fleet credential "mystery" arises (auth works/doesn't contradict user memory): (1) assume session-archive images are corrupted — OCR of transcript frames mangles strings while preserving shape (Jellyfin admin password read as "ResFace_3241" instead of "<REDACTED-rotated-2026-09-10>"; the prefix was garbage, the digits were real; the user's own recall had the right prefix but flipped digits). (2) NEVER brute-force a live auth endpoint — pull the stored hash and verify candidates offline: Jellyfin stores it in jellyfin.db (Users table, Password column) as $PBKDF2-SHA512$iterations=210000$salt$hash where salt AND hash are HEX-ENCODED (32 B salt → 64 hex chars, 128-char hash). A parser that assumes base64 silently tests zero candidates — always print back the parsed lengths (saltbytes/hashbytes) and candidate count before concluding "no match". (3) A single matching hash is conclusive; a live POST adds nothing and just logs auth-failure warnings. _(context: Jellyfin 401 mystery on Dankserver, 2026-09-01. User's correction of the corrupted prefix + digit flip; offline hash check vs stored DB hash decided it. Files: serverAppConfigs-docs/Dankserver/Jellyfin.md, serverAppConfigs-docs/log.md (both corrected).)_
- Rootless podman UID mapping on Dankclaw/Dankserver is CONTINUOUS, not offset: container uid N → host uid 100000 + N - 1 (dank subuid 100000:65536; container 0 → 1000, container 1 → 100000, container 1000 → 100999). Never compute host uid as 100000+N. Before chowning a bind root for a non-root image, read the real mapping from any throwaway container: podman run --rm image cat /proc/self/uid_map ("ID-inside ID-outside range" lines). A wrong chown gives EACCES crash-loops (quadlet auto-restart, container never persists; podman logs empty because --rm). Verified 2026-09-01 with seerr-team/seerr (in-image node uid 1000 → chown 100999 fixed it). _(context: Seerr first boot on Dankclaw crash-looped with EACCES mkdir /app/config/logs after chown 101000; /proc/self/uid_map showed "1 100000 65536" ⇒ container 1000 → host 100999. Second off-by-one incident on this fleet (postgres 100069 arithmetic was the first).)_
- Rootless podman UID mapping on Dankclaw/Dankserver is CONTINUOUS, not offset: container uid N → host uid 100000 + N - 1 (dank subuid 100000:65536; container 0 → 1000, container 1 → 100000, container 1000 → 100999). Never compute host uid as 100000+N. Before chowning a bind root for a non-root image, read the real mapping from any throwaway container: podman run --rm image cat /proc/self/uid_map ("ID-inside ID-outside range" lines). A wrong chown gives EACCES crash-loops (quadlet auto-restart, container never persists; podman logs empty because --rm). Verified 2026-09-01 with seerr-team/seerr (in-image node uid 1000 → chown 100999 fixed it). _(context: Seerr first boot on Dankclaw crash-looped with EACCES mkdir /app/config/logs after chown 101000; /proc/self/uid_map showed "1 100000 65536" ⇒ container 1000 → host 100999. Second off-by-one incident on this fleet (postgres 100069 arithmetic was the first).)_
- Two pitfalls from the 2026-09-01 frankbook/dankserver session: (1) zsh (default login shell on Dankware AND dankserver) expands =word at the start of an unquoted word to the command's path — so echo ==FOO== inside an ssh command fails with zsh:1: =FOO== not found. Quote echo arguments (echo "==FOO==") or use printf in any command line destined for zsh. (2) qBittorrent ≥ 4.2 ignores a plaintext WebUI\Password in qBittorrent.conf — it must be @ByteArray(salt:hash) from PBKDF2-HMAC-SHA512 (100000 iterations, 16-byte salt, 64-byte key, both base64). Also when the WebUI is reached through a port forwarder that preserves the LAN Host header (e.g. socat), set HostHeaderValidation=false and ServerDomains=* or qB rejects the request. Reference implementation: VM_SEED heredoc in deploy_frankbook_stack, scripts/deploy-remote.sh. _(context: frankbook-pro *arr bring-up + dankserver cutover verification, serverAppConfigs repo)_
- Two pitfalls from the 2026-09-01 frankbook/dankserver session: (1) zsh (default login shell on the workstation AND dankserver) expands =word at the start of an unquoted word to the command's path — so echo ==FOO== inside an ssh command fails with zsh:1: =FOO== not found. Quote echo arguments (echo "==FOO==") or use printf in any command line destined for zsh. (2) qBittorrent ≥ 4.2 ignores a plaintext WebUI\Password in qBittorrent.conf — it must be @ByteArray(salt:hash) from PBKDF2-HMAC-SHA512 (100000 iterations, 16-byte salt, 64-byte key, both base64). Also when the WebUI is reached through a port forwarder that preserves the LAN Host header (e.g. socat), set HostHeaderValidation=false and ServerDomains=* or qB rejects the request. Reference implementation: VM_SEED heredoc in deploy_frankbook_stack, scripts/deploy-remote.sh. _(context: frankbook-pro *arr bring-up + dankserver cutover verification, serverAppConfigs repo)_
- FreeToken v0.1.2 on qwen3_5_moe loads ONLY modelopt-style quant configs: block-fp8 (quant_method "fp8" + weight_scale_inv, the Huihui/official-Qwen format) works; every compressed-tensors FP8 checkpoint fails. Two live-verified failure modes (2026-09-01, Dankserver): (1) compressed-tensors per-channel (Ornith-1.5-35B-A3B-FP8) aborts at expert-source resolution — ValueError "Missing MoE expert source layers: {'gate_up': ...}" because expert tensors don't follow mlp.experts.{N}.gate_proj/up_proj/down_proj naming; (2) compressed-tensors block-fp8 128×128 (HauhauCS-Aggressive-fp8) resolves experts but _dequant_fp8_weight multiplies weight × weight_scale directly, so [out/128, in/128] block scales don't broadcast → RuntimeError shape mismatch (tensor a 2048 vs b ...). Both exit cleanly at load (~37 GB fits RAM, no swap-thrash). Before swapping any model onto FreeToken, check config.json quant_method/weight_scale shape; a "healthy after ~6s" from deploy.sh proves nothing (health gate passes while the doomed worker is still starting) — verify via journal Uvicorn startup + a real completion. Evidence: serverAppConfigs-docs/LLM/free[REDACTED].md Quantization Compatibility table. _(context: frank-default-small model-swap attempts on Dankserver, 2026-09-01: Ornith FP8 and HauhauCS FP8 both failed to load on FreeToken v0.1.2; reverted to batsclamp Huihui FP8 (commit 4d5bea4), docs commit fe34f04.)_
- FreeToken v0.1.2 on qwen3_5_moe loads ONLY modelopt-style quant configs: block-fp8 (quant_method "fp8" + weight_scale_inv, the Huihui/official-Qwen format) works; every compressed-tensors FP8 checkpoint fails. Two live-verified failure modes (2026-09-01, Dankserver): (1) compressed-tensors per-channel (Ornith-1.5-35B-A3B-FP8) aborts at expert-source resolution — ValueError "Missing MoE expert source layers: {'gate_up': ...}" because expert tensors don't follow mlp.experts.{N}.gate_proj/up_proj/down_proj naming; (2) compressed-tensors block-fp8 128×128 (HauhauCS-Aggressive-fp8) resolves experts but _dequant_fp8_weight multiplies weight × weight_scale directly, so [out/128, in/128] block scales don't broadcast → RuntimeError shape mismatch (tensor a 2048 vs b ...). Both exit cleanly at load (~37 GB fits RAM, no swap-thrash). Before swapping any model onto FreeToken, check config.json quant_method/weight_scale shape; a "healthy after ~6s" from deploy.sh proves nothing (health gate passes while the doomed worker is still starting) — verify via journal Uvicorn startup + a real completion. Evidence: serverAppConfigs-docs/LLM/free[REDACTED].md Quantization Compatibility table. _(context: frank-default-small model-swap attempts on Dankserver, 2026-09-01: Ornith FP8 and HauhauCS FP8 both failed to load on FreeToken v0.1.2; reverted to batsclamp Huihui FP8 (commit 4d5bea4), docs commit fe34f04.)_
- skill://quadlet-v1-generator-quirks
- skill://quadlet-v1-generator-quirks
- Before swapping a model into an LLM service, preflight the CHECKPOINT SIZE (bytes on disk / weight precision) against the host's usable RAM, not just VRAM — the failure is silent and self-sustaining, not a clean crash. Case: thomsonreuters/Thomson-1.0-Small is Qwen3.6-35B-A3B arch but ships BF16 (~66 GB) vs the Huihui FP8 it replaced (~35 GB); Dankserver has 62 GB system RAM. FreeToken streams MoE experts from host RAM, so 66 GB of weights exceed physical RAM. Compounding traps that turned a fit-failure into an outage: (1) 127 GB swap means the kernel PAGES instead of OOM-killing, so sshd is starved and never gets killed — no recovery window opens on its own; (2) systemd Restart=on-failure makes it a crash-reload-rethrash loop; (3) the unit is enabled + WantedBy=multi-user.target, so even a reboot re-thrashes on boot until the bad MODEL_DIR is masked/reverted — a plain power-cycle does NOT fix it. Correct recovery requires a shell to run systemctl mask --now svc FIRST (stops + breaks the reboot trap), then pull the revert. deploy.sh reports "healthy after Ns" as soon as systemd is active, BEFORE the model finishes loading — never treat that as load success; verify with a real completion. Verdict: BF16 35B-A3B cannot run on a 62 GB box; needs an FP8/GGUF quant or a bigger host. RTX 3090 has no FP8 tensor cores so even FP8 runs via upcast but still fits in RAM. _(context: 2026-09-01 frank-default-small swap to Thomson-1.0-Small on Dankserver caused a sustained swap-thrash outage; SSH locked out, no OOM relief due to 127GB swap. Repo revert staged (commit 1163844), awaiting physical/console recovery.)_
- Before swapping a model into an LLM service, preflight the CHECKPOINT SIZE (bytes on disk / weight precision) against the host's usable RAM, not just VRAM — the failure is silent and self-sustaining, not a clean crash. Case: thomsonreuters/Thomson-1.0-Small is Qwen3.6-35B-A3B arch but ships BF16 (~66 GB) vs the Huihui FP8 it replaced (~35 GB); Dankserver has 62 GB system RAM. FreeToken streams MoE experts from host RAM, so 66 GB of weights exceed physical RAM. Compounding traps that turned a fit-failure into an outage: (1) 127 GB swap means the kernel PAGES instead of OOM-killing, so sshd is starved and never gets killed — no recovery window opens on its own; (2) systemd Restart=on-failure makes it a crash-reload-rethrash loop; (3) the unit is enabled + WantedBy=multi-user.target, so even a reboot re-thrashes on boot until the bad MODEL_DIR is masked/reverted — a plain power-cycle does NOT fix it. Correct recovery requires a shell to run systemctl mask --now svc FIRST (stops + breaks the reboot trap), then pull the revert. deploy.sh reports "healthy after Ns" as soon as systemd is active, BEFORE the model finishes loading — never treat that as load success; verify with a real completion. Verdict: BF16 35B-A3B cannot run on a 62 GB box; needs an FP8/GGUF quant or a bigger host. RTX 3090 has no FP8 tensor cores so even FP8 runs via upcast but still fits in RAM. _(context: 2026-09-01 frank-default-small swap to Thomson-1.0-Small on Dankserver caused a sustained swap-thrash outage; SSH locked out, no OOM relief due to 127GB swap. Repo revert staged (commit 1163844), awaiting physical/console recovery.)_
@@ -57,7 +57,7 @@
- Rootless-podman GPU access on systemd-user-unit-spawned containers (Dankclaw comfyui, 2026-08-31): containers spawned by systemd user units get NO supplementary groups at spawn time, so group_add: render fails with "unable to find group render" (userns name lookup) AND --group-add keep-groups provably preserves an EMPTY group set — torch then dies with "No HIP GPUs are available" even though /dev/kfd and /dev/dri/renderD* are merely passed through (they show as nobody:nogroup inside = unmapped). The only spawn-context-independent fix is udev MODE 0666 on the compute nodes (kfd, renderD*; leave card* display nodes alone). Two gotchas when applying: (1) udevadm trigger --type=devices does NOT enumerate misc-class devices (they have no subsystem) — use udevadm trigger --action=change --sysname-match=kfd; (2) the rule must be KERNEL=="kfd", MODE="0666" with NO SUBSYSTEM= guard, because a SUBSYSTEM=="misc" match never matches. Verified end to end: torch.cuda.is_available() True, device "Radeon 8060S Graphics", port bound same minute.
- Rootless-podman GPU access on systemd-user-unit-spawned containers (Dankclaw comfyui, 2026-08-31): containers spawned by systemd user units get NO supplementary groups at spawn time, so group_add: render fails with "unable to find group render" (userns name lookup) AND --group-add keep-groups provably preserves an EMPTY group set — torch then dies with "No HIP GPUs are available" even though /dev/kfd and /dev/dri/renderD* are merely passed through (they show as nobody:nogroup inside = unmapped). The only spawn-context-independent fix is udev MODE 0666 on the compute nodes (kfd, renderD*; leave card* display nodes alone). Two gotchas when applying: (1) udevadm trigger --type=devices does NOT enumerate misc-class devices (they have no subsystem) — use udevadm trigger --action=change --sysname-match=kfd; (2) the rule must be KERNEL=="kfd", MODE="0666" with NO SUBSYSTEM= guard, because a SUBSYSTEM=="misc" match never matches. Verified end to end: torch.cuda.is_available() True, device "Radeon 8060S Graphics", port bound same minute.
- FLM (frank-default-sightsound, port 7771) routes chat completions per request by exact FLM registry tag: each request locks the NPU, loads the requested model if not resident, generates, releases. Measured swap costs (Aug 30 2026): first request to an unloaded tag pays full cold load (~7-90 s); switching back re-loads the previous model (~10 s) — alternating tags = every request pays swap. CRITICAL gotcha: a tag listed in /v1/models is NOT necessarily loadable — load fails with [ERROR] Failed to load model: No such file '/home/dank/.config/flm/xclbins/PackName/name.xclbin' when the kernel-pack dir isn't symlinked under FLM_XCLBIN_PATH (/home/dank/.config/flm/xclbins). Fix = ln -s /opt/fastflowlm/share/flm/xclbins/PackName ~/.config/flm/xclbins/PackName (verified for Qwen3.5-4B-NPU2; other unlisted packs fail the same way). Unknown tags answer HTTP 200 with flat {"error":"Failed to load tag model!"} and no choices — status-code-only clients silently get empty completions; clients must check the error key, not just HTTP status.
- FLM (frank-default-sightsound, port 7771) routes chat completions per request by exact FLM registry tag: each request locks the NPU, loads the requested model if not resident, generates, releases. Measured swap costs (Aug 30 2026): first request to an unloaded tag pays full cold load (~7-90 s); switching back re-loads the previous model (~10 s) — alternating tags = every request pays swap. CRITICAL gotcha: a tag listed in /v1/models is NOT necessarily loadable — load fails with [ERROR] Failed to load model: No such file '/home/dank/.config/flm/xclbins/PackName/name.xclbin' when the kernel-pack dir isn't symlinked under FLM_XCLBIN_PATH (/home/dank/.config/flm/xclbins). Fix = ln -s /opt/fastflowlm/share/flm/xclbins/PackName ~/.config/flm/xclbins/PackName (verified for Qwen3.5-4B-NPU2; other unlisted packs fail the same way). Unknown tags answer HTTP 200 with flat {"error":"Failed to load tag model!"} and no choices — status-code-only clients silently get empty completions; clients must check the error key, not just HTTP status.
- FLM (FastFlowLM) 1.0.1 serving on frank-default-sightsound (Dankclaw, port 52626) answers unknown model tags with HTTP **200** and an error body, not a 404 and not a silent fallback: /v1/chat/completions → flat {"error":"Failed to load tag model!"} with no choices; /v1/completions → nested OpenAI-shaped server_error (code 500) also under HTTP 200. Status-code-only clients therefore see a successful empty response — any client wiring against this endpoint must check for an error key in the body, not just status. Valid tags route per-request to the exact named model (response echoes the requested tag; no restart needed to switch registry tags). This disproved the older llama3.2:1b-fallback behavior documented from 2026-08-16; corrected live-probe evidence is committed in AGENTS.md + serverAppConfigs-docs/LLM/frank-default-sightsound.md at commit 9b9f0f9 — verify against those before relying on this. General lesson: behavior claims written into docs from inference/incident notes get falsified by direct probes; probe the live endpoint (bogus input + control pair) before committing contract claims to docs or client configs. _(context: Live probe 2026-08-30 on frank-default-sightsound: bogus tag "totally-bogus:9b" against /v1/chat/completions and /v1/completions; control with valid tag gemma4-it:e4b. Docs + serve-script comment corrected in commit 9b9f0f9, deployed healthy 4 s post-deploy.)_
- FLM (FastFlowLM) 1.0.1 serving on frank-default-sightsound (Dankclaw, port 52626) answers unknown model tags with HTTP **200** and an error body, not a 404 and not a silent fallback: /v1/chat/completions → flat {"error":"Failed to load tag model!"} with no choices; /v1/completions → nested OpenAI-shaped server_error (code 500) also under HTTP 200. Status-code-only clients therefore see a successful empty response — any client wiring against this endpoint must check for an error key in the body, not just status. Valid tags route per-request to the exact named model (response echoes the requested tag; no restart needed to switch registry tags). This disproved the older llama3.2:1b-fallback behavior documented from 2026-08-16; corrected live-probe evidence is committed in AGENTS.md + serverAppConfigs-docs/LLM/frank-default-sightsound.md at commit 9b9f0f9 — verify against those before relying on this. General lesson: behavior claims written into docs from inference/incident notes get falsified by direct probes; probe the live endpoint (bogus input + control pair) before committing contract claims to docs or client configs. _(context: Live probe 2026-08-30 on frank-default-sightsound: bogus tag "totally-bogus:9b" against /v1/chat/completions and /v1/completions; control with valid tag gemma4-it:e4b. Docs + serve-script comment corrected in commit 9b9f0f9, deployed healthy 4 s post-deploy.)_
- frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, Dankclaw port 52626) serves the WHOLE local model registry on one port and routes /v1/chat/completions per-request by the EXACT FLM registry tag in the model field; unknown tags do NOT 404 — they silently serve llama3.2:1b instead. Any client wiring must copy tag bytes from live GET /v1/models at wiring time (never hand-type, never use a service alias). Wiring pattern used for pi client models.json on Dankware (2026-08-30): backup file → programmatic python json load/dump adding provider dankclaw-npu with baseUrl http://192.168.0.143:52626/v1, model id = exact live tag gemma4-it:e4b, contextWindow/maxTokens parsed from FLM_CTX_LEN in scripts/frank-default-sightsound.env (not transcribed), input ["text","image","audio"], reasoning true → re-parse to prove valid JSON. Health probe for this service is GET /v1/version (FLM has no /health route). Full runbook: serverAppConfigs-docs/LLM/frank-default-sightsound.md; fleet table + runtime section in AGENTS.md carry the iommu=off-kills-NPU trap (SVA bind needs enabled IOMMU for /dev/accel/accel0).
- frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, Dankclaw port 52626) serves the WHOLE local model registry on one port and routes /v1/chat/completions per-request by the EXACT FLM registry tag in the model field; unknown tags do NOT 404 — they silently serve llama3.2:1b instead. Any client wiring must copy tag bytes from live GET /v1/models at wiring time (never hand-type, never use a service alias). Wiring pattern used for pi client models.json on the workstation (2026-08-30): backup file → programmatic python json load/dump adding provider dankclaw-npu with baseUrl http://192.168.0.143:52626/v1, model id = exact live tag gemma4-it:e4b, contextWindow/maxTokens parsed from FLM_CTX_LEN in scripts/frank-default-sightsound.env (not transcribed), input ["text","image","audio"], reasoning true → re-parse to prove valid JSON. Health probe for this service is GET /v1/version (FLM has no /health route). Full runbook: serverAppConfigs-docs/LLM/frank-default-sightsound.md; fleet table + runtime section in AGENTS.md carry the iommu=off-kills-NPU trap (SVA bind needs enabled IOMMU for /dev/accel/accel0).
- Dankclaw hermes stack (fixed 2026-08-30, commit 56763e0): the hermes gateway is a Signal messaging agent with NO HTTP listener on :8090 — the public route hermes.franksdank.cloud → Caddy internal listener :8017 → reverse_proxy 192.168.0.143:9119 (dashboard login-gated admin UI). The gateway connects OUTBOUND to the signal-cli daemon, a HOST process at /opt/signal-cli-0.14.7 listening on host :8080, via SIGNAL_HTTP_URL=http://host.containers.internal:8080 in dankclaw/compose.yaml. Durable quirk: on this rootless-podman/netavark host, containers can only reach the host LAN IP 192.168.0.143 on PUBLISHED (DNAT'd) ports — connections to un-forwarded host ports from container space get instant Connection-refused even though a listener exists; host.containers.internal (netavark gateway alias) is the verified working path for unpublished host ports. Also durable: hermes, hermes-dashboard, and hermes-webui are compose-managed with NO quadlet units — they do not survive reboots (this is how the agent went silent after the Aug 26 reboot); a reboot requires manual podman-compose -f dankclaw/compose.yaml up -d --no-deps svc or deploy.sh dispatch. The hermes gateway drops to an unmapped uid inside its userns: /home/dank/hermes-data must be world-writable (chmod -R a+rwX) or the drop-priv user cannot traverse it and crash-loops on log writes (/data/logs/agent.log PermissionError).
- Dankclaw hermes stack (fixed 2026-08-30, commit 56763e0): the hermes gateway is a Signal messaging agent with NO HTTP listener on :8090 — the public route hermes.franksdank.cloud → Caddy internal listener :8017 → reverse_proxy 192.168.0.143:9119 (dashboard login-gated admin UI). The gateway connects OUTBOUND to the signal-cli daemon, a HOST process at /opt/signal-cli-0.14.7 listening on host :8080, via SIGNAL_HTTP_URL=http://host.containers.internal:8080 in dankclaw/compose.yaml. Durable quirk: on this rootless-podman/netavark host, containers can only reach the host LAN IP 192.168.0.143 on PUBLISHED (DNAT'd) ports — connections to un-forwarded host ports from container space get instant Connection-refused even though a listener exists; host.containers.internal (netavark gateway alias) is the verified working path for unpublished host ports. Also durable: hermes, hermes-dashboard, and hermes-webui are compose-managed with NO quadlet units — they do not survive reboots (this is how the agent went silent after the Aug 26 reboot); a reboot requires manual podman-compose -f dankclaw/compose.yaml up -d --no-deps svc or deploy.sh dispatch. The hermes gateway drops to an unmapped uid inside its userns: /home/dank/hermes-data must be world-writable (chmod -R a+rwX) or the drop-priv user cannot traverse it and crash-loops on log writes (/data/logs/agent.log PermissionError).
- frank fleet as of 2026-08-30 night: frank-default-large (Dankclaw) = kingjones777 Qwen3.8-Flash-Next ROCmFP4-STRIX 113.47 GiB on the card's pinned kingjones30/ROCmFPX fork at /home/dank/GitPackages/kingjones-ROCmFPX/build (commit 36e9acd) — builds require -DLLAMA_BUILD_WEBUI=OFF here (npm/HF-bucket asset provisioning aborts the link at 98%). --flash-attn on works UNPATCHED on the STRIX model (the QSA F16 ggml_fill assert needing scripts/patches/qwen4exp-qsa-f32-mask.patch is agentionai-FAST-specific); mmap MUST stay on (PLE streams via page cache; --no-mmap = silent cgroup kill); no spec decode; agentionai FAST on build-qwen38 is the rollback (one env commit). frank-default-small (Dankserver) = batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on the same FreeToken v0.1.2 runtime (native block-fp8, identical loader path + pool triple as official FP8); FreeToken's /v1/chat/completions 422s WITHOUT an explicit "model" field (llama.cpp servers don't). Deploy verification trap: a leftover manually-started llama-server holding :7770 makes both the unit crash-loop on bind AND deploy.sh's /health check pass — verify systemctl show -p MainPID matches the live pgrep, not just health. pi models.json frank-default-large contextWindow corrected to 131072 (llama clamps n_ctx_train/parallel to per-slot; the old 200000 oversold). _(context: Repo /home/dank/Projects/serverAppConfigs; commits 3066d81 (large config), aa9c1d0 (small config), 0937cfb (docs). Both swaps validated live end to end.)_
- frank fleet as of 2026-08-30 night: frank-default-large (Dankclaw) = kingjones777 Qwen3.8-Flash-Next ROCmFP4-STRIX 113.47 GiB on the card's pinned kingjones30/ROCmFPX fork at /home/dank/GitPackages/kingjones-ROCmFPX/build (commit 36e9acd) — builds require -DLLAMA_BUILD_WEBUI=OFF here (npm/HF-bucket asset provisioning aborts the link at 98%). --flash-attn on works UNPATCHED on the STRIX model (the QSA F16 ggml_fill assert needing scripts/patches/qwen4exp-qsa-f32-mask.patch is agentionai-FAST-specific); mmap MUST stay on (PLE streams via page cache; --no-mmap = silent cgroup kill); no spec decode; agentionai FAST on build-qwen38 is the rollback (one env commit). frank-default-small (Dankserver) = batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on the same FreeToken v0.1.2 runtime (native block-fp8, identical loader path + pool triple as official FP8); FreeToken's /v1/chat/completions 422s WITHOUT an explicit "model" field (llama.cpp servers don't). Deploy verification trap: a leftover manually-started llama-server holding :7770 makes both the unit crash-loop on bind AND deploy.sh's /health check pass — verify systemctl show -p MainPID matches the live pgrep, not just health. pi models.json frank-default-large contextWindow corrected to 131072 (llama clamps n_ctx_train/parallel to per-slot; the old 200000 oversold). _(context: Repo /home/dank/Projects/serverAppConfigs; commits 3066d81 (large config), aa9c1d0 (small config), 0937cfb (docs). Both swaps validated live end to end.)_
- When deploying frank-default-large changes to Dankclaw, the service unit is managed by systemd as frank-default-large.service (NOT a user quadlet unit). The quadlet directory exists but systemd-level unit is the authoritative one. The deploy script targets the systemd service; user-level systemctl --user list-units shows nothing for frank services.
- When deploying frank-default-large changes to Dankclaw, the service unit is managed by systemd as frank-default-large.service (NOT a user quadlet unit). The quadlet directory exists but systemd-level unit is the authoritative one. The deploy script targets the systemd service; user-level systemctl --user list-units shows nothing for frank services.
- Repo: /home/dank/Projects/serverAppConfigs (the workstation).
- Live hosts: dankclaw (NPU/LLM), dankserver.
- Live hosts: dankclaw (NPU/LLM), dankserver.
- SSH accessible; deploy.sh handles push and dispatch.
- SSH accessible; deploy.sh handles push and dispatch.
## 01a063a7-4c37-7765-8f58-21f7c860aed7
## 01a063a7-4c37-7765-8f58-21f7c860aed7
updated_at: 1788403307
updated_at: 1788403307
Context/workflow:
Context/workflow:
-Workstation/operation host is Dankware; repo path is /home/dank/Projects/serverAppConfigs.
-The workstation is the operation host; repo path is /home/dank/Projects/serverAppConfigs.
- Main deployment hosts identified/reconfirmed: Dankserver at 192.168.0.235 and Dankclaw at 192.168.0.143. Both were reachable via passwordless SSH from Dankware.
- Main deployment hosts identified/reconfirmed: Dankserver at 192.168.0.235 and Dankclaw at 192.168.0.143. Both were reachable via passwordless SSH from the workstation.
- Services are primarily defined by per-host Quadlet .container files in the repo and installed as user-systemd units (`systemctl --user ...`) generating `<name>.service`. Some model/LLM services may be system units rather than user units.
- Services are primarily defined by per-host Quadlet .container files in the repo and installed as user-systemd units (`systemctl --user ...`) generating `<name>.service`. Some model/LLM services may be system units rather than user units.
Configured passwordless SSH among dankclaw/dankware, FrankBook-Pro, dankserver, and Hermes containers; installed and verified Podman stack on FrankBook-Pro macOS host.
Configured passwordless SSH among dankclaw, the workstation, FrankBook-Pro, dankserver, and Hermes containers; installed and verified Podman stack on FrankBook-Pro macOS host.
serverAppConfigs service-status investigation from Dankware: repo manages services on Dankserver (192.168.0.235) and Dankclaw (192.168.0.143) via mostly rootless Podman Quadlet user units; both hosts are reachable by passwordless SSH. Hermes WebUI troubleshooting on Dankclaw found hermes/hermes-webui/hermes-dashboard active, but signal-cli-daemon inactive/disabled, which is likely why Hermes/WebUI cannot reach Signal transport.
serverAppConfigs service-status investigation from the workstation: repo manages services on Dankserver (192.168.0.235) and Dankclaw (192.168.0.143) via mostly rootless Podman Quadlet user units; both hosts are reachable by passwordless SSH. Hermes WebUI troubleshooting on Dankclaw found hermes/hermes-webui/hermes-dashboard active, but signal-cli-daemon inactive/disabled, which is likely why Hermes/WebUI cannot reach Signal transport.
Use this when checking service state across Dankclaw/Dankserver from Dankware or another controller.
Use this when checking service state across Dankclaw/Dankserver from the workstation or another controller.
1. Start from the repo and trust current repo state over memory:
1. Start from the repo and trust current repo state over memory:
- cd /home/dank/Projects/serverAppConfigs
- cd /home/dank/Projects/serverAppConfigs
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Frank