From 327e78b950d907a1575b13ebc38c903a39655c5a Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 10:56:37 -0700 Subject: [PATCH] =?UTF-8?q?docs(vault):=20destroy=20the=20dead=20weight=20?= =?UTF-8?q?=E2=80=94=2053=20pages=20and=20the=20change=20diary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume. --- .../Agent/Agent Docs and Edits.md | 6 +- serverAppConfigs-docs/Agent/OMP Harness.md | 22 +- serverAppConfigs-docs/Agent/index.md | 2 +- .../Container Network Namespaces.md | 4 +- .../Architecture/Deploy State Triage.md | 10 +- .../Architecture/Deployment Flow.md | 3 +- .../Architecture/E2E Suite.md | 2 +- .../Architecture/Network Architecture.md | 12 +- .../Architecture/Quadlet Lifecycle.md | 10 +- .../Architecture/Testing Policy.md | 1 - .../Dankclaw/ComfyUI Qwen-Image Workflows.md | 12 +- serverAppConfigs-docs/Dankclaw/ComfyUI.md | 6 +- .../Dankclaw/Dankclaw Stack.md | 9 +- .../Dankclaw/Frank TTS Engine.md | 4 +- serverAppConfigs-docs/Dankclaw/Frank TTS.md | 10 +- serverAppConfigs-docs/Dankclaw/Gitea.md | 2 +- .../Dankclaw/Hermes-Dashboard.md | 2 +- .../Dankclaw/Hermes-WebUI.md | 8 +- serverAppConfigs-docs/Dankclaw/Hermes.md | 14 +- serverAppConfigs-docs/Dankclaw/Marreta.md | 20 +- serverAppConfigs-docs/Dankclaw/OpenWebUI.md | 23 +- serverAppConfigs-docs/Dankclaw/Postgres.md | 2 +- .../Dankclaw/SearXNG Providers.md | 12 +- serverAppConfigs-docs/Dankclaw/SearXNG.md | 33 +- .../Dankclaw/Swap Configuration.md | 4 +- serverAppConfigs-docs/Dankclaw/Voice Packs.md | 12 +- serverAppConfigs-docs/Dankserver/Calibre.md | 2 +- serverAppConfigs-docs/Dankserver/Copyparty.md | 6 +- .../Dankserver/Dankserver Stack.md | 2 +- serverAppConfigs-docs/Dankserver/Jellyfin.md | 6 +- serverAppConfigs-docs/Dankserver/Plex.md | 2 +- serverAppConfigs-docs/Dankserver/RomM.md | 4 +- .../Desktop/Noctalia and Niri.md | 22 +- .../FrankBook-Pro/ARR-Stack.md | 4 +- .../FrankBook-Pro/prowlarr.md | 6 +- .../FrankBook-Pro/qbittorrent.md | 4 +- .../Infrastructure/Anubis.md | 6 +- .../Authentik Outpost Reference.md | 18 +- .../Infrastructure/Authentik.md | 50 +- .../Infrastructure/FrankBook-Pro.md | 10 +- .../Infrastructure/Grafana.md | 6 +- .../Infrastructure/Homepage Status Probes.md | 4 +- .../Infrastructure/Homepage.md | 18 +- .../Infrastructure/LAN-DNS.md | 4 +- serverAppConfigs-docs/Infrastructure/Loki.md | 2 +- .../Observability Provisioning.md | 16 +- .../Infrastructure/Observability.md | 14 +- .../LLM/DFlash DSpark Speculative Decoding.md | 22 +- .../LLM/Dankclaw Improvements.md | 16 +- .../LLM/Dankclaw System Specs.md | 10 +- .../LLM/Dankserver System Specs.md | 17 +- .../LLM/Dual-R9700-Cooperation.md | 327 --- .../LLM/FastFlowLM NPU Inference.md | 448 ---- serverAppConfigs-docs/LLM/FreeToken Builds.md | 97 - .../LLM/GGUF Compatibility.md | 14 +- .../LLM/Heterogeneous Split Triage.md | 14 +- .../Hybrid CPU GPU Processing Dankserver.md | 225 -- .../LLM/LLM Bench Windows and Measurement.md | 271 -- .../LLM/LLM Performance Diagnosis.md | 16 +- .../LLM/Llama Cpp Multi-Device Splits.md | 287 -- .../LLM/Llama Cpp Serving Sizing.md | 15 +- .../LLM/Model Swap and Rollback.md | 18 +- .../LLM/PCIe Link and Atomics.md | 42 +- .../Ethernet Bandwidth Math.md | 454 ---- .../Expected Performance.md | 97 - .../Qwen3.8 PP Deployment.md | 313 --- .../Pipeline Parallelism/Sample Script Set.md | 261 -- .../LLM/Pipeline Parallelism/index.md | 71 - .../LLM/Quantization Production.md | 24 +- .../Quantization by Hardware Architecture.md | 5 +- .../LLM/RAM CPU Offloading in VLLM.md | 206 -- .../LLM/RDNA4 Container Serving.md | 258 -- .../RDNA4 MoE Expert Offload VRAM Budget.md | 7 +- .../Running Qwen3.6 27B FP8 on Dankclaw.md | 246 -- ...nning Qwen3.6 35B A3B FP8 on Dankserver.md | 210 -- .../Shared Memory Architecture Dankclaw.md | 13 +- .../LLM/Tcclaviger Fork Ops.md | 202 -- serverAppConfigs-docs/LLM/VLLM Index.md | 104 - serverAppConfigs-docs/LLM/VLLM Overview.md | 287 -- .../LLM/dankclaw-iommu-translation-off.md | 12 +- .../dankserver-frontier-moe-feasibility.md | 227 -- .../LLM/dankserver-vram-growth.md | 144 - .../LLM/dealign-ornith-crack.md | 127 - serverAppConfigs-docs/LLM/deployment.md | 178 -- serverAppConfigs-docs/LLM/frank-archon.md | 65 +- serverAppConfigs-docs/LLM/frank-dragoon.md | 47 +- .../LLM/frank-medium-batch-ubatch.md | 227 -- .../LLM/frank-medium-llamacpp-dflash2.md | 734 ----- .../LLM/frank-medium-model-swap.md | 189 -- .../LLM/frank-medium-radiance-mxfp4.md | 638 ----- .../LLM/frank-medium-rocmfpx.md | 221 -- serverAppConfigs-docs/LLM/frank-medium.md | 65 - serverAppConfigs-docs/LLM/frank-npu.md | 165 -- .../LLM/frank-redo-2026-08-17.md | 227 -- serverAppConfigs-docs/LLM/frank-zealot.md | 40 +- .../LLM/freetoken-benchmark-sweep.md | 94 - .../LLM/ikllama-ornith-dankserver.md | 263 -- serverAppConfigs-docs/LLM/index.md | 47 +- .../LLM/kingjones-q8agent-rocmfpx.md | 243 -- .../LLM/ling-3.0-flash-trial.md | 123 - .../LLM/qwen3.8-spec-matrix.md | 141 - .../LLM/qwen3.8-strix-optimizations.md | 232 -- .../LLM/swift15-campaign-worklog.md | 82 - .../LLM/ubatch-benchmark-large.md | 56 - ...Boot Forensics and Config Interrogation.md | 208 -- .../LLM/vLLM Porting and Compile Triage.md | 330 --- .../LLM/vllm_research_findings.md | 236 -- .../LLM/x4-pair-experiment.md | 409 --- .../Operations/Container Host Reachability.md | 2 +- .../Operations/Fleet Audit and Posture.md | 12 +- .../Operations/Fleet Ops Playbook.md | 9 +- .../Operations/Payload and Disk Reclaim.md | 4 +- .../Operations/Remote Job Runner.md | 18 +- .../Operations/SSH and Remote Access.md | 4 +- serverAppConfigs-docs/Operations/index.md | 2 +- .../AI Job Search and Resume Tools.md | 12 +- .../Research/ComfyUI ROCm and Vulkan.md | 4 +- .../Research/Lemonade and sd.cpp.md | 57 - .../Research/Llama Cpp Code Diff.md | 222 -- .../Research/Llama Cpp Runtimes.md | 141 - .../Research/Local TTS and Voice Cloning.md | 467 ---- .../Most Popular Self-Hosted AI Apps.md | 218 -- serverAppConfigs-docs/Research/SGLang.md | 154 -- serverAppConfigs-docs/Research/index.md | 13 +- serverAppConfigs-docs/index.md | 63 - serverAppConfigs-docs/log.md | 2380 ----------------- 126 files changed, 443 insertions(+), 14157 deletions(-) delete mode 100644 serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md delete mode 100644 serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md delete mode 100644 serverAppConfigs-docs/LLM/FreeToken Builds.md delete mode 100644 serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md delete mode 100644 serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md delete mode 100644 serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md delete mode 100644 serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md delete mode 100644 serverAppConfigs-docs/LLM/RDNA4 Container Serving.md delete mode 100644 serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md delete mode 100644 serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md delete mode 100644 serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md delete mode 100644 serverAppConfigs-docs/LLM/VLLM Index.md delete mode 100644 serverAppConfigs-docs/LLM/VLLM Overview.md delete mode 100644 serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md delete mode 100644 serverAppConfigs-docs/LLM/dankserver-vram-growth.md delete mode 100644 serverAppConfigs-docs/LLM/dealign-ornith-crack.md delete mode 100644 serverAppConfigs-docs/LLM/deployment.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-model-swap.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium.md delete mode 100644 serverAppConfigs-docs/LLM/frank-npu.md delete mode 100644 serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md delete mode 100644 serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md delete mode 100644 serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md delete mode 100644 serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md delete mode 100644 serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md delete mode 100644 serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md delete mode 100644 serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md delete mode 100644 serverAppConfigs-docs/LLM/swift15-campaign-worklog.md delete mode 100644 serverAppConfigs-docs/LLM/ubatch-benchmark-large.md delete mode 100644 serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md delete mode 100644 serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md delete mode 100644 serverAppConfigs-docs/LLM/vllm_research_findings.md delete mode 100644 serverAppConfigs-docs/LLM/x4-pair-experiment.md delete mode 100644 serverAppConfigs-docs/Research/Lemonade and sd.cpp.md delete mode 100644 serverAppConfigs-docs/Research/Llama Cpp Code Diff.md delete mode 100644 serverAppConfigs-docs/Research/Llama Cpp Runtimes.md delete mode 100644 serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md delete mode 100644 serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md delete mode 100644 serverAppConfigs-docs/Research/SGLang.md delete mode 100644 serverAppConfigs-docs/log.md diff --git a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md index 0104539..8f08456 100644 --- a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md +++ b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md @@ -21,7 +21,7 @@ Damage catalogue: no longer frontmatter (title/tags/description become body text). 2. A rewrite **appended** instead of merged → duplicate headings and a stray second `---` mid-file. -3. Literal model artifacts in the body: `<|channel>}...`, +3. Literal model artifacts in the body: `<|channel>}...` `+- `, `- - `. 4. Wrapped bullets re-emitted with a `- ` prefix on every physical line. 5. Line-range replacement that dropped a sentence tail → dangling fragment, @@ -46,7 +46,7 @@ Repair order (matters): 2. If it merely duplicated: **salvage unique content first** — cut the one-of-a-kind section, paste it into the copy you keep, THEN delete the duplicate ranges. Never delete first. -3. Re-add missing sentence tails (insert before the orphaned fragment), +3. Re-add missing sentence tails (insert before the orphaned fragment) keeping list continuation lines indented. 4. Re-run all checks above. @@ -57,7 +57,7 @@ safe, but its snapshot tag changes on every edit — re-read before each subsequent edit. `.AGENTTRASH/` is gitignored (`.gitignore:25`), so NEW trash stays local — but do not read that as "the repo has none of it": on 2026-10-18 `git ls-files | grep -c AGENTTRASH` returns **37** tracked paths -(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`, +(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/` `tests/.AGENTTRASH/`), because gitignore never untracks what was committed before the rule. Trash is a temporary hold, not an archive. diff --git a/serverAppConfigs-docs/Agent/OMP Harness.md b/serverAppConfigs-docs/Agent/OMP Harness.md index 4e44a74..239e01b 100644 --- a/serverAppConfigs-docs/Agent/OMP Harness.md +++ b/serverAppConfigs-docs/Agent/OMP Harness.md @@ -16,7 +16,7 @@ this file is repo-managed; edits are harness-side only. **Model ids are the StarCraft unit names (official 2026-10-08, on the wire since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and `frank-zealot` (Dankclaw) — engine, port and card behind each id live on -[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`, +[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon` `frank-zealot`. The SERVICES keep the old names (units, scripts, env files, runbook pages); pages written in the old era name the service, the harness names the model. The wire now matches: each endpoint's @@ -26,7 +26,7 @@ picker/grants, the homepage docs and the raw API all use one id per model note on the endpoint echoing the requested name). Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and -192.168.0.143:7771 each answer with the unit name (`frank-archon`, +192.168.0.143:7771 each answer with the unit name (`frank-archon` `frank-dragoon`, `frank-zealot`). ## Compaction chain arithmetic (verify before ANY window change) @@ -40,11 +40,11 @@ breaks compaction. Where the contract lives: - `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL: - `http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`), - `methodOrder` (`shake → snapcompact → soft → handoff → remote`, + `http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`) + `methodOrder` (`shake → snapcompact → soft → handoff → remote` `config.yml:77-82`), `thresholdPercent` (**95**, `config.yml:84`), idle settings (`idleEnabled`, `idleThresholdTokens: 100000`) -- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`, +- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens` `compactionModel` references - Implementation: omp plugin `~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js` @@ -104,16 +104,16 @@ input. Boot-shape OOM laddering is a different problem ## Per-model sampler config (`compat.extraBody`) `models.yml` has **no per-model sampler keys**. The global knobs are -`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`, +`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty` `repetitionPenalty`, and a knob reaches the wire only when it is set — the -bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`), +bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`) and this workstation's `config.yml` sets **none** of them. The 2026-10-05 note that the unset default is `-1`/"not sent" is retired: there is no `-1` sentinel, an unset key is simply absent. The per-model hook is `compat.extraBody`: **Why it works:** `applyOpenAIExtraBody` runs **last** in the -openai-completions builder — after `applyChatCompletionsCompatPolicy` — via +openai-completions builder — after `applyChatCompletionsCompatPolicy` — `Object.assign(params, extraBody)`. So extraBody overrides global settings and compat-policy output. @@ -191,7 +191,7 @@ assistant turn carrying a long `reasoning_content`, then compare `usage.prompt_tokens` with the flag true vs false. A large delta means the field changed template rendering; zero delta means it was ignored. -Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`), +Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`) never `rm`. ## Static-preamble audit ("the context feels bloated") @@ -205,7 +205,7 @@ What loads statically every session (measure each, don't guess): ```bash python3 - <<'EOF' import glob, re, os - for root in [os.path.expanduser("~/.omp/agent/managed-skills"), + for root in [os.path.expanduser("~/.omp/agent/managed-skills") os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]: n=tot=0 for sk in glob.glob(root+"/*/SKILL.md"): @@ -305,4 +305,4 @@ standalone; fold go/no-go arithmetic skills into their probe skill - [[Agent Docs and Edits]] — editing vault docs and hashline targets safely - [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides - [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit -- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes +- measuring the endpoints this config routes diff --git a/serverAppConfigs-docs/Agent/index.md b/serverAppConfigs-docs/Agent/index.md index 26c4697..c2424cd 100644 --- a/serverAppConfigs-docs/Agent/index.md +++ b/serverAppConfigs-docs/Agent/index.md @@ -9,4 +9,4 @@ docs get edited safely, and how the agent's home is deployed. ## Related -- [LLM](../LLM/index.md) — the fleet these agents operate +- the fleet these agents operate diff --git a/serverAppConfigs-docs/Architecture/Container Network Namespaces.md b/serverAppConfigs-docs/Architecture/Container Network Namespaces.md index e07af61..e69fa7a 100644 --- a/serverAppConfigs-docs/Architecture/Container Network Namespaces.md +++ b/serverAppConfigs-docs/Architecture/Container Network Namespaces.md @@ -47,7 +47,7 @@ network namespace**: `Network=` values are read from the live Quadlet sources in `/home/dank/.config/containers/systemd/*.container`; runtime network names on the -host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`, +host at the same date were `podman`, `dankclaw_default`, `systemd-gitea` `systemd-searxng`, `systemd-open-webui`. Two consequences that are easy to miss: @@ -83,7 +83,7 @@ podman exec cat /proc/net/route # default route → gateway IP podman exec sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?' ``` -Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`, +Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard` `10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns, published) and `:9119` (its own published port) connect; `:8080` and `:8787` are **refused instantly** even though both listeners are bound to `0.0.0.0` in diff --git a/serverAppConfigs-docs/Architecture/Deploy State Triage.md b/serverAppConfigs-docs/Architecture/Deploy State Triage.md index 6ab442a..3618b6c 100644 --- a/serverAppConfigs-docs/Architecture/Deploy State Triage.md +++ b/serverAppConfigs-docs/Architecture/Deploy State Triage.md @@ -36,7 +36,7 @@ possibly zero behavioral change. 2. Probe the target (ssh as `dank`, passwordless sudo present): - `cat ~/.local/state/serverAppConfigs/deploy-state/` - `git -C ~/serverAppConfigs rev-parse HEAD` - - `systemctl is-active/is-enabled `; + - `systemctl is-active/is-enabled ` `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:/health` - the on-host env file already carries the new keys (it landed via the ff-pull). @@ -119,8 +119,8 @@ each, kill stale ones by explicit PID. A killed deploy leaves behind: - the unit **stopped/failed** (`Restart=always` does NOT fire after an - explicit stop — start it yourself); -- the state marker **pending** (pre-deploy or in-flight revision); + explicit stop — start it yourself) +- the state marker **pending** (pre-deploy or in-flight revision) - the e2e gate un-run (the deployer runs it last). Reconcile the marker ONLY after the service itself is verified healthy — a @@ -142,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts numbers — which looks exactly like your change having no effect. Key on container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the cause from `journalctl --since "N min ago" | grep -a "\["`. -See [[vLLM Boot Forensics and Config Interrogation]] for the full capture +See for the full capture procedure. ## Related - [[Deployment Flow]] — the deploy pipeline itself - [[Quadlet Lifecycle]] — unit ownership and drift diagnosis -- [[vLLM Boot Forensics and Config Interrogation]] — journal capture +- journal capture diff --git a/serverAppConfigs-docs/Architecture/Deployment Flow.md b/serverAppConfigs-docs/Architecture/Deployment Flow.md index 44195be..513e654 100644 --- a/serverAppConfigs-docs/Architecture/Deployment Flow.md +++ b/serverAppConfigs-docs/Architecture/Deployment Flow.md @@ -68,7 +68,7 @@ service on the other host. | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver//quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/` to `/home/dank/Containers/` (mapped-uid roots move without chown). Steady state is restart-only. | The shared chat template maps to the inference service on the selected target. -The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler; +The `frank-npu` endpoint is retired () and has no deploy handler an unrelated large-model deployment never touches it. ## Caddy safety @@ -125,6 +125,5 @@ routing. See [[Testing Policy]]. ## See Also - [[Testing Policy]] -- [LLM deployment](../LLM/deployment.md) - [[Caddy]] - [[SearXNG]] diff --git a/serverAppConfigs-docs/Architecture/E2E Suite.md b/serverAppConfigs-docs/Architecture/E2E Suite.md index a03d3f0..7b47684 100644 --- a/serverAppConfigs-docs/Architecture/E2E Suite.md +++ b/serverAppConfigs-docs/Architecture/E2E Suite.md @@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior. *arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a sleeping Mac mini must never fail an unrelated deploy. The project joins only when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook - trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`, + trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**` `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr vhosts) or when an explicit `--project` argument names it. An `11-admin` edit that touches the arr links must therefore be paired with a `19-frankbook` run diff --git a/serverAppConfigs-docs/Architecture/Network Architecture.md b/serverAppConfigs-docs/Architecture/Network Architecture.md index 13e424d..34eb1f3 100644 --- a/serverAppConfigs-docs/Architecture/Network Architecture.md +++ b/serverAppConfigs-docs/Architecture/Network Architecture.md @@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS, Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone -else), and accepted traffic returns to Caddy's internal listener on `:8016`, +else), and accepted traffic returns to Caddy's internal listener on `:8016` which proxies every request whole to the Authentik server with the Host header preserved; Authentik's embedded outpost routes on that Host against each proxy provider's `external_host`, serves the login flow in place for @@ -67,12 +67,12 @@ graph TB ``` Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`): -homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`), +homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`) prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta (`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.` (the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass -unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP), +unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP) books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own auth"). Grafana is **not** exempt even though it signs in through authentik as @@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through `Network=searxng.network` (runtime network `systemd-searxng`), while `searxng.container`, `marreta.container` and `marreta-browser.container` join the tunnel the modern way — `Network=gluetun.container` -(`dankclaw/searxng/quadlet/searxng.container:18`, -`dankclaw/marreta/quadlet/marreta.container:34`, +(`dankclaw/searxng/quadlet/searxng.container:18` +`dankclaw/marreta/quadlet/marreta.container:34` `dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet equivalent of the retired compose `network_mode: service:gluetun`. **Five** containers share the VPN exit, not three, and Marreta's outbound fetches leave through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of -exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`); +exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`) everything else on Dankclaw uses the normal LAN route. This matches KB Pattern 2 in [[Gluetun]]. See [[Container Network Namespaces]] for which addresses are reachable from that diff --git a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md index eaa5769..c447265 100644 --- a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md +++ b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md @@ -36,14 +36,14 @@ done ## 2. Cutover: compose service → Quadlet unit Checklist (precedents to copy: -`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`, +`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/` `dankclaw/observability/quadlet/`): 1. Unit file at `//quadlet/.container`, mirroring compose line for line: `Image=` (`localhost/` prefix for locally built images + `Pull=never`), `ContainerName=` (Caddy and scrape configs address containers by name), `Network=`, `User=0:0` when the image's own user - cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved), + cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved) `Exec=` for compose `command:`, `PublishPort=`, health keys, `[Service] Restart=always` + `RestartSec=10`, `[Install] WantedBy=default.target`. **Never `systemctl --user enable` the generated @@ -56,7 +56,7 @@ Checklist (precedents to copy: `systemctl --user restart .service` → `wait_for_health`; retire any legacy hand-written unit into `~/.AGENTTRASH/` (a leftover `~/.config/systemd/user/.service` that shells out to - `podman compose up` collides BY NAME with the generated unit); + `podman compose up` collides BY NAME with the generated unit) `podman rm -f ` so the unit's container can take the name. 4. Planner: map `//**` → service in `scripts/deploy-plan.py` AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies @@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever. Quickest unblock: `podman compose up -d --force-recreate `. Durable fix = the §2 cutover. **Verify the capability, not the status:** `podman exec - python3 -c "import torch; print(torch.cuda.is_available())"`; + python3 -c "import torch; print(torch.cuda.is_available())"` `ss -ltn | grep ` for the actual bind addresses. ComfyUI incident: 2026-09-16, stale device-less container, 9,146 restarts. History repeats: 2026-08-26 Caddy (no unit), 2026-09-11 five observability services. @@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate `. Durable fix unhealthy but logs look fine, compare the baked `{{.Config.Healthcheck.Test}}` array against the intended command before believing the status. -- **Probe a new key before shipping:** drop a throwaway `probe.container`, +- **Probe a new key before shipping:** drop a throwaway `probe.container` `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key"`, `systemctl --user cat probe.service` (shows the generated ExecStart verbatim), then remove + reload. diff --git a/serverAppConfigs-docs/Architecture/Testing Policy.md b/serverAppConfigs-docs/Architecture/Testing Policy.md index bf02032..7f132ce 100644 --- a/serverAppConfigs-docs/Architecture/Testing Policy.md +++ b/serverAppConfigs-docs/Architecture/Testing Policy.md @@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con ## See Also - [[Deployment Flow]] -- [[index]] diff --git a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md index 97a8f8b..e36aebe 100644 --- a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md +++ b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md @@ -56,8 +56,8 @@ sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed fixed in the file (`1930` in the art workflow, `2026` in the text one) — read from `dankclaw/comfyui/workflows/*.json` node `9`. -Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`, -`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`, +Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader` +`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode` `SaveImage` (ids 3–11). | Workflow | File | Shape | Steps | Made for | @@ -113,9 +113,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats` reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats` 2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and -all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`, -`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`, -`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`, +all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning` +`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF` +`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets` `huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18). @@ -126,7 +126,7 @@ yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18) | 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) | | 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) | -Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`, +Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/` #3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`. Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count + sha256 checks) or a plain `hf download` with the token in the environment — diff --git a/serverAppConfigs-docs/Dankclaw/ComfyUI.md b/serverAppConfigs-docs/Dankclaw/ComfyUI.md index 5681527..b2dfd25 100644 --- a/serverAppConfigs-docs/Dankclaw/ComfyUI.md +++ b/serverAppConfigs-docs/Dankclaw/ComfyUI.md @@ -43,7 +43,7 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ## GPU device access (rootless), and why the unit exists -The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`, +The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=` `dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses `GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman cannot map the group name into the user namespace. The companion udev rule @@ -115,7 +115,7 @@ generated units. The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/` 2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats` reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The -health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`, +health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py` bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd` (`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs `http://127.0.0.1:8188/` inside the container, and the container reports @@ -135,5 +135,5 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd` - [[Dankclaw Stack]] - [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research - [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions -- [[Lemonade and sd.cpp]] — the ggml-family alternative +- the ggml-family alternative - Knowledge base: [[ComfyUI]] (tools) diff --git a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md index 9123e12..093670f 100644 --- a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md +++ b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md @@ -52,11 +52,10 @@ graph LR The LLM rows are bare-metal systemd services, not containers, and this repo owns their scripts, units and knowledge base (see -[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on +). Which payload and runtime sits on each port is era state and lives on the endpoint page, not here. `frank-npu` (the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon` -on the Dankserver port) are both retired — see [[frank-npu]] and -[[frank-medium-radiance-mxfp4]]. +on the Dankserver port) are both retired — see and ## Boot Persistence @@ -72,7 +71,7 @@ Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open- ## See Also -- [[index]] — repository index +- repository index - [[Network Architecture]] — routing and proxy - [[Observability]] — Grafana, Prometheus, logs -- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host +- the model-serving endpoints on this host diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md index 981ea19..8ff5ad8 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md @@ -37,7 +37,7 @@ timestamp: 2026-10-18 ## Why this engine, and not the alternatives -The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in [[Local TTS and Voice Cloning]], which is the source of these numbers): +The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in , which is the source of these numbers): | Measure | Resident `tts-server` (arm A, chosen) | Mainline per-request CLI path (arm B) | |---|---|---| @@ -128,6 +128,6 @@ still open. | Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) | | `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work | | A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design | -| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] | +| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see | | The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this | | A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces | diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS.md b/serverAppConfigs-docs/Dankclaw/Frank TTS.md index b0344c0..4f722fa 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS.md @@ -18,10 +18,10 @@ audio in that cloned voice. Everything else in the voice stack sits behind it the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers never learn a path. -**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`, +**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts` both units `active`, no drop-ins, and the homepage card's `wip` gate came off in the same change. Probes on the deployed endpoint: `/health` answers -`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`, +`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}` and two different voices rendered the same sentence into **different** bytes (24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a pass-through of one default. The operator harness routes its `speech` role here; @@ -147,7 +147,7 @@ A `/health` probe job would therefore be structurally never-green, and because `LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set lives on [[Prometheus]]), a permanently red `frank-tts` target would page on Signal every few minutes for an endpoint that is fine. The engine cannot be -scraped at all: it publishes nothing. Status lives in `systemctl --user status`, +scraped at all: it publishes nothing. Status lives in `systemctl --user status` `podman healthcheck run`, the two unit journals, and the homepage card, which probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the 2026-10-18 deploy. @@ -155,7 +155,7 @@ probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the ## The harness speech role The operator harness reaches this endpoint through -`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`, +`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)` `baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api `openai-speech` on the **model** row (naming a runner api at *provider* level fails the whole file's validation and disables every custom provider, which is @@ -171,7 +171,7 @@ harness bundle and measured the day of the flip: `speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list, and it never reaches an HTTP request. So a harness call arrives voice-less by default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared - pack answers a voice-less request and the journal marks that line `(default)`, + pack answers a voice-less request and the journal marks that line `(default)` while an **unknown** name still 400s with the list that exists. - **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not select a pack; it hands the vocalizer a value outside its enum. (The first diff --git a/serverAppConfigs-docs/Dankclaw/Gitea.md b/serverAppConfigs-docs/Dankclaw/Gitea.md index dac5aad..fa20d5a 100644 --- a/serverAppConfigs-docs/Dankclaw/Gitea.md +++ b/serverAppConfigs-docs/Dankclaw/Gitea.md @@ -33,7 +33,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net | `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) | | `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* | -The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md) +The LLM fleet knowledge base: ## Management diff --git a/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md b/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md index 2ca8d6b..0cd8b33 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md @@ -30,7 +30,7 @@ entrypoint does not start the dashboard. The dashboard is a separate process, so it runs in its own Quadlet unit ([`dankclaw/hermes/quadlet/hermes-dashboard.container`](../../dankclaw/hermes/quadlet/hermes-dashboard.container): image `localhost/hermes-agent:latest`, `Pull=never`, `Exec=hermes dashboard ---host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`, +--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always` `RestartSec=5`). All three Hermes containers mount the same `/data` bind mount. Unlike its two siblings — `hermes.container` and `hermes-webui.container` both diff --git a/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md b/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md index f790365..3e57e20 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md @@ -40,7 +40,7 @@ from another LAN host: `/health` → 200, `/` → 302) — **Vhost:** ## Why a container, not a host process -The gateway's state files are owned by the container-mapped uid (`109999`, +The gateway's state files are owned by the container-mapped uid (`109999` from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` → `dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 = 109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless @@ -82,12 +82,12 @@ the sibling dashboard on `9119` is not. - Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994, shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only - *(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable + - (unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable without `sudo`)*; the account can do nothing except authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel while the transport stays up for port forwarding. - sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31) - scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`, + scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no` `X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are present under `Match User tunnel` (`cat` 2026-10-18). - Key options on the authorized key pin the blast radius: @@ -130,7 +130,7 @@ only if inactive (it is never restarted on deploy, so in-flight browser sessions survive), re-tags the image alias, then restarts `hermes.service`, `hermes-dashboard.service` and `hermes-webui.service` in the **user** manager of `dank` (linger must be on for -them to come up at boot — `loginctl show-user dank -p Linger` → `yes`, +them to come up at boot — `loginctl show-user dank -p Linger` → `yes` 2026-10-18), so gateway/dashboard/webui restarts stay isolated from inference deploys. Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live diff --git a/serverAppConfigs-docs/Dankclaw/Hermes.md b/serverAppConfigs-docs/Dankclaw/Hermes.md index 461ab30..bebf1d9 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes.md @@ -18,7 +18,7 @@ bindings, 2026-10-18). Note the **container name drift**: the unit is `podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so `podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no `ContainerName=` key even though its header comment says "container name: -hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via +hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached `hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A record; the name resolves from the Pi-hole hosts file, `dankclaw/pihole/addn-hosts:39`) — **Docs:** @@ -37,7 +37,7 @@ designed wiring; the remedy for both halves is in the log line **Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted at `/data` in all three Hermes containers (gateway, webui, dashboard — -`hermes.container:36`, `hermes-webui.container:46`, +`hermes.container:36`, `hermes-webui.container:46` `hermes-dashboard.container:24`). Each container additionally gets an anonymous volume at `/opt/data`, which is re-created empty on every container recreation — anything that lives only there @@ -80,7 +80,7 @@ is not durable. The tracked, sanitized configuration baseline is `frank-dragoon` — verified from the host with `curl http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is a rootless pasta publish - and is IPv4-only); + and is IPv4-only) `dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of 2026-10-18, that endpoint serves only `frank-archon`** (`GET http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name @@ -115,7 +115,7 @@ per session; the browser itself lives in a dedicated daemon container, the digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows `127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image -`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`, +`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…` `dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal gateway and the webui server — run in the host network namespace (`podman inspect` → `net=host` for both, 2026-10-18), so they can @@ -248,7 +248,7 @@ What actually breaks, and the fix: effort: none # mandatory, see below ``` - Before copying this: which id is served on which port is endpoint state — see + Before copying this: which id is served on which port is endpoint state — [[frank-zealot]] and [[frank-archon]]. The provider/model pairing above is the shape that was landed in 2026; re-derive it from the live config before reusing it. @@ -282,7 +282,7 @@ Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth on match, 401 on miss. No username field. `/health` answers 200 unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with `{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w -%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via +%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth those**; probe login with a deliberately wrong password (expect 401) and the real one (expect 200). Details on the webui surface: [[Hermes-WebUI]]. @@ -311,7 +311,7 @@ login with a deliberately wrong password (expect 401) and the real one instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18; off-network ingress is SSH tunnel only — user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth - *(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*, + - (unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)* forward remote end localhost:8787). diff --git a/serverAppConfigs-docs/Dankclaw/Marreta.md b/serverAppConfigs-docs/Dankclaw/Marreta.md index 308a056..0ff243d 100644 --- a/serverAppConfigs-docs/Dankclaw/Marreta.md +++ b/serverAppConfigs-docs/Dankclaw/Marreta.md @@ -19,7 +19,7 @@ returns a stripped, readable version of the page — no paywall chrome, no tracking, no cookie walls. It is a Laravel 13 app on the `serversideup/php:8.4-fpm-nginx` image (`composer.json` requires `php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` — -), +) with an optional headless-browser companion for JavaScript-heavy pages. **Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained) @@ -31,7 +31,7 @@ with an optional headless-browser companion for JavaScript-heavy pages. - Fetches a target URL with a cascade of strategies: plain HTTP (curl), a headless browser (Lightpanda) for JS pages, and the Internet Archive (Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php` - files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`, + files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy` `WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`). - Strips paywall/consent/tracking markup and re-renders clean HTML. - Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` / @@ -69,7 +69,7 @@ graph LR | `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) | Both units are boot-persistent (`WantedBy=default.target`) and self-healing -(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`, +(`Restart=always`, `RestartSec=10` — `marreta.container:62-69` `marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live 2026-10-18 both report `systemctl --user is-enabled` → `generated` and `is-active` → `active`, and `podman inspect` reports marreta **healthy** @@ -87,14 +87,14 @@ The fix is a single env var, verified against the serversideup source: - The image sets `ENV NGINX_HTTP_PORT="8080"` and the site template is `listen ${NGINX_HTTP_PORT} default_server;`. -- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build); +- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build) the entrypoint `10-init-webserver-config.sh` runs `envsubst` over every `NGINX_*` variable **at container start**. - So `Environment=NGINX_HTTP_PORT=8085` in the quadlet relocates the listen port off the collision with no custom build. Marreta then publishes **nothing** of its own — the netns owner (Gluetun) -publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`), +publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`) exactly as it publishes `8081:8080` for SearXNG. Caddy dials `192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085` @@ -127,7 +127,7 @@ on the mechanism: `BrowserFetchStrategy.php` only calls (`dankclaw/Caddyfile:159-163`). 2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`). -3. **Post-auth routing** — authenticated traffic returns to `http://:8017`, +3. **Post-auth routing** — authenticated traffic returns to `http://:8017` where `@marreta host marreta.franksdank.cloud` routes to `reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). @@ -164,7 +164,7 @@ Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0 — that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and this container sets no `TRUSTED_PROXIES`, so the default is what runs (`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL` -starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`, +starts with `https://` (`app/Providers/AppServiceProvider.php:23-24` `URL::forceScheme('https')`). With `APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded scheme and client IP through the whole chain, so there is no redirect loop @@ -234,7 +234,7 @@ netns down and would otherwise orphan it). Deployed via `deploy.sh dankclaw --service gluetun --service marreta --service caddy` (gluetun first to recreate the tunnel with the new `8082:8085` publish, which also cycles SearXNG's netns ref). Authentik `ProxyProvider` `marreta` -registered (pk 13, `external_host=https://marreta.franksdank.cloud/`, +registered (pk 13, `external_host=https://marreta.franksdank.cloud/` `internal_host=http://caddy:8017`). Results: - Unauth edge → Anubis challenge page (not 502). TLS cert issued via DNS-01. @@ -245,14 +245,14 @@ registered (pk 13, `external_host=https://marreta.franksdank.cloud/`, (`GET 127.0.0.1:9222/json/version` → `Lightpanda/1.0`). - SearXNG unaffected: `:8081` → 200 after the tunnel recreate. -> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`, +> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY` > which 500s every app route (see Secrets above). Fixed by re-keying with the > `base64:` prefix before any encrypted data was persisted. ## Citations - Marreta upstream: (README, - `compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`, + `compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php` `lang/` directory listing). - serversideup/php fpm-nginx port mechanism: `serversideup/docker-php` `src/variations/fpm-nginx/Dockerfile` (`NGINX_HTTP_PORT="8080"`) and diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 42e277d..5e6d7e6 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -111,7 +111,7 @@ The Ollama connection is disabled (`ollama.enable=false`). - Working dir for imports: `/app/backend`; package name `open_webui` (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set - BEFORE importing any `open_webui` module (import-time hard requirement); + BEFORE importing any `open_webui` module (import-time hard requirement) the secret lives at `/app/backend/.webui_secret_key`. - **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3` module (`podman exec open-webui python3 -c "import sqlite3; @@ -122,8 +122,8 @@ The Ollama connection is disabled (`ollama.enable=false`). pydantic `ModelModel` instances (attribute access, not `.get()`), with NO `status`/`enabled` attribute — use `is_active` from the DB. - **API auth for verification:** `create_token` lives in - `open_webui.utils.auth` (NOT `open_webui.utils.utils`); - `get_verified_user` requires an `id` claim (not just `sub`); + `open_webui.utils.auth` (NOT `open_webui.utils.utils`) + `get_verified_user` requires an `id` claim (not just `sub`) `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the container: @@ -148,7 +148,7 @@ The Ollama connection is disabled (`ollama.enable=false`). `ClientConnectorError`: a network-reachability problem (host LAN IP vs `host.containers.internal`), not a model problem. -**Durability:** the model table + connections are DB-only (not git-managed); +**Durability:** the model table + connections are DB-only (not git-managed) a fresh data volume loses them — including the public access grant above. For reproducibility, add an idempotent first-start seed script or document the setup in the service doc. @@ -156,7 +156,7 @@ the setup in the service doc. ## API access (auth-gated) Every `/api/*` endpoint requires an authenticated session (SSO login JWT) -or a per-user API key (`auth.enable_api_keys=true`; keys are minted per +or a per-user API key (`auth.enable_api_keys=true`; keys are minted account under Profile → API Keys). Anonymous web visitors get the `pending` role (`ui.default_user_role=pending`), which cannot chat. Verified 2026-10-07 from loopback `:3005` and over the public vhost: @@ -166,13 +166,12 @@ bogus bearer token); only `/api/config`, `/api/version` and `/health` `POST /api/chat/completions` all return **401** with no key (also with a bogus bearer token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of them expose the model list. An external program needs an account + API key — there is -no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`, +no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}` `/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and `POST /api/chat/completions` → 401; `/api/config` additionally publishes `features.enable_login_form=false`, `features.enable_signup=false` and `oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside. - ## Web search Chat web search is served by [[SearXNG]] on this host, wired in the quadlet @@ -186,7 +185,7 @@ Chat web search is served by [[SearXNG]] on this host, wired in the quadlet The Anubis→Authentik guard chain is a browser mechanism (proof-of-work + session flow); a plain HTTP client cannot solve it, and the internal port bypasses Caddy entirely. -- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`, +- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX` a hardcoded Python constant — no config knob). The `pass_ip` list is checked FIRST and short-circuits the whole botdetection chain, and OpenWebUI's bridge egress (`10.89.x`) is in `pass_ip`, so the cap never applies. Verified: an @@ -249,7 +248,6 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`). restore the password form: set `ui.enable_login_form` back to `true` in the Admin Panel (or DB) and drop the two env vars from the quadlet. - ## Boot persistence (Quadlet) The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state). @@ -296,7 +294,7 @@ TTS is the fleet's one OpenAI-compatible speech endpoint — `http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url` in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That port is the rootless quadlet `frank-tts` (this repo: -`dankclaw/voiceservice/quadlet/frank-tts.container`, +`dankclaw/voiceservice/quadlet/frank-tts.container` `scripts/frank-tts.env`), and its shim resolves `voice=` against `/home/dank/voices//reference.wav` — a name without a reference is a 400 listing the voices that exist, never a silent fallback. @@ -310,7 +308,7 @@ setting). Its engine select takes `""` (Default) or `browser-kokoro`; when `browser-kokoro` is chosen the page synthesizes locally and rejects any voice outside its fixed pack — so `narrator1` fails with `Voice: "narrator1" not found. Should be one of: af_heart, …` **without a -single request reaching the server**. Personal audio settings are per +single request reaching the server**. Personal audio settings are browser: no `user.settings` row and no container env carries an audio override, yet the toast still appears ⇒ the choice lives in that browser's own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**, @@ -325,7 +323,6 @@ the failing click — that contrast is the instrument: a server-path failure leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves neither. - ## Retired instances - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind. @@ -355,7 +352,7 @@ neither. began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three id-keyed objects had to move in the same window as the alias flip: `model.id`+`name` (`frank-zealot` → `frank-zealot`, `params` - `{"context":{"length":200704}}` and the `meta.description` kept), + `{"context":{"length":200704}}` and the `meta.description` kept) `access_grant` (`resource_type='model'`, `resource_id`, principal `user`/`*`, permission `read`), and `openai.api_configs` per-backend `model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1 diff --git a/serverAppConfigs-docs/Dankclaw/Postgres.md b/serverAppConfigs-docs/Dankclaw/Postgres.md index e6b2a02..9a468e4 100644 --- a/serverAppConfigs-docs/Dankclaw/Postgres.md +++ b/serverAppConfigs-docs/Dankclaw/Postgres.md @@ -88,7 +88,7 @@ never enter this repo. See [[Gitea]] and [[Authentik]]. ## Boot persistence (Quadlet) -Both units carry `Restart=always` (`gitea-postgres.container:28`, +Both units carry `Restart=always` (`gitea-postgres.container:28` `authentik-postgres.container:38`) and `[Install] WantedBy=default.target` (`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed under `~/.config/containers/systemd/`, and report `generated` / `active` with diff --git a/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md b/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md index 8fe2e29..d439aa1 100644 --- a/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md +++ b/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md @@ -143,7 +143,7 @@ engines: disabled: false ``` -Key per-engine keys (each overwrites the engine-module default; see +Key per-engine keys (each overwrites the engine-module default `settings_engines.html`): | Key | Effect | @@ -164,7 +164,7 @@ Key per-engine keys (each overwrites the engine-module default; see *different* category. When the primary `google` gets CAPTCHA'd it suspends, but `google2` still returns results, and vice-versa. -**Parser caveat (read before fiddling with `parsers`):** the modern `google`, +**Parser caveat (read before fiddling with `parsers`):** the modern `google` `duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers: [scrapy_html]` under `name: google` mostly does nothing. To make Google @@ -376,14 +376,14 @@ done **2. `/config` REST endpoint** lists every engine + enabled/disabled + categories — an inventory of what is live. It answers 200 on this instance today -(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`, +(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled` `categories`, `timeout`, `shortcut`, and it is the source of the engine counts on [[SearXNG]]). `?engines=` isolates a single provider; `/debug` adds engine diagnostics. **3. `check.searx.space`** — the hosted official checker probes every engine of your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a -plain HTTP probe misses) and posts results on searx.space. Add your instance via +plain HTTP probe misses) and posts results on searx.space. Add your instance a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs to reach the SearXNG port. @@ -417,10 +417,10 @@ misleading). already replaced — checked 2026-10-18; see [[SearXNG]]). It is also immune to the shared-VPN-exit-IP blocking this page is about: it authenticates, so reputation of the exit IP is irrelevant. -5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see +5. **Verify first** — `/json` + `?engines=google` and the searx-checker to exactly which engines are down and why. Confirm the running SearXNG version (check whether `parsers`/`rate_limit` exist in the pinned build). -6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per +6. **Use conservative request tuning** — Dankclaw now allows 5 seconds upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds (`:27`), and retries once (`:32`). This addresses modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only diff --git a/serverAppConfigs-docs/Dankclaw/SearXNG.md b/serverAppConfigs-docs/Dankclaw/SearXNG.md index b03662a..cb26302 100644 --- a/serverAppConfigs-docs/Dankclaw/SearXNG.md +++ b/serverAppConfigs-docs/Dankclaw/SearXNG.md @@ -108,7 +108,7 @@ engines: disabled: true ``` -The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas); +The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas) the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked `secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is censused below. @@ -154,7 +154,7 @@ what keeps the `trusted_proxies` entry meaningful. ## Unblocking Google & Search Providers -The current request policy allows a modest amount of VPN latency (`5.0s` per +The current request policy allows a modest amount of VPN latency (`5.0s` request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting retries to one (`:32`). This can recover slow providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared @@ -169,10 +169,10 @@ upstream request volume. SearXNG **suspends** offending engines — which is wha up as "Google went unavailable". The default suspension is shorter than the folklore: see the table in [[SearXNG Providers]]. This is upstream, not local. -Fixes (cheapest first): tune the `outgoing` block (`request_timeout`, +Fixes (cheapest first): tune the `outgoing` block (`request_timeout` `retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a fresh `name` so a CAPTCHA ban on one leaves the other returning results, and — -if Google still re-blocks — route the Google engines at a clean-IP proxy via +if Google still re-blocks — route the Google engines at a clean-IP proxy per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full root cause, the suspend-model table, the `engines:` config, and the `/json` verification recipe. @@ -212,17 +212,16 @@ root cause, the suspend-model table, the `engines:` config, and the (`searxng.container:21`), and that is one of the variables SearXNG substitutes at load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a config change, out of scope for docs. -- Keep the valkey cache container on the network SearXNG can actually reach — see +- Keep the valkey cache container on the network SearXNG can actually reach — [[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's netns), so the network that matters is the one Gluetun and `searxng-valkey` share. - ## Kagi (paid API engine) Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native -`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`), +`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`) which sets `Authorization: Bearer ` (`kagi.py:103`) and issues -`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`), +`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`) and declares `require_api_key: True` (`kagi.py:63`). - **Enabled**: `weight: 2` alongside bing/duckduckgo, categories @@ -231,10 +230,10 @@ and declares `require_api_key: True` (`kagi.py:63`). `enabled: true`, categories `general, web` (2026-10-18). - **Cost control**: `paging: false` — exactly one API credit per aggregated query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0) - (`shared/searxng/settings.yml:74-77`; upstream paging default is `True`, + (`shared/searxng/settings.yml:74-77`; upstream paging default is `True` `kagi.py:67`, so the `false` here is a real override). -- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`; - `deploy-remote.sh` bakes the real key into the live settings.yml via +- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"` + `deploy-remote.sh` bakes the real key into the live settings.yml `scripts/searxng-render-settings.py`, reading `/opt/searxng/.env` (owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`). @@ -262,15 +261,15 @@ and declares `require_api_key: True` (`kagi.py:63`). - 262 engines in the build, **75 enabled**, categories of the enabled set led by `other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7). -- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`, - `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`, - `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`, - `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`, +- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi` + `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi` + `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news` + `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*` `youtube`, `vimeo`, `dailymotion`, `bing videos`. -- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`, +- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar` `brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`. - The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target - engines that **no longer exist in this build** — `/config` lists no `startpage`, + engines that **no longer exist in this build** — `/config` lists no `startpage` `startpage images` or `startpage news`, so those overrides are no-ops kept as intent. *(labelled 2026-10-18)* diff --git a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md index eab3eae..ba11142 100644 --- a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md +++ b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md @@ -144,8 +144,8 @@ frank-dragoon `health=200` throughout. The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at `dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only -`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`, -`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`, +`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning` +`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch` `20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`. The old swapfile has therefore been purged; nothing on the host still holds it. diff --git a/serverAppConfigs-docs/Dankclaw/Voice Packs.md b/serverAppConfigs-docs/Dankclaw/Voice Packs.md index aebeafa..7730a7d 100644 --- a/serverAppConfigs-docs/Dankclaw/Voice Packs.md +++ b/serverAppConfigs-docs/Dankclaw/Voice Packs.md @@ -23,7 +23,7 @@ voice is how a wrong-voice endpoint stays wrong for weeks. state that stays out of the repo, mounted read-only at `/voices` by the façade unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in `scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:** -arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]]. +arm A `qwentts.cpp`, decided in . Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade container runs as uid 10000 and reads the mount from outside its own uid map. @@ -137,7 +137,7 @@ invariant to every crop in the tracked pack list. 1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop offset is a claim about one exact file, so the digest goes in the env with it. 2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once: - `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`, + `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -` build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the `.rvq` falls under the floor. @@ -188,15 +188,15 @@ loud, not a bug. Work through it in order: 1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec` prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , … codebooks @ … Hz` on every run, and the pinned tree's - `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`; + `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk` `codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`. 2. Decide whether the new number is intended. A changed `enc_dim` means the old `.spk` files are unreadable by the new engine, so **every** pack must be rebuilt, not just the failing one. 3. Update the four arithmetic lines in `scripts/voice-packs.env` - (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`, + (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES` and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and** - `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`, + `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL` `VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes without moving the commit is the mistake to avoid. 4. Rebuild all three packs with `--force` (the retired directories keep the old @@ -226,7 +226,7 @@ loud, not a bug. Work through it in order: narrator1 and narrator2). The builder's first real run is therefore a migration, and it should be a `--force` run per voice so the old directories are retired rather than left half-updated. -- The three source files (`/home/blank/audiosample.mp3`, +- The three source files (`/home/blank/audiosample.mp3` `audiosample2.m4b`, `audiosample3.m4b`, digests pinned in `scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so diff --git a/serverAppConfigs-docs/Dankserver/Calibre.md b/serverAppConfigs-docs/Dankserver/Calibre.md index fed101c..84e2a15 100644 --- a/serverAppConfigs-docs/Dankserver/Calibre.md +++ b/serverAppConfigs-docs/Dankserver/Calibre.md @@ -80,7 +80,7 @@ image's own nginx (the desktop), so the content server runs on internal 8090. **Auth model (calibre 9.x):** users live in a SQLite userdb whose `pw` column is **plaintext** — the file is mode 600. The single admin account is bootstrapped once by the deploy handler -(`deploy_calibre_server` in `scripts/deploy-remote.sh`) via +(`deploy_calibre_server` in `scripts/deploy-remote.sh`) `calibre-server --manage-users -- add`, the password piped on stdin with `printf` (a trailing newline from `echo` lands in the stored password and breaks every login). The desktop's basic auth (`CUSTOM_USER`/`CUSTOM_PASSWORD`) diff --git a/serverAppConfigs-docs/Dankserver/Copyparty.md b/serverAppConfigs-docs/Dankserver/Copyparty.md index 87fa51c..f828c77 100644 --- a/serverAppConfigs-docs/Dankserver/Copyparty.md +++ b/serverAppConfigs-docs/Dankserver/Copyparty.md @@ -83,7 +83,7 @@ The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum co ## Volumes -Live deployment: the whole LUKS array is served at URL `/w` via +Live deployment: the whole LUKS array is served at URL `/w` `Volume=/mnt/entropy:/w:ro` in the quadlet unit ([[Entropy Volume]]) — read-only **at the mount level**, so no containerized consumer can write into the array through this path regardless of account permissions. The ac edition @@ -111,7 +111,7 @@ convention; see [[#Command-line interface (ac edition)]] above. ``` `${COPAY_ACCOUNT}`/`${COPAY_PASSWORD}` are **never** written into the unit: -they live in the host-local `/home/dank/Containers/copyparty/env` (0600), +they live in the host-local `/home/dank/Containers/copyparty/env` (0600) loaded via `[Service] EnvironmentFile=` so systemd expands them into `Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD}` at start. The password still lands in the container argv — visible to anyone who can run @@ -178,6 +178,6 @@ Upgrade to v1.20.19+ to cover all known advisories. The dirkey fix (v1.20.17) ma ## Citations - GitHub Security Advisories, `9001/copyparty` — - https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19); + https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19) each row's patched version read from `first_patched_version` in the `https://api.github.com/advisories/` response. diff --git a/serverAppConfigs-docs/Dankserver/Dankserver Stack.md b/serverAppConfigs-docs/Dankserver/Dankserver Stack.md index 31f3972..370c2d5 100644 --- a/serverAppConfigs-docs/Dankserver/Dankserver Stack.md +++ b/serverAppConfigs-docs/Dankserver/Dankserver Stack.md @@ -54,5 +54,5 @@ The following services ran on Dankserver and are now retired from this stack: ## See Also -- [[index]] — repository index +- repository index - [[Network Architecture]] — routing and proxy diff --git a/serverAppConfigs-docs/Dankserver/Jellyfin.md b/serverAppConfigs-docs/Dankserver/Jellyfin.md index 1af373a..be05ddc 100644 --- a/serverAppConfigs-docs/Dankserver/Jellyfin.md +++ b/serverAppConfigs-docs/Dankserver/Jellyfin.md @@ -14,7 +14,7 @@ serving music, movies, and shows over LAN. It replaced the earlier research brief; it runs alongside [[Plex]], which remains the primary server for some clients. -**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`, +**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096` public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain → [[Authentik]] outpost (post-auth route proxies `localhost:8096`). @@ -67,7 +67,7 @@ user to verify the password. - LDAP provider `Jellyfin LDAP` (Base DN `dc=franksdank,dc=cloud`, bind/search mode `direct`) + application `Jellyfin`. - LDAP outpost `Jellyfin LDAP Outpost` → container `authentik-outpost-ldap` - (quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`), + (quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`) publishing `389`/`636` on the Dankclaw host for cross-host binds. - Service account `svc-jellyfin-ldap` (DN `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud`). @@ -80,7 +80,7 @@ user to verify the password. **Jellyfin side** (plugin config, set via admin → Plugins → LDAP-Auth): - Server `192.168.0.143:389` (plain LDAP over the trusted LAN), Base DN `dc=franksdank,dc=cloud`, bind user = the service-account DN, search - attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`, + attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn` `CreateUsersFromLdap=true`, all folders enabled. **Gotcha — the bind credential is a PASSWORD, not a token.** The LDAP bind is diff --git a/serverAppConfigs-docs/Dankserver/Plex.md b/serverAppConfigs-docs/Dankserver/Plex.md index 99cbac4..682dde1 100644 --- a/serverAppConfigs-docs/Dankserver/Plex.md +++ b/serverAppConfigs-docs/Dankserver/Plex.md @@ -60,7 +60,7 @@ root runs as the `dank` user, and the deployer adopts the data root with ## Integration points - **Caddy:** Plex is exempt from the Anubis bot challenge - (plain-HTTP legacy clients cannot solve JS challenges) — see + (plain-HTTP legacy clients cannot solve JS challenges) — [[Anubis]] "Exempt by construction". - **Tautulli** ([[Tautulli]]) is the companion: it polls PMS on the appnet for statistics and notifications and makes no sense without Plex. diff --git a/serverAppConfigs-docs/Dankserver/RomM.md b/serverAppConfigs-docs/Dankserver/RomM.md index dec123e..9999074 100644 --- a/serverAppConfigs-docs/Dankserver/RomM.md +++ b/serverAppConfigs-docs/Dankserver/RomM.md @@ -32,7 +32,7 @@ Anubis → Authentik chain ([[Caddy]] routes it post-auth to ## Architecture Current RomM releases require MariaDB plus Redis (or Valkey) for the task -queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit), +queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit) not valkey. ```mermaid @@ -114,7 +114,7 @@ scan: ## Deployment (Quadlet) Since 2026-09-01 the trio runs as repo-tracked quadlet units, deployed with -`./scripts/deploy.sh dankserver --service romm` (any of `romm`, +`./scripts/deploy.sh dankserver --service romm` (any of `romm` `romm-mariadb`, `romm-redis` triggers the full ordered cutover): - Units: `dankserver/romm/quadlet/{romm,romm-mariadb,romm-redis}.container` diff --git a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md index 097c5c3..62e6538 100644 --- a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md +++ b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md @@ -45,14 +45,14 @@ Verified: `msg status` returns live JSON state; unknown panel ids return an error listing all valid panel ids (clipboard, control-center, launcher, polkit, session, setup-wizard, test, tray-drawer, wallpaper). -Verified v5 msg verb grammar (keybinds-safe): `panel-toggle `, -`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, -`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, +Verified v5 msg verb grammar (keybinds-safe): `panel-toggle ` +`session lock|suspend|lock-and-suspend|logout|reboot|shutdown` +`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute` `media next|previous|playPause` (graceful no-op, rc 0, with no active -player), `brightness-up [target] [step]`, `brightness-down`, -`settings-open [context]`, `settings-toggle [context]`, `config-reload`, -`wallpaper-get [connector]`, `wallpaper-set `, `wallpaper-next`, -`color-scheme-get|set`, `theme-mode-get`, `templates-apply`, +player), `brightness-up [target] [step]`, `brightness-down` +`settings-open [context]`, `settings-toggle [context]`, `config-reload` +`wallpaper-get [connector]`, `wallpaper-set `, `wallpaper-next` +`color-scheme-get|set`, `theme-mode-get`, `templates-apply` `screenshot-fullscreen`. **Wallpaper directory semantics:** empty/unset `[wallpaper] directory` @@ -81,9 +81,9 @@ Custom palettes live at `~/.config/noctalia/palettes/.json` synced. Missing/invalid file → silent fallback to the builtin palette. Palette JSON shape = the community file's shape: `dark`/`light` maps of -`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`, +`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface` `mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block -(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`, +(`foreground`, `background`, `normal.{black..white}`, `bright.{…}` `cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme -o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the *generator* format, not this one. Copy an existing community JSON and mutate @@ -114,7 +114,7 @@ Procedure: `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with `color-scheme-get`. 5. Re-render every configured template: `noctalia msg templates-apply`. - Regenerates `~/.config/ghostty/themes/noctalia`, + Regenerates `~/.config/ghostty/themes/noctalia` `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm, starship, etc. Symlinked-into-dotfiles paths show up as modified files worth committing. @@ -216,7 +216,7 @@ Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel). scale, transform, logical pos/size) and `niri --version`. Output **names are connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings — the EDID strings printed before them are decoration. On blankpad the live -config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`, +config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl` and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the repo path (that is what gets committed); check with `ls -ld`. diff --git a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md index 4ffaf42..be71434 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md +++ b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md @@ -38,7 +38,7 @@ live call at a time instead of guessing payload shapes: - `'Base Url' must not be empty` when keys were camelCase → **PascalCase is required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys. - `DefaultMetadataProfileId must be > 0` → fetch real ids from - `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`); + `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`) `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠ profiles. - `targetType null` → configContract not registered in that build (this @@ -87,7 +87,7 @@ settings values live in `prowlarr.db` → `Applications.Settings` JSON (the pipeline hides a miss and you conclude "empty field" when the grep itself failed. 4. Decision framing: when posture = gated-with-zero-users, present it as a - decision menu (None behind the front-door wall vs External/authentikarr), + decision menu (None behind the front-door wall vs External/authentikarr) not a silent change — first-account-wins means whoever logs in first owns the instance. diff --git a/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md b/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md index b204f8a..d58bf40 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md +++ b/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md @@ -70,7 +70,7 @@ http://prowlarr:9696/{indexerId}/download?link=...&file=... (torrent handoff) The global API key is in `config.xml` (``); per-application keys are only used by Prowlarr's own sync calls, NOT by the feeds. The arr download clients authenticate to qBittorrent with the permanent WebUI password stored -in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — see +in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — [[qBittorrent]] for why that password must be pinned in the conf file. **Readarr caveat:** Readarr is abandoned upstream; its qBittorrent client @@ -97,13 +97,13 @@ successor; the indexer row itself is healthy. - Categories in the indexer row must be a subset of what the feed's `caps` advertises; extras log "does not support the following categories" and some feeds reject them. -- **A freshly created indexer row defaults to all-off.** If `enableRss`, +- **A freshly created indexer row defaults to all-off.** If `enableRss` `enableAutomaticSearch`, or `enableInteractiveSearch` aren't set, the app logs `Searching indexers for [X]. 0 active indexers` (Lidarr, verified). Search proves TV + music chains 2026-08-30: Sonarr returned 76 live releases for an anime term, Lidarr 18 for YOASOBI – THE BOOK, all via the Prowlarr-Nyaa feed. -- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`); +- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`) GET responses carry `fields: []` — real settings values live in `prowlarr.db` → `Applications.Settings` JSON (read-only inspection). - Instance name/`InstanceName=Prowlarr` in `config.xml` does NOT prefix the diff --git a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md index 65a8446..221637b 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md +++ b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md @@ -13,7 +13,7 @@ qBittorrent is the torrent download client for the *arr stack. The stack runs on not on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr, and Readarr via category tagging. -**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`, +**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080` published by the [[Gluetun]] unit because qBittorrent shares that container's network namespace (a netns-sharer publishes no ports of its own). @@ -121,7 +121,7 @@ Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for `podman machine ssh`. **Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112 -(NOT core — `core` is the podman-machine VM user, reached only via +(NOT core — `core` is the podman-machine VM user, reached only `podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped 2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a low MaxAuthTries: when password-bootstrapping use `-o diff --git a/serverAppConfigs-docs/Infrastructure/Anubis.md b/serverAppConfigs-docs/Infrastructure/Anubis.md index cab9647..71d01c1 100644 --- a/serverAppConfigs-docs/Infrastructure/Anubis.md +++ b/serverAppConfigs-docs/Infrastructure/Anubis.md @@ -93,7 +93,7 @@ and `hermesadmin.` (app-enforced passwords, no public DNS for the first, (`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts (`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so it is exempt by absence — and therefore unauthenticated on the LAN at -`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`, +`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.` `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.` (`dankclaw/Caddyfile:283-423`). @@ -140,7 +140,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at instead of allowed, the browser's fetch throws and it renders its own error page (iOS/macOS browsers phrase it as "the request failed and the interceptors did not return an alternative response") instead of the - login flow. Verify with a JS-capable client end-to-end — mint via + login flow. Verify with a JS-capable client end-to-end — mint `make-challenge`, solve, submit with browser-exact query params — before declaring a challenge chain healthy. 7. **Single-file bind mount + git pull = stale policy.** The policy is @@ -158,7 +158,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at answers any request lacking `X-Real-Ip` with a 500 whose log line reads `[misconfiguration] X-Real-Ip header is not set`. The real chain always sets it (every guarded Caddy listener carries `header_up X-Real-Ip - {remote_host}` — the bats seam enforces the line), so users never see + {remote_host}` — the bats seam enforces the line), so users never this. But any direct probe of `:8923` (scratch container, curl, a new reverse-proxy hop) MUST set the header or it looks like a policy failure. Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`. diff --git a/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md b/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md index 6139884..a3e1f4e 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md @@ -23,8 +23,8 @@ read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost `7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9 members in `authentik_outposts_outpost_providers`), all with `internal_host = http://caddy:8017`, all carrying the **same three flow -UUIDs** (`default-authentication-flow`, -`default-provider-authorization-implicit-consent`, +UUIDs** (`default-authentication-flow` +`default-provider-authorization-implicit-consent` `default-provider-invalidation-flow`), and each backed by an Application (the `slug` column below): @@ -67,7 +67,7 @@ Two provisioning rules, each verified live by an outage: reload** — including hosts whose providers were fine (verified 2026-09-02 when three providers provisioned without any flows took down every guarded vhost after a server restart). Backfill all three flows (copy from pk 1: - `default-authentication-flow`, + `default-authentication-flow` `default-provider-authorization-implicit-consent` (the implicit-consent authorization flow, `default-invalidation-flow`) and the retry loop recovers on its next 3-second cycle. The silent killer: while the outpost @@ -94,7 +94,7 @@ implementation diverges from upstream in ways that bit us; all verified live: contains `"authorization_code"` (enum value with underscore), the token endpoint rejects every exchange with `invalid_request`. - **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes - `parse_token_request` see no `grant_type` → `unsupported_grant_type`; + `parse_token_request` see no `grant_type` → `unsupported_grant_type` POST-body client secrets are not accepted — authenticate the confidential client with `Authorization: Basic base64(client_id:client_secret)`. - **Provider resolution is by `client_id` query param** on the global @@ -107,7 +107,7 @@ implementation diverges from upstream in ways that bit us; all verified live: becomes the *expression source* instead of the context; every call raises a swallowed `PropertyMappingExpressionException` and the claim is skipped. Scope-grant filtering has the same class of bug: `ScopeMapping` has no - provider FK (linkage lives in the `ProviderPropertyMapping` through model), + provider FK (linkage lives in the `ProviderPropertyMapping` through model) so `filter(provider=...)` always yields an empty queryset and tokens asked to carry scopes get zero. Net effect: userinfo/introspection answers with `{sub}` only — email/groups/entitlements claims cannot be delivered by this @@ -156,7 +156,7 @@ Hard-won rules (the first is now contested by the live database): and retest. - `ak shell` import path for the provider model in 2026.8: `from authentik.providers.ldap.models import LDAPProvider`. - *(unverified 2026-10-18: importing it outside a configured Django shell fails + - (unverified 2026-10-18: importing it outside a configured Django shell fails with `ImproperlyConfigured`, and no `ak shell` was driven against the live auth service for this check)* @@ -188,13 +188,13 @@ is a `password` field on the `default-user-settings` prompt Two ORM gotchas: `Flow.stages.add()` fails on this through model (`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is -MTI), so create the `FlowStageBinding` row directly with an explicit `order`; +MTI), so create the `FlowStageBinding` row directly with an explicit `order` and a prompt field is a `Prompt` row linked through `PromptStageField` (`prompt_stage` + `prompt`; the order comes from `Prompt.order`). **A blank password must not set an empty password.** `set_password("")` would make the account log in with no password, so a validation policy -(`default-user-settings-no-empty-password`, bound to the prompt via +(`default-user-settings-no-empty-password`, bound to the prompt `PromptStageValidationPolicy`) denies the form when `prompt_data["password"]` is `""` and allows it when set. Verified with the `PolicyEvaluator` (empty → deny, set → allow, absent → allow) and end-to-end by driving @@ -213,7 +213,7 @@ three `*_flow_id` columns; the proxy table is `authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2 table in 2026.8), the LDAP table is -`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`; +`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id` `authentik_core_application` attaches to a provider through `provider_id` (its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost` is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError` diff --git a/serverAppConfigs-docs/Infrastructure/Authentik.md b/serverAppConfigs-docs/Infrastructure/Authentik.md index 2d1578f..d43def8 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik.md @@ -15,20 +15,20 @@ application-level authentication with its proxy outpost. Not wrapped: Gitea, OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) — they handle their own session. Grafana is the exception that is both: it is a guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2 -relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`, +relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132` `dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]). -- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`; - the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`, +- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container` + the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"` 2026-10-18) - **Host:** dankclaw (`192.168.0.143`) - **Name:** `auth.franksdank.cloud` -- **Stack: five quadlet units, no compose.** `authentik-postgres`, - `authentik-server`, `authentik-worker` (`deploy_authentik`, +- **Stack: five quadlet units, no compose.** `authentik-postgres` + `authentik-server`, `authentik-worker` (`deploy_authentik` `scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost` - (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`, + (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*` `deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and - `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`, + `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap` `scripts/deploy-remote.sh:1020-1032`). All five report `generated` + `active`, `Restart=always`, `WantedBy=default.target`, with linger on (`systemctl --user`, 2026-10-18). The "3 compose services" wording is the @@ -36,7 +36,7 @@ relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`, - **LDAP outpost:** serves LDAP for Jellyfin — see the section below. **Split off from this page:** [[Authentik Processes]] owns the per-container -facts for `authentik-postgres`, `authentik-server` and `authentik-worker`; +facts for `authentik-postgres`, `authentik-server` and `authentik-worker` [[Authentik Outpost Reference]] owns the provider/flow/token lookup material — the live proxy-provider inventory, the proxy-provider provisioning rules, the OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow @@ -136,7 +136,7 @@ routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients. -The live proxy-provider inventory (the per-vhost `external_host` → app mapping), +The live proxy-provider inventory (the per-vhost `external_host` → app mapping) the two provisioning rules the outpost enforces, and this fork's OAuth2 relying-party quirks are reference lookup: [[Authentik Outpost Reference]]. @@ -144,7 +144,7 @@ relying-party quirks are reference lookup: [[Authentik Outpost Reference]]. The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container` → `authentik-outpost-ldap.service`, one of the five units in this stack — -serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`, +serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active` `Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64 single-platform digest** `ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9` @@ -159,7 +159,7 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect .digest'` and pin that digest. - Ports **389** (ldap) and **636** (ldaps) publish on the LAN - (`PublishPort=389:3389`, `636:6636`, + (`PublishPort=389:3389`, `636:6636` `dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp` shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds @@ -196,7 +196,7 @@ every outpost session (see the ORM playbook). core quadlets, starts postgres health-gated (`wait_container_healthy authentik-postgres 120`), then the server (300 s health wait) and the worker. The outposts are separate handlers: `--service authentik-outpost` and - `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`, + `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032` dispatch at `:1579-1588`). 2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema. @@ -210,7 +210,7 @@ every outpost session (see the ORM playbook). 4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create the app with the correct redirect URI, and paste the `client_id` / `client_secret` into the app's SSO settings. The relying parties that exist - today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14); + today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14) **Gitea has no provider row at all** (live provider table, 2026-10-18) — it authenticates itself and stays off the chain by design (`dankclaw/Caddyfile:110-120`). @@ -243,7 +243,7 @@ every outpost session (see the ORM playbook). The embedded outpost does not re-evaluate group membership per request. Each completed authorize writes one row to `authentik_providers_proxy_proxysession` whose `session_data` JSON holds the claims (`groups` included) that get served -for that browser session until the row expires (observed TTLs 1–24 h per +for that browser session until the row expires (observed TTLs 1–24 h row). A session authorized **before** a user was added to a group keeps serving the old group list — Caddy-side RBAC header gates then 403 despite correct membership, and no authentik event is generated (the 403 happens at @@ -309,7 +309,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal 000 (connection refused) — the post-auth listener lives in the caddy container's netns; probe inside it or grep the config. Non-browser clients cannot solve the JS challenge — before flipping, audit service-to-service -traffic: ingestion/push paths must target internal addresses (`loki:3100`), +traffic: ingestion/push paths must target internal addresses (`loki:3100`) never a public hostname of a guarded service. Exempt-by-construction categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP endpoints, services with their own complete auth — document the exemption @@ -336,14 +336,14 @@ EOF `sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18: the exact banner text and the 229-object count — no `ak shell` was driven against the live auth service for this audit)*. -- Import paths (each wrong guess costs a failed exec): `User`, `Group`, +- Import paths (each wrong guess costs a failed exec): `User`, `Group` `Application`, `ApplicationEntitlement`, `PropertyMapping` → `authentik.core.models`; `ProxyProvider` → - `authentik.providers.proxy.models` (NOT `authentik.outposts.models`); - `LDAPProvider` → `authentik.providers.ldap.models`; + `authentik.providers.proxy.models` (NOT `authentik.outposts.models`) + `LDAPProvider` → `authentik.providers.ldap.models` `authentik.core.models.application` does NOT exist as an import path; there is no `authentik.policies.group` module in this build. - *(unverified 2026-10-18: these module paths were not re-executed — importing + - (unverified 2026-10-18: these module paths were not re-executed — importing outside a configured Django shell fails with `ImproperlyConfigured`)* - The live database's table and column names (the 2026.x renames, the MTI parent keys, the read-only `psql` audit recipe) are reference lookup: @@ -376,7 +376,7 @@ downstream — `category: user` panels from `services.json`. - `openwebui_users` / `openwebui_admins` → ApplicationEntitlements **"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the - `Open WebUI` application (`authentik_core_applicationentitlement`, + `Open WebUI` application (`authentik_core_applicationentitlement` 2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles` claim *(unverified 2026-10-18: the group→entitlement binding and the mapping expression were not read; the mapping itself is not visible under that name @@ -392,7 +392,7 @@ downstream — artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 — and every one carries the `@franksdank.cloud` address the template prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked - *(unverified 2026-10-18)*. Template: `username` + + - (unverified 2026-10-18)*. Template: `username` + `@franksdank.cloud` + display name + `is_active=True` + the password set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this @@ -400,7 +400,7 @@ downstream — the row carried no MFA device, token, entitlement, or proxy session, and events survive a user delete because `Event.user` is a denormalized JSON field, not a FK). -- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`, +- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins` `openwebui_admins`, `authentik Admins`, `authentik Read-only`. - Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty, Plex, Seerr, Marreta) need no per-user group: any authenticated user @@ -429,8 +429,8 @@ More field gotchas (each cost a failed exec): `ApplicationEntitlement.applicatio can be null (guard with `getattr`); `LDAPProvider` has `search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/ `organization` — the backing table -(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`, -`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`, +(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id` +`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number` `uid_start_number`, `search_mode`, `bind_mode`, `mfa_support` (`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes` (scopes are linked through `ProviderPropertyMapping`, which is why @@ -484,5 +484,5 @@ podman run --rm --network container: --entrypoint /bin/busybox \ - [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin - [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme - Research: `.docaudit/research/auth.md` — the full provider comparison - *(the file is gone from the repo and from the host checkout as of + - (the file is gone from the repo and from the host checkout as of 2026-10-18)* diff --git a/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md b/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md index c712434..2894f1b 100644 --- a/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md +++ b/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md @@ -47,7 +47,7 @@ torrent. WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s The private key is hashed into the VM env file and never enters the repo. -NAT-PMP inbound forwarding is available on Proton (gluetun supports it via +NAT-PMP inbound forwarding is available on Proton (gluetun supports it `FIREWALL_INPUT_PORTS` + `NATPMP=true`) — deliberately deferred: the e2e gate passes outbound-only, and swarm peers needing an inbound port is a seeding-era concern, not a grab-only one. @@ -86,7 +86,7 @@ are unaffected. got `Unauthenticated`); the live conf was patched and `scripts/deploy-remote.sh` fixed to write the `_PBKDF2` key. Gotcha (live, 2026-09-01): after qB 5.x ran, its conf rewrite left a hash line - 4.6.7 rejects (`invalid credentials` for every login, including the arrs); + 4.6.7 rejects (`invalid credentials` for every login, including the arrs) rewriting the same `@ByteArray` line fresh (same password, 100k iterations) restored login. Any future version flip: regenerate the line. - `HostHeaderValidation=false` + `ServerDomains=*`: the socat forwarder @@ -98,7 +98,7 @@ are unaffected. path (no auto-TMM), so the default itself points at `Finished Torrents`: every grab lands there regardless of client behavior; human review moves files onward; nothing writes to the entropy volume. -- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`, +- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr` `tv-sonarr` (all → `Finished Torrents`). - **Download clients** (gateway `10.88.0.1:8080`, user `admin`): all five test green against live qB — sonarr (`tv-sonarr`), radarr, lidarr, readarr @@ -109,12 +109,12 @@ are unaffected. as "Authentication failure" — that kills runtime auth too, not just API creates; the fix was pinning qB itself (below), after which readarr tests 200 like the rest. -- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`, +- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7` not `latest`): `latest` resolved to 5.x, whose `204` login breaks Readarr's frozen client; 4.6.7 answers the legacy `200` + `"Ok."` that every client accepts. Pinned in `frankbook-pro/qbittorrent/quadlet/qbittorrent.container`. -- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`, +- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync` gateway ports). Readarr deliberately omitted: prowlarr's indexer-push test to readarr's `/api/v1/indexer/test` 500s (EOL schema drift). Legacy `Prowlarr-*` indexer rows carried in the migrated dankserver DBs diff --git a/serverAppConfigs-docs/Infrastructure/Grafana.md b/serverAppConfigs-docs/Infrastructure/Grafana.md index a7e6519..a10bb1a 100644 --- a/serverAppConfigs-docs/Infrastructure/Grafana.md +++ b/serverAppConfigs-docs/Infrastructure/Grafana.md @@ -46,7 +46,7 @@ The Prometheus datasource is declared in git, not clicked into `grafana.db`: address Caddy proxies to, never the loopback-published port — and carries `timeInterval: 15s` so Grafana's rate windows cannot be shorter than the scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned -alongside it by `dashboards/provider.yml` (provider `fleet`, +alongside it by `dashboards/provider.yml` (provider `fleet` `updateIntervalSeconds: 30`, `allowUiUpdates: true`). A Loki datasource is still **not** provisioned — the tree contains exactly one @@ -70,7 +70,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026- Basic client auth, form-encoded token body, and the fork's userinfo-claims bug that leaves identity at the `sub` hash only) is in [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana: - The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`, + The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true` `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so users land as Viewer with `login = `; no email/groups/role claims reach Grafana until the evaluator bug is fixed — do not write role mappings @@ -79,7 +79,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026- the host without the database or an admin password. *(unverified 2026-10-18)*) - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That file is a gluetun env file and it carries `HOME=/root`; systemd imports it - for the service, podman then looks for its configuration under `/root`, + for the service, podman then looks for its configuration under `/root` cannot read it, and the unit dies at start with `stat /root/.config: permission denied`. The unit reads `/home/dank/Containers/grafana/env`, which the deployer seeds with exactly diff --git a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md index 83284d8..530b541 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md @@ -51,7 +51,7 @@ See [[Container Host Reachability]] for the netns rules. published nothing → structurally unprobeable from the host). 3. Probe from the **correct vantage**: from a bridge-net container the host is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on - `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`), + `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`) which belongs to the default `podman` net (`10.88.0.0/16`). Probe with bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so they expand **inside** `podman exec bash -c`. @@ -83,7 +83,7 @@ python3 -m http.server 8899 --bind 127.0.0.1 ``` Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s** -(panels render after the async fetch; checking immediately returns 0 panels), +(panels render after the async fetch; checking immediately returns 0 panels) extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/ `.desc-inputs`/`.desc-use` from the card's `details` object, other cards a plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for diff --git a/serverAppConfigs-docs/Infrastructure/Homepage.md b/serverAppConfigs-docs/Infrastructure/Homepage.md index 243a56b..03da983 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage.md @@ -22,7 +22,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat **Docs page shipped (2026-09-17):** `/docs` serves a static user documentation page (`shared/homepage/docs.html`) — access walkthrough, the running-model guide, media how-tos, and troubleshooting. Per-service `/docs//` pages -remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`, +remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium` `frank-zealot`) moved from admin-only to the **user** page with live model descriptions and a `docs` link to `/docs#models`. @@ -59,7 +59,7 @@ without which the picker showed "No results found" for every family account "equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra (same era)", frank-zealot "equivalent to last year's frontier level -(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic); +(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic) pinned in `homepage_config.bats`. **Protoss rename + Model Information component (2026-10-08):** the model @@ -91,7 +91,7 @@ same renames, api-id mapping, supervision and compaction notes. app card now carries hex-clipped art beside its name (`services.json` `art` field, files under `img/apps/`): 19 brand marks tinted amber from the Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323 -initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard), +initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard) node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy, Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model cards render in the order zealot → dragoon → archon (registry order is the @@ -143,8 +143,8 @@ per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red, Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow, Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service -fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`), -`warn` on OpenWebUI (searches ride the Seerr→SearXNG path), +fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`) +`warn` on OpenWebUI (searches ride the Seerr→SearXNG path) `model.compaction` (the amber "set this as compaction model" callout frank-zealot carries). frank-dragoon's card portrait is the SC2 unit portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole @@ -175,7 +175,7 @@ titles are chamfered plates with the diamond glyph inside the plate. Fonts: body is **Share Tech Mono** (legible retro terminal), headings are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all three pages load the pair from Google Fonts. Four tiles that had -reverted to initials got pictograms: Calibre-Web an open book (orange), +reverted to initials got pictograms: Calibre-Web an open book (orange) RomM a purple GameBoy, Copyparty a blue folder with upload arrow, Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`). frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching @@ -197,7 +197,7 @@ state — the only animation left is the speaking-voice pulse. The result is flat navy panels on crisp 1px borders: same Protoss direction, half the pixels. -**Model ids official (2026-10-08):** the card ids `frank-zealot`, +**Model ids official (2026-10-08):** the card ids `frank-zealot` `frank-dragoon`, `frank-archon` are now the harness's official model ids in `models.yml`/`config.yml` (see [[OMP Harness]]); the systemd services keep the `frank-archon`/`frank-dragoon`/ @@ -205,7 +205,7 @@ systemd services keep the `frank-archon`/`frank-dragoon`/ one name per model. **Api ids renamed on the wire (2026-10-09):** the promise in -`docs.html` is kept — the served model ids are now `frank-zealot`, +`docs.html` is kept — the served model ids are now `frank-zealot` `frank-dragoon`, `frank-archon`, so the docs page no longer carries `api id:` annotations, and the curl/opencode/Pi examples send `frank-zealot`. The systemd services keep the old names. `docs.html` @@ -216,7 +216,7 @@ ids, which did not change. clip-paths replaced with `border-radius` (the clipped corners read as gaps in the 1px border); section-title diamonds moved from absolute positioning to a fixed-width inline glyph (aligned, wrap-safe); closed -WIP items grey out (dim type, desaturated art, muted border via +WIP items grey out (dim type, desaturated art, muted border `:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track (`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms beat; the two SC2 source oggs are trashed); privacy notes became diff --git a/serverAppConfigs-docs/Infrastructure/LAN-DNS.md b/serverAppConfigs-docs/Infrastructure/LAN-DNS.md index bcb668d..1954613 100644 --- a/serverAppConfigs-docs/Infrastructure/LAN-DNS.md +++ b/serverAppConfigs-docs/Infrastructure/LAN-DNS.md @@ -188,7 +188,7 @@ sudo systemctl restart systemd-resolved && resolvectl flush-caches ``` The `~` (route-only) prefix is load-bearing: it routes only those domains to -Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`; +Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143` `resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`. **Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If @@ -236,7 +236,7 @@ Endpoints (all POST, JSON body): `.content.records[]` as some docs claim — parse with `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`. - Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey). -- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`, +- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name` `type`, `content`, `ttl`, `prio`). Quirks that bite (verified live 2026-08-31): diff --git a/serverAppConfigs-docs/Infrastructure/Loki.md b/serverAppConfigs-docs/Infrastructure/Loki.md index fe374d9..16a2727 100644 --- a/serverAppConfigs-docs/Infrastructure/Loki.md +++ b/serverAppConfigs-docs/Infrastructure/Loki.md @@ -55,7 +55,7 @@ The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to ` The service answers and the API works; there is simply no data in it. Measured on 2026-10-18: - `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`. -- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`, +- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal` `chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) — `find /opt/loki/data -type f` counted **8 files**, all WAL segments, cluster-seed and delete-request bookkeeping, **no chunk objects** diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index 97e9823..3e91553 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -21,7 +21,7 @@ committed. | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules | | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) | -Targets and datasources reference **container names** (`grafana:3000`, +Targets and datasources reference **container names** (`grafana:3000` `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never loopback-published ports; `alloy` is **not scraped at all** (no job in the config — its `127.0.0.1:12345` UI is only probed by the homepage timer, @@ -47,7 +47,7 @@ from `deploy_observability`: - `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`) `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus -`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`), +`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`) `shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and `shared/grafana/**` to `grafana` (`:261-264`); changes under `dankclaw/observability/` select the touched unit and its dependency consumer @@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint. **Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`. Current delivered set: **4 alert rules + 2 recording rules** — -`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, +`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff` `LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and -`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`), +`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`) confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) reports 4 alerting + 2 recording. Rules reload with the prometheus service. @@ -101,7 +101,7 @@ curl -s -u admin: http://127.0.0.1:3002/api/datasources | python3 -m json.to The two Prometheus queries were run on 2026-10-18 and are what every number on this page comes from: 4 alerting + 2 recording rules, and `up` for -`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, +`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki` `frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs listed in [[Prometheus]]. The Grafana query needs the local admin password and was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is @@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect). ## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18) -Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, +Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container` loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach via `host.containers.internal`) → signal-cli, addressed by number `+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked -2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`, +2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093` `curl -s http://192.168.0.143:18093/health` → `ok`, and `journalctl --user -u signal-relay` ends with successful `sent to +12062476336:` lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and @@ -148,7 +148,7 @@ state lives in Prometheus' process, not on disk — a restart re-stamps `activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not evidence of a fresh fault. (Alertmanager by contrast **does** persist: `--storage.path=/alertmanager` on a host volume keeps silences and the -notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`, +notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42` `/opt/alertmanager/data`.) Remaining known downs are the structural ones above (HTML-scraped probes, Gitea diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 41de35e..b8f3bea 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -51,13 +51,13 @@ Dankserver runs one member of this family, node-exporter, as its own unit ## Exposure The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18) -shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`, +shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100` `127.0.0.1:12345` and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard `anubis:8923`, and the post-auth handlers route by container name on -`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`), -`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`), +`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`) +`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`) `prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than @@ -70,13 +70,13 @@ and the podman bridge subnet `10.89.0.0/16` before the DROP ## Deployment -`./scripts/deploy.sh dankclaw --service ` runs `deploy_observability`, +`./scripts/deploy.sh dankclaw --service ` runs `deploy_observability` which installs the unit, copies that service's config -(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`, +(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config` `copy_alloy_config`, `copy_grafana_provisioning`, plus `install_signal_relay` with the alertmanager deploy), then waits on the loopback -health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`, -`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`, +health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy` +`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy` `/node-exporter`: `/metrics`). Startup order follows the data dependencies — `grafana` after `prometheus`, `alloy` after `loki` (`grafana.container:32-33`, `alloy.container:38-39`). See diff --git a/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md b/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md index 134b851..0e193ef 100644 --- a/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md +++ b/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md @@ -100,7 +100,7 @@ The DFlash drafter proposes, then ngram-mod runs as an additional implementation `--spec-type` takes a comma list, and the gate is `ctx_dft` in `common/speculative.cpp` (`common_speculative_init_result`): **draftless** types (the ngram family — simple, map-k, map-k4v, mod, cache) are added -unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`, +unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash` `draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences: - A chain may hold ONE context-backed drafter plus any number of ngram types. @@ -201,7 +201,7 @@ Speculative decoding is lossless under rejection sampling: the target verifies e On the mid-August 2026 Dankclaw shape `draft-dflash` was the benchmarked winner at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those three figures are that retired llama.cpp era's property ([[frank-dragoon]] owns -the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]). +the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see ). ## Draft Models @@ -257,7 +257,7 @@ DSpark took `ROCm error: an illegal memory access` inside the shared draft context with two concurrent 16k prompts; DFlash corrupted multi-slot into premature EOS and was retired first. The timeline, the crash class, the "concurrency is the trigger, not size" finding and the `PARALLEL=1` workaround -are on [[qwen3.8-spec-matrix]]; the shape they were measured on is +are on ; the shape they were measured on is [[frank-dragoon]]'s mid-August 2026 era. ## A/B protocol on a live llama.cpp endpoint @@ -303,7 +303,7 @@ Protocol: Read the right counters — per request `timings.draft_n` / `draft_n_accepted` / `predicted_per_second`; cumulative `/metrics` -`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`; +`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total` per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0` while tokens flow = the proposer is NOT engaged, whatever the flags say. A zero-draft config is a useful negative control: it proves the other config's @@ -367,7 +367,7 @@ states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk. **Step 4 — acceptance reality checks.** Treat 40–60 % as the honest band for a matched pair; the arm that anchors the bottom of it on this fleet is the -depth-7 `draft-dflash` row of [[frank-medium-llamacpp-dflash2]]'s spec table +depth-7 `draft-dflash` row of 's spec table (accepted share, rate and VRAM for every depth are in that table). Cheapest proxy for `p` at temp 0 when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights @@ -381,10 +381,10 @@ drafting. **Step 5 — check the attachment surface BEFORE the arithmetic.** Closed engines refuse external drafters by design: halogen-flash-server's `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable -name is a deliberate 400 (a silent downgrade would make comparisons lie); +name is a deliberate 400 (a silent downgrade would make comparisons lie) there is no device routing in its loopback token protocol, so a drafter can never sit on another card. Pure llama.cpp DOES route the draft: -`--spec-draft-device/-devd` (default follows `--device`), `-ngld`, +`--spec-draft-device/-devd` (default follows `--device`), `-ngld` `-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`. One process sees only the devices its backends were built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot @@ -411,20 +411,16 @@ measurement that would settle it. - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS. - DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. -- The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by [[RAM CPU Offloading in VLLM]]. +-The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by . - `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist. - Use `rejection_sample_method` default `strict` for lossless output. ## See Also - [[Quantization by Hardware Architecture]] -- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] -- [[Running Qwen3.6 27B FP8 on Dankclaw]] - [[Dankclaw System Specs]] - [[Dankserver System Specs]] -- [[VLLM Overview]] -- [[deployment]] - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic -- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands +- where the drafter's KV actually lands - [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/Dankclaw Improvements.md b/serverAppConfigs-docs/LLM/Dankclaw Improvements.md index a671ec3..3fad616 100644 --- a/serverAppConfigs-docs/LLM/Dankclaw Improvements.md +++ b/serverAppConfigs-docs/LLM/Dankclaw Improvements.md @@ -30,8 +30,8 @@ taken in. - **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm wins prefill and Vulkan wins decode, and the decode gap widens with context: - the 64k A/B is [[qwen3.8-strix-optimizations]] §4, the 262k A/B is - [[qwen3.8-spec-matrix]] §High-Context Backend Comparison. The "HIP + the 64k A/B is §4, the 262k A/B is + §High-Context Backend Comparison. The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S. - **Power profile**: the `/sys/class/drm/card0/device/power_profile` file does not exist on this machine (checked 2026-08-15). GPU DPM is already @@ -39,7 +39,7 @@ taken in. - **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What `frank-zealot` runs now is its own Current shape — this row is not it. - **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default` - (ngram-mod) with draft-mtp first — timeline on [[qwen3.8-spec-matrix]]; the + (ngram-mod) with draft-mtp first — timeline on ; the spec setting each endpoint runs today is on its own page. - **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded once on [[Dankclaw System Specs]] (Memory). @@ -54,8 +54,8 @@ taken in. That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as "dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is -recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT; the 76.2 GiB row is an -era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`, +recorded in §Why Vulkan + GTT; the 76.2 GiB row is an +era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal` the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read 2026-10-18. @@ -125,9 +125,9 @@ All require root and a reboot for the modprobe change. ## Sources - [[frank-dragoon]] — measured backend/spec numbers -- [[deployment]] — deploy and update process +- deploy and update process - AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS) -- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`, +- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt` `common/arg.cpp`, `common/common.h` - NotebookCheck Ryzen AI MAX+ 395 analysis @@ -135,5 +135,3 @@ All require root and a reboot for the modprobe change. - [[frank-dragoon]] - [[DFlash DSpark Speculative Decoding]] -- [[deployment]] -- [[index]] diff --git a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md index 6b42b4b..0bd631a 100644 --- a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md @@ -32,7 +32,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp | Device ID | 0x1586 | | Architecture | gfx1151 (Strix Halo) | | GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `mem_info_gtt_total` = 122,402,963,456 B read 2026-10-18; rocm-smi's smaller SVM carve-out is not the budget) | -| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ([[ling-3.0-flash-trial]] §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale | +| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ( §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale | ### NPU @@ -44,9 +44,9 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp | Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) | | Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) | | Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` | -| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] | +| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See | -The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot]], 2026-09-17 → 10-01). +The NPU endpoint is retired (2026-09-11; ). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see ); the Gemma4-12B QAT rung it once pointed at is retired history (, 2026-09-17 → 10-01). ### ROCm @@ -80,7 +80,7 @@ The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name | Interface | Type | Speed | IP | Notes | |-----------|------|-------|----|-------| | eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carried the RPC path to Dankserver during the 2026-08-17 pipeline-parallelism trial; no RPC listener runs today (`ss -ltn` shows no :50052, 2026-10-18). Default route via 192.168.0.1. | -| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. | +| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See section 0. | | docker0 | bridge | - | 172.17.0.1 | Usually down. | The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at @@ -112,5 +112,3 @@ cable between claw and server is required for 10G. - [[frank-dragoon]] - [[Shared Memory Architecture Dankclaw]] -- [[frank-redo-2026-08-17]] -- [[FastFlowLM NPU Inference]] diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 682ffa3..c8289e0 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -30,7 +30,7 @@ read 2026-10-18). | Property | Value | |----------|-------| | Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 | -| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) | +| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see ) | | GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3` (both bridges: `current_link_width=8`, `current_link_speed=32.0 GT/s`, read 2026-10-18); ~31.5 GB/s per card. Each card sits behind an on-board PCIe switch (`01:00.0`/`02:00.0`, `05:00.0`/`06:00.0` — "Navi 10 XL Upstream/Downstream Port of PCI Express Switch"), so the GPU endpoint itself trains **x16** Gen5 while the CPU-facing leg stays x8 | | Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board (2026-09-20 census; not re-readable today — `lspci -vv` needs root) *(unverified 2026-10-18)* | | Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 | @@ -55,7 +55,7 @@ read 2026-10-18). | Architecture | gfx1201 (RDNA4) | | VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) | | Link | PCIe 5.0, endpoint-trained **x16** each behind the on-board switch; the CPU-facing leg is x8 (`00:01.1` / `00:01.3`, both `current_link_width=8` at 32.0 GT/s, read 2026-10-18). No XGMI. Separate IOMMU groups — `03:00.0` → group **15**, `07:00.0` → group **20** (`/sys/bus/pci/devices/.../iommu_group`, read 2026-10-18) | -| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see [[Dual-R9700-Cooperation]] | +| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see | | PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) | | iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2 (rocm-smi GPU[2], 512 MiB). The retired llama.cpp trios hid it with `HIP_VISIBLE_DEVICES=0,1`; the live radiance engine instead enumerates all three AQL devices and ignores it ("3 devices [0:gfx1201 1:gfx1201 2:gfx1036], using 2 of 3, ignoring gfx1036") | @@ -104,8 +104,7 @@ RCCL-backed rebuild needs no new host packages. | Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact | The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to -hijack `libhsa-runtime64` were purged on 2026-09-15 — see -[[frank-medium-rocmfpx]]. +hijack `libhsa-runtime64` were purged on 2026-09-15 — ### Active Services @@ -117,12 +116,12 @@ host-network podman container answering `curl 127.0.0.1:7770/health` → The services this page once called active are gone: `frank-medium.service` (llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung -ran 2026-09-16 → 09-20 and is documented in [[frank-medium-llamacpp-dflash2]]; +ran 2026-09-16 → 09-20 and is documented in `frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap. Its archived unit was moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the vllm-radiance replica trio to `~/.AGENTTRASH/20260916-vllm-radiance-retire/` — neither path exists today (checked 2026-10-18), so the archive location is -unverified. See [[deployment]]. +unverified. See . ## Hardware Relevance @@ -134,7 +133,7 @@ unverified. See [[deployment]]. tokens at 32:33, ~6.3/~5.2 GiB headroom) is history. Either way a batch or context change needs a re-fit and not a guess. - 246 GiB system RAM is **not** headroom: the live shape pins a 43.88 GiB host - pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`, + pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928` `scripts/frank-archon.env:184`) plus the 47.68 GiB n-gram/PLE table in RAM, and `free -g` showed **90 GiB available** with the engine loaded (read 2026-10-18). The frank-medium claim that "nothing in the serving path is @@ -147,6 +146,6 @@ unverified. See [[deployment]]. ## See Also - [[frank-archon]] — the active serving runbook -- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed -- [[deployment]] — the deploy flow +- why the PCIe path matters, and what the swap changed +- the deploy flow - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory diff --git a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md deleted file mode 100644 index 4eef7ca..0000000 --- a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md +++ /dev/null @@ -1,327 +0,0 @@ ---- -type: reference -title: Dual R9700 Cooperation -description: "Why two R9700s could not run tensor-parallel on the old board, and what the board swap unblocked." -tags: - - frank-medium - - rocm - - rccl - - pcie-atomics - - tensor-parallelism - - dankserver -timestamp: 2026-09-16 ---- - -# Dual R9700 Cooperation - -Dankserver's two R9700s run **one tensor-parallel engine across both cards** — -today that is `frank-archon` under radiance 1.0.13, TP=2 -([[frank-archon]]). When this page was written the endpoint -was `frank-medium`: first llama.cpp's layer split -([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20 -one vLLM-radiance process at `tensor_parallel_size 2` -([[frank-medium-radiance-mxfp4]], 2026-09-20 → 09-26). This page records why -true tensor parallelism was impossible on the previous board (TUF GAMING B650-E -WIFI), and what the 2026-09-20 board swap changed. - -## The short answer - -**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose -bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden -hostcall-buffer parameter and ROCr refuses to supply it to a device without -atomics, so *no* RCCL collective or P2P operation could be dispatched on that -card — see the [measured matrix](#measured-matrix) and the -[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below. - -On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which -bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page -recommended, at B850 rather than X870E money. Both cards now hang off CPU root -ports (`00:01.1`, `00:01.3`), every bridge on both chains reads -`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** — -the exact precondition this page's own test plan named, and it is now -[measured](#measured-verdict-2026-09-20) rather than assumed. - -What that bought: - -- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted - and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp - `--split-mode tensor` booted and served against an RCCL-backed build — so a - collective HAS been dispatched on this host since the swap. -- **TP=2 is the production shape.** It wins on every axis measured, by 2.1x on - decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day - of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is - retired 2026-09-26, and what runs today is the iq4r publisher under radiance - 1.0.13 ([[frank-archon]]). -- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode - could not carry the DFlash2 drafter, which is the honest caveat on the - comparison below. -- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each): - card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1, - ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now - and M.2_2 is free. - -## The gate, in source - -```cpp -void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp:181 - std::vector link_attrs; - link_attrs.push_back(std::make_pair(LinkAttribute::kLinkAtomicSupport, 0)); - if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) { - if (link_attrs[0].second == 1) { - info_.pcie_atomics_ = true; // no else branch - } - } -} -``` - -The flag is a **queried hardware property**, consumed at -`rocvirtual.cpp:3954` (the hostcall/AQL dispatch path), in `rocblit.cpp` (SDMA -copy paths) and exported as `hipDeviceProp.hostNativeAtomicSupported`. There is -no environment variable, module parameter, or sysfs toggle around it — verified -by grepping ROCclr for `getenv`/`HSA_*`/`ROCR_*`/`AMD_*` in the gate's -neighborhood: none. Writing the endpoint's Device Control 2 bit 6 by hand -(`setpci -s 0b:00.0 CAP_EXP+28.w=0440`) does nothing because the value is -consumed at driver probe time. - -What has to be true upstream of it: every PCIe bridge on the GPU's path to the -CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path -includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability -bit**, so no software can enable it. - -### The same gate, as this board reports it (2026-09-16) - -`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` → -`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` → -`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`: - -| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link | -|---|---|---|---|---| -| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 | -| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 | -| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 | -| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` | -| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) | -| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 | - -Three things fall out of that table, none of them in the source alone: - -- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the - chipset, not the platform, which is why a slot on CPU lanes is the fix. -- **The endpoint's own bit differs between the cards, and that is the kernel's - doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses - to set Requester Enable when any bridge cannot route atomics. ROCclr's - `pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it - makes `lspci -vv -s | grep AtomicOpsCtl` a one-command diagnosis: on - this host card 0 reads `ReqEn+` and card 1 `ReqEn−`. -- **Card 1's effective link is x1**, not the x16 its own capability block - advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a - bandwidth fact rather than an atomics one, and it only matters for - collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at - x4. The layer split's boundary copy does not care. - -## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO) - -`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` → -`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` → -`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only -LAN/SATA/WiFi: - -| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link | -|---|---|---|---|---| -| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 | -| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 | -| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 | -| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) | -| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 | -| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 | -| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 | -| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) | - -- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the - kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no - `Routing-` bridge. Same one-command diagnosis, opposite answer. -- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited - by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per - card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8". -- `rocm-smi --showtopo` reported `PCIE` between the cards (Navi 48 has no XGMI), - they were in separate IOMMU groups (15 and 20 — re-read 2026-10-18: still 15 - and 20), and both root ports had ACS `ReqRedir+`, so this page inferred P2P - would hairpin at the root complex. **That inference is superseded:** the live - engine reports the two cards reaching each other's memory directly, which - [[frank-archon]] and [[Dankserver System Specs]] record. Plain P2P traffic - measured 45.27 GB/s bidirectional on the *old* board — DMA was never the gate, - atomics were. -- Kernel 7.0.0-31-generic with `iommu=pt` then; this host runs 7.0.0-38-generic - with `iommu=pt` today (`/proc/cmdline`, read 2026-10-18). Both were ruled out - as causes ([above](#ruled-out-empirically-do-not-re-derive)) and this host's - IOMMU posture is unchanged. - -**Still unproven (as of 2026-09-20).** The gate that matters is ROCclr's -`checkAtomicSupport()` reading the `kLinkAtomicSupport` link attribute; the ReqEn -bit is the same cause seen from the kernel side, but no RCCL kernel had been -launched since the swap. Until the 2-rank probe runs — or a `--split-mode row` -boot that does *not* log `internal AllReduce init failed ... falling back to -meta-backend butterfly` — "collectives work here" was a strong expectation, not a -measurement. **RCCL never got that measurement**; what settled the operational -question instead is production traffic — radiance's TP=2 wire carries cross-rank -all-reduces over the direct P2P path and prints its own precision, and the -measured `wht6` banner with its exact-step bound is on [[frank-archon]]. - -## Measured matrix - -Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no -model. Rank 0 = card #1 (CPU lanes), rank 1 = card #2 (chipset): - -| Operation | rank 0 (card #1) | rank 1 (card #2) | -|---|---|---| -| `broadcast` | OK | **FAIL** `DistBackendError` | -| `all_reduce` | OK | **FAIL** | -| `barrier` | OK | **FAIL** | -| `send` / `recv` (P2P) | blocked, then timeout | **FAIL** | -| `all_gather` (after failure) | group unusable | — | - -The `AMD_LOG_LEVEL=3` trace names the cause: `Pcie atomics not enabled, hostcall -not supported` → `AQL dispatch failed!` → `hipLaunchKernel` returns -`hipErrorIllegalState` for `ncclDevKernel_Generic_4(...)`. Nothing in vLLM or -RCCL mentions atomics; the runtime log is the only place it appears. - -**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism -as an option, because vLLM's PP stage handoff is exactly that primitive. - -## Software escapes: closed, and the last one is now moot - -Every row below was written under the old board. The last row — the only door -that was ever open — stopped mattering on 2026-09-20, because the hardware gate -it existed to route around no longer applies to this host. - -| Idea | Finding | -|---|---| -| vLLM's IPC custom allreduce | **Gated off by a table keyed to CUDA sm versions** — `{9.0, 10.0, 10.3, 10.7}`; ROCm reports capability `12.0` for gfx1201, so the gate is `False`. `custom_ar` itself is compiled into the image. | -| ...and if that table were patched | Still insufficient: `broadcast`, `barrier`, and the TP logits `all_gather` have no non-RCCL path. | -| The radiance fork's own allreduce | Already present and *on by default* (`RADIANCE_USE_R4D_AR`, size gate `RADIANCE_AR_MAX_KB`): a one-shot P2P push allreduce wrapping `CudaCommunicator.all_reduce`, exchanging IPC handles over the **gloo** group. With the gate raised it covers prefill and decode, and `ncclDevKernel` disappears from the profile. But it does not intercept `PyNcclCommunicator.__init__`'s **warmup allreduce**, nor `pynccl_comm.all_gather` — both direct RCCL calls. | -| NCCL symmetric memory | Unavailable: the shipped RCCL advertises NCCL 2.30.4 but exposes no `ncclDevComm*` API, so torch's symmetric-memory path cannot run. | -| Prefill/decode disaggregation | Needs a KV-transfer connector plus coordination collectives; both out. | -| gloo for GPU work | CPU/TCP collectives — orders of magnitude too slow for per-layer TP. | -| **Rebuild RCCL so its kernels do not declare a hostcall buffer** | **OPEN — unverified.** The claim is that the hostcall requirement is triggered by device-side `printf`/`assert`, and that `-mprintf-kind=buffered` avoids it. Counter-evidence: RCCL's device source at `src/device/` contains **no live** `printf`, `assert`, or `__assertfail` (checked), so what makes its kernels declare `HiddenHostcallBuffer` is unexplained. The claim traces to search-result summaries, not a primary source. Cheap to settle once a host is available: recompile librccl with `-mprintf-kind=buffered`, dump a kernel's declared arguments (`roc-obj-extract` + `llvm-readelf --notes`) and look for `hidden_hostcall_buffer`; if it is gone, a 2-rank probe should dispatch. **Stop and mask the service first** — see the note below. | - -## Bandwidth: why x4 is enough - -A 27B TP=2 allreduce at batch 4 moves roughly -$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$ -per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**, -or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8 -~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound. -That is why the cheap option (the M.2 riser) and the expensive one (the board -swap) were both considered adequate on bandwidth alone — and why the board that -landed, at x8/x8, clears the arithmetic with room to spare. - -## Ruled out empirically (do not re-derive) - -- Kernel versions 7.0.0-28 / -30 / -31 — byte-identical failure. -- `iommu=pt`, and `amd_iommu=off` is unrelated (that was a Dankclaw prefill knob). -- P2P hardware: 45.274 GB/s bidirectional between the cards works fine — plain - memory traffic is not the problem; *atomic* routing is. -- A from-source librccl rebuild: identical failure at the identical call path. -- Every NCCL transport/algorithm knob. -- A one-card test proves nothing: NCCL short-circuits at `world_size=1`, which - is exactly why the two replicas work at all. -- Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 — - different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure. - -## Measured verdict (2026-09-20) - -Both cooperation modes were run on this host on 2026-09-20 — the same salted -10-prompt battery (`scripts/bench-conversation.py`), the service stopped and -masked between arms. The arm table is [[frank-medium-radiance-mxfp4]]'s bench of -record (arms R = radiance TP=2 + DFlash2 as deployed, I = the llama.cpp layer -split that preceded it, L = llama.cpp `--split-mode tensor` against RCCL); this -page keeps only the verdict. **TP=2 won on every axis — roughly 2.1× decode and -3.6× at 4-way concurrency over the layer split — and replaced the layer split the -day of measurement.** - -Arm L carries a caveat no flag can remove: llama.cpp's tensor mode could not load -the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight -pre-allocated on the non-primary device has no backend once `n_copies == 1`), so -it decoded **without** speculation while the other arms decoded with it; and part -of R's margin over I is the 4-bit MXFP4 checkpoint, not the parallelism. - -Two numbers elsewhere on this page are therefore not interchangeable: - -- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce, - with RCCL off, on the old board. It was never a TP decode number, and its - paired decode figure is in "The third path" below. -- The butterfly tensor arm and the RCCL tensor arm a day later are the same shape - family on different collective paths; the measured pair lives with - [[frank-medium-radiance-mxfp4]]'s arms. - -RCCL needs PCIe atomics on *every* participating GPU — the gate the old board -failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is -why TP=2 runs today rather than merely "being unblocked": the image ships TheRock -RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so -radiance could not have used RCCL on this host even with atomics routed. Ubuntu's -RCCL 2.27 does carry gfx1201 kernels, and that is what arm L linked against. - -**Rule for any future multi-GPU collective experiment: stop and mask the -production service first**, per the benchmarking policy, or run it on a host that -is not serving users. The production service is `frank-archon`, a **root** unit, -so on Dankserver that is `sudo systemctl stop frank-archon && sudo systemctl -mask frank-archon` (`scripts/frank-archon.service`; never `disable` — that -deletes the unit file, and `mask` is what the bench policy prescribes). The -2026-09-20 wording below — `systemctl --user stop frank-medium && systemctl ---user mask frank-medium`, frank-medium being a rootless quadlet — is retired -with that service, which today reports `not-found` (read 2026-10-18). - -This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live -service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a -power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the -GPU driver is a plausible cause and this host has precedent for going -unreachable until a power cycle. - - -## The third path (production in the layer-split era): llama.cpp's layer split - -Everything above is about *collectives*, and vLLM cannot proceed without them -in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage -handoff is a p2p `send`/`recv` of the hidden state, the primitive this host -also fails. **llama.cpp's `--split-mode layer` needs no collective at all**: -layers are distributed across the cards and the boundary activation is a -plain per-backend copy, once per forward pass. No RCCL kernel is ever launched -on the second card, so the atomics gate never comes up. - -That is what `frank-medium` ran from 2026-09-16 to 09-20 -([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards, -262144 context. Its measured layer-split rates are that page's; this host's -tensor arm under the butterfly fallback measured 272 t/s prefill / 26.3 tok/s -decode. **That tensor number is not a prediction for an RCCL-backed build** — it -is what tensor split costs when the allreduce degrades to a meta-backend -butterfly, and it is the number the board-swap path exists to retire. - -The ranking therefore changed twice. On 2026-09-16: software already gets both -cards onto one model, so the hardware fix stopped being a capability question -and became a performance one — *tensor* parallelism, ~40 % decode, plus one -pooled KV instead of two independent 122k-token pools. On 2026-09-20 the -hardware half of that question was answered by the board swap; the llama.cpp -measurement half never ran, and `GGML_HIP_RCCL` still defaults `OFF` in -`scripts/build-llama-rdna4.sh:41` (re-read 2026-10-18) for exactly the reason -this page documents — moot while no llama.cpp shape is deployed on this host. - -## What would change the answer - -- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot - since 2026-09-20.** That door only mattered while the hardware gate could not - be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis - and its counter-evidence stay in the escape table for the record. -- The symmetric-kernel path inside RCCL, if a release ever ships the device API - (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock - nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does - not. -- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic - routing~~ — arrived as new silicon instead. -- A GPU-side workaround: none found in ROCclr, and the gate is per-device, - queried once at probe. - -## See Also - -- [[Pipeline Parallelism/index|Pipeline Parallelism]] — two-machine research (the PP variant of this problem) -- [[Dankserver System Specs]] — hardware inventory -- [[Observability]] — the scrape/dashboard/alert stack around these endpoints diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md deleted file mode 100644 index e1a4edc..0000000 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ /dev/null @@ -1,448 +0,0 @@ ---- -type: reference -title: FastFlowLM NPU Inference -description: "The AMD NPU-first runtime: fit analysis for the XDNA2, published benchmarks, install state, and test runbook." -tags: - - fastflowlm - - npu - - xdna2 - - dankclaw - - research -timestamp: 2026-09-03 ---- - -> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology. -> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration. -> The `frank-zealot` name and port 52625 then belonged to the vLLM -> service on Dankclaw (nameistoken Quark W8A8); this page is archive. -> -> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today -> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625 -> (`ss -ltn`, read 2026-10-18). The runtime is still installed -> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`, -> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is -> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound, -> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no -> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on -> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]] -> for the retirement record. Everything below is the 2026-08 research record. - -# FastFlowLM NPU Inference - -FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion. - -## Verdict (2026-08-16) - -- The NPU is a complement, not a replacement. It fits dense models up to about 9B. -- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small. -- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw. -- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge). -- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results. -- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`. - -## What FastFlowLM Is - -FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0. - -Key properties, per the project README and docs: - -- Runs fully on the NPU. No GPU or CPU load during inference. -- CLI mode (`flm run `) and server mode (`flm serve `). -- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625. -- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face. -- Context lengths up to 256k tokens on listed models. -- Vision, audio, embedding, and MoE model support. -- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use. -- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`. - -## NPU Hardware on Dankclaw - -Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU -with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory -via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at -4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]]. - -Live verification on Dankclaw, 2026-08-16: - -``` -$ flm validate -[FLM] Fetching models from: /opt/fastflowlm/share/flm/model_list.json -[Linux] Kernel: 7.0.0-29-generic -[Linux] NPU: /dev/accel/accel0 with 8 columns -[Linux] NPU FW Version: 1.1.2.65 -[Linux] amdxdna version: 0.7 -[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity. -``` - -- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18). -- Firmware 1.1.2.65 meets the 1.1.0.0 minimum. -- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`). -- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`. -- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working". - -## Memory Model - -The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half. - -Budget math for this fleet: - -| Machine | RAM | NPU budget (approx. half) | -|---------|-----|---------------------------| -| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB | -| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU | -| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB | - -The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]] -and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate` -(it owns the current number). - -Weight sizes (approximate, Q4-class): - -| Model | Quant | Weights | Fits ~15 GiB budget? | -|-------|-------|---------|----------------------| -| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes | -| Qwen3-8B | Q4_1 | ~5 GB | yes | -| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge | -| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB | -| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists | - -Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run. - -## Supported Models (Qwen family, plus gpt-oss) - -All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima. - -| Tag | Base model | Context max | Notes | -|-----|-----------|-------------|-------| -| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle | -| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle | -| `qwen3:4b` | Qwen3-4B | 32k | tool calling | -| `qwen3:8b` | Qwen3-8B | 32k | tool calling | -| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking | -| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling | -| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision | -| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | | -| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision | -| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle | -| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling | -| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling | -| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling | -| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling | -| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM | - -Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA. - -## Published Benchmarks (FLM, vendor numbers) - -All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed. - -Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length: - -| Model | 1k | 2k | 4k | 8k | 16k | 32k | -|-------|-----:|-----:|-----:|-----:|-----:|-----:| -| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 | -| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 | -| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 | -| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 | - -Prefill t/s for the same run: - -| Model | 1k | 2k | 4k | 8k | 16k | 32k | -|-------|-----:|-----:|-----:|-----:|-----:|-----:| -| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 | -| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 | -| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 | -| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 | - -Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**: - -| Metric | 1k | 2k | 4k | 8k | 16k | 32k | -|--------|-----:|-----:|-----:|-----:|-----:|-----:| -| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 | -| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 | - -- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current - PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at - 14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade - the worker's model. - -The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it. - -## Fit to the Frank Program - -| Our model | NPU verdict | -|-----------|-------------| -| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. | -| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. | -| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. | -| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. | - -Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either. - -## Install State on Dankclaw (verified 2026-08-16) - -- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0. -- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed. -- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.) -- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed. -- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`. -- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below). -- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot. -- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16. - -## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU) - -Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB. - -| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % | -|-----|-----------:|----------:|-------:|------------:|-----------:|---------:| -| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 | -| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 | -| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a | - -Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client. - -Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested. - -## Model Bakeoff (measured 2026-08-16, 4 models) - -All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom). - -| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) | -|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------| -| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 | -| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 | -| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** | -| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 | - -Notes: - -- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder. -- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load. -- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s. -- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it. - -## Promotion (2026-08-16) - -`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused -twice after this window - see the banner - and the NPU unit today is `frank-npu`, -masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), -wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. -`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap -`FLM_MODEL` in the env to switch. - -### Boot policy (2026-08-17) - -The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw: - -``` -sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service -sudo systemctl daemon-reload -sudo systemctl enable frank-zealot -``` - -Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom). - -The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`). - -Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it. - -Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept. - -## Parallel Streams (measured 2026-08-16) - -The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency. - -Concurrent-request test, gpt-oss-20b, 128-token cap per stream: - -| Streams | Wall s | Aggregate t/s | Per-stream wall s | -|--------:|-------:|--------------:|-------------------| -| 1 | 10.6 | 12.1 | 10.6 | -| 2 | 21.1 | 12.0 | 10.8 / 21.1 | -| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 | - -The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency. - -Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots. - -## Test Runbook -This runbook targets a retired endpoint; keep it as the procedure if the NPU is -ever revived. - -**Host RAM availability.** The gate is half of system RAM by default - a relic of the -install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's -box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM -first, or stop -the service the way the bench policy requires (`sudo systemctl stop -frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would -otherwise restart it). The NPU shares the unified pool with everything else on -the box. - -```mermaid -graph TD - A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine] - B --> C{NPU visible to XRT?} - C -- no --> D[Fix driver or firmware path] - C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose] - E --> F{Generates tokens?} - F -- no --> D - F -- yes --> G[4. flm serve on a scratch port, never 7770] - G --> H[5. Bench decode + prefill t/s] - H --> I{Speed acceptable?} - I -- no --> J[Record results, stop] - I -- yes --> K[6. Attempt gpt-oss:20b] - K --> L{Loads in ~15 GiB?} - L -- no --> J - L -- yes --> M[Bench, record] - M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped] - N --> O[Record, decision] -``` - -1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead. -2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device. -3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit. -4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770. -5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page. -6. Step up in size: 9B, then gpt-oss-20b. -7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load. - -## Registering packs absent from the catalog (NPU2) - -Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model) -to the FLM registry. Registry facts: user registry -`/home/dank/.config/flm/model_list.json`, shape -`{"model_path": "models", "models": {"": {"": {name, url, -file_url, size, flm_min_version, files[], vlm, default_context_length, -max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve -resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the -env var set lists custom tags (`⏬` = not downloaded). - -1. Back up the registry JSON first (`cp -a`). -2. Insert the entry via a python json edit + atomic `os.replace`. Set - `files` to the FULL HF file list — **including `vision_weight.q4nx` even - for text-only packs**: the qwen3.5-family loader opens it unconditionally - and a missing file makes the first request HANG silently (no error log). -3. **Do not use `flm pull ` for custom entries** — it throws - `json.exception out_of_range key ':' not found` while building - the download list. Place files manually into - `/home/dank/.config/flm/models//` with - `curl -sfL "https://huggingface.co//resolve/main/?download=true"` - — plain `raw/main/` returns 135-byte LFS pointer stubs, not - weights; verify byte counts against the HF tree API before trusting a - download. -4. Kernel packs: copy from the same-shape family pack (shape-match - `config.json` on hidden_size/num_hidden_layers/head_dim first) into - `/opt/fastflowlm/share/flm/xclbins//` via `sudo cp -a` - (root-owned dir). -5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so - the new process loads with all files present. -6. Verify with a REAL completion request (`model: ":"`) — - `/v1/models` alone proves registration, not loadability. - -Hard constraints: only ONE `flm serve` process can hold the NPU hardware -context — a second concurrent serve process fails every load with -`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status -0x2000003); never run scratch/verify instances while production holds the -NPU. Tag switching happens INSIDE one serve process; the first request -after a model switch pays ~6–11 s cold-load cost (visible in -`usage.load_duration`). Over ssh, never `pkill -f ""` with a -pattern present in your own command line — use bracket-guarded patterns -(`[q]wen3.5:4b`) or exact PIDs. - -## Tag load-failure diagnosis (200 with an error body) - -`POST /v1/chat/completions` answers **HTTP 200** with a flat -`{"error":"Failed to load model!"}` and no `choices`; `/v1/completions` -returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP -200. A status-code-only client reads this as a successful empty completion — -**always check for the `error` key**. Two distinct causes, distinguished via -`journalctl -u --full`: - -1. **Unknown tag** (not in the registry): immediate error, nothing loaded. - Correct behavior, not a bug. -2. **Registered tag whose kernel pack is missing**: FLM resolves the tag, - downloads missing weight files on the fly (first request hangs ~90–110 s), - then fails `Failed to load model: No such file - '/home/dank/.config/flm/xclbins//.xclbin'`; subsequent - attempts fail fast (~2 s). Root cause: the serve script exports - `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every - kernel lookup resolves under the user dir, which only contains symlinks - someone linked. **A tag listed in `/v1/models` is NOT proof it is - loadable on this host.** Fix: one symlink per missing pack - (`ln -s /opt/fastflowlm/share/flm/xclbins/ - /home/dank/.config/flm/xclbins/`) — verified live: after the - symlink the first request loaded and answered in 7.4 s wall. - -Switch-cost behavior: the `model` field routes per-request (no restart -needed); a request whose model differs from the resident one triggers a full -reload (7–11 s wall for an 8-token completion on ~4B-class packs); -alternating tags pays the swap each time. Single-lane NPU: concurrent -requests queue FIFO behind the current lock holder (`NPU Locked!` → -`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep -the journal for `Failed to load model` (the quoted xclbin path names the -missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks = -installed) against `/opt/fastflowlm/share/flm/xclbins/` (available). - -## Incident: qwen3.6-moe unreachable (2026-08-16) - -Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window. - -1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`. -2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below. -3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load. - -Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16. - -### Unknown-tag fallback (FLM behavior) - -When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`: - -- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs. -- FLM then loads the 1B model in place of the requested model. -- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error. - -**Correction (live-probed 2026-08-30):** the fallback no longer fires on the -current serve path — an unknown tag now answers HTTP 200 with a flat -`{"error":"Failed to load model!"}` body and nothing is loaded or served; -`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200. - -Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model. - -### Request compatibility: the `developer` role (found 2026-08-16) - -pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection: - -- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template. -- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`. -- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug. -- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`. -- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error. - -## Risks and Open Questions - -- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path. -- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only. -- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot - of Dankclaw killed it. The fleet law today is the opposite: every deployed - container/service ships a boot-persistent, self-healing unit, and the retired - NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it - cannot come back at boot. -- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this. -- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags. -- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then - served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the - `amd_iommu=off` decision that day and the unit is masked today. The remaining - NPU content (benchmarks, runbook steps) is research on a device that is again - usable since 2026-10-18 but serves nothing. - -## Citations - -1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16. -2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16. -3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16. -4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16. -5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16. -6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16. -7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html -8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`. -9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection. diff --git a/serverAppConfigs-docs/LLM/FreeToken Builds.md b/serverAppConfigs-docs/LLM/FreeToken Builds.md deleted file mode 100644 index 87caf65..0000000 --- a/serverAppConfigs-docs/LLM/FreeToken Builds.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -type: reference -title: FreeToken Builds -description: "Building the FreeToken runtime here: the CUDA and ROCm builds and the dependency trap each one hit." -tags: [freetoken, build, dankclaw, vllm, mtp] -timestamp: 2026-10-05T00:00:00-07:00 ---- - -# FreeToken Builds - -FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken -vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with -different dependency traps. Both were verified end-to-end 2026-09-11/12. - -## Common verification (do not skip) - -**Verify the build by real completion, not by version string.** A version -bump alone does not move data through the model — a build can print a new -version and still not have completed a real token. Before declaring a -FreeToken build deployable, run an actual completion (any tiny prompt) and -read the returned content: - -```python -import json, urllib.request -req = urllib.request.Request("http://:/v1/chat/completions", - json.dumps({"model": "frank-zealot", "messages": [{"role": "user", - "content": "Hi"}]}).encode(), {"Content-Type": "application/json"}) -print(json.load(urllib.request.urlopen(req, timeout=600))) -``` - -A 200 with a real `choices[0].message.content` is the only build green. - -## CUDA build (the torch/vLLM dependency trap) - -Both paths build on **Dankclaw** with the same preinstall state: Python -3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo -download.pytorch.org/whl/nightly/cu130, installed with -`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev+cu130" ---index-url https://download.pytorch.org/whl/nightly/cu130` — use -`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev -version). Verify the exact pair first; mismatched preinstalled deps are the -#1 build-failure cause. - -Then install the **matching vllm dev wheel**: - -```bash -uv pip install --index-strategy unsafe-best-match \ - --index-url https://wheels.vllm.ai//vllm-0.6.6.dev0+g.cu130-cp312-cp312-linux_x86_64.whl \ - --extra-index-url https://wheels.vllm.ai/nightly/cu130 -``` - -The wheel URL pattern is fixed: -`https://wheels.vllm.ai//vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl` -— read it back from vLLM's CI-run page (`run details → -build_and_install_package` job, "install the vllm wheel" step). **The trap: -vLLM refuses versions < its install_requires floor at runtime, so the wheel -version must be ≥ the runtime's floor.** A wheel built at a lower base -(0.5.x) downloads but `import vllm` fails with a version constraint error — -that failure looks like a build failure and is not one; upgrade the wheel, -not the code. - -If `uv pip install -e . --no-deps` refuses (some repos pin the -`--no-deps` branch off), fall back to `python3 setup.py` inside the venv, -after `python -m venv --copies .venv && source .venv/bin/activate` and -`uv pip install --no-build-isolation "torch==…"` — the venv must NOT -re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI). - -## ROCm build (gfx1151 via TheRock nightly) - -Same venv discipline, ROCm wheels from TheRock's nightly index: - -```bash -uv pip install --index-strategy unsafe-best-match \ - --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \ - "torch==2.10.0.dev+rocm" "triton==3.6.0" "pytorch-triton==3.6.0" -``` - -Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`, -repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two -prebuilt extension wheels into the tree (the repo's `setup.py` skips -compiling them when found). The ROCm build also needs -`install-fastrak-gfx1151.sh` from the repo's scripts and -`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's -x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub -`franksdank/freetoken-ops-gfx1151` — fetch it once and point -`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless -the extension API changed. - -`--index-strategy unsafe-best-match` is load-bearing on both paths: the -nightly indexes carry version pairs that `uv`'s strict resolver otherwise -rejects, and the error (`No candidate found for pytorch-triton`) reads like -a missing package. - -## Related - -- [[Model Swap and Rollback]] — the swap/deploy story these builds serve -- [[Llama Cpp Serving Sizing]] — the harness measurement discipline diff --git a/serverAppConfigs-docs/LLM/GGUF Compatibility.md b/serverAppConfigs-docs/LLM/GGUF Compatibility.md index f66f42e..d0fd028 100644 --- a/serverAppConfigs-docs/LLM/GGUF Compatibility.md +++ b/serverAppConfigs-docs/LLM/GGUF Compatibility.md @@ -14,7 +14,7 @@ Studio refuse custom types at load (`unknown tensor type`); a fork-private type means exactly one engine can serve the file. Measured on this fleet repeatedly: `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type -tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]] +tables live in and (ggml_type 100–108, ftype families 100–106 / 110–113 / 118). ## 1. Read the header, do not guess @@ -69,14 +69,14 @@ filenames lie about quantization and about MTP/vision heads. ``` Validate before trusting: the parse is right iff the tensor-info block ends - somewhere sane and names look real (`blk.0.attn_q.weight`, + somewhere sane and names look real (`blk.0.attn_q.weight` `token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or offset — do not work around it. 3. **Diff what engines actually bind on**: `general.architecture`; tensor count; tensor name list in order; per-tensor `ggml_type` and dims; geometry - KVs (`.block_count`, `.embedding_length`, `.feed_forward_length`, - `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`, + KVs (`.block_count`, `.embedding_length`, `.feed_forward_length` + `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length` `.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors (`visual`, `v_` prefixes). A same-family fine-tune quantized by the same tool usually comes out byte-identical in every one of these — that is the @@ -117,8 +117,8 @@ re-download. ## Related -- [[Llama Cpp Runtimes]] — which fork ships which custom types -- [[Llama Cpp Code Diff]] — the enum tables and code-level differences -* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and +- which fork ships which custom types +- the enum tables and code-level differences +* the era that served the ftype-115 file, and [[frank-dragoon]] for what runs there now - [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads diff --git a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md index 9952905..0f493b3 100644 --- a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md +++ b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md @@ -93,7 +93,7 @@ of bytes), so it is tempting. Before proposing it: (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between - models); + models) - a plain drafter costs **K sequential forwards per round**, so it must be 10–50× faster than the target. Break-even with `r = t_draft/t_target`: `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5` @@ -107,10 +107,10 @@ of bytes), so it is tempting. Before proposing it: (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`. The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device` -(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268), +(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268) `--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v` (`-ctkd`/`-ctvd`, 4111/4124) all exist in -`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp), +`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp) read 2026-10-18). ## 6. Expressing a split (llama.cpp, the usual vehicle) @@ -122,8 +122,8 @@ read 2026-10-18). - buffer types match by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values; -- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`, - `common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`; +- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe` + `common/arg.cpp:2764,2771`); device-side MoE placement is `-ot` - several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the @@ -155,7 +155,7 @@ Rank and report; prefer the boring answer when it wins: 1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link. 2. **One process, weights-resident, KV co-located with attention on the - fast pool** — promising and quantifiable (+% decode at long context); + fast pool** — promising and quantifiable (+% decode at long context) needs a dual-backend build and a baseline measurement. 3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check @@ -169,5 +169,5 @@ Rank and report; prefer the boring answer when it wins: ## Related - [[PCIe Link and Atomics]] — the link and collective walls underneath -- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes +- the deployed split shapes - [[Llama Cpp Serving Sizing]] — KV and pool arithmetic diff --git a/serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md b/serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md deleted file mode 100644 index 017109f..0000000 --- a/serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md +++ /dev/null @@ -1,225 +0,0 @@ ---- -type: reference -description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing." -tags: - - llm -timestamp: 2026-08-28 ---- - -# Hybrid CPU GPU Processing Dankserver - -## Overview - -Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon]]). - -## Architecture - -### Model Weights - -In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2). - -### KV Cache - -KV cache moves between GPU and CPU: - -1. Active KV cache pages stay in GPU VRAM -2. Inactive pages swap to CPU RAM -3. Swapped pages write to disk swap space if RAM is full -4. Pages reload to GPU when needed - -### Compute Distribution - -- GPU handles all matrix multiplication and attention computation -- CPU handles tokenization, detokenization, and input processing -- CPU handles KV cache swap operations -- GPU worker processes run on CPU cores via multiprocessing - -## Optimal Settings for Dankserver - -### GPU Memory Utilization - -Set `--gpu-memory-utilization` based on your GPU: - -| GPU | Recommended Value | -|-----|-------------------| -| 24 GB (RTX 3090/4090) | 0.7-0.8 | -| 40 GB (A100) | 0.8-0.9 | -| 80 GB (A100 80GB) | 0.85-0.95 | -| 80 GB (H100) | 0.9-0.95 | - -Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s. -The live engine does not use this knob at all — it claims an explicit per-card -pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`, -`scripts/frank-archon.env:230-231`). - -Lower values leave more GPU memory for KV cache. This reduces swap frequency. - -### Swap Space / KV Offload - -Current vLLM has no `--swap-space` engine argument, and the CPU KV-offload knob pair -is owned by [[RAM CPU Offloading in VLLM]] (that page carries the upstream -`CacheConfig` verification). In Podman, mount the host cache read-write for KV offload -writes. The table below shows historical `--swap-space` guidance as a sizing reference. - -| System RAM | Recommended Swap | -|------------|------------------| -| 32 GB | 8-16 GB | -| 64 GB | 16-32 GB | -| 128 GB | 32-64 GB | -| 256 GB | 64-128 GB | - -More swap space allows longer contexts and more concurrent requests. - -### Tensor Parallelism - -Use tensor parallelism for large models: - -``` -vllm serve meta-llama/Llama-3.1-70B-Instruct \ - --tensor-parallel-size 4 \ - --gpu-memory-utilization 0.85 -``` - -### KV Cache Data Type - -Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache). - -``` -vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ - --kv-cache-dtype fp8 \ - --gpu-memory-utilization 0.9 -``` - -FP8 KV cache reduces memory usage by 50% compared to BF16. - -## NUMA Optimization - -Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies: - -``` -vllm serve model \ - --tensor-parallel-size 4 \ - --numa-bind -``` - -This pins each GPU worker to its nearest NUMA node. This reduces memory access latency. - -For explicit NUMA binding: - -``` -vllm serve model \ - --tensor-parallel-size 4 \ - --numa-bind \ - --numa-bind-nodes 0 0 1 1 -``` - -## CPU Resource Requirements - -VLLM needs CPU cores for: - -- 1 API server process -- 1 engine core process -- N GPU worker processes (one per GPU) - -Minimum physical cores = 2 + N (where N is GPU count). - -For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read -2026-10-18): - -- Minimum 4 physical CPU cores by the 2 + N rule -- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off) - -## Performance Tuning - -### Chunked Prefill - -Chunked prefill is enabled by default. Tune with: - -``` -vllm serve model \ - --max-num-batched-tokens 8192 -``` - -Higher values improve throughput. Lower values improve latency. - -### CUDA Graphs - -CUDA graphs improve throughput. Disable for faster startup: - -``` -vllm serve model \ - --enforce-eager -``` - -### Compilation Cache - -Reuse torch.compile artifacts: - -``` -export VLLM_CACHE_ROOT=~/.cache/vllm -vllm serve model -``` - -Copy cache between boots for faster startup. - -### Skip Memory Profiling - -Skip memory profiling on repeated boots: - -``` -vllm serve model \ - --kv-cache-memory -``` - -## Monitoring - -Check GPU memory: - -``` -rocm-smi -``` - -Check for preemption warnings in VLLM logs. - -Check CPU utilization: - -``` -htop -``` - -## Example Commands - -### Single GPU, Medium Model - -``` -vllm serve Qwen/Qwen3.6-35B-A3B \ - --gpu-memory-utilization 0.85 \ - --max-num-batched-tokens 4096 \ - --max-num-seqs 128 -``` - -### Single GPU, FP8 Model - -``` -vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ - --kv-cache-dtype fp8 \ - --gpu-memory-utilization 0.9 \ - --max-num-batched-tokens 8192 -``` - -### Multi-GPU, Large Model - -``` -vllm serve meta-llama/Llama-3.1-70B-Instruct \ - --tensor-parallel-size 4 \ - --gpu-memory-utilization 0.9 \ - --max-model-len 8192 -``` - -## See Also - -- [[VLLM Overview]] -- [[VLLM Index]] -- [[RAM CPU Offloading in VLLM]] -- [[Shared Memory Architecture Dankclaw]] -- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] diff --git a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md deleted file mode 100644 index 1668a50..0000000 --- a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -type: reference -title: LLM Bench Windows and Measurement -description: "Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting." -tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet] -timestamp: 2026-10-05T00:00:00-07:00 ---- - -# LLM Bench Windows and Measurement - -Fleet law first: **STOP the production service before any GPU experiment** -(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the -GPU and corrupts results; an unisolated GPU experiment has precedent for -wedging a host until power cycle. Reference harnesses: -`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`, -`scripts/autoresearch/frank-archon-arms.sh`. - -## 1. The window structure (one config per arm, a driver owns the window) - -1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND - no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts - EXITED containers and can never clear — `rm -f` the exited one first) AND - port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01: - `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s - then polls `/health` gets **200 from the still-running PRODUCTION engine** - — the arm "passes" with production's boot log, then `podman exec` dies - with `container state improper`. -2. **Boot the arm as the service, not a scratch port** (no scratch ports on - LLM hosts — one card, one server). The serve script takes its config via - env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`) - pointing at a staged copy; the checkout is never mutated. -3. **Fast-fail the health wait**: poll `podman ps | grep -qx ` inside - the loop — a refused boot dies in one poll instead of 20 min. Capture the - serve script's FULL stdout on failure (a `tail -8` loses engine-init root - causes; raise happens above the tail). -4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots, - SpecDecoding lines) — they survive measure crashes. -5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills), - `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver - before the next boot; restore production from the EXIT trap and verify the - **conjunction oracle**: health 200 + `is-active` + `is-enabled` + - mask-symlink clean. A bare `systemctl start` is not a restore oracle. - (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is - enough: systemd never auto-restarts an explicitly stopped unit.) -6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for - mixed speeds. - -**Destructive tails (2026-10-01):** a scratch script whose tail does -`systemctl stop` + `podman rm -f ` fires later if a -probe loop survived the driver kill — it can destroy a just-deployed -production engine (signature: deploy exits green, then seconds-to-minutes -later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not -crashed). Before ANY deploy: `pgrep -af` every scratch script of the session -and kill strays by explicit PID; killing the dev-side job does NOT kill the -host-side `deploy-remote.sh` (two of them once managed the same unit). - -Two failure shapes inside the window that read as "the bench died": - -- **Stale reads from a stopping container.** `podman stop` answers while the - engine inside is still flushing; a `podman logs` taken in that window - returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or - `podman wait`) before reading logs, or you will write "no results" into - the table for a run that has them. -- **A D-state worker wedges the stop.** A bench child that wedged in the - kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and - then the container is `Stopping` forever. Kill by container first - (`podman kill ` → SIGKILL path), verify the process is gone from - `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the - window script proceed to a redeploy with a zombie bench worker holding - VRAM. - -## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth) - -**What llama-bench cannot express — decide BEFORE designing:** no -`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove -activation buffers fit beside the DEPLOYED KV — only the real unit answers -that), no `--parallel` (no slot shape, no concurrency), no `--spec-*` -(`tg` numbers are the drafterless baseline, useful as a control only), and a -warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per -cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at -~zero depth and meaningless for long conversations. - -**The separator trap:** `/` separates devices WITHIN one configuration, `,` -separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the -single value 36 → everything on one card. `--tensor-split 36,29` (server) → -`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`. - -**Harness shape:** one invocation per cell (an infeasible cell calls -`exit(1)` and kills every later cell of a cross-product invocation — and -infeasibility is usually the question); TSV row per cell as it completes + -`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi ---showmeminfo vram` VRAM trace per cell (a cell that allocated less than -intended looks fast); resume by skipping non-empty cell outputs; stop the -service in the harness, restore from `trap … EXIT`, verify afterwards. Match -the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`). - -**Run under a transient unit, not a shell:** - -```bash -sudo systemd-run --unit= --collect --no-block --property=Type=oneshot \ - --property=LimitMEMLOCK=infinity --property=User=dank \ - /home/dank/serverAppConfigs/scripts/.sh -``` - -Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has -8 MiB and fails the load for an unrelated reason. This also survives ssh -disconnects. **Boot rungs under the production environment, not a bench -shell** — a rootless shell probe booted 3×262144 bf16 that production refused -(systemd `User=` + memlock change what the pinned-expert path holds): -`sudo systemd-run --uid= --unit= -p LimitMEMLOCK=infinity -p -Restart=no env VARFILE=… /path/serve.sh`. - -**Verify before burning hours:** dry-run against a STUB binary with -`SERVICE=` (the trap/stop never reached), tiny grid, -including one forced-`exit 3` cell — prove the TSV row, failure capture, -resume logic. Validate the real flag set in ~1 s by pointing `-m` at a -nonexistent file. **Reading:** bracket the sweep (reference cell first AND -last); check per-cell `stddev_ts` (a difference smaller than spread is not a -result); single-sequence decode is usually flat across batch/ubatch (prefill -knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating -smaller-batch cells); report what the grid cannot prove (drafter, slots, real -context). - -## 3. HTTP benches of server-side parameters - -`llama-bench` benches the bench binary, not the running server — server-side -params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need -HTTP: - -- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure - `prompt_tokens / wall_time`. -- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and - spec acceptance all stabilize over thousands of tokens); measure - `predicted_per_second` from `timings` or `completion_tokens / wall_time`; - temperature 0. - -Known fleet harnesses: `scripts/bench-threads-batch-http.py`, -`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`. -sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3 -reasoning parser streams zero deltas — use `/v1/completions`. - -## 4. The measurement law (under speculative decoding this is absolute) - -**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token -mapping; chunk counting reported 8.3 tok/s for a config whose server counters -said 23–26 — a 3x false regression that nearly abandoned the change that -worked. Always: - -```python -decode_tok_s = usage.completion_tokens / (wall - ttft) -# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk -``` - -vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a -measurement (6.2 → 13.1 across adjacent windows on identical settings) — use -them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean -acceptance length`, per-position rates). Warm pass + ≥3 runs, median; -temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions -(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's -repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text -before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample -nearly shipped the wrong setting. - -## 5. Honest methodology - -- **Match the corpus to the workload.** Token soup under-reports speculative - decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft - settings; acceptance, not step time, moves throughput. Build the corpus - from the repo's own sources: `{"prompt": "\n\n"}`, - `--custom-output-len`. -- **Salt every run** with a unique per-arm token in the first bytes — it is - the prefix-cache key (and a proxy routing key). Without it arm 2 measures - arm 1's warm cache. -- **Two workload shapes need two clients:** distinct prompts (batch - behaviour, prefill parallelism) and sequential turns with growing history - (TTFT of the first streamed chunk per turn — the only measurement that - exposes prefix-cache behaviour). These pull in opposite directions: a cold - batch spreads prefill across replicas, a conversation's next turn only pays - if it lands where its prefix is cached. -- **Read the engine's ledger, not the card:** `Available KV cache memory` / - `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per - request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line - is a LIFETIME average across every prompt the replica ever saw — send the - same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold - → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not - full; it does NOT say a prefix survived. The KV budget is NOT monotonic in - chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure - three points before treating a knob as a lever. -- **Soak before believing:** 15–30 min of concurrent families, count failures - AND latency; report hit/miss split and medians with min/max — the median of - a bimodal distribution describes neither mode. -- **Verification that is not the benchmark:** run the client the fleet - actually uses; and a deployed unit is not proven until an unattended reboot - brings it back. -- **Record the "no" answers** with numbers and mechanism — placement balance, - concurrency admission and chunk-size tweaks all measured flat or worse - here; the wall was arithmetic (cached prefixes + live sequences share one - pool). - -## 6. Sweep hygiene (the arms that produced wrong verdicts) - -- **Prefetch uniformity:** every arm in a sweep carries the same - expert-prefetch setting. The 2026-09-29 campaign measured losing arms with - prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec - 12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed - verdict. -- **Knob-landed preflight:** for every `-e` knob, confirm - `podman exec printenv ` — a knob the fork silently ignores - produces a "parity" arm that is really the default (v1 of the gather sweep - lost all 3 arms this way; serve-script hooks are additive and can drop - vars). -- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was - deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers - compare against default-off, not "incumbent + one flag". -- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than - autofit (remainder) so pool numbers are reproducible; compute need = - tokens × measured B/token (hybrid models: KV dtype does NOT double mamba - states — measure, don't assume 2x). -- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly - deployed a 2x-prefill-tax config. Per-depth tables, never a single median — - one fork's gather win grows with depth (54 t/s @1k, 39.5 @short). -- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy - `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them. - -## 7. Bash laws that ate windows (each cost an arm) - -1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word → - `SPEC=3: command not found`, nothing runs. Pass literal assignments: - `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty. -2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside - an if/else it aborts ONLY that compound statement — the script CONTINUES, - every arm tears down clean with zero measurements. Detection law: `ls` - the per-arm `.measure.txt` files before trusting any summary; arm echoes - are not evidence. -3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u` - it aborts AFTER the harness stopped production. Two statements. -4. `sed -i` renames the file; already-running bash keeps the OLD inode — - mid-flight patches miss arms launched in the preceding minutes. Patch - early or run a make-up window. -5. Never heredoc python through ssh→zsh→bash — write to a file, ship it - (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting - fails silently. Same trap in `git commit -m`: backticks become command - substitution. -6. No comments INSIDE a backslash-continued command (the continuation joins - the comment); verify multi-line command construction with `bash -n` AND a - render test. - -## 8. Results location and recovery (frank-archon arms) - -| copy | path | -|---|---| -| per-arm evidence | `$ARMS_DIR` = `~/.jobs//` (`.rc`, `.done` beside it) | -| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `.out/.env/.overlay/.rc/.metrics` | -| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` | -| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) | -| fallback | `~/.omp/agent/sessions///.{shake,bash}.log` (permanent) | - -`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests — -constants 99.9/88.8 identical across arms by construction; **identical -numbers across arms == stub or bug, never a measurement.** After compaction, -re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session -logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per -window. - -## Related - -- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept -- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes -- [[LLM Performance Diagnosis]] — profiling and attribution methods -- [[Remote Job Runner]] — detached execution of long sweeps -- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 7ccd078..9a493df 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -21,7 +21,7 @@ the cost; the problem is per-token expert movement and KV/attention tuning is ruled out entirely. A genuinely context-bound server shows a clear downward slope. Record cold vs warm separately — a small gap means the cache is working. The canonical flat run (fp8 expert-offload shape, a few hundred to -sixteen thousand tokens of context) is in [[Tcclaviger Fork Ops]]'s profile. +sixteen thousand tokens of context) is in 's profile. ## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer) @@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s): No amount of link arithmetic would have produced those two conclusions: the suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk — -whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also +whose own boot log names the cause (`FULL_DECODE_ONLY` override). It predicted a negative result in advance (spec decode cannot help a shape whose per-token cost scales with the token count) which matched an unexplained flat MTP measurement. @@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing. - One `max_tokens=200` request is production traffic and fine; anything that pins the card is a bench → stop-and-mask first. Check WHICH instance you are reading (`systemctl status | head -12`; task ids jump across - restarts). The engine's own ledger (`KV self size`, `n_slots`, + restarts). The engine's own ledger (`KV self size`, `n_slots` `n_ctx_slot`) outranks your model of it. ## 5. The fork's data path (before blaming the link) @@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as draft depth grows. Check the threshold's premise: a default tuned at 12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone -does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required); +does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required) check for the skip line before concluding autotuning ran. Gather-geometry knobs read at import time → each arm is a full reload. @@ -189,7 +189,7 @@ analyze; `--analyze-only`). Measure with a **sequential-turn client**, not only a batch benchmark — a batch of distinct prompts parallelises prefill across replicas and hides the problem. The two failure modes: round-robin loses every other cache hit -(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned); +(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned) pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s pinned). **The rule that gets both: route by request shape** — a request carrying more than one message is a continuation (agent loop re-sending its @@ -209,13 +209,13 @@ is not a per-request verdict (probe directly: same prompt twice to an idle replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes; `vllm bench serve` needs `--tokenizer `; verify metric names against the live endpoint before dashboards. Test shape: drive the routing -function directly with synthetic bodies (stdlib-only policy module); +function directly with synthetic bodies (stdlib-only policy module) conversation identity is the FIRST N KiB of prompt — a shorter tested prefix hashes differently per turn and pinning silently never happens. ## Related -- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law +- the window discipline and usage-counter law - [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms -- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound +- the compile fix for the pointwise-glue bound diff --git a/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md b/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md deleted file mode 100644 index 896f0ab..0000000 --- a/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md +++ /dev/null @@ -1,287 +0,0 @@ ---- -type: reference -title: Llama Cpp Multi-Device Splits -description: "What llama.cpp can express across devices: the no-collective layer-split path, balancing, and drafter KV sizing." -tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative] -timestamp: 2026-10-05T00:00:00-07:00 ---- - -# Llama Cpp Multi-Device Splits - -## 1. Expressibility law — classify the arrangement before running anything - -llama.cpp expresses exactly two things across devices, and they are not -interchangeable: - -| arrangement | how | expressible? | -|---|---|---| -| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts /` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) | -| **Per-tensor placement** — attention+KV on A, experts on B | `-ot =` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). | - -Upstream anchors, re-read 2026-10-18 against `master`: -`src/llama-context.cpp:517-526` computes `pipeline_parallel` with -`!model.has_tensor_overrides()` as a required term; -`ggml/src/ggml-backend.cpp:1089` carries the verbatim abort -`pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)` -(the `:941` anchor above is where it sat in the `37b53fd` build this was -measured on); and `common/arg.cpp` defines `-sm/--split-mode` (2821), -`-fit/--fit` (2882), `--spec-type` (4295), `--spec-draft-device` (4259), -`--spec-draft-n-max` (4161), `-fa/--flash-attn` (1758) — every flag in §4 is -real upstream, not fork-only -([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)). - -Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1` -with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a -cross-backend pair → model load fails with only `failed to load model`; -`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU -is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host -a card can be the trunk OR the offload host, never the complement of an -AMD-resident peer. Pure Vulkan changes nothing here — the gate is the -override, not the driver mix. - -If the plan needs tensors on a specific device: stop, the answer is no for -any vendor mix. Expressible alternatives, in order: (1) two independent -endpoints, one model per device; (2) single-vendor layer split; (3) device + -CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork -was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can -land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB). - -Layer-split arithmetic to check before promising: a whole layer carries its -experts (MoE) — compute GiB/layer, not per param-group; only some layers carry -KV (every 4th on the hybrid), so a share of layers is only a share of KV; big -non-repeating tensors (PLE tables) land on the main device and cannot move — -they cap how small the primary can get. - -## 2. No collectives? The atomics wall and the layer-split answer - -Decide with one line before designing: - -```bash -sudo lspci -vv -s | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever -lspci -tv # a Routing- bridge is a hardware wall -``` - -`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a -device-backend `send`/`recv` of the hidden state -(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that -fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no -collective: layers distribute in order, only the boundary activation crosses -PCIe once per forward pass. Full probe/repair detail: -[[PCIe Link and Atomics]]. - -## 3. Build: pin by feature, bundle the runtime at the exact depth - -- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 → - `ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way - (`ls /…/spec_decode/ # dflash/ vs dflash2/`). -- `-DCMAKE_HIP_ARCHITECTURES=` — a sibling arch links, loads, then - segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor - mode degrades to the butterfly fallback; layer split unaffected). -- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one - directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth - (`../lib/rocblas/library`, `../lib/hipblaslt/library/`); flattened, - the engine loads every GiB, answers `/health`, and dies on the FIRST request - with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU - arch`. `mkdir -p DST && cp -r SRC/. DST/`, then - `patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`. -- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not - coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs - `glslc` + libvulkan headers. **Device names are not what you expect:** - `--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND - `Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes - llama.cpp print the valid set and exit (self-diagnosing). -- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its - CMakeCache. - -## 4. Serve flags that matter on a split - -```bash -HIP_VISIBLE_DEVICES=0,1 llama-server \ - --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \ - --gpu-layers 999 --fit off --flash-attn on \ - --ctx-size --parallel 1 --cache-type-k f16 --cache-type-v f16 \ - --jinja --metrics --cache-prompt \ - --spec-type draft-dflash --spec-draft-model \ - --spec-draft-device ROCm1 --spec-draft-n-max 7 -``` - -- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set - on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx - is the concurrency policy, concurrent agent loops queue. -- Pin the drafter to the card holding the target's LAST layers - (`--spec-draft-device`); under tensor mode an unpinned draft joins the - meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must - be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7). -- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe - low/medium/high/xhigh and map harness levels before declaring ready. - -## 5. Split balancing: measure, never interpolate - -**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.** -qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on -a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that -layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a -single step; the best integer split can sit 0.2 GiB from balance while its -neighbours are 1+ GiB off. An affine fit reproduces the deployed point and -overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as -a gate; treat a NEGATIVE fitted constant as a unit bug). - -Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute -buffers are **~2x heavier on the output/drafter card** — 512→1536 added -~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV), -`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits — -only the distribution moves; use that to sanity-check a suspicious row. - -**Procedure (one split per load):** - -```bash -WORK=/tmp/split.env; cp scripts/serve-.env $WORK -export FRANK_MEDIUM_ENV_FILE=$WORK -for ts in 39,26 38,27 37,28 36,29 35,30; do - sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK - nohup bash scripts/serve-.sh >/tmp/split.log 2>&1 & - # poll /health (or process exit), then: - rocm-smi --showmeminfo vram | grep 'VRAM Total Used' # sample ~6 s after health - pkill -x llama-server; sleep 6 -done -``` - -Load → sample → kill, NO requests sent (allocation only). Run under `nohup` -and poll from a separate call (a load is 30–90 s). Pick the smallest -inter-card gap that clears the per-card floor with room to spare; one `-ts` -step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket, -not to solve. Prefer balance: the busy card's headroom bounds the context -(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier -card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` → -`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does -not prevent a boundary-copy fault. - -## 6. Drafter KV: sized from the TARGET, bounded by its window policy - -`common/speculative.cpp` builds the draft context with -`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — `common/speculative.cpp:2663` on -`master`, re-read 2026-10-18 — **no flag caps it.** So: - -``` -draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype -``` - -…unless the drafter declares a sliding window. Measured at 262,144 target -ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter -with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the -balance point, and swapping them invalidated an already-balanced split. - -**Failure signature:** the engine loads the whole model, answers `/health`, -and dies on the FIRST draft decode with `ROCm error: out of memory` / -`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` — -a sizing failure, not a broken drafter; under a self-healing unit it presents -as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`, -`key_length` from the drafter's GGUF header first. A compact reader (the -three format traps: no KV alignment — v3 alignment applies only to the tensor -section; arrays are `type(9) → element_type(u32) → count(u64)`, element type -FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY -is the v2/v3 file order, NOT gguf.h's 2/3): - -```python -import struct, sys -def rd(path): - f = open(path, 'rb'); assert f.read(4) == b'GGUF' - struct.unpack('