From a34322aeb036d0941d13eb1ba77a83cf38f7c760 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 9 Oct 2026 13:17:48 -0700 Subject: [PATCH] refactor(skills): consolidate managed skill library 94 -> 25 The managed-skill library was regenerated: 94 narrow per-incident skills replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/ endpoint ops). Git renders this as mass delete + mass add because whole directories changed name; old content stays in history at 12c5e88. Surface check: the 25 on disk are exactly the skills the harness advertises this session. --- .../amd-multigpu-cooperation-triage/SKILL.md | 71 -------- .../anubis-guard-chain-flip/SKILL.md | 25 --- .../arr-auth-posture-probe/SKILL.md | 20 --- .../arr-stack-api-config/SKILL.md | 30 ---- .../arr-v1-api-validator-peeling/SKILL.md | 29 --- .../authentik-claim-cache-purge/SKILL.md | 38 ---- .../authentik-flow-ui-automation/SKILL.md | 46 ----- .../authentik-outpost-verification/SKILL.md | 34 ---- .../authentik-provision-fleet-user/SKILL.md | 42 +++++ .../SKILL.md | 52 ------ .../autoresearch-llm-bench-window/SKILL.md | 76 ++++++++ .../bench-llama-server-params/SKILL.md | 69 ------- .../SKILL.md | 29 +++ .../command-center-dotfiles-import/SKILL.md | 35 ---- .../SKILL.md | 64 ------- .../compose-to-quadlet-cutover/SKILL.md | 72 -------- .../container-config-drift-crashloop/SKILL.md | 82 --------- .../SKILL.md | 33 ---- .../dankclaw-homepage-asset-verify/SKILL.md | 13 ++ .../defuse-shell-rc-secrets/SKILL.md | 21 +++ .../detached-fleet-job-launch/SKILL.md | 46 +++++ .../SKILL.md | 72 -------- .../fandom-wiki-media-fetch/SKILL.md | 19 ++ .../fleet-container-ownership-audit/SKILL.md | 91 ---------- .../fleet-llm-payload-purge-refresh/SKILL.md | 117 ------------ .../fleet-model-cache-purge-refresh/SKILL.md | 73 -------- .../fleet-monitoring-gap-audit/SKILL.md | 79 -------- .../fleet-second-llm-endpoint/SKILL.md | 97 ---------- .../SKILL.md | 64 ------- .../flm-npu-pack-registration/SKILL.md | 27 --- .../flm-tag-load-failure-diagnosis/SKILL.md | 34 ---- .../frank-medium-batch-split-retune/SKILL.md | 95 ---------- .../frank-primary-decode-tuning/SKILL.md | 22 +++ .../frankbook-qbittorrent-e2e-verify/SKILL.md | 57 ------ .../freetoken-build-dankclaw/SKILL.md | 55 ------ .../freetoken-dankclaw-build/SKILL.md | 39 ---- .../freetoken-hf-model-swap/SKILL.md | 36 ---- .../freetoken-rocm-build/SKILL.md | 49 ----- .../SKILL.md | 94 ---------- .../grafana-provisioning-as-code/SKILL.md | 92 ---------- .../gui-hang-remote-diagnosis/SKILL.md | 46 ----- .../halogen-flash-server-ops/SKILL.md | 67 ------- .../halogen-kv-pool-fit/SKILL.md | 26 --- .../hermes-config-home-drift/SKILL.md | 18 -- .../hermes-live-config-edit/SKILL.md | 21 --- .../hermes-local-aux-routing/SKILL.md | 87 --------- .../hermes-webui-secret-rotation/SKILL.md | 20 --- .../SKILL.md | 94 ---------- .../hf-checkpoint-profile/SKILL.md | 26 +++ .../hf-gguf-payload-fetch-verify/SKILL.md | 98 ---------- .../hf-large-artifact-publication/SKILL.md | 40 +++++ .../homepage-smoke-verify-loop/SKILL.md | 21 +++ .../SKILL.md | 32 ---- .../homepage-theme-smoke/SKILL.md | 15 ++ .../html-change-report/SKILL.md | 34 ---- .../ik-llama-hybrid-moe-performance/SKILL.md | 29 --- .../imagegen-openwebui/SKILL.md | 43 ----- .../incident-window-reconstruction/SKILL.md | 92 ---------- .../lan-client-split-dns-fix/SKILL.md | 36 ---- .../live-llm-benchmark-hygiene/SKILL.md | 70 -------- .../live-llm-telemetry-no-bench/SKILL.md | 87 --------- .../llama-bench-fleet-sweep/SKILL.md | 51 ------ .../llama-bench-host-sweep/SKILL.md | 109 ------------ .../llama-cpp-spec-decode-ab/SKILL.md | 58 ------ .../llama-flash-attn-qsa-fix/SKILL.md | 73 -------- .../llama-server-arch-verify/SKILL.md | 50 ++++++ .../SKILL.md | 97 ---------- .../SKILL.md | 62 ------- .../llamacpp-drafter-kv-split-sizing/SKILL.md | 168 ------------------ .../llamacpp-endpoint-model-swap/SKILL.md | 28 +++ .../llamacpp-layer-split-rdna4/SKILL.md | 113 ------------ .../SKILL.md | 70 -------- .../SKILL.md | 132 -------------- .../SKILL.md | 88 --------- .../llamacpp-serve-unit-boot-loop/SKILL.md | 27 +++ .../llamacpp-vram-accounting/SKILL.md | 77 -------- .../llm-bench-noise-discipline/SKILL.md | 27 +++ .../llm-fleet-model-swap/SKILL.md | 41 ----- .../llm-model-swap-fit-preflight/SKILL.md | 50 ------ .../llm-replica-routing-prefix-cache/SKILL.md | 100 ----------- .../SKILL.md | 67 ------- .../niri-output-rotation-pinning/SKILL.md | 96 ---------- .../noctalia-config-remote-edit/SKILL.md | 28 --- .../noctalia-custom-palette-recolor/SKILL.md | 67 ------- .../noctalia-reference/SKILL.md | 32 ---- .../noctalia-template-offline-render/SKILL.md | 45 ----- .../omp-harness-window-fit-audit/SKILL.md | 24 +++ .../omp-per-model-sampler-config/SKILL.md | 60 ------- .../openwebui-picker-acl-repair/SKILL.md | 29 +++ .../pcie-link-capability-probe/SKILL.md | 122 ------------- .../SKILL.md | 101 ----------- .../pico8-asset-standards/SKILL.md | 46 ----- .../podman-netavark-host-port-probe/SKILL.md | 29 --- .../SKILL.md | 51 ------ .../podman-recreate-anon-volume-wipe/SKILL.md | 28 --- agent/managed-skills/porkbun-dns-ops/SKILL.md | 26 --- .../published-port-cutover-gate/SKILL.md | 88 --------- .../quadlet-v1-generator-quirks/SKILL.md | 29 --- .../radiance-frank-primary-benching/SKILL.md | 32 ++++ .../radiance-precision-selection/SKILL.md | 31 ++++ .../radiance-quantize-and-score/SKILL.md | 26 +++ .../radiance-w4-campaign/SKILL.md | 18 ++ .../radiance-w4-convert-playbook/SKILL.md | 29 +++ .../SKILL.md | 109 ------------ .../rootless-quadlet-log-retrieval/SKILL.md | 42 ----- .../spec-decode-ab-single-gpu-host/SKILL.md | 77 -------- .../spec-decode-worth-it-triage/SKILL.md | 70 -------- .../ssh-fish-host-remote-exec/SKILL.md | 16 -- .../ssh-key-bootstrap-no-sshpass/SKILL.md | 64 ------- .../SKILL.md | 77 -------- .../SKILL.md | 88 --------- .../stale-signal-audit/SKILL.md | 71 -------- .../SKILL.md | 33 ---- .../static-smoke-server-verify/SKILL.md | 22 +++ .../status-surface-dead-signal-audit/SKILL.md | 56 ------ .../subagent-yield-failure-recovery/SKILL.md | 27 +++ .../swapfile-resize-safe-ordering/SKILL.md | 62 ------- .../vhost-proxy-health-probes/SKILL.md | 30 ---- .../wol-wake-sleeping-fleet-host/SKILL.md | 46 ----- 119 files changed, 740 insertions(+), 5816 deletions(-) delete mode 100644 agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md delete mode 100644 agent/managed-skills/anubis-guard-chain-flip/SKILL.md delete mode 100644 agent/managed-skills/arr-auth-posture-probe/SKILL.md delete mode 100644 agent/managed-skills/arr-stack-api-config/SKILL.md delete mode 100644 agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md delete mode 100644 agent/managed-skills/authentik-claim-cache-purge/SKILL.md delete mode 100644 agent/managed-skills/authentik-flow-ui-automation/SKILL.md delete mode 100644 agent/managed-skills/authentik-outpost-verification/SKILL.md create mode 100644 agent/managed-skills/authentik-provision-fleet-user/SKILL.md delete mode 100644 agent/managed-skills/authentik-proxy-provider-app-link/SKILL.md create mode 100644 agent/managed-skills/autoresearch-llm-bench-window/SKILL.md delete mode 100644 agent/managed-skills/bench-llama-server-params/SKILL.md create mode 100644 agent/managed-skills/closed-engine-runtime-investigation/SKILL.md delete mode 100644 agent/managed-skills/command-center-dotfiles-import/SKILL.md delete mode 100644 agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md delete mode 100644 agent/managed-skills/compose-to-quadlet-cutover/SKILL.md delete mode 100644 agent/managed-skills/container-config-drift-crashloop/SKILL.md delete mode 100644 agent/managed-skills/dankclaw-guard-chain-layer-diagnosis/SKILL.md create mode 100644 agent/managed-skills/dankclaw-homepage-asset-verify/SKILL.md create mode 100644 agent/managed-skills/defuse-shell-rc-secrets/SKILL.md create mode 100644 agent/managed-skills/detached-fleet-job-launch/SKILL.md delete mode 100644 agent/managed-skills/e2e-arr-rawip-reachability-triage/SKILL.md create mode 100644 agent/managed-skills/fandom-wiki-media-fetch/SKILL.md delete mode 100644 agent/managed-skills/fleet-container-ownership-audit/SKILL.md delete mode 100644 agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md delete mode 100644 agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md delete mode 100644 agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md delete mode 100644 agent/managed-skills/fleet-second-llm-endpoint/SKILL.md delete mode 100644 agent/managed-skills/fleet-service-precommit-iteration/SKILL.md delete mode 100644 agent/managed-skills/flm-npu-pack-registration/SKILL.md delete mode 100644 agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md delete mode 100644 agent/managed-skills/frank-medium-batch-split-retune/SKILL.md create mode 100644 agent/managed-skills/frank-primary-decode-tuning/SKILL.md delete mode 100644 agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md delete mode 100644 agent/managed-skills/freetoken-build-dankclaw/SKILL.md delete mode 100644 agent/managed-skills/freetoken-dankclaw-build/SKILL.md delete mode 100644 agent/managed-skills/freetoken-hf-model-swap/SKILL.md delete mode 100644 agent/managed-skills/freetoken-rocm-build/SKILL.md delete mode 100644 agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md delete mode 100644 agent/managed-skills/grafana-provisioning-as-code/SKILL.md delete mode 100644 agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md delete mode 100644 agent/managed-skills/halogen-flash-server-ops/SKILL.md delete mode 100644 agent/managed-skills/halogen-kv-pool-fit/SKILL.md delete mode 100644 agent/managed-skills/hermes-config-home-drift/SKILL.md delete mode 100644 agent/managed-skills/hermes-live-config-edit/SKILL.md delete mode 100644 agent/managed-skills/hermes-local-aux-routing/SKILL.md delete mode 100644 agent/managed-skills/hermes-webui-secret-rotation/SKILL.md delete mode 100644 agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md create mode 100644 agent/managed-skills/hf-checkpoint-profile/SKILL.md delete mode 100644 agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md create mode 100644 agent/managed-skills/hf-large-artifact-publication/SKILL.md create mode 100644 agent/managed-skills/homepage-smoke-verify-loop/SKILL.md delete mode 100644 agent/managed-skills/homepage-status-probe-namespace-triage/SKILL.md create mode 100644 agent/managed-skills/homepage-theme-smoke/SKILL.md delete mode 100644 agent/managed-skills/html-change-report/SKILL.md delete mode 100644 agent/managed-skills/ik-llama-hybrid-moe-performance/SKILL.md delete mode 100644 agent/managed-skills/imagegen-openwebui/SKILL.md delete mode 100644 agent/managed-skills/incident-window-reconstruction/SKILL.md delete mode 100644 agent/managed-skills/lan-client-split-dns-fix/SKILL.md delete mode 100644 agent/managed-skills/live-llm-benchmark-hygiene/SKILL.md delete mode 100644 agent/managed-skills/live-llm-telemetry-no-bench/SKILL.md delete mode 100644 agent/managed-skills/llama-bench-fleet-sweep/SKILL.md delete mode 100644 agent/managed-skills/llama-bench-host-sweep/SKILL.md delete mode 100644 agent/managed-skills/llama-cpp-spec-decode-ab/SKILL.md delete mode 100644 agent/managed-skills/llama-flash-attn-qsa-fix/SKILL.md create mode 100644 agent/managed-skills/llama-server-arch-verify/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-cross-backend-device-split/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-device-split-feasibility/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-drafter-kv-split-sizing/SKILL.md create mode 100644 agent/managed-skills/llamacpp-endpoint-model-swap/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-layer-split-rdna4/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-layer-split-vram-rebalance/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-mtp-vision-endpoint-bringup/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-multigpu-layer-split-cutover/SKILL.md create mode 100644 agent/managed-skills/llamacpp-serve-unit-boot-loop/SKILL.md delete mode 100644 agent/managed-skills/llamacpp-vram-accounting/SKILL.md create mode 100644 agent/managed-skills/llm-bench-noise-discipline/SKILL.md delete mode 100644 agent/managed-skills/llm-fleet-model-swap/SKILL.md delete mode 100644 agent/managed-skills/llm-model-swap-fit-preflight/SKILL.md delete mode 100644 agent/managed-skills/llm-replica-routing-prefix-cache/SKILL.md delete mode 100644 agent/managed-skills/llm-serving-benchmark-methodology/SKILL.md delete mode 100644 agent/managed-skills/niri-output-rotation-pinning/SKILL.md delete mode 100644 agent/managed-skills/noctalia-config-remote-edit/SKILL.md delete mode 100644 agent/managed-skills/noctalia-custom-palette-recolor/SKILL.md delete mode 100644 agent/managed-skills/noctalia-reference/SKILL.md delete mode 100644 agent/managed-skills/noctalia-template-offline-render/SKILL.md create mode 100644 agent/managed-skills/omp-harness-window-fit-audit/SKILL.md delete mode 100644 agent/managed-skills/omp-per-model-sampler-config/SKILL.md create mode 100644 agent/managed-skills/openwebui-picker-acl-repair/SKILL.md delete mode 100644 agent/managed-skills/pcie-link-capability-probe/SKILL.md delete mode 100644 agent/managed-skills/pcie-link-degradation-vs-idle-triage/SKILL.md delete mode 100644 agent/managed-skills/pico8-asset-standards/SKILL.md delete mode 100644 agent/managed-skills/podman-netavark-host-port-probe/SKILL.md delete mode 100644 agent/managed-skills/podman-netns-addressability-probe/SKILL.md delete mode 100644 agent/managed-skills/podman-recreate-anon-volume-wipe/SKILL.md delete mode 100644 agent/managed-skills/porkbun-dns-ops/SKILL.md delete mode 100644 agent/managed-skills/published-port-cutover-gate/SKILL.md delete mode 100644 agent/managed-skills/quadlet-v1-generator-quirks/SKILL.md create mode 100644 agent/managed-skills/radiance-frank-primary-benching/SKILL.md create mode 100644 agent/managed-skills/radiance-precision-selection/SKILL.md create mode 100644 agent/managed-skills/radiance-quantize-and-score/SKILL.md create mode 100644 agent/managed-skills/radiance-w4-campaign/SKILL.md create mode 100644 agent/managed-skills/radiance-w4-convert-playbook/SKILL.md delete mode 100644 agent/managed-skills/rccl-pcie-atomics-collective-failure/SKILL.md delete mode 100644 agent/managed-skills/rootless-quadlet-log-retrieval/SKILL.md delete mode 100644 agent/managed-skills/spec-decode-ab-single-gpu-host/SKILL.md delete mode 100644 agent/managed-skills/spec-decode-worth-it-triage/SKILL.md delete mode 100644 agent/managed-skills/ssh-fish-host-remote-exec/SKILL.md delete mode 100644 agent/managed-skills/ssh-key-bootstrap-no-sshpass/SKILL.md delete mode 100644 agent/managed-skills/stale-compose-container-quadlet-migration/SKILL.md delete mode 100644 agent/managed-skills/stale-signal-and-hollow-gate-audit/SKILL.md delete mode 100644 agent/managed-skills/stale-signal-audit/SKILL.md delete mode 100644 agent/managed-skills/stash-supersession-audit-before-drop/SKILL.md create mode 100644 agent/managed-skills/static-smoke-server-verify/SKILL.md delete mode 100644 agent/managed-skills/status-surface-dead-signal-audit/SKILL.md create mode 100644 agent/managed-skills/subagent-yield-failure-recovery/SKILL.md delete mode 100644 agent/managed-skills/swapfile-resize-safe-ordering/SKILL.md delete mode 100644 agent/managed-skills/vhost-proxy-health-probes/SKILL.md delete mode 100644 agent/managed-skills/wol-wake-sleeping-fleet-host/SKILL.md diff --git a/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md b/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md deleted file mode 100644 index 232f5f7..0000000 --- a/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -name: amd-multigpu-cooperation-triage -description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives." ---- - -# AMD Multi-GPU Cooperation Triage - -When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix". - -## 1. Check atomic-op routing on the whole path - -RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*: - -```cpp -void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp - link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0}); - if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) - if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else, no override -} -``` - -```bash -# Endpoint capability + negotiated link, and EVERY upstream bridge -for d in ; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done -lspci -tv # is the card on a CPU root port or behind the chipset? -``` - -- Endpoint must show `AtomicOpsCtl: ReqEn+`. -- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe. - -Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`. - -## 2. Map the board before recommending anything - -Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4. - -## 3. Prove what the runtime actually needs (do it OFF the production host) - -Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging: - -```python -torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT", - world_size=2, rank=r, timeout=timedelta(seconds=25)) -# probe: broadcast, all_reduce, barrier, send/recv, all_gather -``` - -- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall. -- `world_size=1` always succeeds; one-card success proves nothing. -- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/'`). -- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened. - -## 4. Don't be fooled by "collective-free" claims - -Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out. - -## 5. The option set, ranked - -| Option | Works when | Cost | Trade | -|---|---|---|---| -| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient | -| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser | -| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode | -| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today | -| Patch the custom-AR table | never | — | other collectives still die | -| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host | - -## Pitfalls - -- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`). -- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show ` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP. -- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`). diff --git a/agent/managed-skills/anubis-guard-chain-flip/SKILL.md b/agent/managed-skills/anubis-guard-chain-flip/SKILL.md deleted file mode 100644 index 7d84f67..0000000 --- a/agent/managed-skills/anubis-guard-chain-flip/SKILL.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -name: anubis-guard-chain-flip -description: "Runbook for moving a Caddy-fronted service behind the Anubis + Authentik embedded-outpost guard chain on Dankclaw (provider-first ordering, validate gate, verification probes)." ---- - -# Anubis + Authentik guard-chain flip (Dankclaw) - -Move a public vhost from `reverse_proxy :` onto: client → TLS edge → Anubis challenge (:8923) → Authentik embedded outpost decision → return to internal `http://:8017` listener → Host-matched handler → real upstream. - -## Order (never reorder) -1. **Provider first.** Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app: - - Write a Python snippet to a local /tmp file that imports `authentik.providers.proxy.models.ProxyProvider`, constructs it with `name`, `external_host="https:///"` (exact, trailing slash), `internal_host="http://caddy:8017"`, sets `p.pk = None`, then `p.save()`. - - Pipe raw: `ssh -o BatchMode=yes dankclaw 'podman exec -i authentik-server ak shell' < /tmp/x.py`. Heredoc quoting through ssh SyntaxErrors; the raw-stdin pattern is the proven one (ak shell does `exec(sys.stdin.read())`). -2. **Route flip in ONE commit/deploy**: vhost becomes `reverse_proxy anubis:8923 { header_up X-Real-Ip {remote_host} }`; inside the `http://:8017` listener add `@ host ` + `handle @ { reverse_proxy }`. -3. Deploy via the repo deploy flow (commit first; deploy pushes main itself). - -## Verification (all three) -- Unauthenticated probe from client: `curl -s https:///` → HTTP 200 challenge page, grep body for challenge markers (`challenge|not a bot|anubis`). A 502 here instead of the challenge = chain mis-wired. -- Post-auth routing proof: grep the LIVE config inside the container — `podman exec caddy sh -c "grep -n -A3 /etc/caddy/Caddyfile"` — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0. -- Outpost reachability: `podman exec authentik-server getent hosts caddy` resolves the compose DNS alias (10.89.1.x) that internal_host dials. - -## Pitfalls -- Probing `http://localhost:8017/` from the host returns HTTP 000 (connection refused): the post-auth listener lives in the caddy container's netns and is not bound on host loopback. Probe inside the netns or grep the config instead. -- Non-browser clients cannot solve the JS challenge. Before flipping, audit service-to-service traffic: ingestion/push paths (e.g. Alloy → `loki:3100`) must target internal addresses; any backend datasource URL pointing at a public hostname of a guarded service breaks the moment you flip. -- Exempt-by-construction categories that must stay unguarded: the IdP itself (`auth.` — its login flow can't sit behind a challenge), git smart-HTTP endpoints (Gitea git clients), services with their own complete auth. Document the exemption in the vhost comment. diff --git a/agent/managed-skills/arr-auth-posture-probe/SKILL.md b/agent/managed-skills/arr-auth-posture-probe/SKILL.md deleted file mode 100644 index 4896429..0000000 --- a/agent/managed-skills/arr-auth-posture-probe/SKILL.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -name: arr-auth-posture-probe -description: "Verify a self-hosted *arr-style service's real auth posture by behavioral API probes instead of config-flag inference; includes the sed-masks-a-miss pitfall and wrong-host probe pitfall" ---- - -# Arr-auth posture probe - -Ground truth for "is this arr/gate open?" comes from behavior, not config flags. `AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in `config.xml` is ambiguous on paper; on the wire it means a login screen with **zero users** — first-account-wins. - -## Recipe -1. Confirm which host runs the service before probing (`podman ps` on the target, not the box you ssh'd into first — arrs live on Dankserver, chat/searxng on Dankclaw). -2. Probe an API endpoint that bypasses the SPA shell, locally and through the vhost: - `curl -s -o /dev/null -w "%{http_code}" http://:/api/v3/system/status` - - SPA roots (`/`) always serve a 200 shell — never conclude "open" or "gated" from `/`. - - 401 = gated; 200 + JSON = API fully open (anyone on the net can drive it with curl). -3. For config truth, dump the whole file (`podman exec cat /config/config.xml`), not a piped `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the pipeline hides a miss and you conclude "empty field" when the grep itself failed. -4. API keys from config.xml are local-only secrets; they stop being secret the moment a vhost serves the API unauthenticated. - -## Decision framing for the user -When posture = gated-with-zero-users, present it as a decision menu (None behind the front-door wall vs External/authentikarr), not a silent change — first-account-wins means whoever logs in first owns the instance. diff --git a/agent/managed-skills/arr-stack-api-config/SKILL.md b/agent/managed-skills/arr-stack-api-config/SKILL.md deleted file mode 100644 index a7475e6..0000000 --- a/agent/managed-skills/arr-stack-api-config/SKILL.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -name: arr-stack-api-config -description: "Configure/download-client and indexer wiring for the *arr stack on Dankserver via live API calls — schema-first, test-before-enable; covers Sonarr/Radarr v3 vs Lidarr/Readarr v1 API split and Prowlarr v2 Applications/sync model." ---- - -# Arr-stack API config (Dankserver) - -## API version split -- Sonarr/Radarr: `/api/v3/...`; Lidarr/Readarr/Prowlarr: `/api/v1/...`. -- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object (GET it, patch `settings`, PUT whole object back). An array body fails with `Path: $ | BytePositionInLine: 1`. -- API keys live in each container's config: `podman exec grep -o "[^<]*" /config/config.xml | cut -d">" -f2`. - -## Schema-first probing (validators are free documentation) -Send the FULL schema template object with fields filled; v1 validators reject partials: -- `'Fields' must not be empty` -> send full schema copy, not `fields:[]`. -- `'Base Url' must not be empty` when keys were camelCase -> PascalCase required (`BaseUrl`, `ApiKey`). -- `targetType null` -> configContract not registered in that build (this Prowlarr build has no Prowlarr-plugin indexer; integration is Torznab-family only). -- `testingPackageId` in a PUT body downgrades validation failures to warnings. - -## Prowlarr v2 model (inverted from v1) -Indexers live ON THE ARR SIDE; Prowlarr syncs indexer configs via `ApplicationSync` commands (`POST /api/v1/command {"name":"ApplicationSync","body":{"applicationId":N}}`). Applications settings keys must be PascalCase or sync throws NullConfig. Torznab feed paths can answer with the SPA HTML shell when absent — verify by byte count (HTML shell ~1946 bytes, 1 ms) vs XML, not just status code. - -## DB as arbiter -When endpoint shapes are opaque, stop the container and edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly, then start. Indexers table columns are Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the enable flag column is `Enable` (not EnableRss). - -## Verification rules -- `POST /downloadclient` returns the COLLECTION, not the created object — verify via GET. -- Test endpoints return verdicts in the BODY at HTTP 200 (`{}` = pass, list = fail) — judge from bodies, not status codes. -- `syncLevel` is enum ApplicationSyncLevel — omit or use int 2; string rejected. -- Command POSTs (MoviesSearch etc.) run server-side and respond only when the work finishes: curl `-m` timeouts (HTTP 000) during search/grab are NOT failures; poll queue/history afterward instead of re-firing. Warm small commands (RefreshMonitoredDownloads) answer in ~20 ms — a hang means live work in progress, not a dead API. diff --git a/agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md b/agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md deleted file mode 100644 index 59bc197..0000000 --- a/agent/managed-skills/arr-v1-api-validator-peeling/SKILL.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -name: arr-v1-api-validator-peeling -description: "When driving Sonarr/Radarr (v3) vs Lidarr/Readarr/Prowlarr (v1 .NET) APIs over HTTP: probe endpoints by status code, peel validators one live call at a time, and treat config.xml on disk as truth — v1 rejects literal enum strings that break AuthorizationMiddleware." ---- - -# *arr v1 API validator peeling - -Use when creating root folders, profiles, or settings via the *arr HTTP APIs, or when a v1 (.NET) arr 500-sprawls after a config edit/restart. - -## API version split -- Sonarr/Radarr: `/api/v3/...`. Lidarr/Readarr/Prowlarr: `/api/v1/...` (Prowlarr port 9696, NOT 8090 — that's Jackett). -- Probe endpoints by status code first: **empty body = endpoint absent**, not a failure. v1 has no `/api/v1/profile` endpoint at all. - -## Peel validators one live call at a time -POST a minimal body and let the validator error name the next required field, instead of guessing payload shapes: -- `DefaultMetadataProfileId must be >0` → that field is required; fetch real ids from `GET /api/v1/metadataprofile` (returns rows like `(1,'Standard')`). -- `'Name' must not be empty` → include `name`. -- `Quality Profile does not exist` → the id you sent doesn't exist; list real ones first (`GET /api/v1/qualitydefinition` returns the quality catalog, ids ≠ profiles). -Then POST the full body with **existing** ids: `{"name":"music","path":"/music","defaultQualityProfileId":1,"defaultMetadataProfileId":1}`. - -## Auth-config pitfall (v1 .NET apps) -- `None` + `Enabled` = self-contradictory: UI Save permanently disabled, and any save throws `DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every page after restart. -- The valid no-auth representation is **absence of the `` key**, not the literal string `Disabled`. Fix = delete the key from `/config/config.xml` (`sed -i "//d"`), restart container. -- Tell-tale: one v1 app boots clean while siblings 500-sprawl → diff their config.xml auth blocks; the healthy one shows the correct shape. - -## General rules -- Container `State=running` ≠ app serving; read `podman logs --tail 12 ` before theorizing. -- When a UI gate blocks saving, go under it: these apps keep plaintext XML config on disk (`/config/config.xml`) — read/edit it directly via `podman exec`. -- API keys live in each container's `/config/config.xml` ``; grep them live rather than trusting docs. diff --git a/agent/managed-skills/authentik-claim-cache-purge/SKILL.md b/agent/managed-skills/authentik-claim-cache-purge/SKILL.md deleted file mode 100644 index 3a6b254..0000000 --- a/agent/managed-skills/authentik-claim-cache-purge/SKILL.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -name: authentik-claim-cache-purge -description: "Fix persistent 403s at Caddy RBAC header gates after authentik group changes: purge the embedded outpost's proxysession claim cache and verify silent re-auth" ---- - -# Authentik outpost claim-cache purge - -Use when a user gets persistent 403s at a Caddy `header_regexp` RBAC gate (e.g. `/admin`) even though their authentik group membership is correct, and re-login did not help. - -## Mechanism (authentik 2026.8, this fleet) -- The embedded outpost does NOT re-evaluate groups per request. Each completed authorize writes one row to `authentik_providers_proxy_proxysession`; `session_data` JSON holds the claims (incl. `groups`) served for that browser session until the row expires (observed TTLs 1–24 h per row). -- A session minted before the group add keeps serving the old group list → gate 403s forever until expiry. -- The 403 happens at Caddy and generates ZERO authentik events — silence in `authentik_events_event` is expected. -- Re-login does not fix it: the browser's outpost cookie re-maps to the same stale row. -- `proxysession.user_id` is null; map rows to users via `session_data->'claims'->>'preferred_username'`. - -## Diagnosis (run on dankclaw) -```bash -podman exec authentik-postgres psql -U authentik -d authentik -At -c "SELECT session_data->'claims'->>'preferred_username' AS user, session_data->'claims'->>'groups' AS cached_groups, expires FROM authentik_providers_proxy_proxysession ORDER BY expires;" -``` -Stale row = `cached_groups` missing the new group while repo/Caddy gate expects it. - -## Fix -```bash -podman exec authentik-postgres psql -U authentik -d authentik -At -c "WITH d AS (DELETE FROM authentik_providers_proxy_proxysession WHERE session_data->'claims'->>'preferred_username' IN ('') RETURNING 1) SELECT 'deleted='||count(*) FROM d;" -``` -- The user's authentik django login session is untouched → next visit re-authorizes SILENTLY (no password prompt) with fresh claims. -- NEVER rotate `AUTHENTIK_SECRET_KEY` for this — it invalidates every outpost session fleet-wide. - -## Verification without the user's password -Simulate their browser: delete a TEST user's proxysession rows (e.g. `qa-e2e-admin`), keep its e2e storage state on disk, then run the admin spec bypassing setup (setup ALWAYS fresh-logins, which would mask the path): -```bash -cd tests/e2e && npx playwright test --project=admin --no-deps -``` -Passing with no login prompt = silent re-auth proven. `--no-deps` is required because project dependencies would rerun setup. - -## Doc -Mechanism + SQL live in `serverAppConfigs-docs/Infrastructure/Authentik.md`, section "Outpost claim cache (proxysession)" (commit ea0bba3). diff --git a/agent/managed-skills/authentik-flow-ui-automation/SKILL.md b/agent/managed-skills/authentik-flow-ui-automation/SKILL.md deleted file mode 100644 index 704ba9d..0000000 --- a/agent/managed-skills/authentik-flow-ui-automation/SKILL.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -name: authentik-flow-ui-automation -description: Drive the authentik two-stage login through real browser UI for E2E tests on Anubis+outpost-guarded routes; hardened against the Lit stage re-render race and Anubis PoW budgets ---- - -Drive authentik's two-stage login (identification → password) through a real browser UI for E2E tests on routes guarded by the Anubis + Authentik chain (Caddy → Anubis :8923 → outpost :9000). Proven in `tests/e2e/lib/login.ts` (`akLogin`) against authentik 2026.x. - -## Hard rules - -1. **Fresh context per login.** Never reuse a context or page across logins, and never retry a failed login inside the same context — a half-finished flow poisons authentik's `pending_user` stage and the next login hangs on a "Continue as " interstitial that never completes. Each login = new `browser.newContext()`. -2. **Real Chrome UA.** Anubis denies obvious headless UAs; set `userAgent` to a real Chrome UA in `use:` (config-level) or `launchOptions`. Headless chromium + REAL_UA solves the PoW fine. -3. **Shadow-DOM locators.** authentik stages are Lit web components; locate inputs with `page.locator('input[name=username]')` etc. (Playwright pierces shadow roots) — never `page.$`. - -## The Lit re-render race (the big one) - -The flow executor re-renders a stage right after mount. A programmatic `fill()` can succeed and then be WIPED by the re-render; the submit then POSTs an empty form ("This field is required") and `waitForURL` times out while the page still shows the password stage with an empty field. - -Hardened sequence per stage: - -```ts -for (let attempt = 1; attempt <= 3; attempt++) { - await page.fill('input[name=password]', pass); - // VERIFY the value survived the re-render before submitting - if ((await page.inputValue('input[name=password]')) !== pass) continue; - await page.keyboard.press('Enter'); // native form path; avoids button re-render races - try { - await page.waitForURL(u => !u.pathname.includes('/if/flow/'), { timeout: 30_000 }); - return; // past this stage - } catch { /* maybe wiped; retry */ } -} -throw new Error(`akLogin: stage never advanced after 3 attempts`); -``` - -- Submit with **Enter**, not a button click — the button can be re-created between snapshot and click. -- Verify-after-fill is the load-bearing step; without it you silently submit empty forms. -- The identification stage has the same race; apply the same fill-verify-Enter there. - -## Anubis PoW budgets - -The first guarded-vhost hit in a fresh context runs an Anubis proof-of-work; on a cold/loaded box it can outlive default assertion budgets. Give title/URL asserts after `goto` 60 s budgets. With `COOKIE_DOMAIN` set fleet-wide (Anubis ≥ v1.27), every later guarded vhost in the same context skips the challenge (~0.5 s per vhost in the crawl specs). - -## See also - -- `serverAppConfigs-docs/Architecture/E2E Suite.md` — project layout, credentials model (`tests/e2e/.env.local`, git-ignored), run commands, gotcha list. -- `serverAppConfigs-docs/Infrastructure/Anubis.md` — chain behavior, v1.27 gotchas (X-Real-Ip mandatory, DENY = HTTP 200 body). -- `arr-auth-posture-probe` for non-browser API-posture checks of guarded routes. diff --git a/agent/managed-skills/authentik-outpost-verification/SKILL.md b/agent/managed-skills/authentik-outpost-verification/SKILL.md deleted file mode 100644 index 7f57756..0000000 --- a/agent/managed-skills/authentik-outpost-verification/SKILL.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -name: authentik-outpost-verification -description: "Verify authentik embedded-outpost and proxy-provider state on a live host by probing from inside the container network namespace; covers meta-header routing, rootless podman observation artifacts, and 2026.x schema quirks (renamed columns, ak shell swallowing piped stdin)." ---- - -# Authentik outpost/provider verification (live host) - -## When to use -Guarded apps behind Anubis+authentik return an error page after authentication, or you need ground truth on outpost/provider state on a host running authentik 2026.x with podman quadlet units. - -## Core insight: probe from inside the network namespace -- Rootless podman gives each container its own netns; **host `ss -ltn` cannot see listeners inside other containers' namespaces**. A missing listener on the host is an observation artifact, not a fault — verify by dialing *from inside* the namespace. -- Working probe (distroless images have no shell, so `podman exec sh` fails): - ``` - podman run --rm --network container: --entrypoint /bin/busybox docker.io/library/busybox:latest \ - wget -q -T 4 -t 1 -O /dev/null -S "" 2>&1 | head -8 - ``` -- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings triggers globbing (`no matches found`). - -## Meta-header routing truth -- The outpost routes by **meta-header middleware**, not URL-path lookup. A request to `/outpost.goauthentik.io/login` **without** `X-authentik-meta-outpost/-provider/-app` headers is a path that genuinely does not exist → a 404 there is *correct behavior*, not a fault. -- Probe the auth hop exactly as Caddy sends it: full meta-header set (`X-authentik-meta-outpost: `, `X-authentik-meta-provider: `, `X-authentik-meta-app: name=`). If a full-meta-header request still 404s, the outpost's URL space is not mounted in the answering process — that is the real failure point, and it means the transport is aimed at a URL space that doesn't exist there (embedded outposts mount their URL space at `authentik_host`; check the outpost row's `_config.authentik_host` before theorizing). -- Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding. - -## Schema quirks in 2026.x -- `ak shell` is interactive-only; piped stdin is swallowed (prints import banner, discards output). For live DB truth use: - ``` - podman exec authentik-postgres psql -U authentik -d authentik -x -c "select * from authentik_outposts_outpost" - ``` -- Columns are renamed vs older versions: `Provider.objects.filter(type=...)` raises FieldError (type moved to subclass tables); the outpost `enabled` column may not exist under that name; provider pk is `oauth2provider_ptr_id`. **Query with `select *` and stop guessing column names.** -- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman version — the field is `.HostConfig.NetworkMode`. - -## Decision fork when the auth hop 404s -A) point the guarded apps' auth hop at where auth actually lives (the vhost matching `authentik_host`), or B) flip the outpost to standalone (own container, own URL space, new quadlet unit + `AUTHENTIK_TOKEN`). This is an architecture decision for the user — present both with blast radius, don't pick silently. diff --git a/agent/managed-skills/authentik-provision-fleet-user/SKILL.md b/agent/managed-skills/authentik-provision-fleet-user/SKILL.md new file mode 100644 index 0000000..fc6eeec --- /dev/null +++ b/agent/managed-skills/authentik-provision-fleet-user/SKILL.md @@ -0,0 +1,42 @@ +--- +name: authentik-provision-fleet-user +description: "Provision or audit an Authentik user on Dankclaw (auth.franksdank.cloud): ak shell ORM drive, group set, behavioral login verify, stale-claim caveat." +--- + +# Provision an Authentik user on Dankclaw + +Source of truth: `serverAppConfigs-docs/Infrastructure/Authentik.md` ("User provisioning playbook"). Authentik objects are NOT repo-managed — no deploy, no file edits. + +1. Inspect first, mirror the newest human user's groups: +```bash +ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' 2>&1 | sed -n '/objects imported/,$p' +from authentik.core.models import User, Group +for u in User.objects.all().order_by("username"): + print(u.username, u.email, sorted(u.groups.values_list("name", flat=True))) +print(sorted(Group.objects.values_list("name", flat=True))) +EOF +``` +2. Create (idempotent guard; password goes in the heredoc = stdin, never argv, so it never lands in `ps`): +```python +from authentik.core.models import User, Group +u = User.objects.filter(username="X").first() +if not u: + u = User(username="X", email="X@franksdank.cloud", name="X", is_active=True) + u.set_password("...") + u.save() +for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"): # human-user precedent: ridley + u.groups.add(Group.objects.get(name=g)) +``` +3. Verify through the exact PasswordStage code path: +```python +from django.contrib.auth import authenticate +assert authenticate(request=None, username="X", password="...") +assert authenticate(request=None, username="X", password="nope") is None +``` + +## Rules +- No `User.create_user`; `is_superuser` is a group-derived property, not a constructor kwarg (raises `TypeError`). `path` defaults to `users`. +- Guarded-vhost access = any authenticated user (all proxy providers use implicit-consent); groups are consumed downstream: `homepage-admins` (Caddy `/admin` header gate), `openwebui_users`/`_admins` (roles claim), `jellyfin_users`/`_admins` (LDAP outpost). EXCLUDE admin groups from a plain user; ask before granting `homepage-admins`. +- Group add after a login → Caddy RBAC can 403 on the outpost's cached claims; purge that user's rows: `DELETE FROM authentik_providers_proxy_proxysession WHERE session_data->'claims'->>'preferred_username' = 'X';` (in the `authentik-postgres` container) — next visit re-authorizes silently. +- Jellyfin visibility rides the LDAP outpost cache; expect a lag. +- Look up flows by slug, not name. Do not rotate `AUTHENTIK_SECRET_KEY` to fix claim issues. diff --git a/agent/managed-skills/authentik-proxy-provider-app-link/SKILL.md b/agent/managed-skills/authentik-proxy-provider-app-link/SKILL.md deleted file mode 100644 index f15b757..0000000 --- a/agent/managed-skills/authentik-proxy-provider-app-link/SKILL.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -name: authentik-proxy-provider-app-link -description: "Diagnose and fix an authentik embedded-outpost proxy provider that never authenticates (login wall / raw flow served on the app host): the provider must be linked to an Application, have a non-null invalidation_flow, and an exact external_host match." ---- - -# authentik proxy provider not authenticating (login wall) - -Symptom: a service behind the authentik embedded outpost (Caddy -> Anubis -> outpost -> app) never lets an authenticated user through. The browser is parked on a login page served ON THE APP'S OWN HOST, e.g. `https://app.example.com/if/flow/default-authentication-flow/?next=/` or `https://app.example.com/flows/-/default/authentication/`. The IdP session cookie lives on `auth.example.com` and is never sent to `app.example.com`, so login is demanded forever. - -## Root causes (check in this order) - -1. **Provider not linked to an Application (most common).** The embedded outpost's config API filters `ProxyProvider.objects.filter(application__isnull=False)`. A provider created with `ProxyProvider.objects.create(...)` and no Application is never returned to the outpost, so the outpost never loads it and the request falls through to raw flow serving. - - FIX: create an Application bound to the provider: - ```python - app = Application.objects.create(name="Marreta", slug="marreta", launch_url="https://marreta.franksdank.cloud") - app.provider = proxy_provider; app.save() - proxy_provider.application = app; proxy_provider.save() - ``` -2. **Null `invalidation_flow`.** The Rust outpost serde requires a formatted UUID; a null value aborts `get_outpost` with `invalid type: null, expected a formatted UUID string at line 1 column N` and takes down EVERY provider on that outpost. - - FIX: `proxy_provider.invalidation_flow = InvalidationFlow.objects.get(name="default-provider-invalidation-flow")`. -3. **`external_host` mismatch.** Must equal the Host header the outpost sees. Trailing-slash mismatch (`https://x` vs `https://x/`) = no match. Match the fleet convention (no trailing slash). -4. **Missing `authentication_flow` / `authorization_flow`.** Set to the fleet defaults (`default-authentication-flow`, `default-provider-authorization-implicit-consent`). - -## Diagnose - -```bash -# 1. Compare broken vs working provider field set (ak shell, as the authentik user): -sudo -u authentik AUTHENTIK_POSTGRESQL_HOST=127.0.0.1 AUTHENTIK_POSTGRESQL_PORT=5432 \ - AUTHENTIK_POSTGRESQL_NAME=authentik AUTHENTIK_POSTGRESQL_USER=authentik \ - AUTHENTIK_POSTGRESQL_PASSWORD='' ak shell -``` -```python -for p in ProxyProvider.objects.all(): - print(p.name, p.external_host, bool(p.application), p.authentication_flow_id, p.authorization_flow_id, p.invalidation_flow_id) -``` -```bash -# 2. The serde error (this is the smoking gun for cause #2): -journalctl --user -u authentik-outpost.service --since '-10 min' | grep -iE 'error|invalid|panic' -# 3. Confirm the outpost loaded the provider after the fix: -journalctl --user -u authentik-outpost.service --since '-2 min' | grep -i 'loaded app' -``` - -## Apply + restart - -After any provider/application change: `systemctl --user restart authentik-server.service authentik-outpost.service` (rootless, user dank on Dankclaw). Then verify the OAuth dance: -```bash -curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://app.example.com/ -# expect 302 -> /outpost.goauthentik.io/start?rd=... (NOT a raw /flows/ URL) -``` - -## Provenance note -On the Dank fleet these providers are NOT repo-managed (no blueprint). Fixes applied via `ak shell` are live-only. Capture any new provider+application as a repo blueprint or setup script to avoid drift. diff --git a/agent/managed-skills/autoresearch-llm-bench-window/SKILL.md b/agent/managed-skills/autoresearch-llm-bench-window/SKILL.md new file mode 100644 index 0000000..518b224 --- /dev/null +++ b/agent/managed-skills/autoresearch-llm-bench-window/SKILL.md @@ -0,0 +1,76 @@ +--- +name: autoresearch-llm-bench-window +description: "How to run an iterative config-tuning window on a Dank-fleet LLM endpoint: harness pattern, noise-floor-first discipline, bracket mapping, and the frank-primary laws measured in the 2026-10-06 segment." +--- + +# Autoresearch bench window for a fleet LLM endpoint + +Use when running (or re-running) an iterative config-tuning segment against a +Dank-fleet LLM endpoint (frank-primary/frank-large/frank-small): build the +harness, establish the noise floor, then bracket-map one flag at a time. + +## Harness pattern (already built for frank-primary — reuse, don't rebuild) + +Committed at `autoresearch.sh` (dev box) + `scripts/autoresearch-frank-primary.sh` +(dankserver) in serverAppConfigs. The pattern generalizes to any fleet endpoint: + +1. Dev-box wrapper scp's the candidate env file + bench script to the target + host, then ssh-runs the bench. Never push the candidate config to origin + (the scp dirties the host checkout on purpose). +2. Host script, in order: stop the production unit (relinked units: `systemctl + stop`, NEVER mask — mask refuses a symlinked unit file) → boot a scratch + container (distinct name, e.g. `arbench`) on the SAME devices with the + candidate env — mirror the production podman contract exactly, differing + only in container name and a FRESH prefix-cache tier (wiped per run) so cold + measurements stay cold → poll the health endpoint until it answers (the + listener binds AFTER the full load, so a bare curl is honest; allow ~2-3 min + cold boot, 720s timeout) → run ONE self-contained deterministic request → + parse the engine's timings object into METRIC lines → tear down the scratch → + restore production. +3. The restore MUST be an EXIT trap (`restore() { podman rm -f scratch; + systemctl start ; }`) so a failed bench never leaves the endpoint + down. Verify after EVERY run: `systemctl is-active`, health curl until it + returns JSON (200 right after start is NOT enough — it can bind 60-90s + later), and no scratch leftover. +4. Bench request: fixed prompt built from a sorted repo-corpus glob with a + fixed random seed (deterministic → stable prompt_n), greedy sampling + (temperature 0) for a fixed decode length, sent locally (127.0.0.1) to + remove LAN jitter. Read prefill/decode/draft-acceptance from the engine's + own timings object — never roll a custom bench without asking. +5. Scratch containers must die by teardown; never leave a script whose tail + stops/removes a production unit (the stale-probe-tail lesson). + +## Discipline (the laws that make the numbers mean anything) + +1. **Noise floor FIRST.** Before iterating, replicate an identical config 2-3x + and treat that spread as the floor. Fleet endpoints here: decode ±3%, + prefill ±2%. A single-run "win" under the floor is NOT a win — this exact + mistake cost a false 33.06 keep in the 2026-10-06 segment. +2. **One flag change per iteration**; log every run honestly (keep/discard/ + crash); discards auto-revert the worktree. +3. **Bracket-map, don't climb.** Test both directions of every lever around + its deployed value, INCLUDING boot failures — crashes are envelope data + ("0.82/2048/wht6/40960 is the seqs=3-viable quartet" came from four crash + walls). Test downward too: 40960 MiB host pool is a floor, not slack. +4. **Repro before recording a mechanism.** An anomalous result next to an + abandoned bench may be contamination (boot race). Re-run clean once; only + then name a law. The MNBT 2560 collapse (2.6 t/s) was reproduced clean to + rule this out. +5. **Watch for truncation.** predicted_n < max_tokens means the generation + died mid-run — that config is pathological regardless of the rate. +6. **When the sweep ties, tie-break on real-traffic properties** (e.g. draft + acceptance for multi-turn workloads), never on probe artifacts, and prefer + the config that matches the documented runbook. + +## frank-primary specifics (2026-10-06 segment, commit c80c250) + +- MNBT=2048 is a RESONANCE POINT at MTP-on shapes, not an optimum: 2560 + collapses decode to 2.6-7.7 t/s (acceptance normal, verify stalls ~130ms/step, + truncated output), 1536 truncates at 57/256, 4096 won't boot at seqs=3. Do + not re-test. +- HOST_POOL_MIB 40960 is the load-bearing floor (36864 crashes boot). +- Spec depth 1-3 and seqs 2-3 all tie within noise; 0 and 4 are real + regressions (29.57 / 24.57 t/s vs 31.72 baseline). +- Levers known flat: GPU_HEADROOM, HOST_POOL upward, maxlen above 200k. +- Final kept config = Frank's original (spec=3, seqs=2, maxlen=200k, wht6, + 0.82, 2048, 40960, fp8 KV, TP=2), baseline decode 31.72 t/s at 43.6k depth. diff --git a/agent/managed-skills/bench-llama-server-params/SKILL.md b/agent/managed-skills/bench-llama-server-params/SKILL.md deleted file mode 100644 index 1ec996e..0000000 --- a/agent/managed-skills/bench-llama-server-params/SKILL.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -name: bench-llama-server-params -description: "Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode" ---- - -# Benchmarking llama.cpp Server-Side Parameters - -## Problem -`llama-bench` benchmarks the bench binary itself, not the running server. -It cannot measure the effect of server-side parameters like `--threads-batch`, -`--ubatch-size`, `--spec-type`, or `--parallel`. - -## Solution: HTTP-Based Benchmarking - -### Prefill throughput (prompt processing) -1. Start the server with the target parameter value -2. Send a large prompt via the chat API with `max_tokens=1` -3. Measure `prompt_tokens / wall_time` from the response usage - -`max_tokens=1` is correct here — you want only prompt processing time, -not generation. The server crashes on `max_tokens=0`. - -### Decode throughput (generation) -1. Start the server with the target parameter value -2. Send a prompt that generates a long response -3. Use `max_tokens=16384` minimum (16k) — **never `max_tokens=1`** for decode - benchmarking. One token tells you nothing about sustained generation rate. -4. Measure `predicted_per_second` from the response `timings` object, or - compute `completion_tokens / wall_time` - -### Why at least 16k tokens for decode -- Decode throughput varies over the generation (warmup, cache growth, MoE - routing). A single token is noise. -- Speculative decoding acceptance rates stabilize over thousands of tokens. -- Short generations finish before the draft head reaches steady state. -- 16k tokens gives a stable average at ~22 tok/s (≈12 min per run). - -### Key Pattern -```python -# Decode benchmark — measure generation throughput -body = json.dumps({ - "model": "", - "messages": [{"role": "user", "content": ""}], - "max_tokens": 16384, # 16k minimum for decode; use max_tokens=1 ONLY for prefill - "temperature": 0, # deterministic for reproducibility - "stream": False -}) -# Measure wall time from request send to response receipt -# Get predicted_per_second from timings, or completion_tokens / wall_time -``` - -### When to Use -- `--threads-batch` (prefill): use `max_tokens=1` -- `--ubatch-size` (decode): use `max_tokens=16384+` -- `--spec-type` / speculative decoding: use `max_tokens=16384+` (decode only) -- Any server-side parameter that `llama-bench` cannot target - -### Benchmarking policy (MANDATORY) -1. **STOP the production service** before running any benchmark -2. **Mask** the service (`systemctl mask`, never `disable` — on these hosts - `disable` deletes the unit file from `/etc/systemd/system/`) -3. Run the bench -4. **Unmask + restart** the service when done -5. **Verify health** before reporting results - -### Example Scripts -- `scripts/bench-threads-batch-http.py` — prefill sweep (max_tokens=1) -- `scripts/bench-decode-threads-batch.py` — decode sweep (max_tokens=256+) -- `scripts/bench-spec-decode.py` — speculative decoding config sweep diff --git a/agent/managed-skills/closed-engine-runtime-investigation/SKILL.md b/agent/managed-skills/closed-engine-runtime-investigation/SKILL.md new file mode 100644 index 0000000..419a8ee --- /dev/null +++ b/agent/managed-skills/closed-engine-runtime-investigation/SKILL.md @@ -0,0 +1,29 @@ +--- +name: closed-engine-runtime-investigation +description: "Recipe for diagnosing/tuning a closed-source LLM runtime (halogen-class) on this fleet: docs, env-lever enumeration from the binary, ledger scraping, cache-assisted A/B under the bench law." +--- + +# Investigating a closed-source LLM runtime (halogen-class) + +Use when a containerized closed-source engine (e.g. halogen on frank-large) underperforms and the repo docs don't explain it. Fleet law: stop the production unit BEFORE any GPU experiment (`sudo systemctl stop `; mask FAILS on relinked units — stop alone, redeploy after). + +## 1. Ground truth before flags +- Boot ledger + per-request timings live in `podman logs ` (container-lifetime only) UNLESS the serve script captures stdout on disk (foreground `podman run`). A `podman run -d ... > LOG` redirect captures ONLY the container ID — a one-line hex file is the tell. +- `serve_api:`-style log lines carry per-request `prompt/prefill/rounds/commit/detok` — build the depth-vs-rate curve from REAL traffic before any bench. Shallow-healthy + deep-collapsed = a depth-linear term (KV reads), not a broken engine. +- If the engine's public repo exists, read its FLAGS/docs (e.g. `docs/FLAGS.md`) AND the bench tool source (e.g. `tools/halogen-bench.py`) — bench tools reveal which regimes the published numbers never measured (halogen's tg rows are shallow-only). + +## 2. Enumerate the undocumented levers +The binary self-announces every env var it reads. The image often lacks `strings`: +``` +podman run --rm --entrypoint /bin/sh -c \ + 'grep -aoE "_[A-Z0-9_]+" /usr/local/bin/ | sort -u' +``` +Classify by kernel family (attention vs linear-state vs GEMV) against the suspected term. Only test levers with public precedent (e.g. issue #12 proved HALOGEN_FA_TQ) or an obvious name match; defaults are the publisher's measured winners — an A/B must show >=~15% to adopt. + +## 3. Cache-assisted A/B (cheap reps) +Boot each probe arm as its own container (env is read at startup) with the prompt cache ON (explicit MB >= one full-entry at ~70 KB/token): rep 1 pays the cold prefill, rep 2+ resumes (~2 s) — depth measurements then cost decode-time only. Publish the probe on host LOOPBACK (127.0.0.1:), run the driver ON the host (python3 stdlib is enough), read the engine's own `timings` (prompt_ms/cache_n/predicted_ms/draft_n_accepted). Never benchmark beside a live engine on the same GPU. + +## 4. Adopt or revert +- Winner: env line into the serve env + test-seam update + docs, then `deploy.sh --service ` (restart restores everything; no unmask needed after stop-only). +- Loser: `podman rm -f` probes, start the unit, revert nothing. +- Either way: record the measured before/after in the runbook; report the depth behavior upstream if the publisher invites benchmark data — their bench gaps (depth, carve-topologies) are exactly what issues like #12 close. diff --git a/agent/managed-skills/command-center-dotfiles-import/SKILL.md b/agent/managed-skills/command-center-dotfiles-import/SKILL.md deleted file mode 100644 index f95d221..0000000 --- a/agent/managed-skills/command-center-dotfiles-import/SKILL.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -name: command-center-dotfiles-import -description: "Runbook for deploying the command-center repo (dotfiles env layer + OMP harness) to fleet machines via full-replica rsync and deploy.sh, including override-package mechanics and ssh/sshpass transport gotchas; executed end-to-end on blankpad 2026-09-09." ---- - -# Command-center fleet deployment (repo = deployment) - -Repo `~/.omp` is itself the deployment: packages are home-relative trees under `dotfiles//`; `scripts/deploy.sh` symlinks live files into the repo checkout. `git pull` + `deploy.sh` is the whole propagation story. - -## Transport decision (agreed with Frank, executed 2026-09-09) -- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to `@:/home//.omp/`, NOT clone-from-Gitea — `main` can be ahead of `origin/main`; push is a user-only trigger, so clone would ship stale state. -- Deploy runs on-target as the target user: `bash /home//.omp/scripts/deploy.sh` (resolves repo via `git -C $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be present at target path — full-replica transport guarantees this). - -## Pre-overwrite audit (MANDATORY before first sync on a non-blank host) -Diff every file that `deploy.sh --check` reports as DRIFT against the repo copy BEFORE letting quarantine-and-replace run. Host-specific divergence belongs in an override package, not in quarantine: -1. Pull divergent live files from target (`ssh ... 'cat ' > local-staged-copy`). -2. Secret-audit staged copies (grep token/secret/password/key shapes) before anything enters git. -3. Compose `dotfiles/zz-/` on the source machine from those live copies; commit to repo first. -4. Ship package dir via rsync, then run deploy sync — override deploys LAST and wins over base packages. - -## deploy.sh mechanics (verified on blankpad) -- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it first, always. -- Default mode = sync: adopt identicals silently, link new, quarantine-and-replace divergents into `.quarantine///` (gitignored; never rm). -- **Override-target bug (fixed cc2c652):** `zz-` packages nest base-package dirs internally (`zz-blankpad/niri/.config/...`); target derivation must strip the nested package segment, else overrides deploy to phantom `$HOME/niri/.config/...` while real targets keep pointing at canonical copies (visible symptom: check shows 0 actions but `readlink ~/.zshrc` points at base `zsh/.zshrc`). -- systemd units auto-enable only when ExecStart binary resolves on that host; dconf package loads via `dconf load /`, never symlinks. -- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once post-transfer even before first sync. - -## ssh/sshpass transport gotchas (bit us live) -- Resolve target via mDNS FQDN (`avahi-resolve -4 -n .local`); bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or firewall default-drop, not auth failure — needs local/console fix on target. -- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; env passed via tool `env` param did not propagate to the sshpass child process. -- Remote login shell may be **fish**: never send bash syntax (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash /path/script.sh` or `bash -c "..."`. -- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A here-doc attached to the ssh command is consumed by `cat` — the intended script body becomes the file content, silently. Verify shipped bytes with md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat` drained it (false DIVERGED). - -## Verification of a completed deploy -Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped = idempotent steady state. Spot-check `readlink` on override files points into `dotfiles/zz-/`, not base packages. diff --git a/agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md b/agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md deleted file mode 100644 index bb96372..0000000 --- a/agent/managed-skills/compose-stack-bringup-groundtruth/SKILL.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -name: compose-stack-bringup-groundtruth -description: "Bring up a compose-defined container stack on a rootless podman host by verifying reality first (mounts, images, in-image interfaces) instead of trusting compose env/flag guesses; includes the copyparty ac-edition argv-interface recipe and boot-persistence audit." ---- - -# Compose-stack bring-up with ground truth (rootless podman hosts) - -Use when standing up (or debugging) a compose-defined service stack on a rootless -podman host where the stack has never run, or answers with surprise statuses -(SIGSEGV at boot, HTTP 500 on `/`, "manifest unknown"). - -## Procedure - -1. **Audit reality before pressing buttons.** `podman ps -a` runs per-user — - check every namespace (`sudo podman ps -a`) and native units - (`systemctl list-units`) before declaring a service absent; a service may run - as a native systemd unit or snap, not a container. Compare compose mount source - paths against the real filesystem (`ls /mnt/...`): compose files often name - phantom paths that never existed (e.g. `/mnt/media/*` vs the real LUKS array at - `/mnt/entropy`). Fix compose through git first, deploy second. - -2. **Pre-create bind-mount sources.** Rootless podman fails container creation - with `statfs /opt//...: no such file or directory` when a bind source is - missing. `sudo mkdir -p` + `chown 1000:1000` every compose volume source - before the first `podman-compose up -d`. - -3. **Staged bring-up.** Infra first (`up -d mariadb redis`), poll healthchecks via - `podman inspect --format "{{.State.Health.Status}}"`, then `up -d` the rest. - -4. **Never guess image tags — query ground truth.** ghcr.io answers **403 on the - bearer-token request for nonexistent/private repos** ("manifest unknown" / token - len 0 = repo or tag does not exist, not an auth problem). Query the registry - API (`ghcr.io/token?scope=repository:/:pull` then - `/v2///tags/list`) before pinning a tag. - -5. **Read the interface from inside the image** before guessing flags/envs: - `podman exec cat /z/cpp.sh`, `cat /z/initcfg`, - `podman exec python3 -m copyparty --help-accounts`. A service answering - HTTP 500 with its own styled error page is *running* — it is misconfigured, - not down. Copyparty ac edition ground truth: entrypoint - `/bin/ash /z/cpp.sh -c /z/initcfg` then `exec python3 -m copyparty "$@"`; - accounts/volumes are **argv flags** (`-a USER:PASS`, `-v SRC:DST:PERMS`), not - `COPAY_*` envs (zero `COPAY_` references in the package). With no accounts and - no volumes configured it serves cwd read/write for everyone — an - unauthenticated world-writable share of the whole mount. - -6. **Beware remote-shell quoting traps.** Remote zsh eats bare `===`/`?` tokens - (glob qualifiers/globs); grep patterns with nested quotes over ssh can be - quote-eaten into silent zero-match. A zero-match from a mangled pattern is not - absence — re-verify the target exists before concluding it. - -7. **Boot-persistence audit last.** On rootless podman hosts with - `podman-user-generator`, containers started via podman-compose get generated - `container-.service` units at first boot (evidence: an existing unit for - one container proves the generator works). Verify after first reboot: - `systemctl --user is-enabled .service`. - -## Gotchas -- jellyfinetype SIGSEGV at boot with `user: "1000:1000"` = unmapped uid inside the - rootless userns (host uid 1000 == container uid 0); drop the user line and let it - run as container-root (= host deploy user). -- `up -d --force-recreate` applies compose changes that change command/env; probe with - a poll loop (`for i in $(seq...)`) because fresh containers answer `000`/refused for - the first seconds while indexing. diff --git a/agent/managed-skills/compose-to-quadlet-cutover/SKILL.md b/agent/managed-skills/compose-to-quadlet-cutover/SKILL.md deleted file mode 100644 index e4247bc..0000000 --- a/agent/managed-skills/compose-to-quadlet-cutover/SKILL.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -name: compose-to-quadlet-cutover -description: "Convert a compose-only service to a boot-persistent rootless Quadlet unit on the Dankclaw/Dankserver fleet — checklist plus the traps (CMD-SHELL healthchecks in distroless images, --format docker, EnvironmentFile HOME=/root, loopback publish posture)" ---- - -# Compose service → boot-persistent Quadlet unit (this fleet) - -Use when a service in `dankclaw/compose.yaml` (or `dankserver/`) must survive a -reboot. Precedents to copy: `dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/` -and `dankclaw/observability/quadlet/` (five units, 2026-09-11). - -## Checklist - -1. **Unit file** at `dankclaw//quadlet/.container`, mirroring the - compose service line for line: `Image=`, `ContainerName=` - (Caddy and scrape configs address containers by name), `Network=dankclaw_default`, - `User=0:0` when the image's own user cannot write a rootless-owned bind mount, - `Volume=` for each mount (`:ro` preserved), `Exec=` for the compose `command:`, - `PublishPort=`, `HealthCmd`/`HealthInterval`/`HealthTimeout`/`HealthRetries`/ - `HealthStartPeriod`, `[Service] Restart=always` + `RestartSec=10`, - `[Install] WantedBy=default.target`. Never `systemctl --user enable` the - generated unit — `[Install]` plus linger IS the boot persistence. -2. **Exposure**: publish on `127.0.0.1` unless something off-host genuinely - needs it. Caddy reaches containers by name over the shared network, and the - homepage status timer probes loopback, so the 0.0.0.0 publish is almost - always gratuitous and is a guard-chain bypass. -3. **Handler** in `scripts/deploy-remote.sh`: `install_quadlet ` → - `systemctl --user restart .service` → `wait_for_health` on the - loopback endpoint. Rebuild locally built images first - (`build_compose_image`), and seed any per-service env file (see traps). -4. **Planner**: map `dankclaw//*` and `shared//*` to the service in - `scripts/deploy-plan.py`; keep dependencies ordered in `SERVICE_ORDER`. - Add a `scripts/test-changed.sh` case *above* the generic - `dankclaw/*/quadlet/*.container` fallback. -5. **Retire the compose entry** in the same change — two owners for one - container name is how a stale duplicate comes back. -6. **Seam test** `tests/_quadlet.bats`: boot persistence, restart policy, - no wildcard publish, container name still matching Caddy/scrape references, - health command, deployer installs+restarts+waits, never `rm`, never - `enable`. Migrate any older assertion that read the compose file. -7. **Deploy** per service (`./scripts/deploy.sh dankclaw --service `), - then prove: `is-active`/`is-enabled` = active/generated, container health - `healthy`, the endpoint answers, `list-dependencies default.target` shows the - unit, and off-host probes on the previously published ports now refuse. - -## Traps that cost real time - -- **`HealthCmd` = CMD-SHELL.** podman runs it as `/bin/sh -c`; a distroless - image has no shell, so the check fails forever. `--health-cmd` will not take - exec form either (a JSON array is stored as one literal argument). Put - `HEALTHCHECK ... CMD ["binary","arg"]` in the image and **build with - `--format docker`** — the default OCI format has no HEALTHCHECK field and - drops it with only a warning. -- **Never point a unit's `EnvironmentFile` at the repo root `.env`.** That file - is a gluetun env file carrying `HOME=/root`: systemd sets HOME for the - service, podman then looks for its config under `/root`, and the unit dies - with `stat /root/.config: permission denied`. Use a per-service file under - `/home/dank/Containers//env`, seeded by the deploy handler with a - single-variable filter and validated so a missing secret fails loudly. -- **`Network=host` exporters** that a containerized Prometheus scrapes must - bind the host's LAN address (containers cannot reach the host loopback) plus - `127.0.0.1` for the homepage probe — two `--web.listen-address` flags, never - the wildcard. -- **alloy's UI** listens on the container's loopback and 1.x refuses the `ui` - config block, so the unit must pass `--server.http.listen-addr=0.0.0.0:12345` - before its port can be published; without it the service has no reachable - health endpoint and its status card reads down forever. -- **Locally built images** need `Pull=never` and `Image=localhost//:tag`; - registry-prefixed refs do not resolve from local storage on these hosts. -- Compose-managed containers are replaced in place by `podman run --replace - --rm`; verify afterwards that the container no longer carries - `io.podman.compose.project`. diff --git a/agent/managed-skills/container-config-drift-crashloop/SKILL.md b/agent/managed-skills/container-config-drift-crashloop/SKILL.md deleted file mode 100644 index 0ac86c3..0000000 --- a/agent/managed-skills/container-config-drift-crashloop/SKILL.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -name: container-config-drift-crashloop -description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order." ---- - -# A container that restarts forever while its definition is correct - -## When to use - -A container is crash-looping or perpetually "unhealthy", but the file that -defines it (compose service, unit, deploy script) looks right. Also when a -service has been quietly down for days and nobody noticed, because a restart -policy kept something running. - -The cause is usually **config drift**: the running container was created from an -older definition, and the restart policy restarts it instead of recreating it. -A compose/unit edit does not touch an existing container. - -## Diagnose (5 minutes, in this order) - -1. **Who started it?** The label names the owner, not your guesswork: - - ``` - podman inspect --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}' - ``` - - `podman-compose@…` means compose owns it, whatever unit files exist. - Compare `created` with when the definition last changed. A container older - than the edit that should have replaced it is the bug. - -2. **What is it ACTUALLY running?** Read the argv, not the summary fields: - - ``` - podman inspect --format '{{.Config.CreateCommand}}' - ``` - - Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device` - flags are present — a false negative that can send you hunting the wrong - root cause. `CreateCommand` is the truth; grep it for the flag you care - about (`--device`, `--group-add`, `-e`, mounts). - -3. **What does the process say?** `podman logs | grep -iE 'error|cannot|Runtime'`. - A capability error (`No HIP GPUs are available`) alongside a missing device - flag in step 2 confirms drift rather than a code or image fault. Count the - damage: `podman inspect --format '{{.RestartCount}}'`. - -## Fix - -Quickest unblock: recreate from the definition (`podman compose up -d ---force-recreate `), which is what should have happened automatically. - -Durable fix, when the host is governed by a repo that requires boot-persistent -units (this one does): migrate the service to a Quadlet. Order that worked: - -1. build the image and `install` the unit file to `~/.config/containers/systemd/` -2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a - leftover `~/.config/systemd/user/.service` collides by name with the - unit the Quadlet generator produces -3. `podman rm -f ` — the stale container holds the name the unit's - container needs, and it carries the old config -4. `systemctl --user daemon-reload && systemctl --user restart .service` -5. delete the compose service so only one owner remains per container name - -## Verify the capability, not the status - -`Up` and even `healthy` can be true of a half-configured container. Check the -thing that was broken, from inside: - -``` -podman exec python3 -c "import torch; print(torch.cuda.is_available())" -ss -ltn | grep # confirm the bind addresses, not the intent -``` - -And confirm boot persistence the way the generator exposes it: -`systemctl --user is-enabled .service` → `generated` is healthy; an -explicit `enable` fails on generated units. - -## Related reading in this repo - -- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces -- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident -- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice diff --git a/agent/managed-skills/dankclaw-guard-chain-layer-diagnosis/SKILL.md b/agent/managed-skills/dankclaw-guard-chain-layer-diagnosis/SKILL.md deleted file mode 100644 index 6d7c63a..0000000 --- a/agent/managed-skills/dankclaw-guard-chain-layer-diagnosis/SKILL.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -name: dankclaw-guard-chain-layer-diagnosis -description: "When a user reports a guarded dankclaw page (homepage/vhost) broken but the server may be healthy: verify each layer of the Anubis→Authentik→Caddy chain server-side and isolate client-side vs server-side." ---- - -# Dankclaw Guard-Chain Layer Diagnosis - -Use when a user reports a guarded page (e.g. `www.franksdank.cloud` homepage) "broken / links gone / not loading" and you must determine whether the fault is server-side or client-side/session. - -## Chain topology -`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home / @data handlers) → app/static` -- Public vhosts `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`. -- Post-auth `http://:8017` listener routes by `@ host `. -- Homepage static: `/srv/homepage/` (index.html, app.js, style.css, services.json, status.json, deployed.js). - -## Layer-by-layer checks (run from the host via ssh) -1. **Caddy unit + guard containers**: `systemctl --user is-active caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE 'anubis|authentik|caddy'` — expect `Up ... (healthy)`. -2. **Anubis challenge page validity**: `curl -sk https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143` — a healthy challenge contains `"anubis_challenge"` JSON with `difficulty` and the main JS. If present, the guard serves fine. -3. **Authentik outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep -iE 'error|exception|traceback|500|provider'` — routine dramatiq task logs are normal. -4. **@home / @data from INSIDE the caddy container** (the only way to reach 8016/8017): - `podman exec caddy sh -c 'wget -q -S -T5 --header "Host: www.franksdank.cloud" -O- http://127.0.0.1:8017/'` - and the same for `/services.json`, `/status.json`. Expect 200 + body. -5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json` parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`). - -## Gotchas (cost me calls) -- **BusyBox wget in the caddy container**: NO `-H` flag. Use `--header "Host: ..."`. -- **8016/8017 live in the caddy container's netns**: `curl http://127.0.0.1:8017` from the HOST returns `000` (connection refused) — EXPECTED, not a fault. Test from inside the container. -- **anubis container is distroless**: `podman exec anubis sh` fails (`sh: executable not found`). Inspect via its logs / the served challenge page instead. -- **x-real-ip masking**: Anubis logs `x-real-ip: 10.89.1.22` (Caddy container podman IP) for ALL clients — rootless-podman published-port SNAT. Pre-existing; clients indistinguishable by IP; clearance is cookie-based. Do NOT mistake this for the bug. -- **Homepage render**: `app.js` does `Promise.all([fetch services.json, fetch status.json])`; on ANY fetch failure the catch sets `#summary` to `status unavailable: ` and the grid stays EMPTY — which a user reads as "links gone / not loading." A 30-service status.json with a 9-service services.json renders fine (stateByName lookup). - -## Decision -If every layer above is healthy (challenge valid, outpost clean, @home/@data 200, files render), the fault is client-side/session. STOP thrashing the live system (repeated failed browser challenges share the masked 10.89.1.22 identity). Get the authenticated view: ask for the exact symptom (challenge spinning? Authentik login loop? empty grid? "status unavailable" text?) or the Authentik login to drive the page in a real browser. diff --git a/agent/managed-skills/dankclaw-homepage-asset-verify/SKILL.md b/agent/managed-skills/dankclaw-homepage-asset-verify/SKILL.md new file mode 100644 index 0000000..b56cad0 --- /dev/null +++ b/agent/managed-skills/dankclaw-homepage-asset-verify/SKILL.md @@ -0,0 +1,13 @@ +--- +name: dankclaw-homepage-asset-verify +description: "Add/verify homepage assets on Dankclaw: Caddy @data allowlist, deploy copy step, e2e panel census, live content-type check." +--- + +Every new top-level homepage asset directory (e.g. `shared/homepage/img/`, `shared/homepage/sfx/`) needs FOUR coordinated changes, all in one commit: + +1. **Caddy allowlist** — `dankclaw/Caddyfile` `@data path` matcher must include `//*`. Without it the post-auth `@home` catch-all answers with `index.html` (`200 text/html` for a PNG/OGG — silent breakage). +2. **Deploy copy** — `scripts/deploy-remote.sh` `deploy_homepage()` copies `shared/homepage//` with the `cp -r ... /srv/homepage/` merge pattern beside the fonts/img/sfx blocks. +3. **Test pins** — `tests/homepage_caddy.bats` grep-pins the new `@data` path; if the asset changes panel rendering, the e2e census in `tests/e2e/specs/10-homepage.spec.ts` must count the new render style (it derives expected names from services.json: `.panel .name` for normal cards, `.mi-name` for fused Model Information cards — a new fused component breaks the panel census otherwise, which is exactly how the 2026-10-08 Protoss deploy failed its e2e gate). +4. **Deploy both services** — `deploy.sh dankclaw --service homepage` and `--service caddy` (Caddyfile change), then live-verify from a browser tab: `fetch('//')` must return `200` with the correct `content-type` (`image/jpeg`, `audio/ogg`), not `text/html`. + +Checker note: `status.json` names follow `services.json` automatically (the deployed checker re-reads it via the homepage deploy's restart of homepage-status.timer), so renaming cards needs no checker change. diff --git a/agent/managed-skills/defuse-shell-rc-secrets/SKILL.md b/agent/managed-skills/defuse-shell-rc-secrets/SKILL.md new file mode 100644 index 0000000..c7609fe --- /dev/null +++ b/agent/managed-skills/defuse-shell-rc-secrets/SKILL.md @@ -0,0 +1,21 @@ +--- +name: defuse-shell-rc-secrets +description: "Find literal API keys/tokens in shell rc files and replace them with 600-perm secret files + loader lines, syncing the canonical tool token file." +--- + +# Defuse shell-rc secrets + +Symptom: `export KEY=...` literal in `.zshrc`/`.bashrc` — world-readable in every process env, one `git add -A` from a dotfiles repo. + +## Procedure (repeat per host) +1. Sweep: `grep -n "export.*KEY\|export.*TOKEN" ~/.zshrc ~/.zprofile ~/.profile ~/.bashrc 2>/dev/null` (run per host; when batching hosts in one ssh compound command, label which output came from which host — misattribution wastes turns). +2. Extract WITHOUT echoing: `mkdir -p ~/.config/secrets && chmod 700` it, then `sed -n 's/^export NAME="\(.*\)"$/\1/p' ~/.zshrc > ~/.config/secrets/` (quote variant and unquoted variant are different sed patterns — verify byte counts with `wc -c`, a 0-byte file means the pattern missed). +3. `chmod 600` every secret file; sync any canonical tool file the ecosystem expects (e.g. `~/.cache/huggingface/token`). +4. Replace the rc literal (hashline `PUT N.=N:` range — never echo the secret in the op): `[ -f ~/.config/secrets/ ] && export NAME="$(cat ~/.config/secrets/)"`. +5. Verify: `zsh -n ~/.zshrc` (or `bash -n`); keep a dated `.bak` of the rc on remote hosts; shred any /tmp transfer copies immediately (`shred -u`). + +## Laws +- A token pasted into chat lands in transcripts and model context — treat chat as a leak channel; give the user the `read -rs` snippet instead. +- scp of a secret FILE (encrypted channel, no argv) is fine; the token is never a command-line argument. +- After replacing a legacy HF token, probe scope non-destructively: `HfApi().whoami()` (legacy read tokens show `type: access_token` without canWrite; the decisive probe is `create_repo(exist_ok=True)` on the real target repo). +- Verify extraction byte-count before deleting/trusting; a successful-looking sed can write an empty file. diff --git a/agent/managed-skills/detached-fleet-job-launch/SKILL.md b/agent/managed-skills/detached-fleet-job-launch/SKILL.md new file mode 100644 index 0000000..9d07dce --- /dev/null +++ b/agent/managed-skills/detached-fleet-job-launch/SKILL.md @@ -0,0 +1,46 @@ +--- +name: detached-fleet-job-launch +description: Launch a long-running remote job on Dank fleet hosts so it survives ssh/logind teardown +--- + +# Detached long-running jobs on Dank fleet hosts + +## Law +`ssh host 'setsid nohup CMD & '` is NOT detached. systemd-logind deactivates +`session-N.scope` when the ssh command returns and SIGKILLs the whole cgroup: +no journal entry, no OOM trace, python block-buffers stdout so the log stays +0 bytes. setsid/nohup do not escape the cgroup. + +## Canonical launch +``` +ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \ + systemd-run --user --collect --unit=NAME \ + -p SendSIGKILL=no \ + -p StandardOutput=append:/home/dank/logs/NAME.log \ + -p StandardError=append:/home/dank/logs/NAME.log \ + sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"' +``` +- transient user unit = outside every session scope; survives ssh teardown + (requires `Linger=yes`, true for `dank` on both hosts) +- `echo $? > …rc` inside the unit = completion sentinel the unit itself writes +- `-u` / unbuffered output so the log is live +- verify launch in the same ssh call: `systemctl --user is-active NAME.service` + +## Watcher (one async job, never poll) +``` +ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \ + echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log' +``` +Run it async with timeout 0; the result is injected on completion. + +## Forensics when a detached job died +1. `ps -p PID` — gone. +2. Log 0 bytes → buffered stdout lost at SIGKILL. +3. `journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)"` — + timestamps matching the death window confirm the scope kill (vs OOM, which + leaves kernel lines). + +## Notes +- zsh remote shells: `echo ===X===` fails (`=cmd` expansion); quote sentinels. +- The HF upload of 2026-10-08 (113.7 GiB, hf_xet) died this way ~2 min in; + relaunch as above succeeded. diff --git a/agent/managed-skills/e2e-arr-rawip-reachability-triage/SKILL.md b/agent/managed-skills/e2e-arr-rawip-reachability-triage/SKILL.md deleted file mode 100644 index 4bcbe37..0000000 --- a/agent/managed-skills/e2e-arr-rawip-reachability-triage/SKILL.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -name: e2e-arr-rawip-reachability-triage -description: "Triage an e2e admin-SSO failure on a *arr raw IP (192.168.0.112:8989) by probing from the box the browser runs on, instead of assuming the Mac is asleep and burning retries waking it." ---- - -# e2e raw-IP *arr failures — probe from the browser's box, not a peer - -`tests/e2e/specs/11-admin.spec.ts` navigates the real browser to -`http://192.168.0.112:8989/` (Sonarr's forwarded IP on FrankBook-Pro). When it -fails with `ERR_CONNECTION_TIMED_OUT` the instinct is "the Mac is asleep". -Sometimes true. Not always — and waking it is not the only recovery. - -## The decisive question - -**Who cannot reach the port?** Playwright runs on the dev box (blankpad), not on -a fleet host. A healthy probe from dankclaw proves nothing about the browser's -path. - -```bash -# 1. From the box that RUNS the browser: -curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/ - -# 2. From a LAN peer, same port: -ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \ - --max-time 6 http://192.168.0.112:$p/; done' - -# 3. Liveness of the Mac itself (ICMP, not TCP): -ping -c 2 -W 3 192.168.0.112 -``` - -## Reading the matrix - -| peer probe | blankpad probe | ping | meaning | -|---|---|---|---| -| 200 | 200 | ok | transient — re-run the spec | -| timeout | timeout | fails | Mac asleep; wake, ports return in seconds | -| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP to those ports blocked. Waking and re-running cannot help | - -## If it is the third row, confirm the Mac is not at fault - -Through the dankclaw jump (`ssh dankclaw "ssh dank@192.168.0.112 ..."`): - -```bash -pgrep -fl socat # forwarders bound to 192.168.0.112:port -netstat -an | grep -c 8989 # listening -curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8989/ # 200 = app fine -/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate -``` - -Forwarders up + loopback 200 + `Firewall is disabled` ⇒ the Mac is innocent. The -block is between the dev box and the Mac, so stop optimising the Mac. - -## What to do instead - -Don't hold a deploy hostage to a known-environmental red gate: - -1. Verify the deployed service **directly** — its own ledger/health/real - completion — and state that the e2e gate was skipped or partially red, and - why (`deploy.sh --skip-e2e` prints exactly that warning, so the skip is - recorded, not hidden). -2. Report the residual gap plainly: *"29/30 e2e pass; 11-admin fails on Sonarr's - raw IP from the dev box; peer probe returns 200, so the deploy is verified by - other means and the gate gap is a dev-box network path issue."* -3. Do not weaken or skip the spec to make it green — it is guarding a real - SSO-plus-RBAC contract. - -## Note on the sleep case - -The classic flake is real: with the Mac asleep, ports refuse from **everywhere** -including peers, ping latency runs ~150 ms, and an SSH/ping wake restores -200/302 within seconds. That is the cheap first check — just don't stop there -when the peer probe is already green. diff --git a/agent/managed-skills/fandom-wiki-media-fetch/SKILL.md b/agent/managed-skills/fandom-wiki-media-fetch/SKILL.md new file mode 100644 index 0000000..41287b0 --- /dev/null +++ b/agent/managed-skills/fandom-wiki-media-fetch/SKILL.md @@ -0,0 +1,19 @@ +--- +name: fandom-wiki-media-fetch +description: Fetch audio/image media from a Fandom wiki via its MediaWiki API when the wiki HTML pages 403 curl or hang headless browsers. +--- + +Goal: get direct media URLs (and files) from a starcraft.fandom.com-style Fandom wiki. + +Why this route: Fandom HTML pages return a ~5 KB JS shell to `curl` (403 via `read ...:raw`) and time out headless Chromium (`browser.open`/`goto` >30 s on quotation megapages). The MediaWiki API is plain JSON and answers plain curl. + +Steps: +1. Find the content page's wikitext: + `curl -sL -A "Mozilla/5.0" "https://.fandom.com/api.php?action=parse&page=&prop=wikitext&format=json&formatversion=2"` + Top-level pages are often tabber/transclusion shells — follow the `{{:Subpage}}` / `{{TemplateName}}` references with the same call (e.g. `StarCraft_unit_quotations` → `/Protoss` → `Template:QuotesSC1Zealot`). +2. Extract media names from the wikitext. StarCraft quotes use `{{sm2|File.ogg|displayed quote}}` inside `{{UnitQuoteBox|select=...}}` fields — match the quote text to pick the exact file. +3. Resolve direct CDN URLs (works for images too): + `curl -sL -A "Mozilla/5.0" ".../api.php?action=query&prop=imageinfo&iiprop=url|size&format=json&formatversion=2&titles=File:A.ogg|File:B.jpg"` + → `query.pages[].imageinfo[0].url` = `https://static.wikia.nocookie.net/...` — downloadable with plain `curl -sfL`. +4. To discover art on a unit page: `action=query&prop=images&imlimit=100&titles=`, filter titles (icon/art/portrait/Head/Render), then resolve the keepers via imageinfo. Note splash/head art naming: `File:Zealot SC2 Art1.jpg`, `File:Archon SC2 Head1.jpg`; some obvious guesses (SC1-era units' SC2 splash) do not exist — verify with imageinfo, absent pages return `missing` with no `imageinfo`. +5. Verify downloads (`file sfx/*.ogg` → `Ogg data, Vorbis audio`) — a 403 body saves as a success with `-o` and no `-f`. diff --git a/agent/managed-skills/fleet-container-ownership-audit/SKILL.md b/agent/managed-skills/fleet-container-ownership-audit/SKILL.md deleted file mode 100644 index 86d22c6..0000000 --- a/agent/managed-skills/fleet-container-ownership-audit/SKILL.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -name: fleet-container-ownership-audit -description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them." ---- - -# Finding the containers that will not survive a reboot - -Use when auditing a Podman fleet for boot persistence, when a service is -"up" but stopped working at some past reboot, or when a compose-managed stack is -being converted to Quadlet units. This is the *detection* half; the conversion -checklist is separate. - -## The one-line test - -Every running container on a boot-persistent host should carry a -`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and -rootless Podman does not restart bare containers after the explicit stop that -shutdown performs — so it will be missing tomorrow morning. - -```bash -for c in $(podman ps --format '{{.Names}}'); do - printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')" -done -``` - -Compare the result against the host's installed units -(`ls ~/.config/containers/systemd/*.container`) and the repo's -`//quadlet/`. Three outcomes, three meanings: - -- **label present** → the unit owns it; boot persistence comes from `[Install] - WantedBy=default.target` plus `loginctl show-user -p Linger` = yes. -- **label absent, compose service exists** → the classic gap: it dies at reboot. - Also check the publish address while you are there; compose forms often bind - the wildcard, which puts an unauthenticated UI/API on every interface. -- **label present AND a compose service with the same name** → a second owner. - The runtime is fine, but a stray `podman compose up` can create a duplicate - that fights the unit for the container name and the port. Remove the compose - service and leave an explanatory comment; that is the repo's convention. - -## Traps that make this hard to see - -- **Editing a compose service does not replace a running container.** - `restart: unless-stopped` will happily keep an old container — old image, old - device list, old mounts — alive forever. One instance ran 9,146 restart - attempts on stale config before anyone looked. If the container predates the - edit, recreate it explicitly; do not debug the service. -- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.** - It can read empty on a container that demonstrably has the devices. Read the - real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside: - `podman exec python3 -c 'import torch; print(torch.cuda.is_available())'`. -- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape - that flips from `connection refused` to `unsupported Content-Type "text/html"` - means the service came up and only the scrape is wrong. Read the error, not - just up/down — and never "fix" a red probe by deleting the job, because an - absent scrape and a healthy one look identical. -- **A hand-written `~/.config/systemd/user/.service` that shells out to - `podman compose up` collides by name with the Quadlet the generator produces - for `.container`.** Retire it to a dated trash directory before - installing the unit. -- **`is-enabled` saying `generated` is the healthy state** for Quadlet units; - `systemctl --user enable` fails on them. - -## Migration shape that works - -1. Unit file in the repo at `//quadlet/.container`: - `Image=` with the `localhost/` prefix for locally built images, `Pull=never`, - explicit `PublishPort` (loopback unless the service is the documented LAN - exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed, - `Restart=always`, `[Install] WantedBy=default.target`. -2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator - splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle - that fails silently. -3. `deploy_()` in the remote handler: install the unit, retire any - compose-era unit into `~/.AGENTTRASH/`, `podman rm -f ` so the - unit's own can take the name, daemon-reload, restart, wait for health. -4. Delete the compose service (one owner per container name) and leave a comment - naming the unit, so the next reader does not re-add it. -5. Planner: add a path rule (`//**` -> service) **and** a - `service_is_allowed` entry, or the shared dispatcher rejects the service - before anything runs. -6. Migrate the seams: tests that read the compose file for devices, command - flags or groups must read the unit instead — and resolve file paths from the - tooling's own declarations, never from `find ... | head -1`. - -## When the last service leaves a compose file - -The repo's compose-validity seam requires at least one service, so a file whose -services have all migrated fails its own test. Either keep the file as -comments and relax that seam deliberately, or delete the file and fix the -planner's compose-diff path — decide with the owner, because the deploy tooling -reads it. diff --git a/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md b/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md deleted file mode 100644 index bc205be..0000000 --- a/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -name: fleet-llm-payload-purge-refresh -description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap." ---- - -# Purge and refresh LLM model payloads (Dank fleet) - -Use when asked to clear / reclaim / refresh model caches on a host that is -*serving* users, or to prove that a payload is up to date. - -## 1. Inventory before touching anything - -```bash -du -sh /home/dank/models/* | sort -h # what is actually there -``` - -Then find which of those the live services mount — never delete a payload a unit -mounts without an explicit, in-this-conversation instruction: - -```bash -grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \ - /home/dank/.config/containers/systemd/*.container \ - /etc/systemd/system/frank-*.service -``` - -On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn -checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF + -mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not** -`models/checkpoints` — verify, because `models/checkpoints` looks like cache but -has held Flux weights whose upstream is licence-gated. - -## 2. Confirm each payload is re-fetchable BEFORE deleting it - -Query the Hub anonymously; a gated repo 403s even with the host token: - -```bash -curl -s "https://huggingface.co/api/models//tree/main?recursive=true&blobs=true" \ - | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]" -``` - -An error/empty result means it is NOT safe to delete. Also filter by **quant -name**, never by `endswith('.gguf')` — mradermacher ships large quants as -`.partNofM` fragments which that test hides. - -## 3. Fingerprint, then delete - -Record byte sizes (and sha256 when the file matters) **before** removal: that is -the only way to answer "did the refresh actually change anything" afterwards. - -Per this repo's policy `rm` needs explicit operator permission; otherwise stage -into `~/.AGENTTRASH/-/`. Note that a same-filesystem trash reclaims -nothing — say so rather than implying space was freed. Root-owned leftovers (a -trashed swapfile) need `sudo`, and check `swapon --show` first. - -## 4. Refresh, per service - -**frank-large (halogen)** — the engine fetches for you. Clear the directory but -leave it **existing and empty**, then start the unit: - -```bash -rm -rf /home/dank/models/halogen/* # keep the directory itself -systemctl --user start frank-large.service -journalctl --user -u frank-large.service -f | grep -a halogen: -``` - -The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD` -(`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`, -resuming if interrupted. **Trap:** podman refuses a bind mount whose source -directory is missing — `Error: statfs /home/dank/models/halogen: no such file or -directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty -directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load. - -**frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start -with weights missing, so pull them yourself and verify: - -```bash -DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF -# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf -curl -fL -C - --retry 10 -o "$DEST/" \ - "https://huggingface.co//resolve/main/" # -f is mandatory -sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to the LFS oid + GGUF magic -``` - -`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always -`-f`, always compare byte counts to the Hub's declared size, always check the LFS -sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from -argv entirely (it is world-readable in `/proc//cmdline`). - -## 5. Verify, then record - -```bash -curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # frank-large -curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health # frank-small -curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models -# a REAL completion on both, then halogen's boot ledger: -journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything" -``` - -Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`, -`gitea`, `loki` — `curl -sk --resolve :443:192.168.0.143`) because a purge -touches state the whole host shares. - -## 6. Things that bite - -- **Sidecars are versioned in place.** Equal file sizes are not proof the repo is - unchanged, and a stale local sidecar is invisible from its name. Compare local - sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB - overlay update was caught. -- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The - A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding - cache churn there risks the OOM/state failures this fleet has already seen. -- **Reboot is the repair of last resort** if rootless podman's pause process dies - during the work (`invalid internal status … could not find any running - process`); `podman system migrate` panics in 5.7.0. -- Because a peer session may be committing to `serverAppConfig` concurrently, - re-read a doc region immediately before editing it, and check that the edit - response's neighbouring lines are the ones you expect. diff --git a/agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md b/agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md deleted file mode 100644 index 6e37eb3..0000000 --- a/agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -name: fleet-model-cache-purge-refresh -description: "Clear and re-fetch model payloads on a Dank-fleet host that serves production LLM endpoints — inventory, protect live payloads, verify every fetch, restore the service, and use the refresh to detect upstream updates." ---- - -Clearing a model cache on a serving host is a two-sided operation: the operator wants no stale -payload, and the host must keep serving. Do it in this order, on the target host. - -## 1. Inventory before touching anything - -```bash -du -sh /home/dank/models/* | sort -h # payload dirs, biggest last -grep -rlE "models/|" /home/dank/.config/containers/systemd/ /etc/systemd/system/*.service -``` - -Classify every dir: **live service payload** (referenced by a unit/env), **retired runtime -payload**, **unreferenced model dir**, **tool/script dir**, **logs** (evidence — keep). Check the -service env files (`scripts/*.env`) for `MODEL_DIR`/`MODEL` paths, not just unit `Volume=` lines: -a payload can be mounted by path and named by env. `models/logs` and small tool dirs are not cache. - -## 2. Record fingerprints first - -`ls -l --block-size=1 ` gives sizes; for the LLM payloads compare each to the Hub's declared -size (`/api/models//tree/main?recursive=true&blobs=true`). They can differ — see §6. If a -file is irreplaceable upstream (delisted revision, gated repo) say so before deleting it. - -## 3. Protect live payloads by delete-then-refetch, never delete-only - -Deleting a live payload leaves the endpoint unable to start. Know the recovery path *before* -deleting: some engines fetch their own weights (halogen: `HALOGEN_DOWNLOAD`, triggered when the -checkpoint is absent), others need a manual fetch (llama.cpp GGUF services: the serve script -refuses to start without the weights). - -**Bind-mount trap:** podman refuses a mount whose source directory is missing — -`Error: statfs /home/dank/models/: no such file or directory`, with the unit looping on -`Restart=on-failure`. Recreate the directory **empty** (`mkdir -p`), and the engine's first-start -fetch runs. - -## 4. Fetch with verification (never trust an exit code) - -`curl -o FILE URL` exits 0 on 401/403 and writes the error body — a gated repo produced three -153-byte shards with identical sha256s and a cheerful DONE line. Always: `-fL -C -`, compare the -byte count against the Hub's declared size, and compare `sha256sum` against the LFS oid. Public -repos means **no token in argv** (it is world-readable in `/proc//cmdline`). - -**Parts trap:** mradermacher-style repos ship large quants as `.part1ofN … .partNofN` — a filter on -`endswith('.gguf')` makes those quants look absent. Fetch the parts, verify each, then -`cat part1 part2 part3 > name.gguf` in numeric order; the assembled file's size equals the sum -(concatenation is deterministic, so per-part hashes verify the whole). - -## 5. Restore, then verify behaviour — not health alone - -Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), `reset-failed`, `start`; -root units need `sudo`. Watch the engine's own fetch/load lines in the journal, then check -`/health`, `/v1/models`, and a **real completion** on every restored endpoint. Read the engine's -memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits. - -## 6. A refresh is also an update check - -Clearing and re-fetching only guarantees freshness *at fetch time* — if upstream has not changed, -the bytes come back identical, which is itself the finding. Compare recorded fingerprints against -the new local sizes and the Hub's declared sizes: sidecars are often versioned **in place**, so a -stale copy is invisible from its file name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 -while the main checkpoint returned byte-identical). - -## 7. Close out - -- Clear test payloads once the benchmark using them is done, and keep the payload the *next* - experiment needs. -- Leave the operator's decision points explicit: unreferenced-but-licensed dirs, other services' - caches, and whether they want the payloads absent at their next run or fresh as of now. -- Record sizes/verdicts in the vault (runbook + `log.md`); never `rm` without explicit permission, - and remember a same-filesystem trash reclaims no space. diff --git a/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md b/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md deleted file mode 100644 index 0476867..0000000 --- a/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -name: fleet-monitoring-gap-audit -description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses." ---- - -# Auditing a homelab fleet's monitoring for what *should* be there - -Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose -services are declared in a git repo. The two failure modes this catches are -different, and only the first announces itself. - -## 1. A service that should exist and doesn't is invisible - -A missing exporter shows up as **nothing**: no failing target if the job was -configured for a host that no longer serves, no alert because the series is -simply absent. So check existence, not health: - -```bash -# Every container the host actually runs, including dead ones -podman ps -a --format '{{.Names}} {{.Status}}' -# Every container that is supposed to be boot-persistent -ls ~/.config/containers/systemd/*.container -systemctl --user is-enabled .service # expect: generated -``` - -A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic -case: rootless Podman does not restart bare containers after the stop that -shutdown performs, so a compose-only service dies at the first reboot and stays -dead. Check the compose file for the name — if it is there and has no unit, the -repo's own rule is being violated. Fix by writing the unit (mirror the same -service on the other host), deleting the compose entry (one owner per container -name), and making sure the deploy path can reach it: - -- planner: service in the target's service set + a path rule for its directory -- dispatcher allowlist: `:` -- handler: install from `"$TARGET/..."`, never a hardcoded peer's path - -## 2. Classify every down target — do not report a count - -`targets?state=active` gives health **and** the last error. Two down targets can -mean opposite things: - -| Symptom | Meaning | -|---|---| -| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) | -| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine | -| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter | -| target absent entirely | see §1 | - -```bash -curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c " -import json,sys -for t in json.load(sys.stdin)['data']['activeTargets']: - if t['health'] != 'up': - print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])" -# and probe each by hand before believing the scraper -curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http:///metrics -``` - -Say which are down and why. "7 of 11 up" without the four reasons is how a real -outage hides behind three config defects — that is exactly what one fleet did: -its crash-looping GPU service was one of four failures reported as a number. - -## 3. Check the exposure posture while you are in there - -An exporter that binds the wildcard publishes an unauthenticated API on every -interface. Expect exactly two listeners per host-exporter — loopback for -host-local probes, the host's own LAN address for the Prometheus container: - -```bash -ss -ltn | grep # must show the two explicit addresses, never 0.0.0.0 -``` - -## 4. Leave the decisions that are not yours - -Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**: -disable a probe to make a dashboard green, or "fix" a crash-looping service whose -correct posture depends on what the owner wants — capture the evidence (restart -count, exit code, last error line) and hand it over intact. diff --git a/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md b/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md deleted file mode 100644 index 4014ffb..0000000 --- a/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -name: fleet-second-llm-endpoint -description: "Add a second LLM endpoint to a host that already serves one on the Dank fleet (new port, unit shape, co-tenancy budget, harness provider/roles, monitoring, cutover, verification) — with the port-bind race, the reasoning_effort template trap, and the tools/--jinja trap that silently break subagents." ---- - -# Adding a second LLM endpoint to a Dank fleet host - -Use when a host must serve two models at once (e.g. an iGPU runtime plus a -second card), or when one endpoint is moving and a new one takes the freed -hardware. Rebuilt from the 2026-09-16 Dankclaw split (halogen `frank-large` on -:7770 + llama.cpp `frank-small` on :7771). - -## 0. Decide the shape before touching hardware - -- **One provider per HOST:PORT** in `~/.omp/agent/models.yml` (on the box where -the harness runs, not on the servers). A second endpoint is a new provider -block, not a second model in the existing one. -- **Read the budget as an inequality, not a wish.** For a unified-memory runtime -the engine's own fit check is the arbiter (`budget = host − pinned weights − -reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV by -counting only the full-attention layers — and remember `--parallel N` DIVIDES -`--ctx-size`, it does not multiply. -- **A co-tenant is a budget line, not a footnote.** Free host RAM by changing -the POOL or the ARENA (the reserve is a fit-check term, never an allocation), -and drop optional charges (a vision tower is ~2 GiB) rather than hoping. - -## 1. Repo first, and retire in the same commit - -1. New service files: `scripts/.env` (documented tunables), `.service` - (root unit for a native binary; a Quadlet only for a container), - `serve-.sh` (defaults + hard guards for missing artifacts). -2. Planner: add the service to `SERVICE_ORDER` AND the target's service set in - `scripts/deploy-plan.py`, map its paths (including "the unit/env/serve trio - selects only me"), add it to the inventories in `deploy.sh` and - `deploy-remote.sh`, and the dispatch case. -3. Retire what the cutover obsoletes into `.AGENTTRASH/` with a suffix — and - **check what pointed at it**: bench harnesses, a chat template only the old - runtime read, and `scripts/test-changed.sh` mappings are the usual victims - (a mapping to a test file retired earlier fails the pre-commit wrapper - silently for weeks). -4. Seams: one bats file per service asserting the *invariants* (`pool == 2 × - ctx`, KV bytes ≤ card budget, port separation), not just constant values. - A negative grep must strip comments (`grep -vE '^\s*#' file | grep -q ...`) - or it will match the prose that explains the very thing you removed. - -## 2. The port-bind race (rootless podman) - -pasta binds the published port at container CREATE. A predecessor that has just -stopped still holds it, so a naive handler starts a doomed container: - -``` -Deactivated successfully <- predecessor gone -pasta: Listen failed for HOST TCP port */PORT: Address already in use -exit 126 <- only Restart=on-failure saves you -``` - -Order the handler: **retire the old manager's unit → stop THIS service's own -unit (no-op on a first cutover) → wait for every socket on the port to clear -(listener, accepted, TIME_WAIT) → install → start.** Without the second step a -redeploy deadlocks against its own gate. - -## 3. Wire the surfaces, and refuse impossible signals - -- Harness roles: point the small/fast + image roles at the new endpoint, and - move the vision role with the vision tower (only one endpoint should own - each capability). -- `shared/homepage/services.json`: a card per endpoint, probed from the - monitor's own host (loopback is legal there; the LAN address otherwise). -- Prometheus: only add a job if the runtime actually exposes Prometheus text. - Halogen publishes `/health`, `/cache` and `/v1/*` and nothing else — scraping - it would be red by construction, so the job is REMOVED with the reason written - down rather than left as a permanent red. - -## 4. Cutover, then verify by driving it - -```bash -./scripts/deploy.sh --service --service # order matters -``` - -Then, per endpoint: `/health`; `/v1/models` reports the alias; a chat completion -returns content AND `reasoning_content`; **a request carrying `tools` returns -`tool_calls`**; an image request is answered (or refused with a named reason); -the runtime's own speculation counters move; and the unit is `enabled` / -`generated` with linger on. Read the engine's boot ledger for a silent clamp. - -Finally spend one real request through the harness per changed role — routing -labels lie, the servers' own journals do not: `journalctl -u ` shows which -endpoint actually served it. - -## Traps that look like model faults - -| symptom | cause | -|---|---| -| `500 tools param requires --jinja flag` | the endpoint's runtime lacks the Jinja chat-template path, so every tool-using agent on it is broken. mainline llama.cpp defaults it on — pin it | -| `500 Jinja Exception: Unexpected reasoning effort ` | the model's own template has a narrower accepted set (Qwen3.8: xhigh/medium/low only) than the harness's levels; fold them in `compat.reasoningEffortMap` | -| the same request works by hand but a subagent dies | subagents run at a different thinking level — check both | -| the container dies instantly although the predecessor "stopped" | the port race above | diff --git a/agent/managed-skills/fleet-service-precommit-iteration/SKILL.md b/agent/managed-skills/fleet-service-precommit-iteration/SKILL.md deleted file mode 100644 index f78ca9e..0000000 --- a/agent/managed-skills/fleet-service-precommit-iteration/SKILL.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -name: fleet-service-precommit-iteration -description: "Iterate on a service that lives in the serverAppConfigs git checkout on a fleet host (dankclaw/dankserver) before committing: ship files for live testing, keep the checkout pullable, then commit and deploy — covers the untracked-files pull trap that recurs every session." ---- - -# Iterating on a fleet service before you commit - -This repo's contract is git → `deploy.sh` → host. But live testing needs the new -file *on the host* before it is committed, and the naive way (copy it into the -host checkout) blocks the very deploy that would install it. This is the loop -that works, plus the traps that cost real time. - -## The loop - -1. Edit in the local repo. Run the seams: `./tests/run.sh`. -2. Ship the working files to the host **for testing**: - `cat local/file | ssh 'cat > /home/dank/serverAppConfigs/'` - (then `chmod +x` if it is executed — scp/`cat >` drops the exec bit, and a - container entrypoint mounted from that path fails with status 126, "command - found but not executable"). -3. Test against the live service; iterate until it works. -4. **Before committing**, prove the host copy is byte-identical to what you are - about to commit: `sha256sum` locally and on the host. If they differ, you are - about to lose work in step 5. -5. Commit and push. -6. Clean the host checkout so the deploy can pull: - - untracked hand copies → move them out of the checkout - (`mkdir -p /tmp/handcopy && mv /tmp/handcopy/`); - - modified tracked files → `git checkout -- ` (this *discards* the - hand copy, which is safe only because of step 4); - - confirm with `git status --short` printing nothing. -7. `./scripts/deploy.sh --service ` — it pushes, pulls ff-only, - installs units/config, restarts, waits for health, then runs the Playwright - e2e gate. - -## Traps, in the order they bite - -- **The pull aborts on hand copies.** `deploy.sh` pulls with ff-only and git - refuses: *"The following untracked working tree files would be overwritten by - merge"*. Any untracked file in the checkout whose path the pull wants to - create stops the deploy — which is exactly the file you just shipped. Always - do step 6. -- **`git checkout --` silently eats uncommitted work.** Without step 4 you can - destroy the only copy of an edit that exists nowhere else. -- **A runtime mounts files *straight out of the checkout*.** Here - `dankserver/frank-medium/**` and `shared/grafana/provisioning/**` are mounted - read-only into containers, so a hand copy takes effect at the next container - restart with no deploy. Test that way, then let the deploy make it official. -- **Root-owned files from a container write** (e.g. a patched `.so` produced - inside a build container) can land in the checkout as `root:root`; `chown` them - back to the deploy user before git operations. -- **A restart is not a verification.** Health endpoints answer from a stale - process, a proxied service can be served by a peer, and a provisioned - dashboard may live in unified storage rather than the SQLite table you grep. - Verify the *specific* claim: the new code path's own output, the file's - checksum on the host, the API that serves the resource. - -## Why not keep a scratch copy on the host? - -Because the deployed artifact must be the committed one. Iterating outside git -(the `/tmp` loop is fine for *probes*) is for measurement scripts; anything the -service actually runs has to land in the checkout, and the checkout has to stay -pullable. Two directories, one rule: probes may live anywhere, runtime state -lives in git. diff --git a/agent/managed-skills/flm-npu-pack-registration/SKILL.md b/agent/managed-skills/flm-npu-pack-registration/SKILL.md deleted file mode 100644 index d9bc4cf..0000000 --- a/agent/managed-skills/flm-npu-pack-registration/SKILL.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -name: flm-npu-pack-registration -description: "Runbook for registering a custom FastFlowLM NPU2 model pack on a tag that isn't in FLM's built-in catalog (Dankclaw :7771), including the LFS-pointer and vision-weight traps and the NPU hwctx contention rule." ---- - -# FLM custom-tag pack registration (Dankclaw NPU endpoint) - -Use when adding a non-catalog NPU2 pack (e.g. an extraction or fine-tuned model) to the FLM registry served by `frank-npu` on port 7771. Companion diagnosis skill: `flm-tag-load-failure-diagnosis`. - -## Registry facts -- User registry: `/home/dank/.config/flm/model_list.json`, shape `{"model_path": "models", "models": {"": {"": {name, url, file_url, size, flm_min_version, files[], vlm, default_context_length, max_prefill_len, details{format:"NPU2", family, ...}, label[]}}}}}`. -- Serve resolves tags through `FLM_CONFIG_PATH` at startup; `flm list` with the env var set lists custom tags (`⏬` = not downloaded). - -## Procedure -1. Back up the registry JSON first (`cp -a`). -2. Insert the entry via a python json edit + atomic `os.replace`. Set `files` to the FULL HF file list — **including `vision_weight.q4nx`** even for text-only packs: the qwen3.5-family loader opens it unconditionally and a missing file makes the first request HANG silently (no error log). -3. **Do not use `flm pull ` for custom entries** — it throws `json.exception out_of_range key ':' not found` while building the download list. Place files manually into `/home/dank/.config/flm/models//` with: - `curl -sfL "https://huggingface.co//resolve/main/?download=true" -o ` - Plain `raw/main/` returns 135-byte LFS pointer stubs, not weights — verify byte counts against the HF tree API before trusting a download. -4. Kernel packs: copy from the same-shape family pack (shape-match `config.json` on hidden_size/num_hidden_layers/head_dim first) into `/opt/fastflowlm/share/flm/xclbins//` via `sudo cp -a` (root-owned dir). -5. Deploy/restart the serve unit (deploy path, not ad-hoc restarts) so the new process loads with all files present. -6. Verify with a REAL completion request on 7771 (`model: ":"`) — `/v1/models` alone proves registration, not loadability. - -## Hard constraints -- Only ONE `flm serve` process can hold the NPU hardware context. A second concurrent serve process fails every load with `DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status 0x2000003). Never run scratch/verify instances while production holds the NPU — verify via the deployed service instead. -- Tag switching happens INSIDE one serve process; first request after a model switch pays ~6–11 s cold-load cost (visible in `usage.load_duration`). -- Over ssh, never `pkill -f ""` with a pattern present in your own command line — it kills your own session; use bracket-guarded patterns (`[q]wen3.5:4b`) or exact PIDs. diff --git a/agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md b/agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md deleted file mode 100644 index cf66834..0000000 --- a/agent/managed-skills/flm-tag-load-failure-diagnosis/SKILL.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -name: flm-tag-load-failure-diagnosis -description: "Diagnose FLM (FastFlowLM) NPU serve failures where a tag listed in /v1/models answers \"Failed to load model: No such file ...xclbin\" — missing kernel-pack symlink under FLM_XCLBIN_PATH is the root cause; includes per-request switch-cost behavior." ---- - -# FLM tag load-failure diagnosis (Dankclaw sight+sound endpoint) - -## Symptom -`POST /v1/chat/completions` on the FLM serve endpoint answers **HTTP 200** with a flat error body `{"error":"Failed to load model!"}` and no `choices`. `/v1/completions` instead returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP 200. A status-code-only client reads this as a successful empty completion — always check for the `error` key. - -Two distinct causes, distinguish via `journalctl -u frank-default-sightsound.service`: - -1. **Unknown tag** (not in the registry): immediate error, nothing loaded. Correct behavior; not a bug. -2. **Registered tag whose kernel pack is missing**: FLM resolves the tag to its pack dir, downloads missing weight files on the fly (first request hangs ~90–110 s downloading), then fails with `Failed to load model: No such file '/home/dank/.config/flm/xclbins//.xclbin'`. Subsequent attempts fail fast (~2 s). - -## Root cause of case 2 -The serve script exports `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so **every** kernel lookup resolves under the user dir — but that dir only contains symlinks for the packs someone linked. A tag listed in `/v1/models` is NOT proof it is loadable on this host. - -## Fix -One symlink per missing pack, run on Dankclaw: -``` -ln -s /opt/fastflowlm/share/flm/xclbins/ /home/dank/.config/flm/xclbins/ -``` -Verified 2026-08-30: `Qwen3.5-4B-NPU2` — after the symlink, first request loaded and answered in 7.4 s wall (cold load + generate). - -## Switch-cost behavior (measured same day) -- The `model` field routes per-request; no restart needed to switch between registry tags. -- A request whose model differs from the resident one triggers a full reload ("Loading model:" in journal before prefill): measured 7–11 s wall for an 8-token completion on ~4B-class packs. Same-tag requests are fast; alternating tags pays the swap each time. -- Single-lane NPU: concurrent requests queue FIFO behind the current lock holder (`NPU Locked!` → `NPU Lock Released!`). - -## Probe recipe (repeatable) -1. `curl -s http://192.168.0.143:52626/v1/models` — registry listing is authoritative for *registered* tags, not loadability. -2. On a failed load, grep the journal for `Failed to load model` (use `--full`) — the quoted xclbin path names the missing pack dir exactly. -3. Compare `ls /home/dank/.config/flm/xclbins/` (symlinks = installed) against `/opt/fastflowlm/share/flm/xclbins/` (all available packs); diff = fixable with symlinks. diff --git a/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md b/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md deleted file mode 100644 index 4d6fae5..0000000 --- a/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -name: frank-medium-batch-split-retune -description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify." ---- - -# frank-medium batch geometry + split rebalance - -Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the -model on frank-medium, or when someone reports that prefill "does not respond" to -those knobs. The batch and the tensor split are **coupled**: the compute buffers are -not symmetric across the two cards, so any batch change moves the balance point. - -## Order of work (do not skip the second half) - -1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold - a shape constant across dozens of configs: - `scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation - per cell, `-r 3 --no-warmup`. -2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by - loading the real service with a per-config env override and reading `rocm-smi`. A - llama-bench number cannot decide the split: it has no `--parallel`, sizes its own - context as `n_prompt + n_gen + n_depth`, and cannot load the drafter. -3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed - point, then **deploy** and verify. - -## 1. The sweep - -```bash -# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is -# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here -sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \ - --property=LimitMEMLOCK=infinity --property=User=dank \ - --setenv=OUT=/home/dank/models/logs/bench-bub \ - --setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \ - /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh -``` - -The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv` -(appended as it goes, so progress is readable), samples VRAM per cell, and restarts -the service from an EXIT trap. A re-run skips cells that already have a `bench.json`. - -Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not -track the request, nothing else in the table means anything. The VRAM columns scaling -monotonically with ubatch is the independent confirmation. - -Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth): -an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046, -384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral -(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every -cell). Test **both** ends of the range: sampling only upward makes a shallow basin look -like a plateau. - -Sanity rule: the same ubatch measured under different logical batches should agree to -under 1 tok/s. If it does not, the table is noise and the first suspect is a stray -GPU user, not the knob. - -## 2. The split, at the deployed shape - -`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env` -to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with -`FRANK_MEDIUM_ENV_FILE=`, polls `/health` for `"status":"ok"`, reads -`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under -`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`. - -**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH -cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the -process dies — a 180 s sleep fails, and even 600 s has failed while the process is still -alive inside the unit. Skipping the gate produces -`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from -"this split does not fit" and will send you chasing a phantom OOM. - -Free bytes to expect, and the arithmetic behind them: -`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs -the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB -against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one -layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the -reachable minimum gap is about one DeltaNet layer, never zero. - -2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32; -**35,30 @512 -> 6.44 / 6.90 (deployed)**. - -## 3. Seam, deploy, verify - -- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured - 1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the - model reproduces the new point. Check that neighbouring splits still FAIL the balance - gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the - slack at one DeltaNet layer. -- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares - summing to the model's 65 blocks, ROCm1 the smaller). -- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`. -- Verify live: the flags in `/proc//cmdline`, `/slots` = 2 x 131072, VRAM matching - the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance` - lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB - lower once the service has served traffic than straight after load. diff --git a/agent/managed-skills/frank-primary-decode-tuning/SKILL.md b/agent/managed-skills/frank-primary-decode-tuning/SKILL.md new file mode 100644 index 0000000..2108d4a --- /dev/null +++ b/agent/managed-skills/frank-primary-decode-tuning/SKILL.md @@ -0,0 +1,22 @@ +--- +name: frank-primary-decode-tuning +description: "Tune frank-primary (radiance TP=2) decode throughput: benchmark harness usage, flag levers, known optima and boot-failure walls." +--- + +# frank-primary decode tuning + +Harness: `bash autoresearch.sh` (dev box) scp's `scripts/frank-primary.env` + `scripts/autoresearch-frank-primary.sh` to dankserver, stops prod (`systemctl stop frank-primary`, NEVER mask — relinked unit), boots scratch container `arbench` on :7770 (fresh prefix-cache tier at `$STATE_DIR/kvcache/arbench` for cold prefill), polls health 720s, sends ONE greedy completion (fixed 140k-char repo-corpus prompt -> 43,627 tokens, 256-token decode), parses `timings` into METRIC lines, tears down, restores prod via EXIT trap. ~4 min/run; decode noise ~0.3%. + +Metric: `METRIC decode_tps` = `timings.predicted_per_second` at 43.6k depth. + +## Optima (measured 2026-10-06, keep these) +- `FRANK_PRIMARY_SPEC_TOKENS=2` (unimodal: 1->31.89, 2->32.40, 3->31.72, 4->24.57 t/s; acceptance collapses past 3) +- `FRANK_PRIMARY_MAX_SEQS=3` with `MAX_MODEL_LEN=200000` (600k KV; +4.2% total; 4 slots regresses) +- `FRANK_PRIMARY_EXPERT_VS_CACHE=0.82` bracket-confirmed (0.78 fails boot, 0.84 -8.9%, 0.87 collapse) +- `TP_WIRE=wht6`, `MAX_BATCHED_TOKENS=2048` — REQUIRED at seqs=3: bf16 wire and MNBT 4096 both fail boot (VRAM) + +## Non-levers (decode-neutral, don't spend runs) +GPU_HEADROOM_MIB (48 = neutral), HOST_POOL_MIB 49152 (neutral), MAX_MODEL_LEN any change (regresses: bigger KV steals resident weights), prefix-cache tiers (prefill-side). + +## Rules +One flag per iteration; keep only > +1% decode over best; a boot failure is `crash` (trap restores prod). Constraint floor: MAX_SEQS*MAX_MODEL_LEN >= 400000. diff --git a/agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md b/agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md deleted file mode 100644 index 7af8e3e..0000000 --- a/agent/managed-skills/frankbook-qbittorrent-e2e-verify/SKILL.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -name: frankbook-qbittorrent-e2e-verify -description: "Verify the FrankBook-Pro gluetun→qBittorrent grab pipeline end-to-end (egress gate inside gluetun, generic SID login, legal test torrent, category path check, data-safe cleanup) after any tunnel/qB/category change" ---- - -# FrankBook-Pro qBittorrent E2E verification - -Use after any change to the FrankBook-Pro torrent pipeline — WireGuard credential rotation, qB version change, gluetun reconfig, category re-seeding. Proves the full grab path (tunnel egress → qB download → category save path → clean removal). All commands run from Dankware via `ssh frankbook-pro`; inside the Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for `podman machine ssh`. - -## 1. Egress gate — MUST run inside gluetun - -```bash -podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json' -``` - -- Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config). Judge by org, never country alone. -- NEVER run this one ssh-context out (e.g. bare `wget` on the Mac host): it silently returns the residential Comcast IP and reads as a hard-gate leak that does not exist. If a check shows a residential IP, first re-run inside the container before diagnosing. - -## 2. Login — generic cookie parse - -The QPW fetch is flaky (~1 in N returns empty); assert length before use. An empty-password run once produced a Forbidden loop that cost hours. - -```bash -QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2) -[ ${#QPW} -eq 20 ] || exit 1 -SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \ - http://127.0.0.1:8080/api/v2/auth/login \ - | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1) -curl -b "$SID" ... # pass the WHOLE match -``` - -- qB 4.x cookie name is plain `SID`; qB 5.x uses `QBT_SID_`. Hard-coding either name breaks the other version's parse (sidlen=0 → 403 cascades). -- Parse-bug 403s never reach qB as failed logins — no ban accrues. Check server-side truth in the log: `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log` (`WebAPI login success/failure` + source IP). Repeated REAL failures accrue toward qB's 5-failures/3600s IP ban. - -## 3. Add a legal test torrent - -Blender open-movie torrents (~300 MB, e.g. Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file (`-F "file=@..."`) and `category=Finished Torrents`. Never leave a real-looking grab behind. - -## 4. Poll and verify - -Poll `GET /torrents/info` (~20 s): state `downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading` at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents` (qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT inherit category paths — one more reason for the 4.6.7 pin). Confirm on disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is normal for the tunnel. - -## 5. Cleanup — delete only your artifact - -```bash -curl -s -o /dev/null -w "%{http_code}" -b "$SID" \ - --data-urlencode "hashes=" \ - --data-urlencode "deleteFiles=true" \ - http://127.0.0.1:8080/api/v2/torrents/delete # expect 200 -``` - -- Target the known hash literally — the user's real grabs share the directory and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir gone, all other entries intact. - -## Version-coupling constraints - -- qB is pinned to 4.6.7 (readarr's EOL client rejects qB ≥ 5's `204` login at runtime). Do not unpin while readarr exists. -- After ANY qB version flip, regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k iterations) — a 5.x-written hash line makes 4.6.7 reject every login. Rewrite while qB is stopped. diff --git a/agent/managed-skills/freetoken-build-dankclaw/SKILL.md b/agent/managed-skills/freetoken-build-dankclaw/SKILL.md deleted file mode 100644 index 7d1852c..0000000 --- a/agent/managed-skills/freetoken-build-dankclaw/SKILL.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -name: freetoken-build-dankclaw -description: "Build FreeToken on Dankclaw (Ubuntu 24.04) with CUDA — installs nvidia-cuda-toolkit, sets CUDA_HOME=/usr, overrides nvcc/CUDA version mismatch" ---- - -# FreeToken Build on Dankclaw - -Dankclaw is Ubuntu 24.04 with ROCm already installed. FreeToken builds for **CUDA** (not ROCm) because the kernel package (`_pinned_tensor`) links against `cudart`. - -## Prerequisites (install if missing) - -```bash -export PATH="/home/dank/.local/bin:$PATH" -export CUDA_HOME=/usr -export FREETOKEN_ALLOW_CUDA_MISMATCH=1 - -# CUDA toolkit (if not present): -sudo apt-get install -y nvidia-cuda-toolkit -# UV (if not present): -curl -LsSf https://astral.sh/uv/install.sh | sh -``` - -## Key environment variables - -|Variable|Value|Why| -|---|---|---| -|`CUDA_HOME`|`/usr`|Ubuntu CUDA toolkit package places nvcc at `/usr/bin/nvcc` and libcudart at `/usr/lib/x86_64-linux-gnu/`| -|`FREETOKEN_ALLOW_CUDA_MISMATCH=1`|Set|nvcc is 12.4 but torch 2.11.0+cu130 ships CUDA 13.0 — safe to override| -|`PATH`|`/home/dank/.local/bin:$PATH`|uv installs here| - -## Build procedure - -```bash -cd /home/dank -git clone https://github.com/FlashML-org/FreeToken.git -cd FreeToken -uv venv && source .venv/bin/activate -uv pip install -e ".[accel]" -``` - -## Verification - -```bash -ft --version # → freetoken version 0.1.2 -python3 -c "import torch; print(torch.__version__)" # → 2.11.0+cu130 -python3 -c "from freetoken.kernel import _pinned_tensor; print(_pinned_tensor.__file__)" -python3 -c "import flashinfer; print(flashinfer.__version__)" # → 0.6.18 -python3 -c "import sgl_kernel; print(sgl_kernel.__version__)" # → 0.4.5 -``` - -## Notes - -- `torch.cuda.is_available()` may return `False` with "Can't initialize NVML" warning — this is the management library, not CUDA runtime. The kernels are built and linked correctly. -- The NVIDIA driver modules may or may not be loaded; they don't affect the build. -- ROCm is present on Dankclaw but FreeToken's kernel build specifically requires CUDA (links `cudart`). diff --git a/agent/managed-skills/freetoken-dankclaw-build/SKILL.md b/agent/managed-skills/freetoken-dankclaw-build/SKILL.md deleted file mode 100644 index 2313878..0000000 --- a/agent/managed-skills/freetoken-dankclaw-build/SKILL.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -name: freetoken-dankclaw-build -description: "Build FreeToken on Dankclaw — installs CUDA toolkit, sets CUDA_HOME, handles nvcc/CUDA version mismatch" ---- - -## Build FreeToken on Dankclaw - -### Prerequisites -- CUDA toolkit: `sudo apt install nvidia-cuda-toolkit` (gives nvcc at /usr/bin, libcudart at /usr/lib/x86_64-linux-gnu/) -- uv: installed at /home/dank/.local/bin/uv (or install via `curl -LsSf https://astral.sh/uv/install.sh | sh`) -- CMake, git, Python >= 3.10 — all present - -### Build steps -```bash -cd /home/dank -git clone https://github.com/FlashML-org/FreeToken.git # or cd into existing -cd FreeToken -uv venv && source .venv/bin/activate -export CUDA_HOME=/usr -export FREETOKEN_ALLOW_CUDA_MISMATCH=1 # nvcc 12.4 + torch cu130 is safe -uv pip install -e ".[accel]" -``` - -### Verification -```bash -ft --version -python3 -c " -from freetoken.checkpoint import ftw -import torch -t = torch.randn(3,4,dtype=torch.float16) -print(ftw._dtype_str(t.dtype), ftw._elsize(t.dtype)) -" -``` - -### Key notes -- FTW format is dtype-agnostic: FP16, BF16 (default), FP32, FP8 all work -- GGUF support: only Gemma4GGUFForCausalLM -- GGML types: F32/F16/BF16 (native views), Q4_0/Q6_K (dequant), Q8_0 (listed, no dequant) -- The NVIDIA driver modules may not be loaded at build time (NVML warning) — not a problem for the build itself diff --git a/agent/managed-skills/freetoken-hf-model-swap/SKILL.md b/agent/managed-skills/freetoken-hf-model-swap/SKILL.md deleted file mode 100644 index 2e0dfab..0000000 --- a/agent/managed-skills/freetoken-hf-model-swap/SKILL.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -name: freetoken-hf-model-swap -description: "Swap the FreeToken model behind frank-default-small on Dankserver: preflight the HF quant format against the loader matrix BEFORE downloading, then download, repoint MODEL_DIR, deploy, and verify with a REAL completion (health 200 is not enough)." ---- - -# FreeToken HF Model Swap (frank-default-small, Dankserver) - -Swap the model behind `frank-default-small` (FreeToken v0.1.2, Dankserver, RTX 3090) by downloading a HF safetensors checkpoint into a model dir, repointing `MODEL_DIR` in `scripts/frank-default-small.env`, and deploying. Verify with a REAL completion — health 200 is NOT enough. - -## Preflight — verify quant format BEFORE downloading - -FreeToken v0.1.2 loads ONLY modelopt-style quant configs for `qwen3_5_moe`. Every compressed-tensors FP8 checkpoint fails (live-verified 2026-09-01: two candidates, 72 GB downloaded, both unloadable). Check before any multi-GB download: - -```bash -curl -sL https://huggingface.co///raw/main/config.json | \ - python3 -c "import json,sys; c=json.load(sys.stdin); q=c.get('quantization_config',{}); print(q.get('quant_method'), q.get('weight_block_size'), list((q.get('weight_scale') or ['?','?'])) if isinstance(q.get('weight_scale'),list) else q.get('weight_scale_inv') is not None)" -``` - -Decision matrix: - -- `quant_method: "fp8"` + `weight_block_size: [128,128]` + `weight_scale_inv` → ✅ modelopt block-fp8 (the Huihui/official-Qwen format; only proven FP8 path) -- `quant_method: "compressed-tensors"` → ❌ FAILS. Per-channel aborts at expert-source resolution (`Missing MoE expert source layers: {'gate_up'}` when expert tensor naming is nonstandard); block-fp8 128×128 aborts in `_dequant_fp8_weight` (multiplies `weight × weight_scale` directly; block scales don't broadcast → shape RuntimeError) -- No quant config (BF16) → loads but 35B = ~70 GB weights >> 62 GB system RAM → sustained swap-thrash that locks out SSH (2026-09-01 outage). Never deploy BF16 for a 35B-class model on this box; a clean preflight compares disk size × precision against free RAM + 24 GB VRAM headroom. -- Also check: architecture `qwen3_5_moe`, expert tensor naming `mlp.experts.{N}.gate_proj/up_proj/down_proj` (read `model.safetensors.index.json` weight_map on HF if unsure — nonstandard prefixes defeat the MoE expert-source provider), config.json ships the chat template you want (FreeToken reads it from the checkpoint), disk space ≥ checkpoint size. - -If the checkpoint fails the matrix, STOP — do not download, do not touch the running service. Find a compliant quant or use a different runtime. - -## Swap steps - -1. Download into `/home/dank/models//` via `snapshot_download` with the freetoken venv python (background the SSH call; poll until file count matches the index). -2. Edit `scripts/frank-default-small.env`: `MODEL_DIR`, description line (name, quant, arch, disk size, host, date), header comment. Keep F84 pool numbers unless the weight mass or KV dtype changed (FP8 35B ≈ 35 GB: `MODEL_CACHE_SIZE=2048`, `NUM_TOKENS=453005`, reasoning `on`; BF16 66 GB fits nothing — see preflight). -3. Run `./tests/run.sh`, commit, push to Gitea origin (git-only flow; never edit on the host). -4. `./scripts/deploy.sh dankserver --service frank-default-small`. The LLM_WAIT window is up to 20 min; load of ~35 GB FP8 is minutes warm. -5. **Verify for real** — the deploy health gate has reported "healthy" while a doomed worker was still starting (twice). Check journal for Uvicorn startup complete, then `curl /v1/models` (root = your model dir, ctx 262144) and a REAL completion: reasoning must parse into `reasoning_content` and content must answer correctly with a natural `finish_reason: stop`. Give reasoning models enough `max_tokens` to finish thinking (a tight budget yields empty content + `finish: length` — that is correct behavior, not a failure). -6. If load fails: journal shows the exact loader error, service exits cleanly (no swap-thrash), `Restart=on-failure` re-loop is harmless. Revert env via `git checkout -- `, redeploy, verify. -7. Update the runbook (`serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` quant matrix + history), `log.md`, and AGENTS.md fleet table; stage superseded model dirs to `~/.AGENTTRASH/-topic/` on the host and list sizes for the user — never `rm`. diff --git a/agent/managed-skills/freetoken-rocm-build/SKILL.md b/agent/managed-skills/freetoken-rocm-build/SKILL.md deleted file mode 100644 index 5886f4b..0000000 --- a/agent/managed-skills/freetoken-rocm-build/SKILL.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -name: freetoken-rocm-build -description: "Verify and execute FreeToken build on ROCm hosts — check gfx1151, ROCm 6.3, hipcc, and dependency wheel availability (torch 2.11, sgl-kernel 0.4.5, flashinfer)" ---- - -# FreeToken ROCm Build - -## When to use -Verify FreeToken can build on an ROCm host (Dankclaw, similar systems) before committing to the build path. - -## Steps - -1. **Check hardware** - ``` - rocminfo | grep 'Name.*gfx' # verify GPU architecture - lsmod | grep amdgpu # driver loaded - ls /opt/rocm/bin/ | grep hipcc # compiler available - ``` - -2. **Check build deps** - ``` - cmake --version # needs >=3.20 - python3 --version # needs >=3.10 - gcc --version # needs >=9 - ninja --version # optional but recommended - ``` - -3. **Verify wheel availability** - ``` - pip index versions torch # needs 2.11-2.12 range - pip index versions sgl-kernel # needs 0.4.5 - pip index versions flashinfer-python # needs >=0.6 - ``` - Check PyPI for `rocm` in wheel filenames, or use PyTorch ROCm index. - -4. **Build** - ``` - git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken - python3 -m venv .venv && source .venv/bin/activate - pip install -e ".[accel]" # installs fi + sgl extras - ``` - -## Known issues -- nvcc missing on Dankclaw — not needed for ROCm build, only for CUDA JIT -- sgl-kernel at versioned endpoint may return 404 — use unversioned `pip index versions` -- triton 3.6.0 pins to Linux only - -## CUDA alternative -If building for CUDA: install `sudo pacman -S cuda` for nvcc, or use pre-built CUDA wheels that ship pre-compiled kernels. diff --git a/agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md b/agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md deleted file mode 100644 index d19acb3..0000000 --- a/agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -name: grafana-prometheus-provisioning-as-code -description: "Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives." ---- - -# Observability as code (Grafana + Prometheus, Dankclaw fleet) - -Use when adding a scrape target, an alert rule, a datasource, or a dashboard to -this homelab. Everything lives in git; Grafana and Prometheus only ever read -what the deployer installs. - -## The triad — every change touches three places - -1. **Config in the repo** - - `shared/prometheus/prometheus.yml` — `scrape_configs`, and `rule_files:` - pointing at `/etc/prometheus/rules/*.yml` - - `shared/prometheus/rules/.yml` — alert/recording rules - - `shared/grafana/provisioning/datasources/prometheus.yml` — datasource - - `shared/grafana/provisioning/dashboards/{provider.yml,.json}` -2. **The copier in `scripts/deploy-remote.sh`** — `copy_prometheus_config` - installs config **and** rules into `/opt/prometheus/config/` (the container - mounts that to `/etc/prometheus`); `copy_grafana_provisioning` installs into - `/opt/grafana/provisioning/` (the grafana unit already mounts it read-only). - Paths are absolute and `run_privileged`. -3. **The planner in `scripts/deploy-plan.py`** — a `shared//**` change - must select the service that reads it (`shared/prometheus` → prometheus, - `shared/grafana` → grafana). Missing mapping = the file deploys nowhere and - nothing reports an error. - -Add a seam in `tests/observability_quadlet.bats` for each change: the file -exists, the config references it, the deployer copies it, and — where two files -must agree (target vs published port) — that they do. - -## Reach targets by container name, never the loopback publish - -Datasources use `http://prometheus:9090` and scrape jobs for co-located -services use `prometheus:9090` / `loki:3100`. Published ports are -`127.0.0.1`-only by design (see the exposure rules), so a browser-facing URL or -a published port in a datasource is wrong from inside the network. LLM -endpoints are the exception: they are on other hosts, so use their LAN address. - -## Three failure modes that produce no error message - -- **A rule file nothing loads evaluates nothing.** Prometheus reports no - problem for a `rule_files` entry that matches nothing, or for a rules file - nobody references. Ship both together and assert the reference in a seam. -- **A provisioned dashboard is not in the legacy `dashboard` table** on the - current Grafana. Reading `grafana.db` for it returns empty and looks like a - failed provision. It lives in unified storage behind - `dashboard.grafana.app`; query the API instead. -- **Lifetime averages are not per-request evidence.** e.g. an engine's - "Prefix cache hit rate: 29 %" is every prompt it ever saw, including - benchmarks. Measure the behaviour you care about with a targeted probe. - -## Verify through the channel the consumer uses - -```bash -# Prometheus: config and rules parse/load -ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/.yml' -ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \ - | python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"' -ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...' # every target health=up -ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...' # 0 firing when healthy - -# Grafana: provisioned objects and a live query (admin/admin on this install) -ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"' -ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/"' -ssh dankclaw 'curl -s -u admin:admin \ - "http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"' -``` - -A target that is `up` proves reachability, not data: also run one query for a -series you expect and confirm it returns a sample. - -## Grafana facts for this install - -- No UI-configured state: `editable: false` on the datasource, fixed - `uid: prometheus` so dashboards reference it without a lookup. -- Dashboards re-read from disk every 30 s (`updateIntervalSeconds`), so a JSON - edit appears without a restart — but the file must still be *installed*, which - means a deploy. -- There is **no Alertmanager and no contact point**: rules evaluate, alerts show - on `/alerts`, in the `ALERTS` metric and on a dashboard panel, and reach - nobody. Say that plainly in docs rather than implying observability. - -## Naming metrics - -Never write a series name from memory into a config, a rule, or a dashboard — -one shipped comment here named a metric that does not exist. Dump the live -names first and paste them: - -```bash -ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u' -``` diff --git a/agent/managed-skills/grafana-provisioning-as-code/SKILL.md b/agent/managed-skills/grafana-provisioning-as-code/SKILL.md deleted file mode 100644 index 4df5e70..0000000 --- a/agent/managed-skills/grafana-provisioning-as-code/SKILL.md +++ /dev/null @@ -1,92 +0,0 @@ ---- -name: grafana-provisioning-as-code -description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative." ---- - -# Grafana provisioning as code (Dankclaw fleet) - -Use when adding or changing a Grafana datasource or dashboard on this fleet, or -when a dashboard "does not appear" after a deploy. - -## Shape of the thing - -Grafana on Dankclaw is an **empty install configured entirely from git**. The -quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a -provisioned artefact exists in exactly three places: - -``` -shared/grafana/provisioning/datasources/.yml # repo: source of truth -shared/grafana/provisioning/dashboards/provider.yml # dashboard loader config -shared/grafana/provisioning/dashboards/.json # dashboards - | deploy-remote.sh: copy_grafana_provisioning() - v -/opt/grafana/provisioning/... # host: installed state - | read-only mount - v -/etc/grafana/provisioning/... # generated: inside container -``` - -Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be -invisible to review, unknown to anyone reading the repo, and lost with the data -volume. - -## Procedure - -1. Write the files under `shared/grafana/provisioning/`. - - Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by - uid, so a random one breaks them), `access: proxy`, and the URL by - **container name** (`http://prometheus:9090`) — the published port is - loopback-only and unreachable from inside the network. - - Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab; - failing an operator who wants a different rate window is the wrong trade) - and a JSON dashboard under a stable `uid`. -2. Add the file(s) to `copy_grafana_provisioning()` in - `scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each - file — follow `copy_prometheus_config` for style). -3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify - with `python3 scripts/deploy-plan.py --target dankclaw --paths `. -4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse, - the datasource URL must be the container name, the unit must mount - `/opt/grafana/provisioning`, and the deployer must call the copier. Then - `./tests/run.sh`. -5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the - provisioning tree, restarts Grafana, then runs the e2e gate). - -## Verify — and the trap that makes the obvious check lie - -A **provisioned dashboard does not appear in the legacy `dashboard` table** of -`grafana.db` on the current Grafana build; dashboards live in unified storage -behind the `dashboard.grafana.app` API group. The datasource *does* land in -`data_source`, and that asymmetry is easy to misread as "datasource worked, -dashboard failed". A `sqlite3`/python query against `dashboard` is a **false -negative**, not evidence. - -Check in this order instead: - -```bash -# 1. provisioning actually ran (these loggers name themselves) -podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards" - -# 2. the dashboard exists (uid is what you set in the JSON) -curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db" - -# 3. the datasource really reaches Prometheus (proves the whole chain) -curl -s -u admin:admin \ - "http://127.0.0.1:3002/api/datasources/proxy/uid//api/v1/query?query=up" -``` - -Grafana publishes on loopback `:3002` only; the off-host path is the guarded -vhost. Its admin login is still the default `admin:admin` — which is why every -assertion above needs credentials, and which is worth changing. - -## Before writing any panel or config comment - -Grep the metric names out of the live endpoint; naming a series that does not -exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the -real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`, -whose *rates* give the hit rate). A comment naming an unqueryable metric, or a -panel that silently renders "No data", is worse than no panel: - -```bash -curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm -``` diff --git a/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md b/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md deleted file mode 100644 index d1a2d5f..0000000 --- a/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -name: gui-hang-remote-diagnosis -description: "Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability" ---- - -# GUI Hang Remote Diagnosis - -Use when the desktop is frozen, VT switches do nothing, but SSH still works. - -## Sequence - -1. **Find the smoking gun** — kernel hung-task reporter already logged it: - ```bash - journalctl -b -p err --since "-2 hours" --no-pager | tail -80 - ``` - Look for `INFO: task X blocked for more than N seconds` and `blocked on a semaphore likely last held by task Y:PID` (modern kernels name the lock holder outright). -2. **Get full stacks** for victim and holder: - ```bash - journalctl -b -k --since "" --no-pager | tail -120 - cat /proc//status | head -5 # State: R (spinning) vs D (blocked) - cat /proc//status | head -5 - ``` - - Victim `D` + holder `R` = livelock/deadlock with lock held. - - `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, not progressing. -3. **Prove whether hardware or software** — GPU/CPU still alive? No Xid errors? - ```bash - nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv - journalctl -b -k --no-pager | grep -i xid - ``` - If those answer instantly with no Xid, it's a driver lock issue, not hardware. -4. **Find the trigger window**: block start = (first hung message timestamp − reported seconds). Then inspect user-session events: - ```bash - journalctl -b --since "" --until "" --no-pager | grep -v audit - ``` -5. **Enumerate collateral**: any other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), USB workers, etc. - -## Recoverability verdict - -- Compositor in `D` state → unkillable; spinning kernel thread holding the lock → cannot be poked. - **Only exit is reboot**: `sync && systemctl reboot --force --force` (double-force skips stop-jobs that wait forever on D-state PIDs). -- If holder is `D` (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first. - -## After diagnosing - -- Map stack symbols to driver internals to name the exact subsystem (e.g. `DifrPrefetchEventDeferredWork` → NVKMS DIFR prefetch). -- Check upstream: search the exact symbol names + driver version; propose targeted off-switches (`NVreg_RegistryDwords`, module params from `/proc/driver/nvidia/params`) rather than blanket driver swaps. diff --git a/agent/managed-skills/halogen-flash-server-ops/SKILL.md b/agent/managed-skills/halogen-flash-server-ops/SKILL.md deleted file mode 100644 index a164936..0000000 --- a/agent/managed-skills/halogen-flash-server-ops/SKILL.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -name: halogen-flash-server-ops -description: "Operate the frank-large halogen-flash-server deployment on Dankclaw (model swaps, restarts, memory sizing, health and client verification, rollback) — use for anything touching frank-large after the 2026-09-11 runtime swap" ---- - -# halogen-flash-server ops (frank-large, Dankclaw) - -frank-large is a **rootless Quadlet container** since 2026-09-11: -`dankclaw/frank-large/quadlet/frank-large.container` in the serverAppConfigs repo, -installed to `~/.config/containers/systemd/`, generated unit `frank-large.service` -in the **user** manager. Full runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. - -## Never use sudo systemctl on it - -It is a user unit. `sudo systemctl stop frank-large` does nothing useful. - -```bash -systemctl --user status frank-large.service -podman logs frank-large | head -40 # version banner + memory ledger -curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs -curl -s http://127.0.0.1:7770/v1/models # must list frank-large -``` - -## Deploy / change flags - -1. Edit the `.container` file in the repo (all tunables are inline `Environment=` lines). -2. `./tests/run.sh` (the quadlet seam covers GPU devices, memlock, ipc/seccomp, the single published port, writable weights mount). -3. Commit, push, then `./scripts/deploy.sh dankclaw --service frank-large`. - -The handler retires the old root unit (moving its symlink to the home trash), pulls and -re-tags the image to the `localhost/` alias, installs the unit, restarts it, and gates on -the download when the checkpoint is absent. Never `systemctl --user enable` a generated unit. - -## Sizing a swap - -Read the startup ledger before and after any change: it prints the locked weights, the KV -pool it chose, the working set, the BIOS iGPU carve-out, and the corrected free-memory -figure. `free`, MemAvailable and every monitor built on them overstate free memory by the -size of the model (~68 GiB) because the kernel counts the locked weights as reclaimable -page cache. - -- KV is ~7.2 GiB per 262,144 positions; the prefill arena is 16.7 GiB at `HALOGEN_MAX_TOK=32768`. -- The device budget stops near 47 GiB *with the weights pinned*, and lower on a machine - carrying more besides this server. `HALOGEN_KV_POOL_FIT=1` lowers the pool to fit and - prints what it chose; a refused start needs a smaller `HALOGEN_KV_POOL_POSITIONS` - (`HALOGEN_KV_SLOTS` will not help — slots share one pool). -- Long prompts crawling with the disk busy = file cache, not memory: lower the pool or - raise `HALOGEN_HOST_RESERVE_GIB`. - -## Client contract traps - -- No server-side sampler defaults: a request without `temperature` decodes GREEDY. -- The per-request budget defaults to 8192 and **includes thinking**; a reasoning model - plus a small budget returns EMPTY content with `finish_reason: "length"`. Cap is 65536 - (a 400 above it), and `max_tokens`/`max_completion_tokens`/`max_output_tokens` must agree. -- `response_format` is a 400 ("no constrained decoding") — as it is on FreeToken/frank-small. - Any caller that needs structured output or a tiny budget needs a different backend. -- A request reserves prompt + `max_tokens` positions against the pool; larger requests queue - and then fail with `503 engine_busy` after 3600 s. - -## Verification after any change - -Run a real completion (never just `/health`), a vision probe if the tower is enabled -(image content part with a `data:` URL; `http(s)` URLs are refused), then one call from the -client that consumes it. Rollback: the llama.cpp era is in `.AGENTTRASH/` (`.llama.cpp`), -the STRIX payload and ROCmFPX build stay on disk — but remove the Quadlet from -`~/.config/containers/systemd/` before restoring the root unit, or both claim port 7770. diff --git a/agent/managed-skills/halogen-kv-pool-fit/SKILL.md b/agent/managed-skills/halogen-kv-pool-fit/SKILL.md deleted file mode 100644 index 70405fd..0000000 --- a/agent/managed-skills/halogen-kv-pool-fit/SKILL.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -name: halogen-kv-pool-fit -description: "Resize halogen-flash-server's KV pool or prefill arena without getting silently clamped or OOMing the boot: pool, arena and vision share one device budget, HALOGEN_HOST_RESERVE_GIB only scales the fit-check ceiling and is NOT an allocation, HALOGEN_KV_POOL_FIT only ever lowers, and the sizing warning prints only when a clamp happened." ---- - -# halogen-flash-server KV pool / arena resizing - -One device budget covers KV pool + prefill arena + vision tower: - -``` -budget = host RAM (121.2) - resident weights (67.7) - HALOGEN_HOST_RESERVE_GIB -need = KV (~28 KiB/position, ~11.0 GiB at 401408 as reserved) + arena + vision (1.97) -arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) -``` - -Measured ledger (Dankclaw, engine's own startup print): 262144/32768/r24 → 96.3 held, 12.1 left, 2081 blocks; 524288/16384/r24 → 94.6/13.8/3742; 524288/32768/r15 → 103.5/7.1/310 (stall NOTE prints); 524288/32768/r16 → identical (reserve is NOT an allocation); **401408/32768/r15 (deployed 2026-09-13) → 100.0/9.1/1396, NOTE gone**. - -Procedure: -1. Edit `HALOGEN_KV_POOL_POSITIONS` / `HALOGEN_MAX_TOK` in `dankclaw/frank-large/quadlet/frank-large.container` only. Never raise `HALOGEN_HOST_RESERVE_GIB` — it only moves the fit-check ceiling (at 524288 the ceiling was 16; engine needs budget ≥ 36.5 there; 18 clamps to 262144). -2. Over budget → `HALOGEN_KV_POOL_FIT` silently LOWERS the pool and deploy still reports healthy. Way over → arena alone OOMs the boot while the unit still reads ACTIVE (max_tok 65536 did this). Always `podman logs frank-large | grep 'kv pool:'` and match the reserved count to the requested one; /health proves nothing about pool size. -3. Under budget → nothing prints; absence of WARNING is not evidence of headroom. -4. After restart, FIRST request can silence the engine ~45-60 s ("has not answered PING", /health engine_unresponsive) — one-time pool first-touch, self-recovers; HealthStartPeriod=300s covers it. Do not restart the unit. A completion retried after that window succeeds instantly. - -To free host memory: cut the pool (262144 hands back ~3.6 from 401408) or the arena (MAX_TOK 16384 = 8.4). Host-left gain comes in below the KV delta (engine keeps freed space as working slack): the 401408 trim modelled +3.4, measured +2.0 GiB but 4.5x contiguous blocks. - -Omp client config on blankpad: frank-large `maxTokens` must be ≤ 65536 (server 400s above the cap) and prompt+max_tokens must fit the pool or the request queues 3600 s then 503s. diff --git a/agent/managed-skills/hermes-config-home-drift/SKILL.md b/agent/managed-skills/hermes-config-home-drift/SKILL.md deleted file mode 100644 index 0f1bcf8..0000000 --- a/agent/managed-skills/hermes-config-home-drift/SKILL.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -name: hermes-config-home-drift -description: "Diagnose Hermes webui/agent model-connection failures caused by HERMES_HOME config-home drift (image VOLUME anonymous volumes resetting on recreation) — verify which home each runtime reads, resolve provider→endpoint with the app's own resolver code, then probe the endpoint." ---- - -# Hermes config-home drift diagnosis - -Symptom: hermes-webui/agent "cannot connect" to models even though the model endpoints answer curl. - -1. **Find the real config home per runtime.** `podman exec printenv HERMES_HOME` for each hermes container. If webui reads `/opt/data` (image default) while the gateway reads `/data`, the webui runs the image-default template config (`provider: auto` → openrouter.ai with no key = connect failure). -2. **Know the recreation trap.** Image `VOLUME` points get a fresh anonymous volume on every container recreation (`podman run --replace --rm`) — edits inside such a mount are lost on restart/redeploy. Compare volume ids before/after a restart to confirm. -3. **Fix:** pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state root), deploy via service-scoped deployer, restart. -4. **Verify with the app's own code**, not guesses: - - `from api import config as C; C.resolve_model_provider("")` → returns `(model_id, provider, base_url)`; bare ids map to owning provider by scanning the `providers:` dict models allow-list; `@provider:model` syntax forces an explicit provider. - - `hermes_cli.providers.resolve_user_provider(name, cfg["providers"])` → ProviderDef with real `base_url`; accepts `api`/`url`/`base_url` keys on named provider entries. -5. **Probe the endpoint with the exact served model id** (FLM/NPU endpoints route only by exact registry tag and answer HTTP 200 with `{"error":"Failed to load model!"}` for unknown tags — status-code checks miss this). - -Key gotcha: named providers (`providers:` dict) are a catalog + connection source; `custom:` connections resolve only via `custom_providers[]` entries. If neither holds an entry for the provider name, the request has no endpoint and fails to connect. diff --git a/agent/managed-skills/hermes-live-config-edit/SKILL.md b/agent/managed-skills/hermes-live-config-edit/SKILL.md deleted file mode 100644 index d7600ce..0000000 --- a/agent/managed-skills/hermes-live-config-edit/SKILL.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -name: hermes-live-config-edit -description: "Runbook for changing Hermes (Dankclaw) live config.yaml — model routing, providers, aux routes — including backup, apply, restart, and verification steps" ---- - -# Editing live Hermes config on Dankclaw - -Hermes-webui/gateway read `/home/dank/Containers/hermes/data/config.yaml` (dir is `drwx------` uid 109999; all reads/edits need `sudo`). The repo file `dankclaw/hermes/config.yaml.example` is a sanitized backup and can lag live — always diff the live file first; never treat the example as truth. - -## Procedure -1. **Backup first:** `sudo cp -a .../config.yaml .../config.yaml.bak-YYYYMMDD` (in-dir trash `.AGENTTRASH/` also exists for removals). -2. **Apply change** with surgical `sudo sed -i` or full-file write; YAML comments in the live file are tool-generated — preserve them. -3. **Sweep stale references** across the whole data dir, not just config.yaml: `sudo grep -rln "" /home/dank/Containers/hermes/data --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are historical — leave them. -4. **Derived caches:** `webui/models_cache.json` is keyed by a config fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it's stale. Either let it regenerate lazily or trash it (mv to data dir `.AGENTTRASH/`) — never hand-edit. -5. **Restart the model-consuming units** (they parse config at startup): `systemctl --user restart hermes-webui.service hermes.service`. `hermes-dashboard.service` is unaffected by routing changes. -6. **Verify behaviorally**, not by status codes: one tiny completion per endpoint with the exact served id — POST `/v1/chat/completions` on 7770 (Dankclaw, llama.cpp serves alias `frank-large`), on Dankserver 7770 (FreeToken REQUIRES an explicit `model` field), and on 7771/v1 (FLM routes by exact registry tag, e.g. `gemma4-it:e4b`; unknown tags answer HTTP 200 with `{"error":"Failed to load model!"}` — status-code checks silently pass). -7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the live shape, redacting secrets (password_hash, phone account stay live-only), then run `./tests/run.sh` before commit. - -## Gotchas -- `provider: custom` + `base_url` entries route by base_url and send the model id verbatim; named providers (`dankclaw-large`, `dankserver-small`, `dankclaw-npu`) resolve via their `api:` URL. -- Renaming a served alias anywhere (serve scripts, env ALIAS) MUST propagate to this config in the same change or Hermes requests 404 on nonexistent models — this exact gap caused the 2026-09-05 outage. diff --git a/agent/managed-skills/hermes-local-aux-routing/SKILL.md b/agent/managed-skills/hermes-local-aux-routing/SKILL.md deleted file mode 100644 index 1bec85a..0000000 --- a/agent/managed-skills/hermes-local-aux-routing/SKILL.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -name: hermes-local-aux-routing -description: "Fix a broken Hermes auxiliary route (title generation, compression, vision, etc.) when the target is a local OpenAI-compatible backend — diagnose by probing the real call path in-container, then repoint the transport" ---- - -# Hermes auxiliary routing against local backends - -Use when a Hermes feature that calls an auxiliary task fails against a local -endpoint on this fleet (Frank-small on Dankserver :7770, frank-large on Dankclaw -:7770). The usual cause is a request shape the local runtime does not implement, -not a routing mistake. - -## Diagnose with the real call path - -Do not guess from the config. Run the real function inside the container — it -uses the caller's actual kwargs, the config-driven provider resolution and the -real response parsing: - -```bash -base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \ - 'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \ - podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \ - podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py' -``` - -Copy the file AFTER any restart (`--replace --rm` discards the container layer), -and import the app's own constants so the probe cannot drift from production: - -```python -import sys; sys.path.insert(0, "/opt/hermes") -from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text -from agent.auxiliary_client import call_llm -resp = call_llm(task="title_generation", messages=[...], max_tokens=64, - temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT}) -``` - -Pass `api_mode=` / extra `extra_body` keys to A/B a fix before touching config. -(The CLI at `/opt/hermes/hermes` uses the system python and dies on -`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.) - -## What actually breaks, and the fix - -- **`response_format` (json_object / json_schema) is a hard 400** on every local - backend here — neither FreeToken nor halogen implements constrained decoding. - Config cannot override it: `call_llm` merges the task's `extra_body` first and - then `update()`s the caller's, so the hardcoded one wins. Only temperature and - max_tokens have reactive retry paths. -- **Fix: move that one task to the Responses transport**, which never forwards - `response_format`: - ```yaml - auxiliary: - title_generation: - provider: dankserver-small - model: frank-small - api_mode: codex_responses # real per-task key - extra_body: - reasoning: - effort: none # mandatory, see below - ``` -- **Disable thinking for any cap-limited task.** The adapter omits reasoning - only when the config says `reasoning.enabled: false`; note that the - `reasoning_effort: none` shorthand produces exactly that, which leaves the - server default (thinking ON) and the 64-token title budget is spent on the - think — empty content, `finish_reason: length`. The explicit - `extra_body.reasoning.effort: none` instead sends - `{"effort":"none","summary":"auto"}`, which the local /v1/responses honors. -- Both local runtimes also answer `/v1/responses` and `/v1/messages`, so no - proxy is needed to use the alternate transport. - -## Verify and land it - -1. Re-run the probe with **no** `api_mode`/reasoning kwargs — it must pass from - config alone; that is the production path. -2. Keep the live file (`/home/dank/Containers/hermes/data/config.yaml`, owned by - uid 109999, needs sudo) and the repo example - `dankclaw/hermes/config.yaml.example` in step; the example is a sanitized - backup and lags otherwise. -3. `./tests/run.sh`, commit, push — a Hermes config change is docs+example only, - no service deploy needed. - -## Gotchas - -- `auxiliary..enabled: false` is checked BEFORE `apply_instant_title`, so - disabling a broken title task also removes the derived title — it does not - degrade gracefully the way a failure does. -- A local call can also fail at the model level: a 200 with an error-text body, - or a chatty non-title answer. Always read the raw content, not the status. diff --git a/agent/managed-skills/hermes-webui-secret-rotation/SKILL.md b/agent/managed-skills/hermes-webui-secret-rotation/SKILL.md deleted file mode 100644 index 8502592..0000000 --- a/agent/managed-skills/hermes-webui-secret-rotation/SKILL.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -name: hermes-webui-secret-rotation -description: Rotate the Hermex (hermes-webui) password on Dankclaw and verify auth behaviorally — covers the dual EnvironmentFile gotcha where only dankclaw/.env is effective at container create. ---- - -# Hermex webui secret rotation (Dankclaw) - -Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON `{"password":"..."}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL) on match, 401 on miss. No username field. `/health` answers 200 unauthenticated and `/` answers 302 — never verify auth via those; always probe login with a deliberately wrong password (expect 401) and the real one (expect 200). - -## Rotation procedure -1. `ssh -o BatchMode=yes dankclaw 'bash -s'` with a stdin script (zsh on the host mangles inline `${...}` even inside single-quoted ssh args — never interpolate curl `-w %{http_code}` inline; pipe the whole probe as a heredoc'd bash script). -2. Update BOTH env files with `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=/"`: - - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile) - - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the quadlet unit's `[Container] EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env` is what the container reads at create; rotating only the root file leaves the running container serving the old secret. -3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads env; no deploy needed if the unit file itself didn't change). -4. Verify: wrong-pw login → 401, right-pw → 200, from both host loopback and a LAN client at `http://192.168.0.143:8787` (webui publishes on the LAN interface since commit 4443897; off-network ingress is SSH tunnel only — user `tunnel`, port **22**, key-only auth, forward remote end localhost:8787). - -## Gotchas -- `sudo podman exec` fails exit 125 here: root's namespace does not see rootless containers — run `podman exec hermes-webui printenv HERMES_WEBUI_PASSWORD` unprivileged (ssh user owns the env files, mode 600). -- If rotating because a secret leaked into transcripts/logs, rotate in place via a script that fetches the live value from the running container (`printenv`) instead of transcribing it. diff --git a/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md b/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md deleted file mode 100644 index a50f460..0000000 --- a/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -name: heterogeneous-accelerator-split-triage -description: "Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed." ---- - -# Heterogeneous accelerator split — triage before building - -Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm. - -## 1. Characterise the link (bytes vs latency) - -```bash -lspci -vv -s | grep -E "LnkCap:|LnkSta:" # capability vs actual -nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv -``` - -Expect **idle-downshift**: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number. - -Compute the per-hop cost against the hidden state (`n_embd x 2 B` for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then: - -- bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. **Bytes are almost never the constraint.** -- the real budget line is **hop COUNT**: every device boundary is a host-side sync. A runtime that splits per *op* can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure `graph splits` in the loader log, do not assume. - -**Design rule: partition by where the weights live, never by where the activations flow.** Resident weights, activations crossing. - -## 2. Get the workload's real bottleneck (KV is usually it) - -KV per position x context = bytes read per token, because attention reads the whole cached context every token: - -``` -KV/token = kv_bytes_per_position * context -``` - -Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, **KV dominates**. Then: - -- the highest-leverage placement is **KV cache on the fastest-memory device, together with its attention compute** — never KV alone (separating them puts `KV/token` over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token); -- check the KV + attention weights actually fit: `kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM`. - -## 3. What placement never buys - -Prefill is compute-bound and expert weights are read **once** per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave **decode +12 %, prefill 192 -> 32 tok/s**. - -A `1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining). - -## 4. Is the second device even a separate pool? - -On a UMA APU, check before assuming additivity: - -```bash -cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small) -cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM -rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees -``` - -A large GTT window means the "GPU memory" is the *same* LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM. - -## 5. Speculative decoding as the "cheap" split - -A drafter on the *other* device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it: - -- the drafter needs the target's **exact tokenizer** for a plain draft model (verify by comparing GGUF `tokenizer.ggml.tokens`, not by trusting the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models); -- a plain drafter costs **K sequential forwards per round**, so it must be 10-50x faster than the target. Compute the break-even: with `r = t_draft/t_target`, `rate_ratio = A/(K*r + 1)`, `A = (1-p^(K+1))/(1-p)`. At `r ≈ 0.4-0.5` (a drafter only ~2x faster) break-even needs `p ≈ 0.7+` at K=3; -- always compare against the **incumbent**, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar; -- check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (`DRAFTER_CAP`-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd-ctvd`. - -## 6. Expressing a split (llama.cpp, the usual vehicle) - -```bash --dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest) --ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (what KV co-location needs) -``` - -- buffer types are matched by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values; -- `-cmoe` / `-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`; -- several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and `GPU_BACKEND_A=OFF` in the CMakeCache of the existing build tells you what needs adding; -- contiguous ranges are the *wrong* split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool. - -## 7. Hazards checklist - -- paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss; -- keep `mmap` on for models with huge embedding tables (anonymous memory gets the process cgroup-killed); -- CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply; -- the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a **baseline of the runtime on that device alone**, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine; -- a split consumes the fast device entirely — check what else it was serving before trading it away; -- benchmark/experiment discipline on production hosts: stop and mask first. - -## 8. Verdict shape - -Rank and report, and prefer the boring answer when it wins: - -1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link; -2. **One process, weights-resident, KV co-located with attention on the fast pool** — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement; -3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check `misses x bytes x tok/s` against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache; -4. list explicitly what does *not* help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements. diff --git a/agent/managed-skills/hf-checkpoint-profile/SKILL.md b/agent/managed-skills/hf-checkpoint-profile/SKILL.md new file mode 100644 index 0000000..170cb20 --- /dev/null +++ b/agent/managed-skills/hf-checkpoint-profile/SKILL.md @@ -0,0 +1,26 @@ +--- +name: hf-checkpoint-profile +description: "Profile a Hugging Face safetensors checkpoint (sizes, dtypes, tensor-name space, gated/license) with compact HTTP-range reads — safe for subagents on small context slots." +--- + +# Profile an HF safetensors checkpoint cheaply + +Goal: inventory a repo (files/bytes, dtype histogram, tensor-name patterns, per-group GiB, config keys, license/access) without pulling gigabytes and without overflowing a subagent's context slot. + +## Endpoints (all public, no token needed even for gated models) +- `https://huggingface.co/api/models/?blobs=true` → `siblings` (name+size), `gated`, `private`, `downloads`, `safetensors` metadata, `tags`. +- `https://huggingface.co/api/models/` for tags/base model; `https://huggingface.co/api/models?author=&limit=100` to find sibling repos (canonical id check: cards often link both `X-1.5-Y` and `X-Y`). +- `https://huggingface.co/api/models//resolve/main/` for `config.json`, `model.safetensors.index.json`, `chat_template.jinja`, `LICENSE`, `generation_config.json`; `/raw/main/` for text. + +## dtype + per-group bytes without downloading shards +Parse the safetensors header over HTTP Range: first 8 bytes = little-endian u64 header length, then that many bytes of JSON (`{name: {dtype, shape, data_offsets}}`). Fetch every shard header with a thread pool (131 shards ≈ seconds), then bytes = prod(shape) × dtype size. Collapse names with `re.sub(r"\.\d+\.", ".N.", k)` (and `\.shard_\d+` → `.shard_N`) to get the pattern space; group-sum by prefix to see where the mass sits (experts / ngram-PLE / MTP / vision / trunk). + +Compare the key set against the base model's index to prove "full-weight finetune, identical names" (a zero-diff key set means per-tensor recipes written for the base apply verbatim, and it reveals fused vs per-expert layout, PLE/ngram shards, MTP head, vision tower, absence of `quantization_config`). + +## Guardrails +- Run python scripts from a clean directory: a stray `/tmp/inspect.py` shadows stdlib `inspect` and kills `concurrent.futures` imports. +- Never put an HF token on a command line (`/proc//cmdline` is world-readable); use `curl -K `. +- `curl -f` / `--fail-with-body` and verify byte counts — a 401/403 writes the error body as the "artifact" and exits 0. + +## Subagent context budget +Subagents on a 131k-token slot die from `exceed_context_size_error` when told to read whole doc pages. Give grep-only instructions (`| head -40`, `-o`, `-c`), demand a line cap ("report in ≤250 lines, no verbatim dumps"), and prefer `git clone --depth 5` + local grep to web-page reads. A peer scout that overflows returns exit 1 with the 400 in the harness log and NO structured output — assume nothing was learned and redo the fact inline. diff --git a/agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md b/agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md deleted file mode 100644 index 0ab1df6..0000000 --- a/agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -name: hf-gguf-payload-fetch-verify -description: "Inventory, fetch, hash-verify and assemble a Hugging Face GGUF payload (including mradermacher's .partNofM fragments and gated mirrors) before pointing an LLM service at it — on the Dank fleet or anywhere else." ---- - -# Fetch and verify an HF GGUF payload - -Use before any model swap where the weights come from Hugging Face, especially -large (>32 GiB) quantised models. The failure modes below all look like success. - -## 1. Inventory — query by quant name, never by extension - -```bash -curl -s "https://huggingface.co/api/models///tree/main?recursive=true&blobs=true" \ - | python3 -c " -import json,sys -for e in json.load(sys.stdin): - p=e.get('path','') - if 'Q4_K_S' in p: # the QUANT, not '.gguf' - print(p, e.get('size',0), (e.get('lfs') or {}).get('oid','-')) -" -``` - -**Large quants ship as `.partNofM` fragments.** `mradermacher/*-i1-GGUF` repos -hold e.g. Q4_K_M as `part1of3/2of3/3of3` (37.00+37.00+36.97 = 110.97 GiB) and -Q4_K_S the same way, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are -single `.gguf` files. Filtering on `endswith('.gguf')` therefore makes Q4_K_S -look like it does not exist — do not conclude a quant is missing from an -extension-filtered listing. - -Check both mradermacher repos: `…-i1-GGUF` (imatrix) and `…-GGUF` (plain), and -the model's own repo. Gated mirrors exist (`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF` -carries the same ladder) and return **403 even with a valid HF token** until -that account accepts the gate — prefer the ungated copy rather than chasing access. - -## 2. Fetch — `-f` is mandatory, secrets stay out of argv - -```bash -# token out of band: mode-600 config, never -H on the command line -printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc -chmod 600 /tmp/hf.curlrc -curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc \ - -o "$DEST/$name" "$BASE/$name" -``` - -- `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes the error body as the - artifact. A gated repo yields identical small files (153 bytes) that hash the - same and a `DOWNLOAD_DONE` in seconds for a 104 GiB payload. -- A token in `-H "Authorization: Bearer …"` is visible in `/proc//cmdline` - to any `ps`/`pgrep -a` on the host. Never `pgrep -a` a running fetch. - -## 3. Verify, then assemble - -```bash -# per part: byte count must equal the Hub's declared size, sha256 must equal lfs.oid -sha256sum "$PARTS"/*.part* | sort - -# concatenate in NUMERIC order, then verify the total and the container magic -cat "$PARTS"/X.gguf.part1of3 "$PARTS"/X.gguf.part2of3 "$PARTS"/X.gguf.part3of3 > "$OUT" -stat -c%s "$OUT" # must equal the sum of the part sizes -head -c 4 "$OUT" # must print GGUF -``` - -The assembled file has no published hash (only the parts do), so the per-part -LFS oids plus an exact total are the verification chain. If a size check fails, -suspect your own arithmetic before the payload: the sum of the parts is the -expected value, and it is easy to mis-add by hand (`python3 -c "print(a+b+c)"`). - -## 4. Housekeeping - -- Failure artifacts (error pages) and stray directories: move to - `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm`; list them and wait for approval to - purge. -- Keep the `.partNofM` fragments after assembly when disk allows: they are the - only copies that can be re-verified against the Hub. Note the total they cost - and ask before purging. -- Never delete the payload being replaced as a side effect of a swap: upstreams - delist quants (both mradermacher and orcarouter have delisted Q4_K_M for - Qwen3.8-Flash-Next-Uncensored), and the local file may be the last copy. - -## 5. Waiting for a long fetch - -Do not `pgrep -f