From 135bb2194de60e56b0467999bd9ca1e9e04f40e7 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 12:27:08 -0700 Subject: [PATCH] agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills - config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set --- agent/config.yml | 12 +- .../amd-multigpu-cooperation-triage/SKILL.md | 71 ++++++++ .../container-config-drift-crashloop/SKILL.md | 82 +++++++++ .../fleet-container-ownership-audit/SKILL.md | 91 ++++++++++ .../fleet-llm-payload-purge-refresh/SKILL.md | 117 ++++++++++++ .../fleet-model-cache-purge-refresh/SKILL.md | 73 ++++++++ .../fleet-monitoring-gap-audit/SKILL.md | 79 ++++++++ .../fleet-second-llm-endpoint/SKILL.md | 97 ++++++++++ .../SKILL.md | 64 +++++++ .../frank-medium-batch-split-retune/SKILL.md | 95 ++++++++++ .../SKILL.md | 94 ++++++++++ .../grafana-provisioning-as-code/SKILL.md | 92 ++++++++++ .../SKILL.md | 94 ++++++++++ .../hf-gguf-payload-fetch-verify/SKILL.md | 98 ++++++++++ .../ik-llama-hybrid-moe-performance/SKILL.md | 29 +++ .../incident-window-reconstruction/SKILL.md | 92 ++++++++++ .../live-llm-benchmark-hygiene/SKILL.md | 70 ++++++++ .../live-llm-telemetry-no-bench/SKILL.md | 87 +++++++++ .../llama-bench-fleet-sweep/SKILL.md | 51 ++++++ .../llama-bench-host-sweep/SKILL.md | 109 ++++++++++++ .../llama-cpp-spec-decode-ab/SKILL.md | 58 ++++++ .../SKILL.md | 97 ++++++++++ .../SKILL.md | 62 +++++++ .../llamacpp-drafter-kv-split-sizing/SKILL.md | 168 ++++++++++++++++++ .../llamacpp-layer-split-rdna4/SKILL.md | 113 ++++++++++++ .../SKILL.md | 70 ++++++++ .../SKILL.md | 132 ++++++++++++++ .../SKILL.md | 88 +++++++++ .../llamacpp-vram-accounting/SKILL.md | 77 ++++++++ .../llm-replica-routing-prefix-cache/SKILL.md | 100 +++++++++++ .../SKILL.md | 67 +++++++ .../niri-output-rotation-pinning/SKILL.md | 96 ++++++++++ .../pcie-link-capability-probe/SKILL.md | 122 +++++++++++++ .../SKILL.md | 101 +++++++++++ .../published-port-cutover-gate/SKILL.md | 88 +++++++++ .../SKILL.md | 109 ++++++++++++ .../spec-decode-ab-single-gpu-host/SKILL.md | 77 ++++++++ .../spec-decode-worth-it-triage/SKILL.md | 70 ++++++++ .../SKILL.md | 77 ++++++++ .../SKILL.md | 88 +++++++++ .../stale-signal-audit/SKILL.md | 71 ++++++++ .../status-surface-dead-signal-audit/SKILL.md | 56 ++++++ agent/memories/--home-blank-.omp--/learned.md | 1 + .../--home-blank-.omp--/raw_memories.md | 87 +++++++++ ...-83e9-2eb95ed25621-noctalia_red_theme_n.md | 4 + ...-8fb6-55879410c3fc-niri_portrait_180_fl.md | 4 + ...-9bd0-a8aa7c9d3676-llama_cpp_parallelis.md | 4 + ...-92f1-14cd7fade072-niri_externals_left_.md | 4 + .../MEMORY.md | 48 +++-- .../learned.md | 72 ++++++++ .../memory_summary.md | 2 +- .../raw_memories.md | 164 +++++++++++++++++ ...-a910-e997e01fbe54-halogen_flash_stall_.md | 4 + ...-bc1c-dc405ca02cbd-dankserver_dual_r970.md | 4 + ...-995c-8c3e7b37e241-ik_llamacpp_3090_str.md | 4 + ...-9392-764f6cfbd59e-agents_md_rewrite_an.md | 4 + ...-b00a-52dd4cbb4ff3-rootless_podman_rest.md | 4 + .../skills/auth-chain-verification/SKILL.md | 4 + .../skills/frank_large/SKILL.md | 17 -- .../frank_large/examples/curl_request.json | 5 - .../skills/frank_large/scripts/check_kpool.sh | 6 - .../frank_large/templates/frank-large.service | 6 - .../skills/kv-pool-verification/SKILL.md | 4 + .../skills/marreta_authentik/SKILL.md | 12 -- .../marreta_authentik/examples/e2e_test.md | 2 - .../scripts/probe_marreta.sh | 5 - .../templates/authentik-provider.yaml | 8 - .../rootless-quadlet-log-retrieval/SKILL.md | 4 + agent/models.yml | 115 ++++++++++-- agent/skills/code-review/SKILL.md | 87 +++++++++ agent/skills/code-review/agents/openai.yaml | 3 + agent/skills/diagnosing-bugs/SKILL.md | 138 ++++++++++++++ .../skills/diagnosing-bugs/agents/openai.yaml | 3 + .../scripts/hitl-loop.template.sh | 44 +++++ agent/skills/find-skills/SKILL.md | 141 +++++++++++++++ agent/skills/grilling/SKILL.md | 28 +++ agent/skills/grilling/agents/openai.yaml | 3 + agent/skills/handoff/SKILL.md | 16 ++ agent/skills/handoff/agents/openai.yaml | 5 + agent/skills/implement/SKILL.md | 15 ++ agent/skills/implement/agents/openai.yaml | 5 + agent/skills/prototype/LOGIC.md | 67 +++++++ agent/skills/prototype/SKILL.md | 26 +++ agent/skills/prototype/UI.md | 112 ++++++++++++ agent/skills/prototype/agents/openai.yaml | 3 + agent/skills/tdd/SKILL.md | 38 ++++ agent/skills/tdd/agents/openai.yaml | 3 + agent/skills/tdd/mocking.md | 59 ++++++ agent/skills/tdd/tests.md | 77 ++++++++ agent/skills/teach/GLOSSARY-FORMAT.md | 35 ++++ agent/skills/teach/LEARNING-RECORD-FORMAT.md | 46 +++++ agent/skills/teach/MISSION-FORMAT.md | 31 ++++ agent/skills/teach/RESOURCES-FORMAT.md | 32 ++++ agent/skills/teach/SKILL.md | 140 +++++++++++++++ agent/skills/teach/agents/openai.yaml | 5 + dotfiles/niri/.config/niri/cfg/animation.kdl | 19 +- dotfiles/niri/.config/niri/cfg/input.kdl | 1 - dotfiles/niri/.config/niri/cfg/layout.kdl | 17 +- dotfiles/niri/.config/niri/config.kdl | 8 + .../niri/.config/niri/cfg/display.kdl | 6 +- .../niri/.config/niri/cfg/keybinds.kdl | 62 +++---- .../niri/.config/niri/cfg/misc.kdl | 22 +-- .../niri/.config/niri/cfg/rules.kdl | 24 +-- 103 files changed, 5333 insertions(+), 170 deletions(-) create mode 100644 agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md create mode 100644 agent/managed-skills/container-config-drift-crashloop/SKILL.md create mode 100644 agent/managed-skills/fleet-container-ownership-audit/SKILL.md create mode 100644 agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md create mode 100644 agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md create mode 100644 agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md create mode 100644 agent/managed-skills/fleet-second-llm-endpoint/SKILL.md create mode 100644 agent/managed-skills/fleet-service-precommit-iteration/SKILL.md create mode 100644 agent/managed-skills/frank-medium-batch-split-retune/SKILL.md create mode 100644 agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md create mode 100644 agent/managed-skills/grafana-provisioning-as-code/SKILL.md create mode 100644 agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md create mode 100644 agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md create mode 100644 agent/managed-skills/ik-llama-hybrid-moe-performance/SKILL.md create mode 100644 agent/managed-skills/incident-window-reconstruction/SKILL.md create mode 100644 agent/managed-skills/live-llm-benchmark-hygiene/SKILL.md create mode 100644 agent/managed-skills/live-llm-telemetry-no-bench/SKILL.md create mode 100644 agent/managed-skills/llama-bench-fleet-sweep/SKILL.md create mode 100644 agent/managed-skills/llama-bench-host-sweep/SKILL.md create mode 100644 agent/managed-skills/llama-cpp-spec-decode-ab/SKILL.md create mode 100644 agent/managed-skills/llamacpp-cross-backend-device-split/SKILL.md create mode 100644 agent/managed-skills/llamacpp-device-split-feasibility/SKILL.md create mode 100644 agent/managed-skills/llamacpp-drafter-kv-split-sizing/SKILL.md create mode 100644 agent/managed-skills/llamacpp-layer-split-rdna4/SKILL.md create mode 100644 agent/managed-skills/llamacpp-layer-split-vram-rebalance/SKILL.md create mode 100644 agent/managed-skills/llamacpp-mtp-vision-endpoint-bringup/SKILL.md create mode 100644 agent/managed-skills/llamacpp-multigpu-layer-split-cutover/SKILL.md create mode 100644 agent/managed-skills/llamacpp-vram-accounting/SKILL.md create mode 100644 agent/managed-skills/llm-replica-routing-prefix-cache/SKILL.md create mode 100644 agent/managed-skills/llm-serving-benchmark-methodology/SKILL.md create mode 100644 agent/managed-skills/niri-output-rotation-pinning/SKILL.md create mode 100644 agent/managed-skills/pcie-link-capability-probe/SKILL.md create mode 100644 agent/managed-skills/pcie-link-degradation-vs-idle-triage/SKILL.md create mode 100644 agent/managed-skills/published-port-cutover-gate/SKILL.md create mode 100644 agent/managed-skills/rccl-pcie-atomics-collective-failure/SKILL.md create mode 100644 agent/managed-skills/spec-decode-ab-single-gpu-host/SKILL.md create mode 100644 agent/managed-skills/spec-decode-worth-it-triage/SKILL.md create mode 100644 agent/managed-skills/stale-compose-container-quadlet-migration/SKILL.md create mode 100644 agent/managed-skills/stale-signal-and-hollow-gate-audit/SKILL.md create mode 100644 agent/managed-skills/stale-signal-audit/SKILL.md create mode 100644 agent/managed-skills/status-surface-dead-signal-audit/SKILL.md create mode 100644 agent/memories/--home-blank-.omp--/rollout_summaries/01a09e49-b17b-74c3-83e9-2eb95ed25621-noctalia_red_theme_n.md create mode 100644 agent/memories/--home-blank-.omp--/rollout_summaries/01a0a798-f61e-71d3-8fb6-55879410c3fc-niri_portrait_180_fl.md create mode 100644 agent/memories/--home-blank-.omp--/rollout_summaries/01a0a7ff-dae2-71fd-9bd0-a8aa7c9d3676-llama_cpp_parallelis.md create mode 100644 agent/memories/--home-blank-.omp--/rollout_summaries/01a0a817-7382-7525-92f1-14cd7fade072-niri_externals_left_.md create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/rollout_summaries/01a09e67-630f-74ed-a910-e997e01fbe54-halogen_flash_stall_.md create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/rollout_summaries/01a0a7ad-42e2-739d-bc1c-dc405ca02cbd-dankserver_dual_r970.md create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/rollout_summaries/01a0a87f-6b17-7492-995c-8c3e7b37e241-ik_llamacpp_3090_str.md create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/rollout_summaries/01a0a902-7f58-77f4-9392-764f6cfbd59e-agents_md_rewrite_an.md create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/rollout_summaries/01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3-rootless_podman_rest.md create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/auth-chain-verification/SKILL.md delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/frank_large/SKILL.md delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/frank_large/examples/curl_request.json delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/frank_large/scripts/check_kpool.sh delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/frank_large/templates/frank-large.service create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/kv-pool-verification/SKILL.md delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/marreta_authentik/SKILL.md delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/marreta_authentik/examples/e2e_test.md delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/marreta_authentik/scripts/probe_marreta.sh delete mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/marreta_authentik/templates/authentik-provider.yaml create mode 100644 agent/memories/--home-blank-Projects-serverAppConfig--/skills/rootless-quadlet-log-retrieval/SKILL.md create mode 100644 agent/skills/code-review/SKILL.md create mode 100644 agent/skills/code-review/agents/openai.yaml create mode 100644 agent/skills/diagnosing-bugs/SKILL.md create mode 100644 agent/skills/diagnosing-bugs/agents/openai.yaml create mode 100644 agent/skills/diagnosing-bugs/scripts/hitl-loop.template.sh create mode 100644 agent/skills/find-skills/SKILL.md create mode 100644 agent/skills/grilling/SKILL.md create mode 100644 agent/skills/grilling/agents/openai.yaml create mode 100644 agent/skills/handoff/SKILL.md create mode 100644 agent/skills/handoff/agents/openai.yaml create mode 100644 agent/skills/implement/SKILL.md create mode 100644 agent/skills/implement/agents/openai.yaml create mode 100644 agent/skills/prototype/LOGIC.md create mode 100644 agent/skills/prototype/SKILL.md create mode 100644 agent/skills/prototype/UI.md create mode 100644 agent/skills/prototype/agents/openai.yaml create mode 100644 agent/skills/tdd/SKILL.md create mode 100644 agent/skills/tdd/agents/openai.yaml create mode 100644 agent/skills/tdd/mocking.md create mode 100644 agent/skills/tdd/tests.md create mode 100644 agent/skills/teach/GLOSSARY-FORMAT.md create mode 100644 agent/skills/teach/LEARNING-RECORD-FORMAT.md create mode 100644 agent/skills/teach/MISSION-FORMAT.md create mode 100644 agent/skills/teach/RESOURCES-FORMAT.md create mode 100644 agent/skills/teach/SKILL.md create mode 100644 agent/skills/teach/agents/openai.yaml diff --git a/agent/config.yml b/agent/config.yml index 625448b..1fa944f 100644 --- a/agent/config.yml +++ b/agent/config.yml @@ -24,16 +24,16 @@ providers: - mojeek - public modelRoles: - smol: llamacpp (dankserver)/frank-small:off slow: llamacpp (dankclaw)/frank-large:medium - vision: llamacpp (dankclaw)/frank-large:auto + vision: llamacpp (dankclaw-3090)/frank-small:xhigh plan: llamacpp (dankclaw)/frank-large:medium designer: llamacpp (dankclaw)/frank-large:medium - commit: llamacpp (dankserver)/frank-small:medium - tiny: llamacpp (dankserver)/frank-small:off - task: llamacpp (dankclaw)/frank-large:medium + commit: llamacpp (dankserver)/frank-medium:medium + tiny: llamacpp (dankclaw-3090)/frank-small:off + task: llamacpp (dankclaw-3090)/frank-small:off advisor: llamacpp (dankclaw)/frank-large:medium - default: llamacpp (dankserver)/frank-small:medium + default: llamacpp (dankserver)/frank-medium:medium + smol: llamacpp (dankclaw-3090)/frank-small:xhigh symbolPreset: nerd composer: shape: claude diff --git a/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md b/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md new file mode 100644 index 0000000..232f5f7 --- /dev/null +++ b/agent/managed-skills/amd-multigpu-cooperation-triage/SKILL.md @@ -0,0 +1,71 @@ +--- +name: amd-multigpu-cooperation-triage +description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives." +--- + +# AMD Multi-GPU Cooperation Triage + +When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix". + +## 1. Check atomic-op routing on the whole path + +RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*: + +```cpp +void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp + link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0}); + if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) + if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else, no override +} +``` + +```bash +# Endpoint capability + negotiated link, and EVERY upstream bridge +for d in ; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done +lspci -tv # is the card on a CPU root port or behind the chipset? +``` + +- Endpoint must show `AtomicOpsCtl: ReqEn+`. +- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe. + +Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`. + +## 2. Map the board before recommending anything + +Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4. + +## 3. Prove what the runtime actually needs (do it OFF the production host) + +Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging: + +```python +torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT", + world_size=2, rank=r, timeout=timedelta(seconds=25)) +# probe: broadcast, all_reduce, barrier, send/recv, all_gather +``` + +- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall. +- `world_size=1` always succeeds; one-card success proves nothing. +- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/'`). +- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened. + +## 4. Don't be fooled by "collective-free" claims + +Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out. + +## 5. The option set, ranked + +| Option | Works when | Cost | Trade | +|---|---|---|---| +| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient | +| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser | +| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode | +| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today | +| Patch the custom-AR table | never | — | other collectives still die | +| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host | + +## Pitfalls + +- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`). +- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show ` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP. +- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`). diff --git a/agent/managed-skills/container-config-drift-crashloop/SKILL.md b/agent/managed-skills/container-config-drift-crashloop/SKILL.md new file mode 100644 index 0000000..0ac86c3 --- /dev/null +++ b/agent/managed-skills/container-config-drift-crashloop/SKILL.md @@ -0,0 +1,82 @@ +--- +name: container-config-drift-crashloop +description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order." +--- + +# A container that restarts forever while its definition is correct + +## When to use + +A container is crash-looping or perpetually "unhealthy", but the file that +defines it (compose service, unit, deploy script) looks right. Also when a +service has been quietly down for days and nobody noticed, because a restart +policy kept something running. + +The cause is usually **config drift**: the running container was created from an +older definition, and the restart policy restarts it instead of recreating it. +A compose/unit edit does not touch an existing container. + +## Diagnose (5 minutes, in this order) + +1. **Who started it?** The label names the owner, not your guesswork: + + ``` + podman inspect --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}' + ``` + + `podman-compose@…` means compose owns it, whatever unit files exist. + Compare `created` with when the definition last changed. A container older + than the edit that should have replaced it is the bug. + +2. **What is it ACTUALLY running?** Read the argv, not the summary fields: + + ``` + podman inspect --format '{{.Config.CreateCommand}}' + ``` + + Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device` + flags are present — a false negative that can send you hunting the wrong + root cause. `CreateCommand` is the truth; grep it for the flag you care + about (`--device`, `--group-add`, `-e`, mounts). + +3. **What does the process say?** `podman logs | grep -iE 'error|cannot|Runtime'`. + A capability error (`No HIP GPUs are available`) alongside a missing device + flag in step 2 confirms drift rather than a code or image fault. Count the + damage: `podman inspect --format '{{.RestartCount}}'`. + +## Fix + +Quickest unblock: recreate from the definition (`podman compose up -d +--force-recreate `), which is what should have happened automatically. + +Durable fix, when the host is governed by a repo that requires boot-persistent +units (this one does): migrate the service to a Quadlet. Order that worked: + +1. build the image and `install` the unit file to `~/.config/containers/systemd/` +2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a + leftover `~/.config/systemd/user/.service` collides by name with the + unit the Quadlet generator produces +3. `podman rm -f ` — the stale container holds the name the unit's + container needs, and it carries the old config +4. `systemctl --user daemon-reload && systemctl --user restart .service` +5. delete the compose service so only one owner remains per container name + +## Verify the capability, not the status + +`Up` and even `healthy` can be true of a half-configured container. Check the +thing that was broken, from inside: + +``` +podman exec python3 -c "import torch; print(torch.cuda.is_available())" +ss -ltn | grep # confirm the bind addresses, not the intent +``` + +And confirm boot persistence the way the generator exposes it: +`systemctl --user is-enabled .service` → `generated` is healthy; an +explicit `enable` fails on generated units. + +## Related reading in this repo + +- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces +- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident +- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice diff --git a/agent/managed-skills/fleet-container-ownership-audit/SKILL.md b/agent/managed-skills/fleet-container-ownership-audit/SKILL.md new file mode 100644 index 0000000..86d22c6 --- /dev/null +++ b/agent/managed-skills/fleet-container-ownership-audit/SKILL.md @@ -0,0 +1,91 @@ +--- +name: fleet-container-ownership-audit +description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them." +--- + +# Finding the containers that will not survive a reboot + +Use when auditing a Podman fleet for boot persistence, when a service is +"up" but stopped working at some past reboot, or when a compose-managed stack is +being converted to Quadlet units. This is the *detection* half; the conversion +checklist is separate. + +## The one-line test + +Every running container on a boot-persistent host should carry a +`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and +rootless Podman does not restart bare containers after the explicit stop that +shutdown performs — so it will be missing tomorrow morning. + +```bash +for c in $(podman ps --format '{{.Names}}'); do + printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')" +done +``` + +Compare the result against the host's installed units +(`ls ~/.config/containers/systemd/*.container`) and the repo's +`//quadlet/`. Three outcomes, three meanings: + +- **label present** → the unit owns it; boot persistence comes from `[Install] + WantedBy=default.target` plus `loginctl show-user -p Linger` = yes. +- **label absent, compose service exists** → the classic gap: it dies at reboot. + Also check the publish address while you are there; compose forms often bind + the wildcard, which puts an unauthenticated UI/API on every interface. +- **label present AND a compose service with the same name** → a second owner. + The runtime is fine, but a stray `podman compose up` can create a duplicate + that fights the unit for the container name and the port. Remove the compose + service and leave an explanatory comment; that is the repo's convention. + +## Traps that make this hard to see + +- **Editing a compose service does not replace a running container.** + `restart: unless-stopped` will happily keep an old container — old image, old + device list, old mounts — alive forever. One instance ran 9,146 restart + attempts on stale config before anyone looked. If the container predates the + edit, recreate it explicitly; do not debug the service. +- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.** + It can read empty on a container that demonstrably has the devices. Read the + real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside: + `podman exec python3 -c 'import torch; print(torch.cuda.is_available())'`. +- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape + that flips from `connection refused` to `unsupported Content-Type "text/html"` + means the service came up and only the scrape is wrong. Read the error, not + just up/down — and never "fix" a red probe by deleting the job, because an + absent scrape and a healthy one look identical. +- **A hand-written `~/.config/systemd/user/.service` that shells out to + `podman compose up` collides by name with the Quadlet the generator produces + for `.container`.** Retire it to a dated trash directory before + installing the unit. +- **`is-enabled` saying `generated` is the healthy state** for Quadlet units; + `systemctl --user enable` fails on them. + +## Migration shape that works + +1. Unit file in the repo at `//quadlet/.container`: + `Image=` with the `localhost/` prefix for locally built images, `Pull=never`, + explicit `PublishPort` (loopback unless the service is the documented LAN + exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed, + `Restart=always`, `[Install] WantedBy=default.target`. +2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator + splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle + that fails silently. +3. `deploy_()` in the remote handler: install the unit, retire any + compose-era unit into `~/.AGENTTRASH/`, `podman rm -f ` so the + unit's own can take the name, daemon-reload, restart, wait for health. +4. Delete the compose service (one owner per container name) and leave a comment + naming the unit, so the next reader does not re-add it. +5. Planner: add a path rule (`//**` -> service) **and** a + `service_is_allowed` entry, or the shared dispatcher rejects the service + before anything runs. +6. Migrate the seams: tests that read the compose file for devices, command + flags or groups must read the unit instead — and resolve file paths from the + tooling's own declarations, never from `find ... | head -1`. + +## When the last service leaves a compose file + +The repo's compose-validity seam requires at least one service, so a file whose +services have all migrated fails its own test. Either keep the file as +comments and relax that seam deliberately, or delete the file and fix the +planner's compose-diff path — decide with the owner, because the deploy tooling +reads it. diff --git a/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md b/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md new file mode 100644 index 0000000..bc205be --- /dev/null +++ b/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md @@ -0,0 +1,117 @@ +--- +name: fleet-llm-payload-purge-refresh +description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap." +--- + +# Purge and refresh LLM model payloads (Dank fleet) + +Use when asked to clear / reclaim / refresh model caches on a host that is +*serving* users, or to prove that a payload is up to date. + +## 1. Inventory before touching anything + +```bash +du -sh /home/dank/models/* | sort -h # what is actually there +``` + +Then find which of those the live services mount — never delete a payload a unit +mounts without an explicit, in-this-conversation instruction: + +```bash +grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \ + /home/dank/.config/containers/systemd/*.container \ + /etc/systemd/system/frank-*.service +``` + +On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn +checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF + +mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not** +`models/checkpoints` — verify, because `models/checkpoints` looks like cache but +has held Flux weights whose upstream is licence-gated. + +## 2. Confirm each payload is re-fetchable BEFORE deleting it + +Query the Hub anonymously; a gated repo 403s even with the host token: + +```bash +curl -s "https://huggingface.co/api/models//tree/main?recursive=true&blobs=true" \ + | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]" +``` + +An error/empty result means it is NOT safe to delete. Also filter by **quant +name**, never by `endswith('.gguf')` — mradermacher ships large quants as +`.partNofM` fragments which that test hides. + +## 3. Fingerprint, then delete + +Record byte sizes (and sha256 when the file matters) **before** removal: that is +the only way to answer "did the refresh actually change anything" afterwards. + +Per this repo's policy `rm` needs explicit operator permission; otherwise stage +into `~/.AGENTTRASH/-/`. Note that a same-filesystem trash reclaims +nothing — say so rather than implying space was freed. Root-owned leftovers (a +trashed swapfile) need `sudo`, and check `swapon --show` first. + +## 4. Refresh, per service + +**frank-large (halogen)** — the engine fetches for you. Clear the directory but +leave it **existing and empty**, then start the unit: + +```bash +rm -rf /home/dank/models/halogen/* # keep the directory itself +systemctl --user start frank-large.service +journalctl --user -u frank-large.service -f | grep -a halogen: +``` + +The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD` +(`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`, +resuming if interrupted. **Trap:** podman refuses a bind mount whose source +directory is missing — `Error: statfs /home/dank/models/halogen: no such file or +directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty +directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load. + +**frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start +with weights missing, so pull them yourself and verify: + +```bash +DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF +# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf +curl -fL -C - --retry 10 -o "$DEST/" \ + "https://huggingface.co//resolve/main/" # -f is mandatory +sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to the LFS oid + GGUF magic +``` + +`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always +`-f`, always compare byte counts to the Hub's declared size, always check the LFS +sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from +argv entirely (it is world-readable in `/proc//cmdline`). + +## 5. Verify, then record + +```bash +curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # frank-large +curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health # frank-small +curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models +# a REAL completion on both, then halogen's boot ledger: +journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything" +``` + +Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`, +`gitea`, `loki` — `curl -sk --resolve :443:192.168.0.143`) because a purge +touches state the whole host shares. + +## 6. Things that bite + +- **Sidecars are versioned in place.** Equal file sizes are not proof the repo is + unchanged, and a stale local sidecar is invisible from its name. Compare local + sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB + overlay update was caught. +- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The + A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding + cache churn there risks the OOM/state failures this fleet has already seen. +- **Reboot is the repair of last resort** if rootless podman's pause process dies + during the work (`invalid internal status … could not find any running + process`); `podman system migrate` panics in 5.7.0. +- Because a peer session may be committing to `serverAppConfig` concurrently, + re-read a doc region immediately before editing it, and check that the edit + response's neighbouring lines are the ones you expect. diff --git a/agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md b/agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md new file mode 100644 index 0000000..6e37eb3 --- /dev/null +++ b/agent/managed-skills/fleet-model-cache-purge-refresh/SKILL.md @@ -0,0 +1,73 @@ +--- +name: fleet-model-cache-purge-refresh +description: "Clear and re-fetch model payloads on a Dank-fleet host that serves production LLM endpoints — inventory, protect live payloads, verify every fetch, restore the service, and use the refresh to detect upstream updates." +--- + +Clearing a model cache on a serving host is a two-sided operation: the operator wants no stale +payload, and the host must keep serving. Do it in this order, on the target host. + +## 1. Inventory before touching anything + +```bash +du -sh /home/dank/models/* | sort -h # payload dirs, biggest last +grep -rlE "models/|" /home/dank/.config/containers/systemd/ /etc/systemd/system/*.service +``` + +Classify every dir: **live service payload** (referenced by a unit/env), **retired runtime +payload**, **unreferenced model dir**, **tool/script dir**, **logs** (evidence — keep). Check the +service env files (`scripts/*.env`) for `MODEL_DIR`/`MODEL` paths, not just unit `Volume=` lines: +a payload can be mounted by path and named by env. `models/logs` and small tool dirs are not cache. + +## 2. Record fingerprints first + +`ls -l --block-size=1 ` gives sizes; for the LLM payloads compare each to the Hub's declared +size (`/api/models//tree/main?recursive=true&blobs=true`). They can differ — see §6. If a +file is irreplaceable upstream (delisted revision, gated repo) say so before deleting it. + +## 3. Protect live payloads by delete-then-refetch, never delete-only + +Deleting a live payload leaves the endpoint unable to start. Know the recovery path *before* +deleting: some engines fetch their own weights (halogen: `HALOGEN_DOWNLOAD`, triggered when the +checkpoint is absent), others need a manual fetch (llama.cpp GGUF services: the serve script +refuses to start without the weights). + +**Bind-mount trap:** podman refuses a mount whose source directory is missing — +`Error: statfs /home/dank/models/: no such file or directory`, with the unit looping on +`Restart=on-failure`. Recreate the directory **empty** (`mkdir -p`), and the engine's first-start +fetch runs. + +## 4. Fetch with verification (never trust an exit code) + +`curl -o FILE URL` exits 0 on 401/403 and writes the error body — a gated repo produced three +153-byte shards with identical sha256s and a cheerful DONE line. Always: `-fL -C -`, compare the +byte count against the Hub's declared size, and compare `sha256sum` against the LFS oid. Public +repos means **no token in argv** (it is world-readable in `/proc//cmdline`). + +**Parts trap:** mradermacher-style repos ship large quants as `.part1ofN … .partNofN` — a filter on +`endswith('.gguf')` makes those quants look absent. Fetch the parts, verify each, then +`cat part1 part2 part3 > name.gguf` in numeric order; the assembled file's size equals the sum +(concatenation is deterministic, so per-part hashes verify the whole). + +## 5. Restore, then verify behaviour — not health alone + +Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), `reset-failed`, `start`; +root units need `sudo`. Watch the engine's own fetch/load lines in the journal, then check +`/health`, `/v1/models`, and a **real completion** on every restored endpoint. Read the engine's +memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits. + +## 6. A refresh is also an update check + +Clearing and re-fetching only guarantees freshness *at fetch time* — if upstream has not changed, +the bytes come back identical, which is itself the finding. Compare recorded fingerprints against +the new local sizes and the Hub's declared sizes: sidecars are often versioned **in place**, so a +stale copy is invisible from its file name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 +while the main checkpoint returned byte-identical). + +## 7. Close out + +- Clear test payloads once the benchmark using them is done, and keep the payload the *next* + experiment needs. +- Leave the operator's decision points explicit: unreferenced-but-licensed dirs, other services' + caches, and whether they want the payloads absent at their next run or fresh as of now. +- Record sizes/verdicts in the vault (runbook + `log.md`); never `rm` without explicit permission, + and remember a same-filesystem trash reclaims no space. diff --git a/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md b/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md new file mode 100644 index 0000000..0476867 --- /dev/null +++ b/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md @@ -0,0 +1,79 @@ +--- +name: fleet-monitoring-gap-audit +description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses." +--- + +# Auditing a homelab fleet's monitoring for what *should* be there + +Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose +services are declared in a git repo. The two failure modes this catches are +different, and only the first announces itself. + +## 1. A service that should exist and doesn't is invisible + +A missing exporter shows up as **nothing**: no failing target if the job was +configured for a host that no longer serves, no alert because the series is +simply absent. So check existence, not health: + +```bash +# Every container the host actually runs, including dead ones +podman ps -a --format '{{.Names}} {{.Status}}' +# Every container that is supposed to be boot-persistent +ls ~/.config/containers/systemd/*.container +systemctl --user is-enabled .service # expect: generated +``` + +A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic +case: rootless Podman does not restart bare containers after the stop that +shutdown performs, so a compose-only service dies at the first reboot and stays +dead. Check the compose file for the name — if it is there and has no unit, the +repo's own rule is being violated. Fix by writing the unit (mirror the same +service on the other host), deleting the compose entry (one owner per container +name), and making sure the deploy path can reach it: + +- planner: service in the target's service set + a path rule for its directory +- dispatcher allowlist: `:` +- handler: install from `"$TARGET/..."`, never a hardcoded peer's path + +## 2. Classify every down target — do not report a count + +`targets?state=active` gives health **and** the last error. Two down targets can +mean opposite things: + +| Symptom | Meaning | +|---|---| +| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) | +| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine | +| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter | +| target absent entirely | see §1 | + +```bash +curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c " +import json,sys +for t in json.load(sys.stdin)['data']['activeTargets']: + if t['health'] != 'up': + print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])" +# and probe each by hand before believing the scraper +curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http:///metrics +``` + +Say which are down and why. "7 of 11 up" without the four reasons is how a real +outage hides behind three config defects — that is exactly what one fleet did: +its crash-looping GPU service was one of four failures reported as a number. + +## 3. Check the exposure posture while you are in there + +An exporter that binds the wildcard publishes an unauthenticated API on every +interface. Expect exactly two listeners per host-exporter — loopback for +host-local probes, the host's own LAN address for the Prometheus container: + +```bash +ss -ltn | grep # must show the two explicit addresses, never 0.0.0.0 +``` + +## 4. Leave the decisions that are not yours + +Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**: +disable a probe to make a dashboard green, or "fix" a crash-looping service whose +correct posture depends on what the owner wants — capture the evidence (restart +count, exit code, last error line) and hand it over intact. diff --git a/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md b/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md new file mode 100644 index 0000000..4014ffb --- /dev/null +++ b/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md @@ -0,0 +1,97 @@ +--- +name: fleet-second-llm-endpoint +description: "Add a second LLM endpoint to a host that already serves one on the Dank fleet (new port, unit shape, co-tenancy budget, harness provider/roles, monitoring, cutover, verification) — with the port-bind race, the reasoning_effort template trap, and the tools/--jinja trap that silently break subagents." +--- + +# Adding a second LLM endpoint to a Dank fleet host + +Use when a host must serve two models at once (e.g. an iGPU runtime plus a +second card), or when one endpoint is moving and a new one takes the freed +hardware. Rebuilt from the 2026-09-16 Dankclaw split (halogen `frank-large` on +:7770 + llama.cpp `frank-small` on :7771). + +## 0. Decide the shape before touching hardware + +- **One provider per HOST:PORT** in `~/.omp/agent/models.yml` (on the box where +the harness runs, not on the servers). A second endpoint is a new provider +block, not a second model in the existing one. +- **Read the budget as an inequality, not a wish.** For a unified-memory runtime +the engine's own fit check is the arbiter (`budget = host − pinned weights − +reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV by +counting only the full-attention layers — and remember `--parallel N` DIVIDES +`--ctx-size`, it does not multiply. +- **A co-tenant is a budget line, not a footnote.** Free host RAM by changing +the POOL or the ARENA (the reserve is a fit-check term, never an allocation), +and drop optional charges (a vision tower is ~2 GiB) rather than hoping. + +## 1. Repo first, and retire in the same commit + +1. New service files: `scripts/.env` (documented tunables), `.service` + (root unit for a native binary; a Quadlet only for a container), + `serve-.sh` (defaults + hard guards for missing artifacts). +2. Planner: add the service to `SERVICE_ORDER` AND the target's service set in + `scripts/deploy-plan.py`, map its paths (including "the unit/env/serve trio + selects only me"), add it to the inventories in `deploy.sh` and + `deploy-remote.sh`, and the dispatch case. +3. Retire what the cutover obsoletes into `.AGENTTRASH/` with a suffix — and + **check what pointed at it**: bench harnesses, a chat template only the old + runtime read, and `scripts/test-changed.sh` mappings are the usual victims + (a mapping to a test file retired earlier fails the pre-commit wrapper + silently for weeks). +4. Seams: one bats file per service asserting the *invariants* (`pool == 2 × + ctx`, KV bytes ≤ card budget, port separation), not just constant values. + A negative grep must strip comments (`grep -vE '^\s*#' file | grep -q ...`) + or it will match the prose that explains the very thing you removed. + +## 2. The port-bind race (rootless podman) + +pasta binds the published port at container CREATE. A predecessor that has just +stopped still holds it, so a naive handler starts a doomed container: + +``` +Deactivated successfully <- predecessor gone +pasta: Listen failed for HOST TCP port */PORT: Address already in use +exit 126 <- only Restart=on-failure saves you +``` + +Order the handler: **retire the old manager's unit → stop THIS service's own +unit (no-op on a first cutover) → wait for every socket on the port to clear +(listener, accepted, TIME_WAIT) → install → start.** Without the second step a +redeploy deadlocks against its own gate. + +## 3. Wire the surfaces, and refuse impossible signals + +- Harness roles: point the small/fast + image roles at the new endpoint, and + move the vision role with the vision tower (only one endpoint should own + each capability). +- `shared/homepage/services.json`: a card per endpoint, probed from the + monitor's own host (loopback is legal there; the LAN address otherwise). +- Prometheus: only add a job if the runtime actually exposes Prometheus text. + Halogen publishes `/health`, `/cache` and `/v1/*` and nothing else — scraping + it would be red by construction, so the job is REMOVED with the reason written + down rather than left as a permanent red. + +## 4. Cutover, then verify by driving it + +```bash +./scripts/deploy.sh --service --service # order matters +``` + +Then, per endpoint: `/health`; `/v1/models` reports the alias; a chat completion +returns content AND `reasoning_content`; **a request carrying `tools` returns +`tool_calls`**; an image request is answered (or refused with a named reason); +the runtime's own speculation counters move; and the unit is `enabled` / +`generated` with linger on. Read the engine's boot ledger for a silent clamp. + +Finally spend one real request through the harness per changed role — routing +labels lie, the servers' own journals do not: `journalctl -u ` shows which +endpoint actually served it. + +## Traps that look like model faults + +| symptom | cause | +|---|---| +| `500 tools param requires --jinja flag` | the endpoint's runtime lacks the Jinja chat-template path, so every tool-using agent on it is broken. mainline llama.cpp defaults it on — pin it | +| `500 Jinja Exception: Unexpected reasoning effort ` | the model's own template has a narrower accepted set (Qwen3.8: xhigh/medium/low only) than the harness's levels; fold them in `compat.reasoningEffortMap` | +| the same request works by hand but a subagent dies | subagents run at a different thinking level — check both | +| the container dies instantly although the predecessor "stopped" | the port race above | diff --git a/agent/managed-skills/fleet-service-precommit-iteration/SKILL.md b/agent/managed-skills/fleet-service-precommit-iteration/SKILL.md new file mode 100644 index 0000000..f78ca9e --- /dev/null +++ b/agent/managed-skills/fleet-service-precommit-iteration/SKILL.md @@ -0,0 +1,64 @@ +--- +name: fleet-service-precommit-iteration +description: "Iterate on a service that lives in the serverAppConfigs git checkout on a fleet host (dankclaw/dankserver) before committing: ship files for live testing, keep the checkout pullable, then commit and deploy — covers the untracked-files pull trap that recurs every session." +--- + +# Iterating on a fleet service before you commit + +This repo's contract is git → `deploy.sh` → host. But live testing needs the new +file *on the host* before it is committed, and the naive way (copy it into the +host checkout) blocks the very deploy that would install it. This is the loop +that works, plus the traps that cost real time. + +## The loop + +1. Edit in the local repo. Run the seams: `./tests/run.sh`. +2. Ship the working files to the host **for testing**: + `cat local/file | ssh 'cat > /home/dank/serverAppConfigs/'` + (then `chmod +x` if it is executed — scp/`cat >` drops the exec bit, and a + container entrypoint mounted from that path fails with status 126, "command + found but not executable"). +3. Test against the live service; iterate until it works. +4. **Before committing**, prove the host copy is byte-identical to what you are + about to commit: `sha256sum` locally and on the host. If they differ, you are + about to lose work in step 5. +5. Commit and push. +6. Clean the host checkout so the deploy can pull: + - untracked hand copies → move them out of the checkout + (`mkdir -p /tmp/handcopy && mv /tmp/handcopy/`); + - modified tracked files → `git checkout -- ` (this *discards* the + hand copy, which is safe only because of step 4); + - confirm with `git status --short` printing nothing. +7. `./scripts/deploy.sh --service ` — it pushes, pulls ff-only, + installs units/config, restarts, waits for health, then runs the Playwright + e2e gate. + +## Traps, in the order they bite + +- **The pull aborts on hand copies.** `deploy.sh` pulls with ff-only and git + refuses: *"The following untracked working tree files would be overwritten by + merge"*. Any untracked file in the checkout whose path the pull wants to + create stops the deploy — which is exactly the file you just shipped. Always + do step 6. +- **`git checkout --` silently eats uncommitted work.** Without step 4 you can + destroy the only copy of an edit that exists nowhere else. +- **A runtime mounts files *straight out of the checkout*.** Here + `dankserver/frank-medium/**` and `shared/grafana/provisioning/**` are mounted + read-only into containers, so a hand copy takes effect at the next container + restart with no deploy. Test that way, then let the deploy make it official. +- **Root-owned files from a container write** (e.g. a patched `.so` produced + inside a build container) can land in the checkout as `root:root`; `chown` them + back to the deploy user before git operations. +- **A restart is not a verification.** Health endpoints answer from a stale + process, a proxied service can be served by a peer, and a provisioned + dashboard may live in unified storage rather than the SQLite table you grep. + Verify the *specific* claim: the new code path's own output, the file's + checksum on the host, the API that serves the resource. + +## Why not keep a scratch copy on the host? + +Because the deployed artifact must be the committed one. Iterating outside git +(the `/tmp` loop is fine for *probes*) is for measurement scripts; anything the +service actually runs has to land in the checkout, and the checkout has to stay +pullable. Two directories, one rule: probes may live anywhere, runtime state +lives in git. diff --git a/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md b/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md new file mode 100644 index 0000000..4d6fae5 --- /dev/null +++ b/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md @@ -0,0 +1,95 @@ +--- +name: frank-medium-batch-split-retune +description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify." +--- + +# frank-medium batch geometry + split rebalance + +Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the +model on frank-medium, or when someone reports that prefill "does not respond" to +those knobs. The batch and the tensor split are **coupled**: the compute buffers are +not symmetric across the two cards, so any batch change moves the balance point. + +## Order of work (do not skip the second half) + +1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold + a shape constant across dozens of configs: + `scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation + per cell, `-r 3 --no-warmup`. +2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by + loading the real service with a per-config env override and reading `rocm-smi`. A + llama-bench number cannot decide the split: it has no `--parallel`, sizes its own + context as `n_prompt + n_gen + n_depth`, and cannot load the drafter. +3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed + point, then **deploy** and verify. + +## 1. The sweep + +```bash +# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is +# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here +sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \ + --property=LimitMEMLOCK=infinity --property=User=dank \ + --setenv=OUT=/home/dank/models/logs/bench-bub \ + --setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \ + /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh +``` + +The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv` +(appended as it goes, so progress is readable), samples VRAM per cell, and restarts +the service from an EXIT trap. A re-run skips cells that already have a `bench.json`. + +Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not +track the request, nothing else in the table means anything. The VRAM columns scaling +monotonically with ubatch is the independent confirmation. + +Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth): +an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046, +384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral +(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every +cell). Test **both** ends of the range: sampling only upward makes a shallow basin look +like a plateau. + +Sanity rule: the same ubatch measured under different logical batches should agree to +under 1 tok/s. If it does not, the table is noise and the first suspect is a stray +GPU user, not the knob. + +## 2. The split, at the deployed shape + +`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env` +to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with +`FRANK_MEDIUM_ENV_FILE=`, polls `/health` for `"status":"ok"`, reads +`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under +`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`. + +**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH +cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the +process dies — a 180 s sleep fails, and even 600 s has failed while the process is still +alive inside the unit. Skipping the gate produces +`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from +"this split does not fit" and will send you chasing a phantom OOM. + +Free bytes to expect, and the arithmetic behind them: +`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs +the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB +against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one +layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the +reachable minimum gap is about one DeltaNet layer, never zero. + +2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32; +**35,30 @512 -> 6.44 / 6.90 (deployed)**. + +## 3. Seam, deploy, verify + +- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured + 1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the + model reproduces the new point. Check that neighbouring splits still FAIL the balance + gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the + slack at one DeltaNet layer. +- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares + summing to the model's 65 blocks, ROCm1 the smaller). +- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`. +- Verify live: the flags in `/proc//cmdline`, `/slots` = 2 x 131072, VRAM matching + the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance` + lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB + lower once the service has served traffic than straight after load. diff --git a/agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md b/agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md new file mode 100644 index 0000000..d19acb3 --- /dev/null +++ b/agent/managed-skills/grafana-prometheus-provisioning-as-code/SKILL.md @@ -0,0 +1,94 @@ +--- +name: grafana-prometheus-provisioning-as-code +description: "Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives." +--- + +# Observability as code (Grafana + Prometheus, Dankclaw fleet) + +Use when adding a scrape target, an alert rule, a datasource, or a dashboard to +this homelab. Everything lives in git; Grafana and Prometheus only ever read +what the deployer installs. + +## The triad — every change touches three places + +1. **Config in the repo** + - `shared/prometheus/prometheus.yml` — `scrape_configs`, and `rule_files:` + pointing at `/etc/prometheus/rules/*.yml` + - `shared/prometheus/rules/.yml` — alert/recording rules + - `shared/grafana/provisioning/datasources/prometheus.yml` — datasource + - `shared/grafana/provisioning/dashboards/{provider.yml,.json}` +2. **The copier in `scripts/deploy-remote.sh`** — `copy_prometheus_config` + installs config **and** rules into `/opt/prometheus/config/` (the container + mounts that to `/etc/prometheus`); `copy_grafana_provisioning` installs into + `/opt/grafana/provisioning/` (the grafana unit already mounts it read-only). + Paths are absolute and `run_privileged`. +3. **The planner in `scripts/deploy-plan.py`** — a `shared//**` change + must select the service that reads it (`shared/prometheus` → prometheus, + `shared/grafana` → grafana). Missing mapping = the file deploys nowhere and + nothing reports an error. + +Add a seam in `tests/observability_quadlet.bats` for each change: the file +exists, the config references it, the deployer copies it, and — where two files +must agree (target vs published port) — that they do. + +## Reach targets by container name, never the loopback publish + +Datasources use `http://prometheus:9090` and scrape jobs for co-located +services use `prometheus:9090` / `loki:3100`. Published ports are +`127.0.0.1`-only by design (see the exposure rules), so a browser-facing URL or +a published port in a datasource is wrong from inside the network. LLM +endpoints are the exception: they are on other hosts, so use their LAN address. + +## Three failure modes that produce no error message + +- **A rule file nothing loads evaluates nothing.** Prometheus reports no + problem for a `rule_files` entry that matches nothing, or for a rules file + nobody references. Ship both together and assert the reference in a seam. +- **A provisioned dashboard is not in the legacy `dashboard` table** on the + current Grafana. Reading `grafana.db` for it returns empty and looks like a + failed provision. It lives in unified storage behind + `dashboard.grafana.app`; query the API instead. +- **Lifetime averages are not per-request evidence.** e.g. an engine's + "Prefix cache hit rate: 29 %" is every prompt it ever saw, including + benchmarks. Measure the behaviour you care about with a targeted probe. + +## Verify through the channel the consumer uses + +```bash +# Prometheus: config and rules parse/load +ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/.yml' +ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \ + | python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"' +ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...' # every target health=up +ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...' # 0 firing when healthy + +# Grafana: provisioned objects and a live query (admin/admin on this install) +ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"' +ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/"' +ssh dankclaw 'curl -s -u admin:admin \ + "http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"' +``` + +A target that is `up` proves reachability, not data: also run one query for a +series you expect and confirm it returns a sample. + +## Grafana facts for this install + +- No UI-configured state: `editable: false` on the datasource, fixed + `uid: prometheus` so dashboards reference it without a lookup. +- Dashboards re-read from disk every 30 s (`updateIntervalSeconds`), so a JSON + edit appears without a restart — but the file must still be *installed*, which + means a deploy. +- There is **no Alertmanager and no contact point**: rules evaluate, alerts show + on `/alerts`, in the `ALERTS` metric and on a dashboard panel, and reach + nobody. Say that plainly in docs rather than implying observability. + +## Naming metrics + +Never write a series name from memory into a config, a rule, or a dashboard — +one shipped comment here named a metric that does not exist. Dump the live +names first and paste them: + +```bash +ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u' +``` diff --git a/agent/managed-skills/grafana-provisioning-as-code/SKILL.md b/agent/managed-skills/grafana-provisioning-as-code/SKILL.md new file mode 100644 index 0000000..4df5e70 --- /dev/null +++ b/agent/managed-skills/grafana-provisioning-as-code/SKILL.md @@ -0,0 +1,92 @@ +--- +name: grafana-provisioning-as-code +description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative." +--- + +# Grafana provisioning as code (Dankclaw fleet) + +Use when adding or changing a Grafana datasource or dashboard on this fleet, or +when a dashboard "does not appear" after a deploy. + +## Shape of the thing + +Grafana on Dankclaw is an **empty install configured entirely from git**. The +quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a +provisioned artefact exists in exactly three places: + +``` +shared/grafana/provisioning/datasources/.yml # repo: source of truth +shared/grafana/provisioning/dashboards/provider.yml # dashboard loader config +shared/grafana/provisioning/dashboards/.json # dashboards + | deploy-remote.sh: copy_grafana_provisioning() + v +/opt/grafana/provisioning/... # host: installed state + | read-only mount + v +/etc/grafana/provisioning/... # generated: inside container +``` + +Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be +invisible to review, unknown to anyone reading the repo, and lost with the data +volume. + +## Procedure + +1. Write the files under `shared/grafana/provisioning/`. + - Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by + uid, so a random one breaks them), `access: proxy`, and the URL by + **container name** (`http://prometheus:9090`) — the published port is + loopback-only and unreachable from inside the network. + - Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab; + failing an operator who wants a different rate window is the wrong trade) + and a JSON dashboard under a stable `uid`. +2. Add the file(s) to `copy_grafana_provisioning()` in + `scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each + file — follow `copy_prometheus_config` for style). +3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify + with `python3 scripts/deploy-plan.py --target dankclaw --paths `. +4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse, + the datasource URL must be the container name, the unit must mount + `/opt/grafana/provisioning`, and the deployer must call the copier. Then + `./tests/run.sh`. +5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the + provisioning tree, restarts Grafana, then runs the e2e gate). + +## Verify — and the trap that makes the obvious check lie + +A **provisioned dashboard does not appear in the legacy `dashboard` table** of +`grafana.db` on the current Grafana build; dashboards live in unified storage +behind the `dashboard.grafana.app` API group. The datasource *does* land in +`data_source`, and that asymmetry is easy to misread as "datasource worked, +dashboard failed". A `sqlite3`/python query against `dashboard` is a **false +negative**, not evidence. + +Check in this order instead: + +```bash +# 1. provisioning actually ran (these loggers name themselves) +podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards" + +# 2. the dashboard exists (uid is what you set in the JSON) +curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db" + +# 3. the datasource really reaches Prometheus (proves the whole chain) +curl -s -u admin:admin \ + "http://127.0.0.1:3002/api/datasources/proxy/uid//api/v1/query?query=up" +``` + +Grafana publishes on loopback `:3002` only; the off-host path is the guarded +vhost. Its admin login is still the default `admin:admin` — which is why every +assertion above needs credentials, and which is worth changing. + +## Before writing any panel or config comment + +Grep the metric names out of the live endpoint; naming a series that does not +exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the +real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`, +whose *rates* give the hit rate). A comment naming an unqueryable metric, or a +panel that silently renders "No data", is worse than no panel: + +```bash +curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm +``` diff --git a/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md b/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md new file mode 100644 index 0000000..a50f460 --- /dev/null +++ b/agent/managed-skills/heterogeneous-accelerator-split-triage/SKILL.md @@ -0,0 +1,94 @@ +--- +name: heterogeneous-accelerator-split-triage +description: "Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed." +--- + +# Heterogeneous accelerator split — triage before building + +Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm. + +## 1. Characterise the link (bytes vs latency) + +```bash +lspci -vv -s | grep -E "LnkCap:|LnkSta:" # capability vs actual +nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv +``` + +Expect **idle-downshift**: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number. + +Compute the per-hop cost against the hidden state (`n_embd x 2 B` for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then: + +- bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. **Bytes are almost never the constraint.** +- the real budget line is **hop COUNT**: every device boundary is a host-side sync. A runtime that splits per *op* can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure `graph splits` in the loader log, do not assume. + +**Design rule: partition by where the weights live, never by where the activations flow.** Resident weights, activations crossing. + +## 2. Get the workload's real bottleneck (KV is usually it) + +KV per position x context = bytes read per token, because attention reads the whole cached context every token: + +``` +KV/token = kv_bytes_per_position * context +``` + +Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, **KV dominates**. Then: + +- the highest-leverage placement is **KV cache on the fastest-memory device, together with its attention compute** — never KV alone (separating them puts `KV/token` over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token); +- check the KV + attention weights actually fit: `kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM`. + +## 3. What placement never buys + +Prefill is compute-bound and expert weights are read **once** per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave **decode +12 %, prefill 192 -> 32 tok/s**. + +A `1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining). + +## 4. Is the second device even a separate pool? + +On a UMA APU, check before assuming additivity: + +```bash +cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small) +cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM +rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees +``` + +A large GTT window means the "GPU memory" is the *same* LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM. + +## 5. Speculative decoding as the "cheap" split + +A drafter on the *other* device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it: + +- the drafter needs the target's **exact tokenizer** for a plain draft model (verify by comparing GGUF `tokenizer.ggml.tokens`, not by trusting the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models); +- a plain drafter costs **K sequential forwards per round**, so it must be 10-50x faster than the target. Compute the break-even: with `r = t_draft/t_target`, `rate_ratio = A/(K*r + 1)`, `A = (1-p^(K+1))/(1-p)`. At `r ≈ 0.4-0.5` (a drafter only ~2x faster) break-even needs `p ≈ 0.7+` at K=3; +- always compare against the **incumbent**, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar; +- check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (`DRAFTER_CAP`-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd-ctvd`. + +## 6. Expressing a split (llama.cpp, the usual vehicle) + +```bash +-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest) +-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (what KV co-location needs) +``` + +- buffer types are matched by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values; +- `-cmoe` / `-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`; +- several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and `GPU_BACKEND_A=OFF` in the CMakeCache of the existing build tells you what needs adding; +- contiguous ranges are the *wrong* split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool. + +## 7. Hazards checklist + +- paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss; +- keep `mmap` on for models with huge embedding tables (anonymous memory gets the process cgroup-killed); +- CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply; +- the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a **baseline of the runtime on that device alone**, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine; +- a split consumes the fast device entirely — check what else it was serving before trading it away; +- benchmark/experiment discipline on production hosts: stop and mask first. + +## 8. Verdict shape + +Rank and report, and prefer the boring answer when it wins: + +1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link; +2. **One process, weights-resident, KV co-located with attention on the fast pool** — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement; +3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check `misses x bytes x tok/s` against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache; +4. list explicitly what does *not* help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements. diff --git a/agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md b/agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md new file mode 100644 index 0000000..0ab1df6 --- /dev/null +++ b/agent/managed-skills/hf-gguf-payload-fetch-verify/SKILL.md @@ -0,0 +1,98 @@ +--- +name: hf-gguf-payload-fetch-verify +description: "Inventory, fetch, hash-verify and assemble a Hugging Face GGUF payload (including mradermacher's .partNofM fragments and gated mirrors) before pointing an LLM service at it — on the Dank fleet or anywhere else." +--- + +# Fetch and verify an HF GGUF payload + +Use before any model swap where the weights come from Hugging Face, especially +large (>32 GiB) quantised models. The failure modes below all look like success. + +## 1. Inventory — query by quant name, never by extension + +```bash +curl -s "https://huggingface.co/api/models///tree/main?recursive=true&blobs=true" \ + | python3 -c " +import json,sys +for e in json.load(sys.stdin): + p=e.get('path','') + if 'Q4_K_S' in p: # the QUANT, not '.gguf' + print(p, e.get('size',0), (e.get('lfs') or {}).get('oid','-')) +" +``` + +**Large quants ship as `.partNofM` fragments.** `mradermacher/*-i1-GGUF` repos +hold e.g. Q4_K_M as `part1of3/2of3/3of3` (37.00+37.00+36.97 = 110.97 GiB) and +Q4_K_S the same way, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are +single `.gguf` files. Filtering on `endswith('.gguf')` therefore makes Q4_K_S +look like it does not exist — do not conclude a quant is missing from an +extension-filtered listing. + +Check both mradermacher repos: `…-i1-GGUF` (imatrix) and `…-GGUF` (plain), and +the model's own repo. Gated mirrors exist (`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF` +carries the same ladder) and return **403 even with a valid HF token** until +that account accepts the gate — prefer the ungated copy rather than chasing access. + +## 2. Fetch — `-f` is mandatory, secrets stay out of argv + +```bash +# token out of band: mode-600 config, never -H on the command line +printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc +chmod 600 /tmp/hf.curlrc +curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc \ + -o "$DEST/$name" "$BASE/$name" +``` + +- `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes the error body as the + artifact. A gated repo yields identical small files (153 bytes) that hash the + same and a `DOWNLOAD_DONE` in seconds for a 104 GiB payload. +- A token in `-H "Authorization: Bearer …"` is visible in `/proc//cmdline` + to any `ps`/`pgrep -a` on the host. Never `pgrep -a` a running fetch. + +## 3. Verify, then assemble + +```bash +# per part: byte count must equal the Hub's declared size, sha256 must equal lfs.oid +sha256sum "$PARTS"/*.part* | sort + +# concatenate in NUMERIC order, then verify the total and the container magic +cat "$PARTS"/X.gguf.part1of3 "$PARTS"/X.gguf.part2of3 "$PARTS"/X.gguf.part3of3 > "$OUT" +stat -c%s "$OUT" # must equal the sum of the part sizes +head -c 4 "$OUT" # must print GGUF +``` + +The assembled file has no published hash (only the parts do), so the per-part +LFS oids plus an exact total are the verification chain. If a size check fails, +suspect your own arithmetic before the payload: the sum of the parts is the +expected value, and it is easy to mis-add by hand (`python3 -c "print(a+b+c)"`). + +## 4. Housekeeping + +- Failure artifacts (error pages) and stray directories: move to + `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm`; list them and wait for approval to + purge. +- Keep the `.partNofM` fragments after assembly when disk allows: they are the + only copies that can be re-verified against the Hub. Note the total they cost + and ask before purging. +- Never delete the payload being replaced as a side effect of a swap: upstreams + delist quants (both mradermacher and orcarouter have delisted Q4_K_M for + Qwen3.8-Flash-Next-Uncensored), and the local file may be the last copy. + +## 5. Waiting for a long fetch + +Do not `pgrep -f