agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
Frank committed
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,71 @@
|
||||
---
|
||||
name: amd-multigpu-cooperation-triage
|
||||
description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives."
|
||||
---
|
||||
|
||||
# AMD Multi-GPU Cooperation Triage
|
||||
|
||||
When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix".
|
||||
|
||||
## 1. Check atomic-op routing on the whole path
|
||||
|
||||
RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*:
|
||||
|
||||
```cpp
|
||||
void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp
|
||||
link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
|
||||
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
|
||||
if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else, no override
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
# Endpoint capability + negotiated link, and EVERY upstream bridge
|
||||
for d in <gpu-btfs>; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done
|
||||
lspci -tv # is the card on a CPU root port or behind the chipset?
|
||||
```
|
||||
|
||||
- Endpoint must show `AtomicOpsCtl: ReqEn+`.
|
||||
- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe.
|
||||
|
||||
Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`.
|
||||
|
||||
## 2. Map the board before recommending anything
|
||||
|
||||
Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4.
|
||||
|
||||
## 3. Prove what the runtime actually needs (do it OFF the production host)
|
||||
|
||||
Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging:
|
||||
|
||||
```python
|
||||
torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT",
|
||||
world_size=2, rank=r, timeout=timedelta(seconds=25))
|
||||
# probe: broadcast, all_reduce, barrier, send/recv, all_gather
|
||||
```
|
||||
|
||||
- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall.
|
||||
- `world_size=1` always succeeds; one-card success proves nothing.
|
||||
- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/<stdin>'`).
|
||||
- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened.
|
||||
|
||||
## 4. Don't be fooled by "collective-free" claims
|
||||
|
||||
Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out.
|
||||
|
||||
## 5. The option set, ranked
|
||||
|
||||
| Option | Works when | Cost | Trade |
|
||||
|---|---|---|---|
|
||||
| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient |
|
||||
| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser |
|
||||
| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode |
|
||||
| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today |
|
||||
| Patch the custom-AR table | never | — | other collectives still die |
|
||||
| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`).
|
||||
- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show <ip>` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP.
|
||||
- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`).
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: container-config-drift-crashloop
|
||||
description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order."
|
||||
---
|
||||
|
||||
# A container that restarts forever while its definition is correct
|
||||
|
||||
## When to use
|
||||
|
||||
A container is crash-looping or perpetually "unhealthy", but the file that
|
||||
defines it (compose service, unit, deploy script) looks right. Also when a
|
||||
service has been quietly down for days and nobody noticed, because a restart
|
||||
policy kept something running.
|
||||
|
||||
The cause is usually **config drift**: the running container was created from an
|
||||
older definition, and the restart policy restarts it instead of recreating it.
|
||||
A compose/unit edit does not touch an existing container.
|
||||
|
||||
## Diagnose (5 minutes, in this order)
|
||||
|
||||
1. **Who started it?** The label names the owner, not your guesswork:
|
||||
|
||||
```
|
||||
podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
|
||||
```
|
||||
|
||||
`podman-compose@…` means compose owns it, whatever unit files exist.
|
||||
Compare `created` with when the definition last changed. A container older
|
||||
than the edit that should have replaced it is the bug.
|
||||
|
||||
2. **What is it ACTUALLY running?** Read the argv, not the summary fields:
|
||||
|
||||
```
|
||||
podman inspect <name> --format '{{.Config.CreateCommand}}'
|
||||
```
|
||||
|
||||
Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device`
|
||||
flags are present — a false negative that can send you hunting the wrong
|
||||
root cause. `CreateCommand` is the truth; grep it for the flag you care
|
||||
about (`--device`, `--group-add`, `-e`, mounts).
|
||||
|
||||
3. **What does the process say?** `podman logs <name> | grep -iE 'error|cannot|Runtime'`.
|
||||
A capability error (`No HIP GPUs are available`) alongside a missing device
|
||||
flag in step 2 confirms drift rather than a code or image fault. Count the
|
||||
damage: `podman inspect --format '{{.RestartCount}}'`.
|
||||
|
||||
## Fix
|
||||
|
||||
Quickest unblock: recreate from the definition (`podman compose up -d
|
||||
--force-recreate <svc>`), which is what should have happened automatically.
|
||||
|
||||
Durable fix, when the host is governed by a repo that requires boot-persistent
|
||||
units (this one does): migrate the service to a Quadlet. Order that worked:
|
||||
|
||||
1. build the image and `install` the unit file to `~/.config/containers/systemd/`
|
||||
2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a
|
||||
leftover `~/.config/systemd/user/<svc>.service` collides by name with the
|
||||
unit the Quadlet generator produces
|
||||
3. `podman rm -f <container>` — the stale container holds the name the unit's
|
||||
container needs, and it carries the old config
|
||||
4. `systemctl --user daemon-reload && systemctl --user restart <svc>.service`
|
||||
5. delete the compose service so only one owner remains per container name
|
||||
|
||||
## Verify the capability, not the status
|
||||
|
||||
`Up` and even `healthy` can be true of a half-configured container. Check the
|
||||
thing that was broken, from inside:
|
||||
|
||||
```
|
||||
podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
|
||||
ss -ltn | grep <port> # confirm the bind addresses, not the intent
|
||||
```
|
||||
|
||||
And confirm boot persistence the way the generator exposes it:
|
||||
`systemctl --user is-enabled <svc>.service` → `generated` is healthy; an
|
||||
explicit `enable` fails on generated units.
|
||||
|
||||
## Related reading in this repo
|
||||
|
||||
- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
|
||||
- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident
|
||||
- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: fleet-container-ownership-audit
|
||||
description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them."
|
||||
---
|
||||
|
||||
# Finding the containers that will not survive a reboot
|
||||
|
||||
Use when auditing a Podman fleet for boot persistence, when a service is
|
||||
"up" but stopped working at some past reboot, or when a compose-managed stack is
|
||||
being converted to Quadlet units. This is the *detection* half; the conversion
|
||||
checklist is separate.
|
||||
|
||||
## The one-line test
|
||||
|
||||
Every running container on a boot-persistent host should carry a
|
||||
`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and
|
||||
rootless Podman does not restart bare containers after the explicit stop that
|
||||
shutdown performs — so it will be missing tomorrow morning.
|
||||
|
||||
```bash
|
||||
for c in $(podman ps --format '{{.Names}}'); do
|
||||
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
|
||||
done
|
||||
```
|
||||
|
||||
Compare the result against the host's installed units
|
||||
(`ls ~/.config/containers/systemd/*.container`) and the repo's
|
||||
`<host>/<service>/quadlet/`. Three outcomes, three meanings:
|
||||
|
||||
- **label present** → the unit owns it; boot persistence comes from `[Install]
|
||||
WantedBy=default.target` plus `loginctl show-user <user> -p Linger` = yes.
|
||||
- **label absent, compose service exists** → the classic gap: it dies at reboot.
|
||||
Also check the publish address while you are there; compose forms often bind
|
||||
the wildcard, which puts an unauthenticated UI/API on every interface.
|
||||
- **label present AND a compose service with the same name** → a second owner.
|
||||
The runtime is fine, but a stray `podman compose up` can create a duplicate
|
||||
that fights the unit for the container name and the port. Remove the compose
|
||||
service and leave an explanatory comment; that is the repo's convention.
|
||||
|
||||
## Traps that make this hard to see
|
||||
|
||||
- **Editing a compose service does not replace a running container.**
|
||||
`restart: unless-stopped` will happily keep an old container — old image, old
|
||||
device list, old mounts — alive forever. One instance ran 9,146 restart
|
||||
attempts on stale config before anyone looked. If the container predates the
|
||||
edit, recreate it explicitly; do not debug the service.
|
||||
- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.**
|
||||
It can read empty on a container that demonstrably has the devices. Read the
|
||||
real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside:
|
||||
`podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available())'`.
|
||||
- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape
|
||||
that flips from `connection refused` to `unsupported Content-Type "text/html"`
|
||||
means the service came up and only the scrape is wrong. Read the error, not
|
||||
just up/down — and never "fix" a red probe by deleting the job, because an
|
||||
absent scrape and a healthy one look identical.
|
||||
- **A hand-written `~/.config/systemd/user/<svc>.service` that shells out to
|
||||
`podman compose up` collides by name with the Quadlet the generator produces
|
||||
for `<svc>.container`.** Retire it to a dated trash directory before
|
||||
installing the unit.
|
||||
- **`is-enabled` saying `generated` is the healthy state** for Quadlet units;
|
||||
`systemctl --user enable` fails on them.
|
||||
|
||||
## Migration shape that works
|
||||
|
||||
1. Unit file in the repo at `<host>/<service>/quadlet/<service>.container`:
|
||||
`Image=` with the `localhost/` prefix for locally built images, `Pull=never`,
|
||||
explicit `PublishPort` (loopback unless the service is the documented LAN
|
||||
exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed,
|
||||
`Restart=always`, `[Install] WantedBy=default.target`.
|
||||
2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator
|
||||
splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle
|
||||
that fails silently.
|
||||
3. `deploy_<svc>()` in the remote handler: install the unit, retire any
|
||||
compose-era unit into `~/.AGENTTRASH/`, `podman rm -f <container>` so the
|
||||
unit's own can take the name, daemon-reload, restart, wait for health.
|
||||
4. Delete the compose service (one owner per container name) and leave a comment
|
||||
naming the unit, so the next reader does not re-add it.
|
||||
5. Planner: add a path rule (`<host>/<dir>/**` -> service) **and** a
|
||||
`service_is_allowed` entry, or the shared dispatcher rejects the service
|
||||
before anything runs.
|
||||
6. Migrate the seams: tests that read the compose file for devices, command
|
||||
flags or groups must read the unit instead — and resolve file paths from the
|
||||
tooling's own declarations, never from `find ... | head -1`.
|
||||
|
||||
## When the last service leaves a compose file
|
||||
|
||||
The repo's compose-validity seam requires at least one service, so a file whose
|
||||
services have all migrated fails its own test. Either keep the file as
|
||||
comments and relax that seam deliberately, or delete the file and fix the
|
||||
planner's compose-diff path — decide with the owner, because the deploy tooling
|
||||
reads it.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
name: fleet-llm-payload-purge-refresh
|
||||
description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap."
|
||||
---
|
||||
|
||||
# Purge and refresh LLM model payloads (Dank fleet)
|
||||
|
||||
Use when asked to clear / reclaim / refresh model caches on a host that is
|
||||
*serving* users, or to prove that a payload is up to date.
|
||||
|
||||
## 1. Inventory before touching anything
|
||||
|
||||
```bash
|
||||
du -sh /home/dank/models/* | sort -h # what is actually there
|
||||
```
|
||||
|
||||
Then find which of those the live services mount — never delete a payload a unit
|
||||
mounts without an explicit, in-this-conversation instruction:
|
||||
|
||||
```bash
|
||||
grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \
|
||||
/home/dank/.config/containers/systemd/*.container \
|
||||
/etc/systemd/system/frank-*.service
|
||||
```
|
||||
|
||||
On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn
|
||||
checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF +
|
||||
mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not**
|
||||
`models/checkpoints` — verify, because `models/checkpoints` looks like cache but
|
||||
has held Flux weights whose upstream is licence-gated.
|
||||
|
||||
## 2. Confirm each payload is re-fetchable BEFORE deleting it
|
||||
|
||||
Query the Hub anonymously; a gated repo 403s even with the host token:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
|
||||
```
|
||||
|
||||
An error/empty result means it is NOT safe to delete. Also filter by **quant
|
||||
name**, never by `endswith('.gguf')` — mradermacher ships large quants as
|
||||
`.partNofM` fragments which that test hides.
|
||||
|
||||
## 3. Fingerprint, then delete
|
||||
|
||||
Record byte sizes (and sha256 when the file matters) **before** removal: that is
|
||||
the only way to answer "did the refresh actually change anything" afterwards.
|
||||
|
||||
Per this repo's policy `rm` needs explicit operator permission; otherwise stage
|
||||
into `~/.AGENTTRASH/<date>-<topic>/`. Note that a same-filesystem trash reclaims
|
||||
nothing — say so rather than implying space was freed. Root-owned leftovers (a
|
||||
trashed swapfile) need `sudo`, and check `swapon --show` first.
|
||||
|
||||
## 4. Refresh, per service
|
||||
|
||||
**frank-large (halogen)** — the engine fetches for you. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit:
|
||||
|
||||
```bash
|
||||
rm -rf /home/dank/models/halogen/* # keep the directory itself
|
||||
systemctl --user start frank-large.service
|
||||
journalctl --user -u frank-large.service -f | grep -a halogen:
|
||||
```
|
||||
|
||||
The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD`
|
||||
(`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`,
|
||||
resuming if interrupted. **Trap:** podman refuses a bind mount whose source
|
||||
directory is missing — `Error: statfs /home/dank/models/halogen: no such file or
|
||||
directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty
|
||||
directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load.
|
||||
|
||||
**frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start
|
||||
with weights missing, so pull them yourself and verify:
|
||||
|
||||
```bash
|
||||
DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
|
||||
# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf
|
||||
curl -fL -C - --retry 10 -o "$DEST/<file>" \
|
||||
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
|
||||
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to the LFS oid + GGUF magic
|
||||
```
|
||||
|
||||
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always
|
||||
`-f`, always compare byte counts to the Hub's declared size, always check the LFS
|
||||
sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from
|
||||
argv entirely (it is world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
## 5. Verify, then record
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # frank-large
|
||||
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health # frank-small
|
||||
curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models
|
||||
# a REAL completion on both, then halogen's boot ledger:
|
||||
journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything"
|
||||
```
|
||||
|
||||
Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`,
|
||||
`gitea`, `loki` — `curl -sk --resolve <host>:443:192.168.0.143`) because a purge
|
||||
touches state the whole host shares.
|
||||
|
||||
## 6. Things that bite
|
||||
|
||||
- **Sidecars are versioned in place.** Equal file sizes are not proof the repo is
|
||||
unchanged, and a stale local sidecar is invisible from its name. Compare local
|
||||
sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB
|
||||
overlay update was caught.
|
||||
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The
|
||||
A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding
|
||||
cache churn there risks the OOM/state failures this fleet has already seen.
|
||||
- **Reboot is the repair of last resort** if rootless podman's pause process dies
|
||||
during the work (`invalid internal status … could not find any running
|
||||
process`); `podman system migrate` panics in 5.7.0.
|
||||
- Because a peer session may be committing to `serverAppConfig` concurrently,
|
||||
re-read a doc region immediately before editing it, and check that the edit
|
||||
response's neighbouring lines are the ones you expect.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
name: fleet-model-cache-purge-refresh
|
||||
description: "Clear and re-fetch model payloads on a Dank-fleet host that serves production LLM endpoints — inventory, protect live payloads, verify every fetch, restore the service, and use the refresh to detect upstream updates."
|
||||
---
|
||||
|
||||
Clearing a model cache on a serving host is a two-sided operation: the operator wants no stale
|
||||
payload, and the host must keep serving. Do it in this order, on the target host.
|
||||
|
||||
## 1. Inventory before touching anything
|
||||
|
||||
```bash
|
||||
du -sh /home/dank/models/* | sort -h # payload dirs, biggest last
|
||||
grep -rlE "models/|<model-name>" /home/dank/.config/containers/systemd/ /etc/systemd/system/*.service
|
||||
```
|
||||
|
||||
Classify every dir: **live service payload** (referenced by a unit/env), **retired runtime
|
||||
payload**, **unreferenced model dir**, **tool/script dir**, **logs** (evidence — keep). Check the
|
||||
service env files (`scripts/*.env`) for `MODEL_DIR`/`MODEL` paths, not just unit `Volume=` lines:
|
||||
a payload can be mounted by path and named by env. `models/logs` and small tool dirs are not cache.
|
||||
|
||||
## 2. Record fingerprints first
|
||||
|
||||
`ls -l --block-size=1 <dir>` gives sizes; for the LLM payloads compare each to the Hub's declared
|
||||
size (`/api/models/<repo>/tree/main?recursive=true&blobs=true`). They can differ — see §6. If a
|
||||
file is irreplaceable upstream (delisted revision, gated repo) say so before deleting it.
|
||||
|
||||
## 3. Protect live payloads by delete-then-refetch, never delete-only
|
||||
|
||||
Deleting a live payload leaves the endpoint unable to start. Know the recovery path *before*
|
||||
deleting: some engines fetch their own weights (halogen: `HALOGEN_DOWNLOAD`, triggered when the
|
||||
checkpoint is absent), others need a manual fetch (llama.cpp GGUF services: the serve script
|
||||
refuses to start without the weights).
|
||||
|
||||
**Bind-mount trap:** podman refuses a mount whose source directory is missing —
|
||||
`Error: statfs /home/dank/models/<dir>: no such file or directory`, with the unit looping on
|
||||
`Restart=on-failure`. Recreate the directory **empty** (`mkdir -p`), and the engine's first-start
|
||||
fetch runs.
|
||||
|
||||
## 4. Fetch with verification (never trust an exit code)
|
||||
|
||||
`curl -o FILE URL` exits 0 on 401/403 and writes the error body — a gated repo produced three
|
||||
153-byte shards with identical sha256s and a cheerful DONE line. Always: `-fL -C -`, compare the
|
||||
byte count against the Hub's declared size, and compare `sha256sum` against the LFS oid. Public
|
||||
repos means **no token in argv** (it is world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
**Parts trap:** mradermacher-style repos ship large quants as `.part1ofN … .partNofN` — a filter on
|
||||
`endswith('.gguf')` makes those quants look absent. Fetch the parts, verify each, then
|
||||
`cat part1 part2 part3 > name.gguf` in numeric order; the assembled file's size equals the sum
|
||||
(concatenation is deterministic, so per-part hashes verify the whole).
|
||||
|
||||
## 5. Restore, then verify behaviour — not health alone
|
||||
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), `reset-failed`, `start`;
|
||||
root units need `sudo`. Watch the engine's own fetch/load lines in the journal, then check
|
||||
`/health`, `/v1/models`, and a **real completion** on every restored endpoint. Read the engine's
|
||||
memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits.
|
||||
|
||||
## 6. A refresh is also an update check
|
||||
|
||||
Clearing and re-fetching only guarantees freshness *at fetch time* — if upstream has not changed,
|
||||
the bytes come back identical, which is itself the finding. Compare recorded fingerprints against
|
||||
the new local sizes and the Hub's declared sizes: sidecars are often versioned **in place**, so a
|
||||
stale copy is invisible from its file name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560
|
||||
while the main checkpoint returned byte-identical).
|
||||
|
||||
## 7. Close out
|
||||
|
||||
- Clear test payloads once the benchmark using them is done, and keep the payload the *next*
|
||||
experiment needs.
|
||||
- Leave the operator's decision points explicit: unreferenced-but-licensed dirs, other services'
|
||||
caches, and whether they want the payloads absent at their next run or fresh as of now.
|
||||
- Record sizes/verdicts in the vault (runbook + `log.md`); never `rm` without explicit permission,
|
||||
and remember a same-filesystem trash reclaims no space.
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: fleet-monitoring-gap-audit
|
||||
description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses."
|
||||
---
|
||||
|
||||
# Auditing a homelab fleet's monitoring for what *should* be there
|
||||
|
||||
Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose
|
||||
services are declared in a git repo. The two failure modes this catches are
|
||||
different, and only the first announces itself.
|
||||
|
||||
## 1. A service that should exist and doesn't is invisible
|
||||
|
||||
A missing exporter shows up as **nothing**: no failing target if the job was
|
||||
configured for a host that no longer serves, no alert because the series is
|
||||
simply absent. So check existence, not health:
|
||||
|
||||
```bash
|
||||
# Every container the host actually runs, including dead ones
|
||||
podman ps -a --format '{{.Names}} {{.Status}}'
|
||||
# Every container that is supposed to be boot-persistent
|
||||
ls ~/.config/containers/systemd/*.container
|
||||
systemctl --user is-enabled <name>.service # expect: generated
|
||||
```
|
||||
|
||||
A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic
|
||||
case: rootless Podman does not restart bare containers after the stop that
|
||||
shutdown performs, so a compose-only service dies at the first reboot and stays
|
||||
dead. Check the compose file for the name — if it is there and has no unit, the
|
||||
repo's own rule is being violated. Fix by writing the unit (mirror the same
|
||||
service on the other host), deleting the compose entry (one owner per container
|
||||
name), and making sure the deploy path can reach it:
|
||||
|
||||
- planner: service in the target's service set + a path rule for its directory
|
||||
- dispatcher allowlist: `<target>:<service>`
|
||||
- handler: install from `"$TARGET/..."`, never a hardcoded peer's path
|
||||
|
||||
## 2. Classify every down target — do not report a count
|
||||
|
||||
`targets?state=active` gives health **and** the last error. Two down targets can
|
||||
mean opposite things:
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) |
|
||||
| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine |
|
||||
| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter |
|
||||
| target absent entirely | see §1 |
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
if t['health'] != 'up':
|
||||
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
||||
# and probe each by hand before believing the scraper
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics
|
||||
```
|
||||
|
||||
Say which are down and why. "7 of 11 up" without the four reasons is how a real
|
||||
outage hides behind three config defects — that is exactly what one fleet did:
|
||||
its crash-looping GPU service was one of four failures reported as a number.
|
||||
|
||||
## 3. Check the exposure posture while you are in there
|
||||
|
||||
An exporter that binds the wildcard publishes an unauthenticated API on every
|
||||
interface. Expect exactly two listeners per host-exporter — loopback for
|
||||
host-local probes, the host's own LAN address for the Prometheus container:
|
||||
|
||||
```bash
|
||||
ss -ltn | grep <port> # must show the two explicit addresses, never 0.0.0.0
|
||||
```
|
||||
|
||||
## 4. Leave the decisions that are not yours
|
||||
|
||||
Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**:
|
||||
disable a probe to make a dashboard green, or "fix" a crash-looping service whose
|
||||
correct posture depends on what the owner wants — capture the evidence (restart
|
||||
count, exit code, last error line) and hand it over intact.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: fleet-second-llm-endpoint
|
||||
description: "Add a second LLM endpoint to a host that already serves one on the Dank fleet (new port, unit shape, co-tenancy budget, harness provider/roles, monitoring, cutover, verification) — with the port-bind race, the reasoning_effort template trap, and the tools/--jinja trap that silently break subagents."
|
||||
---
|
||||
|
||||
# Adding a second LLM endpoint to a Dank fleet host
|
||||
|
||||
Use when a host must serve two models at once (e.g. an iGPU runtime plus a
|
||||
second card), or when one endpoint is moving and a new one takes the freed
|
||||
hardware. Rebuilt from the 2026-09-16 Dankclaw split (halogen `frank-large` on
|
||||
:7770 + llama.cpp `frank-small` on :7771).
|
||||
|
||||
## 0. Decide the shape before touching hardware
|
||||
|
||||
- **One provider per HOST:PORT** in `~/.omp/agent/models.yml` (on the box where
|
||||
the harness runs, not on the servers). A second endpoint is a new provider
|
||||
block, not a second model in the existing one.
|
||||
- **Read the budget as an inequality, not a wish.** For a unified-memory runtime
|
||||
the engine's own fit check is the arbiter (`budget = host − pinned weights −
|
||||
reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV by
|
||||
counting only the full-attention layers — and remember `--parallel N` DIVIDES
|
||||
`--ctx-size`, it does not multiply.
|
||||
- **A co-tenant is a budget line, not a footnote.** Free host RAM by changing
|
||||
the POOL or the ARENA (the reserve is a fit-check term, never an allocation),
|
||||
and drop optional charges (a vision tower is ~2 GiB) rather than hoping.
|
||||
|
||||
## 1. Repo first, and retire in the same commit
|
||||
|
||||
1. New service files: `scripts/<svc>.env` (documented tunables), `.service`
|
||||
(root unit for a native binary; a Quadlet only for a container),
|
||||
`serve-<svc>.sh` (defaults + hard guards for missing artifacts).
|
||||
2. Planner: add the service to `SERVICE_ORDER` AND the target's service set in
|
||||
`scripts/deploy-plan.py`, map its paths (including "the unit/env/serve trio
|
||||
selects only me"), add it to the inventories in `deploy.sh` and
|
||||
`deploy-remote.sh`, and the dispatch case.
|
||||
3. Retire what the cutover obsoletes into `.AGENTTRASH/` with a suffix — and
|
||||
**check what pointed at it**: bench harnesses, a chat template only the old
|
||||
runtime read, and `scripts/test-changed.sh` mappings are the usual victims
|
||||
(a mapping to a test file retired earlier fails the pre-commit wrapper
|
||||
silently for weeks).
|
||||
4. Seams: one bats file per service asserting the *invariants* (`pool == 2 ×
|
||||
ctx`, KV bytes ≤ card budget, port separation), not just constant values.
|
||||
A negative grep must strip comments (`grep -vE '^\s*#' file | grep -q ...`)
|
||||
or it will match the prose that explains the very thing you removed.
|
||||
|
||||
## 2. The port-bind race (rootless podman)
|
||||
|
||||
pasta binds the published port at container CREATE. A predecessor that has just
|
||||
stopped still holds it, so a naive handler starts a doomed container:
|
||||
|
||||
```
|
||||
Deactivated successfully <- predecessor gone
|
||||
pasta: Listen failed for HOST TCP port */PORT: Address already in use
|
||||
exit 126 <- only Restart=on-failure saves you
|
||||
```
|
||||
|
||||
Order the handler: **retire the old manager's unit → stop THIS service's own
|
||||
unit (no-op on a first cutover) → wait for every socket on the port to clear
|
||||
(listener, accepted, TIME_WAIT) → install → start.** Without the second step a
|
||||
redeploy deadlocks against its own gate.
|
||||
|
||||
## 3. Wire the surfaces, and refuse impossible signals
|
||||
|
||||
- Harness roles: point the small/fast + image roles at the new endpoint, and
|
||||
move the vision role with the vision tower (only one endpoint should own
|
||||
each capability).
|
||||
- `shared/homepage/services.json`: a card per endpoint, probed from the
|
||||
monitor's own host (loopback is legal there; the LAN address otherwise).
|
||||
- Prometheus: only add a job if the runtime actually exposes Prometheus text.
|
||||
Halogen publishes `/health`, `/cache` and `/v1/*` and nothing else — scraping
|
||||
it would be red by construction, so the job is REMOVED with the reason written
|
||||
down rather than left as a permanent red.
|
||||
|
||||
## 4. Cutover, then verify by driving it
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh <target> --service <large> --service <small> # order matters
|
||||
```
|
||||
|
||||
Then, per endpoint: `/health`; `/v1/models` reports the alias; a chat completion
|
||||
returns content AND `reasoning_content`; **a request carrying `tools` returns
|
||||
`tool_calls`**; an image request is answered (or refused with a named reason);
|
||||
the runtime's own speculation counters move; and the unit is `enabled` /
|
||||
`generated` with linger on. Read the engine's boot ledger for a silent clamp.
|
||||
|
||||
Finally spend one real request through the harness per changed role — routing
|
||||
labels lie, the servers' own journals do not: `journalctl -u <svc>` shows which
|
||||
endpoint actually served it.
|
||||
|
||||
## Traps that look like model faults
|
||||
|
||||
| symptom | cause |
|
||||
|---|---|
|
||||
| `500 tools param requires --jinja flag` | the endpoint's runtime lacks the Jinja chat-template path, so every tool-using agent on it is broken. mainline llama.cpp defaults it on — pin it |
|
||||
| `500 Jinja Exception: Unexpected reasoning effort <level>` | the model's own template has a narrower accepted set (Qwen3.8: xhigh/medium/low only) than the harness's levels; fold them in `compat.reasoningEffortMap` |
|
||||
| the same request works by hand but a subagent dies | subagents run at a different thinking level — check both |
|
||||
| the container dies instantly although the predecessor "stopped" | the port race above |
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: fleet-service-precommit-iteration
|
||||
description: "Iterate on a service that lives in the serverAppConfigs git checkout on a fleet host (dankclaw/dankserver) before committing: ship files for live testing, keep the checkout pullable, then commit and deploy — covers the untracked-files pull trap that recurs every session."
|
||||
---
|
||||
|
||||
# Iterating on a fleet service before you commit
|
||||
|
||||
This repo's contract is git → `deploy.sh` → host. But live testing needs the new
|
||||
file *on the host* before it is committed, and the naive way (copy it into the
|
||||
host checkout) blocks the very deploy that would install it. This is the loop
|
||||
that works, plus the traps that cost real time.
|
||||
|
||||
## The loop
|
||||
|
||||
1. Edit in the local repo. Run the seams: `./tests/run.sh`.
|
||||
2. Ship the working files to the host **for testing**:
|
||||
`cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
|
||||
(then `chmod +x` if it is executed — scp/`cat >` drops the exec bit, and a
|
||||
container entrypoint mounted from that path fails with status 126, "command
|
||||
found but not executable").
|
||||
3. Test against the live service; iterate until it works.
|
||||
4. **Before committing**, prove the host copy is byte-identical to what you are
|
||||
about to commit: `sha256sum` locally and on the host. If they differ, you are
|
||||
about to lose work in step 5.
|
||||
5. Commit and push.
|
||||
6. Clean the host checkout so the deploy can pull:
|
||||
- untracked hand copies → move them out of the checkout
|
||||
(`mkdir -p /tmp/handcopy && mv <paths> /tmp/handcopy/`);
|
||||
- modified tracked files → `git checkout -- <paths>` (this *discards* the
|
||||
hand copy, which is safe only because of step 4);
|
||||
- confirm with `git status --short` printing nothing.
|
||||
7. `./scripts/deploy.sh <target> --service <svc>` — it pushes, pulls ff-only,
|
||||
installs units/config, restarts, waits for health, then runs the Playwright
|
||||
e2e gate.
|
||||
|
||||
## Traps, in the order they bite
|
||||
|
||||
- **The pull aborts on hand copies.** `deploy.sh` pulls with ff-only and git
|
||||
refuses: *"The following untracked working tree files would be overwritten by
|
||||
merge"*. Any untracked file in the checkout whose path the pull wants to
|
||||
create stops the deploy — which is exactly the file you just shipped. Always
|
||||
do step 6.
|
||||
- **`git checkout --` silently eats uncommitted work.** Without step 4 you can
|
||||
destroy the only copy of an edit that exists nowhere else.
|
||||
- **A runtime mounts files *straight out of the checkout*.** Here
|
||||
`dankserver/frank-medium/**` and `shared/grafana/provisioning/**` are mounted
|
||||
read-only into containers, so a hand copy takes effect at the next container
|
||||
restart with no deploy. Test that way, then let the deploy make it official.
|
||||
- **Root-owned files from a container write** (e.g. a patched `.so` produced
|
||||
inside a build container) can land in the checkout as `root:root`; `chown` them
|
||||
back to the deploy user before git operations.
|
||||
- **A restart is not a verification.** Health endpoints answer from a stale
|
||||
process, a proxied service can be served by a peer, and a provisioned
|
||||
dashboard may live in unified storage rather than the SQLite table you grep.
|
||||
Verify the *specific* claim: the new code path's own output, the file's
|
||||
checksum on the host, the API that serves the resource.
|
||||
|
||||
## Why not keep a scratch copy on the host?
|
||||
|
||||
Because the deployed artifact must be the committed one. Iterating outside git
|
||||
(the `/tmp` loop is fine for *probes*) is for measurement scripts; anything the
|
||||
service actually runs has to land in the checkout, and the checkout has to stay
|
||||
pullable. Two directories, one rule: probes may live anywhere, runtime state
|
||||
lives in git.
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: frank-medium-batch-split-retune
|
||||
description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify."
|
||||
---
|
||||
|
||||
# frank-medium batch geometry + split rebalance
|
||||
|
||||
Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the
|
||||
model on frank-medium, or when someone reports that prefill "does not respond" to
|
||||
those knobs. The batch and the tensor split are **coupled**: the compute buffers are
|
||||
not symmetric across the two cards, so any batch change moves the balance point.
|
||||
|
||||
## Order of work (do not skip the second half)
|
||||
|
||||
1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold
|
||||
a shape constant across dozens of configs:
|
||||
`scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation
|
||||
per cell, `-r 3 --no-warmup`.
|
||||
2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by
|
||||
loading the real service with a per-config env override and reading `rocm-smi`. A
|
||||
llama-bench number cannot decide the split: it has no `--parallel`, sizes its own
|
||||
context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
|
||||
3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed
|
||||
point, then **deploy** and verify.
|
||||
|
||||
## 1. The sweep
|
||||
|
||||
```bash
|
||||
# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is
|
||||
# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here
|
||||
sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
--setenv=OUT=/home/dank/models/logs/bench-bub \
|
||||
--setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \
|
||||
/home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv`
|
||||
(appended as it goes, so progress is readable), samples VRAM per cell, and restarts
|
||||
the service from an EXIT trap. A re-run skips cells that already have a `bench.json`.
|
||||
|
||||
Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not
|
||||
track the request, nothing else in the table means anything. The VRAM columns scaling
|
||||
monotonically with ubatch is the independent confirmation.
|
||||
|
||||
Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth):
|
||||
an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046,
|
||||
384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral
|
||||
(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every
|
||||
cell). Test **both** ends of the range: sampling only upward makes a shallow basin look
|
||||
like a plateau.
|
||||
|
||||
Sanity rule: the same ubatch measured under different logical batches should agree to
|
||||
under 1 tok/s. If it does not, the table is noise and the first suspect is a stray
|
||||
GPU user, not the knob.
|
||||
|
||||
## 2. The split, at the deployed shape
|
||||
|
||||
`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env`
|
||||
to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with
|
||||
`FRANK_MEDIUM_ENV_FILE=<temp>`, polls `/health` for `"status":"ok"`, reads
|
||||
`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under
|
||||
`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`.
|
||||
|
||||
**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH
|
||||
cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the
|
||||
process dies — a 180 s sleep fails, and even 600 s has failed while the process is still
|
||||
alive inside the unit. Skipping the gate produces
|
||||
`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from
|
||||
"this split does not fit" and will send you chasing a phantom OOM.
|
||||
|
||||
Free bytes to expect, and the arithmetic behind them:
|
||||
`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs
|
||||
the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB
|
||||
against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one
|
||||
layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the
|
||||
reachable minimum gap is about one DeltaNet layer, never zero.
|
||||
|
||||
2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32;
|
||||
**35,30 @512 -> 6.44 / 6.90 (deployed)**.
|
||||
|
||||
## 3. Seam, deploy, verify
|
||||
|
||||
- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured
|
||||
1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the
|
||||
model reproduces the new point. Check that neighbouring splits still FAIL the balance
|
||||
gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the
|
||||
slack at one DeltaNet layer.
|
||||
- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares
|
||||
summing to the model's 65 blocks, ROCm1 the smaller).
|
||||
- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`.
|
||||
- Verify live: the flags in `/proc/<pid>/cmdline`, `/slots` = 2 x 131072, VRAM matching
|
||||
the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance`
|
||||
lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB
|
||||
lower once the service has served traffic than straight after load.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: grafana-prometheus-provisioning-as-code
|
||||
description: "Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives."
|
||||
---
|
||||
|
||||
# Observability as code (Grafana + Prometheus, Dankclaw fleet)
|
||||
|
||||
Use when adding a scrape target, an alert rule, a datasource, or a dashboard to
|
||||
this homelab. Everything lives in git; Grafana and Prometheus only ever read
|
||||
what the deployer installs.
|
||||
|
||||
## The triad — every change touches three places
|
||||
|
||||
1. **Config in the repo**
|
||||
- `shared/prometheus/prometheus.yml` — `scrape_configs`, and `rule_files:`
|
||||
pointing at `/etc/prometheus/rules/*.yml`
|
||||
- `shared/prometheus/rules/<name>.yml` — alert/recording rules
|
||||
- `shared/grafana/provisioning/datasources/prometheus.yml` — datasource
|
||||
- `shared/grafana/provisioning/dashboards/{provider.yml,<name>.json}`
|
||||
2. **The copier in `scripts/deploy-remote.sh`** — `copy_prometheus_config`
|
||||
installs config **and** rules into `/opt/prometheus/config/` (the container
|
||||
mounts that to `/etc/prometheus`); `copy_grafana_provisioning` installs into
|
||||
`/opt/grafana/provisioning/` (the grafana unit already mounts it read-only).
|
||||
Paths are absolute and `run_privileged`.
|
||||
3. **The planner in `scripts/deploy-plan.py`** — a `shared/<group>/**` change
|
||||
must select the service that reads it (`shared/prometheus` → prometheus,
|
||||
`shared/grafana` → grafana). Missing mapping = the file deploys nowhere and
|
||||
nothing reports an error.
|
||||
|
||||
Add a seam in `tests/observability_quadlet.bats` for each change: the file
|
||||
exists, the config references it, the deployer copies it, and — where two files
|
||||
must agree (target vs published port) — that they do.
|
||||
|
||||
## Reach targets by container name, never the loopback publish
|
||||
|
||||
Datasources use `http://prometheus:9090` and scrape jobs for co-located
|
||||
services use `prometheus:9090` / `loki:3100`. Published ports are
|
||||
`127.0.0.1`-only by design (see the exposure rules), so a browser-facing URL or
|
||||
a published port in a datasource is wrong from inside the network. LLM
|
||||
endpoints are the exception: they are on other hosts, so use their LAN address.
|
||||
|
||||
## Three failure modes that produce no error message
|
||||
|
||||
- **A rule file nothing loads evaluates nothing.** Prometheus reports no
|
||||
problem for a `rule_files` entry that matches nothing, or for a rules file
|
||||
nobody references. Ship both together and assert the reference in a seam.
|
||||
- **A provisioned dashboard is not in the legacy `dashboard` table** on the
|
||||
current Grafana. Reading `grafana.db` for it returns empty and looks like a
|
||||
failed provision. It lives in unified storage behind
|
||||
`dashboard.grafana.app`; query the API instead.
|
||||
- **Lifetime averages are not per-request evidence.** e.g. an engine's
|
||||
"Prefix cache hit rate: 29 %" is every prompt it ever saw, including
|
||||
benchmarks. Measure the behaviour you care about with a targeted probe.
|
||||
|
||||
## Verify through the channel the consumer uses
|
||||
|
||||
```bash
|
||||
# Prometheus: config and rules parse/load
|
||||
ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/<name>.yml'
|
||||
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \
|
||||
| python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"'
|
||||
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...' # every target health=up
|
||||
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...' # 0 firing when healthy
|
||||
|
||||
# Grafana: provisioned objects and a live query (admin/admin on this install)
|
||||
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"'
|
||||
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/<uid>"'
|
||||
ssh dankclaw 'curl -s -u admin:admin \
|
||||
"http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"'
|
||||
```
|
||||
|
||||
A target that is `up` proves reachability, not data: also run one query for a
|
||||
series you expect and confirm it returns a sample.
|
||||
|
||||
## Grafana facts for this install
|
||||
|
||||
- No UI-configured state: `editable: false` on the datasource, fixed
|
||||
`uid: prometheus` so dashboards reference it without a lookup.
|
||||
- Dashboards re-read from disk every 30 s (`updateIntervalSeconds`), so a JSON
|
||||
edit appears without a restart — but the file must still be *installed*, which
|
||||
means a deploy.
|
||||
- There is **no Alertmanager and no contact point**: rules evaluate, alerts show
|
||||
on `/alerts`, in the `ALERTS` metric and on a dashboard panel, and reach
|
||||
nobody. Say that plainly in docs rather than implying observability.
|
||||
|
||||
## Naming metrics
|
||||
|
||||
Never write a series name from memory into a config, a rule, or a dashboard —
|
||||
one shipped comment here named a metric that does not exist. Dump the live
|
||||
names first and paste them:
|
||||
|
||||
```bash
|
||||
ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u'
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: grafana-provisioning-as-code
|
||||
description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative."
|
||||
---
|
||||
|
||||
# Grafana provisioning as code (Dankclaw fleet)
|
||||
|
||||
Use when adding or changing a Grafana datasource or dashboard on this fleet, or
|
||||
when a dashboard "does not appear" after a deploy.
|
||||
|
||||
## Shape of the thing
|
||||
|
||||
Grafana on Dankclaw is an **empty install configured entirely from git**. The
|
||||
quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a
|
||||
provisioned artefact exists in exactly three places:
|
||||
|
||||
```
|
||||
shared/grafana/provisioning/datasources/<name>.yml # repo: source of truth
|
||||
shared/grafana/provisioning/dashboards/provider.yml # dashboard loader config
|
||||
shared/grafana/provisioning/dashboards/<name>.json # dashboards
|
||||
| deploy-remote.sh: copy_grafana_provisioning()
|
||||
v
|
||||
/opt/grafana/provisioning/... # host: installed state
|
||||
| read-only mount
|
||||
v
|
||||
/etc/grafana/provisioning/... # generated: inside container
|
||||
```
|
||||
|
||||
Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be
|
||||
invisible to review, unknown to anyone reading the repo, and lost with the data
|
||||
volume.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Write the files under `shared/grafana/provisioning/`.
|
||||
- Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by
|
||||
uid, so a random one breaks them), `access: proxy`, and the URL by
|
||||
**container name** (`http://prometheus:9090`) — the published port is
|
||||
loopback-only and unreachable from inside the network.
|
||||
- Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab;
|
||||
failing an operator who wants a different rate window is the wrong trade)
|
||||
and a JSON dashboard under a stable `uid`.
|
||||
2. Add the file(s) to `copy_grafana_provisioning()` in
|
||||
`scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each
|
||||
file — follow `copy_prometheus_config` for style).
|
||||
3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify
|
||||
with `python3 scripts/deploy-plan.py --target dankclaw --paths <file>`.
|
||||
4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse,
|
||||
the datasource URL must be the container name, the unit must mount
|
||||
`/opt/grafana/provisioning`, and the deployer must call the copier. Then
|
||||
`./tests/run.sh`.
|
||||
5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the
|
||||
provisioning tree, restarts Grafana, then runs the e2e gate).
|
||||
|
||||
## Verify — and the trap that makes the obvious check lie
|
||||
|
||||
A **provisioned dashboard does not appear in the legacy `dashboard` table** of
|
||||
`grafana.db` on the current Grafana build; dashboards live in unified storage
|
||||
behind the `dashboard.grafana.app` API group. The datasource *does* land in
|
||||
`data_source`, and that asymmetry is easy to misread as "datasource worked,
|
||||
dashboard failed". A `sqlite3`/python query against `dashboard` is a **false
|
||||
negative**, not evidence.
|
||||
|
||||
Check in this order instead:
|
||||
|
||||
```bash
|
||||
# 1. provisioning actually ran (these loggers name themselves)
|
||||
podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"
|
||||
|
||||
# 2. the dashboard exists (uid is what you set in the JSON)
|
||||
curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"
|
||||
|
||||
# 3. the datasource really reaches Prometheus (proves the whole chain)
|
||||
curl -s -u admin:admin \
|
||||
"http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"
|
||||
```
|
||||
|
||||
Grafana publishes on loopback `:3002` only; the off-host path is the guarded
|
||||
vhost. Its admin login is still the default `admin:admin` — which is why every
|
||||
assertion above needs credentials, and which is worth changing.
|
||||
|
||||
## Before writing any panel or config comment
|
||||
|
||||
Grep the metric names out of the live endpoint; naming a series that does not
|
||||
exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the
|
||||
real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`,
|
||||
whose *rates* give the hit rate). A comment naming an unqueryable metric, or a
|
||||
panel that silently renders "No data", is worse than no panel:
|
||||
|
||||
```bash
|
||||
curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm
|
||||
```
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: heterogeneous-accelerator-split-triage
|
||||
description: "Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed."
|
||||
---
|
||||
|
||||
# Heterogeneous accelerator split — triage before building
|
||||
|
||||
Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm.
|
||||
|
||||
## 1. Characterise the link (bytes vs latency)
|
||||
|
||||
```bash
|
||||
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:" # capability vs actual
|
||||
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
|
||||
```
|
||||
|
||||
Expect **idle-downshift**: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number.
|
||||
|
||||
Compute the per-hop cost against the hidden state (`n_embd x 2 B` for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then:
|
||||
|
||||
- bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. **Bytes are almost never the constraint.**
|
||||
- the real budget line is **hop COUNT**: every device boundary is a host-side sync. A runtime that splits per *op* can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure `graph splits` in the loader log, do not assume.
|
||||
|
||||
**Design rule: partition by where the weights live, never by where the activations flow.** Resident weights, activations crossing.
|
||||
|
||||
## 2. Get the workload's real bottleneck (KV is usually it)
|
||||
|
||||
KV per position x context = bytes read per token, because attention reads the whole cached context every token:
|
||||
|
||||
```
|
||||
KV/token = kv_bytes_per_position * context
|
||||
```
|
||||
|
||||
Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, **KV dominates**. Then:
|
||||
|
||||
- the highest-leverage placement is **KV cache on the fastest-memory device, together with its attention compute** — never KV alone (separating them puts `KV/token` over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token);
|
||||
- check the KV + attention weights actually fit: `kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM`.
|
||||
|
||||
## 3. What placement never buys
|
||||
|
||||
Prefill is compute-bound and expert weights are read **once** per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave **decode +12 %, prefill 192 -> 32 tok/s**.
|
||||
|
||||
A `1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining).
|
||||
|
||||
## 4. Is the second device even a separate pool?
|
||||
|
||||
On a UMA APU, check before assuming additivity:
|
||||
|
||||
```bash
|
||||
cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small)
|
||||
cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
|
||||
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
|
||||
```
|
||||
|
||||
A large GTT window means the "GPU memory" is the *same* LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
|
||||
|
||||
## 5. Speculative decoding as the "cheap" split
|
||||
|
||||
A drafter on the *other* device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it:
|
||||
|
||||
- the drafter needs the target's **exact tokenizer** for a plain draft model (verify by comparing GGUF `tokenizer.ggml.tokens`, not by trusting the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models);
|
||||
- a plain drafter costs **K sequential forwards per round**, so it must be 10-50x faster than the target. Compute the break-even: with `r = t_draft/t_target`, `rate_ratio = A/(K*r + 1)`, `A = (1-p^(K+1))/(1-p)`. At `r ≈ 0.4-0.5` (a drafter only ~2x faster) break-even needs `p ≈ 0.7+` at K=3;
|
||||
- always compare against the **incumbent**, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar;
|
||||
- check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (`DRAFTER_CAP`-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd-ctvd`.
|
||||
|
||||
## 6. Expressing a split (llama.cpp, the usual vehicle)
|
||||
|
||||
```bash
|
||||
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest)
|
||||
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (what KV co-location needs)
|
||||
```
|
||||
|
||||
- buffer types are matched by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values;
|
||||
- `-cmoe` / `-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
|
||||
- several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and `GPU_BACKEND_A=OFF` in the CMakeCache of the existing build tells you what needs adding;
|
||||
- contiguous ranges are the *wrong* split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool.
|
||||
|
||||
## 7. Hazards checklist
|
||||
|
||||
- paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss;
|
||||
- keep `mmap` on for models with huge embedding tables (anonymous memory gets the process cgroup-killed);
|
||||
- CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply;
|
||||
- the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a **baseline of the runtime on that device alone**, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine;
|
||||
- a split consumes the fast device entirely — check what else it was serving before trading it away;
|
||||
- benchmark/experiment discipline on production hosts: stop and mask first.
|
||||
|
||||
## 8. Verdict shape
|
||||
|
||||
Rank and report, and prefer the boring answer when it wins:
|
||||
|
||||
1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link;
|
||||
2. **One process, weights-resident, KV co-located with attention on the fast pool** — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement;
|
||||
3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check `misses x bytes x tok/s` against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache;
|
||||
4. list explicitly what does *not* help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
name: hf-gguf-payload-fetch-verify
|
||||
description: "Inventory, fetch, hash-verify and assemble a Hugging Face GGUF payload (including mradermacher's .partNofM fragments and gated mirrors) before pointing an LLM service at it — on the Dank fleet or anywhere else."
|
||||
---
|
||||
|
||||
# Fetch and verify an HF GGUF payload
|
||||
|
||||
Use before any model swap where the weights come from Hugging Face, especially
|
||||
large (>32 GiB) quantised models. The failure modes below all look like success.
|
||||
|
||||
## 1. Inventory — query by quant name, never by extension
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<org>/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
for e in json.load(sys.stdin):
|
||||
p=e.get('path','')
|
||||
if 'Q4_K_S' in p: # the QUANT, not '.gguf'
|
||||
print(p, e.get('size',0), (e.get('lfs') or {}).get('oid','-'))
|
||||
"
|
||||
```
|
||||
|
||||
**Large quants ship as `.partNofM` fragments.** `mradermacher/*-i1-GGUF` repos
|
||||
hold e.g. Q4_K_M as `part1of3/2of3/3of3` (37.00+37.00+36.97 = 110.97 GiB) and
|
||||
Q4_K_S the same way, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are
|
||||
single `.gguf` files. Filtering on `endswith('.gguf')` therefore makes Q4_K_S
|
||||
look like it does not exist — do not conclude a quant is missing from an
|
||||
extension-filtered listing.
|
||||
|
||||
Check both mradermacher repos: `…-i1-GGUF` (imatrix) and `…-GGUF` (plain), and
|
||||
the model's own repo. Gated mirrors exist (`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`
|
||||
carries the same ladder) and return **403 even with a valid HF token** until
|
||||
that account accepts the gate — prefer the ungated copy rather than chasing access.
|
||||
|
||||
## 2. Fetch — `-f` is mandatory, secrets stay out of argv
|
||||
|
||||
```bash
|
||||
# token out of band: mode-600 config, never -H on the command line
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
|
||||
chmod 600 /tmp/hf.curlrc
|
||||
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc \
|
||||
-o "$DEST/$name" "$BASE/$name"
|
||||
```
|
||||
|
||||
- `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes the error body as the
|
||||
artifact. A gated repo yields identical small files (153 bytes) that hash the
|
||||
same and a `DOWNLOAD_DONE` in seconds for a 104 GiB payload.
|
||||
- A token in `-H "Authorization: Bearer …"` is visible in `/proc/<pid>/cmdline`
|
||||
to any `ps`/`pgrep -a` on the host. Never `pgrep -a` a running fetch.
|
||||
|
||||
## 3. Verify, then assemble
|
||||
|
||||
```bash
|
||||
# per part: byte count must equal the Hub's declared size, sha256 must equal lfs.oid
|
||||
sha256sum "$PARTS"/*.part* | sort
|
||||
|
||||
# concatenate in NUMERIC order, then verify the total and the container magic
|
||||
cat "$PARTS"/X.gguf.part1of3 "$PARTS"/X.gguf.part2of3 "$PARTS"/X.gguf.part3of3 > "$OUT"
|
||||
stat -c%s "$OUT" # must equal the sum of the part sizes
|
||||
head -c 4 "$OUT" # must print GGUF
|
||||
```
|
||||
|
||||
The assembled file has no published hash (only the parts do), so the per-part
|
||||
LFS oids plus an exact total are the verification chain. If a size check fails,
|
||||
suspect your own arithmetic before the payload: the sum of the parts is the
|
||||
expected value, and it is easy to mis-add by hand (`python3 -c "print(a+b+c)"`).
|
||||
|
||||
## 4. Housekeeping
|
||||
|
||||
- Failure artifacts (error pages) and stray directories: move to
|
||||
`~/.AGENTTRASH/YYYYMMDD-topic`, never `rm`; list them and wait for approval to
|
||||
purge.
|
||||
- Keep the `.partNofM` fragments after assembly when disk allows: they are the
|
||||
only copies that can be re-verified against the Hub. Note the total they cost
|
||||
and ask before purging.
|
||||
- Never delete the payload being replaced as a side effect of a swap: upstreams
|
||||
delist quants (both mradermacher and orcarouter have delisted Q4_K_M for
|
||||
Qwen3.8-Flash-Next-Uncensored), and the local file may be the last copy.
|
||||
|
||||
## 5. Waiting for a long fetch
|
||||
|
||||
Do not `pgrep -f <script>` where the pattern appears in your own command line —
|
||||
the loop matches itself and spins its full iteration count while the download is
|
||||
fine. Watch the artifact instead:
|
||||
|
||||
```bash
|
||||
while [ "$(stat -c%s "$DEST/$name" 2>/dev/null || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done
|
||||
```
|
||||
|
||||
## 6. Before choosing a smaller quant
|
||||
|
||||
A smaller quant buys memory and ~3–5 % decode, and costs quality. It does **not**
|
||||
change what the runtime can express: on llama.cpp, per-tensor cross-device
|
||||
placement is refused whenever `--override-tensor` is set
|
||||
(`has_tensor_overrides()` gates `pipeline_parallel`, which sets `n_copies`), and a
|
||||
layer split can only move contiguous layer ranges. State which of the two you are
|
||||
buying — headroom or a design change — before spending 100 GiB of bandwidth.
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: ik-llama-hybrid-moe-performance
|
||||
description: Diagnose and tune ik_llama.cpp hybrid CPU/GPU MoE decode/prefill on the dankclaw 3090+Strix Halo frank-large deployment; includes the barrier-split root-cause method and the systemd-unit test-interference trap
|
||||
---
|
||||
|
||||
# ik_llama.cpp hybrid MoE performance campaign (dankclaw frank-large)
|
||||
|
||||
Use when tuning or debugging decode/prefill speed of the ik_llama.cpp hybrid deployment (RTX 3090 + Strix Halo LPDDR, CPU-resident MoE experts) on dankclaw, or when a config experiment gives surprising numbers.
|
||||
|
||||
## Measured envelope (2026-09-16)
|
||||
|
||||
- Decode 3.8-4.3 tok/s, flat across thread counts (T1 ≈ T16). Prefill ~192 tok/s at 1.8k-32k prompts = the CPU AVX512-VNNI compute ceiling for CPU experts (batch amortizes expert reads; compute-bound, NOT memory-bound).
|
||||
- Decode root cause: ~100 CPU↔GPU backend splits/token; GPU at 5% SM during decode (launch/sync-bound over the PCIe x4 dock); T16 profile shows ~90% of cycles in `pause` (ggml_barrier spin) because threads wait on the serial critical path. Memory placement (mmap/anon/THP/deferred) does NOT matter — all variants land 3.8-4.6.
|
||||
- CUDA-graph capture aborts: qwen4exp's hc broadcasts use per-layer `ggml_repeat_4d` whose CUDA path does a capture-illegal op. Until ik gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs stay dead. Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels 4x slower than ik iqk) — do not switch for speed.
|
||||
- MTP/speculative impossible for mradermacher i1 GGUFs: the NextN head is stripped (no blk.48 tensors).
|
||||
|
||||
## Tuning levers, ranked
|
||||
|
||||
1. `-ncmoe 42` puts 6 layers' experts on the 3090: decode 4.54, prefill (small prompt) 32 — but VRAM 22/24.6 GiB at 262k ctx; needs an OOM-margin decision from the operator.
|
||||
2. `--prefetch-experts` is a no-op when the page cache is resident.
|
||||
3. `-thp` huge-mmap usually fails (`Cannot allocate memory`) and falls back silently.
|
||||
|
||||
## Traps
|
||||
|
||||
- The frank-large root systemd unit has Restart=always: `pkill`ing llama-server respawns it in ~4 s. For manual experiments: `sudo systemctl stop frank-large.service` first (root unit — NOT `systemctl --user`, that is stale halogen-era docs); verify `pgrep -xc llama-server` == 1 and 'HTTP server listening' in the current log before trusting any measurement.
|
||||
- Any `-ot`/`-ncmoe` override disables mmap for the whole model load in ik (`create_tensors` returns false).
|
||||
- `-ncmoe N` without `GGML_CUDA_NO_PINNED_WEIGHTS=1` page-locks ~108 GiB (co-tenant killer — the 2026-09-15 incident).
|
||||
- `--defer-experts` defers only PLE; under `-ngl 999` the experts still target CUDA0 → cudaMalloc OOM (~79.7 GiB on a 24 GiB card).
|
||||
- Never run `-j16` builds concurrently with decode measurements — both results are garbage.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: incident-window-reconstruction
|
||||
description: "Reconstruct which command/config was running at the instant an incident symptom first appeared, from on-disk artifacts (log mtimes + per-run durations + journal first-failure time) when memory and /tmp are gone."
|
||||
---
|
||||
|
||||
# Incident window reconstruction
|
||||
|
||||
Use when the question is **"what was running when it broke?"** — an incident,
|
||||
outage, resource blow-up, or unexplained service death — and the answer must
|
||||
come from artifacts, not recollection. Typical triggers: a host went down during
|
||||
an experiment, a service died mid-run, someone asks for the parameters in effect
|
||||
at the moment of failure.
|
||||
|
||||
## Principle
|
||||
|
||||
Three independent clocks, cross-checked — never recollection:
|
||||
|
||||
1. **Per-run duration** — every script wraps commands in `time`, so the log
|
||||
holds `real 4m19.129s`.
|
||||
2. **Log mtime** — the last write of that log.
|
||||
3. **Journal first-failure instant** — the first error line from the affected
|
||||
service.
|
||||
|
||||
`start ≈ mtime − real` for the *last* run in a log, and each run nests inside
|
||||
the previous one's window. The run whose `[start, end]` contains the journal's
|
||||
first-failure instant is the answer.
|
||||
|
||||
## Collect
|
||||
|
||||
```bash
|
||||
# 1. ordering and clocks of every artifact (log mtimes)
|
||||
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
|
||||
|
||||
# 2. durations + per-run section markers + the exact command lines
|
||||
ssh <host> 'grep -ah "^real" <logdir>/*.log'
|
||||
ssh <host> 'for f in <names>; do echo "## $f"; \
|
||||
grep -aE "^####|^\+ /path/to/binary" <logdir>/$f.log \
|
||||
| sed "s/\(.\{130\}\)/\1\n /g"; \
|
||||
grep -aE "^real|_EXIT=" <logdir>/$f.log; done'
|
||||
|
||||
# 3. first failure of the affected service, and what preceded it
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -aiE "Started|Created container" | tail -6'
|
||||
```
|
||||
|
||||
## Traps (all hit for real)
|
||||
|
||||
- **`/tmp` does not survive a reboot.** Launch experiment scripts from a
|
||||
persistent path (e.g. `~/models/...`) and log beside them; if the repair path
|
||||
is a reboot (common), `/tmp` scripts *and* any sampler output written there are
|
||||
gone. On the Dank fleet: `/home/dank/models/logs/<tag>.log` survived while
|
||||
`/tmp/x4*.sh` did not.
|
||||
- **A log's mtime is not proof the run finished.** A log truncated (or rotated)
|
||||
while a script still holds the fd keeps growing from that point; a 6-line log
|
||||
with an `ALL_DONE` marker means the earlier bulk was removed, not that nothing
|
||||
ran.
|
||||
- **Long command lines are display-truncated by agent tools.** Wrap before
|
||||
reading: `sed 's/\(.\{130\}\)/\1\n /g'`. Otherwise you will report
|
||||
parameters that were never used.
|
||||
- **Off-by-one file names silently skip evidence.** `x4b.log` vs
|
||||
`x4b-experiment.log` in a `for f in …; do [ -f $f.log ]` loop made half the
|
||||
artifacts invisible. List the directory and match exactly.
|
||||
- **Overlap is a hypothesis, not an observation.** Two resource-heavy
|
||||
invocations in the same minute may or may not have been resident at once;
|
||||
say so explicitly unless a sampler or cgroup counter proves it.
|
||||
- **Do not invent the cause.** Reconstruct the *window* (evidence) and keep the
|
||||
*mechanism* separate: grep the proof that would establish it (`journalctl -k |
|
||||
grep -i oom` for a killed process, driver logs for a GPU fault). If it is not
|
||||
there, publish "trigger not established" — an invented cause is worse than an
|
||||
open question because the next agent plans against it.
|
||||
|
||||
## Fleet-specific sensitivity
|
||||
|
||||
On this fleet, container healthchecks shell out to the **`podman` CLI**, so the
|
||||
first `invalid internal status … could not find any running process` in the user
|
||||
journal is an *early, per-minute* indicator of rootless-podman state death — and
|
||||
it timestamps the incident to within one healthcheck interval. That is how the
|
||||
18:58:30 Dankclaw break was pinned to the bench still loading at the time.
|
||||
|
||||
## Worked example (2026-09-16, Dankclaw)
|
||||
|
||||
| evidence | value |
|
||||
|---|---|
|
||||
| log mtimes | x4e-run 18:58:29, x4f-run 18:59:56 |
|
||||
| durations | E1 `real 4m53.946s` (llama-cli spam), F `real 5m24.785s` (exit 143 = killed) |
|
||||
| journal | first `invalid internal status` 18:58:30 |
|
||||
| ⇒ in flight | **F**, launched 18:54:31 (inside its 18:54:31–18:59:56 window) |
|
||||
|
||||
Answer delivered: the exact flags of F, plus the note that the preceding
|
||||
`llama-cli` had written 3.19 GB of spinner lines and that the `/tmp` VRAM sampler
|
||||
was gone — so the overlap of the two 111 GiB-class invocations stayed marked as
|
||||
[INFERENCE] rather than becoming a claim.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: live-llm-benchmark-hygiene
|
||||
description: "Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification."
|
||||
---
|
||||
|
||||
# Benchmarking a live LLM server honestly
|
||||
|
||||
Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance,
|
||||
any OpenAI-compatible endpoint). The point is to avoid the four confounds that
|
||||
turned a night's numbers into fiction before each one was fixed.
|
||||
|
||||
## 1. Match the corpus to the workload
|
||||
|
||||
Synthetic token soup (`vllm bench serve --dataset-name random`) under-reports
|
||||
speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
|
||||
code, and acceptance — not step time — is what moves throughput. For a coding
|
||||
fleet, build the corpus from the repo's own sources: file body plus an
|
||||
instruction that requires reading it. Then `tok/update` and TPOT mean something.
|
||||
|
||||
## 2. Salt every run so the prefix cache cannot help
|
||||
|
||||
vLLM runs with `--enable-prefix-caching`, and a repeated corpus silently
|
||||
measures a warm cache. Put a unique per-run token in the first bytes (it is both
|
||||
the engine's cache key and, if a proxy routes on it, the routing key). Without
|
||||
this, the second run of any comparison is not comparable to the first.
|
||||
|
||||
## 3. Measure turns, not only batches
|
||||
|
||||
A batch of distinct prompts parallelises prefill and hides the real cost of a
|
||||
conversation: an agent loop re-sends its context every turn. Two workloads need
|
||||
two clients:
|
||||
|
||||
- **batch/throughput** — `vllm bench serve --dataset-name custom --dataset-path
|
||||
<jsonl>` with `--custom-output-len`, several concurrency levels.
|
||||
- **sequential turns** — a small client that sends turn after turn with growing
|
||||
history and a large shared prefix, timing **TTFT of the first streamed chunk**
|
||||
per turn. This is the only measurement that exposes prefix-cache behaviour.
|
||||
|
||||
## 4. Read the engine's own ledger for capacity
|
||||
|
||||
Do not infer memory from the card's size. The server prints the truth at boot:
|
||||
|
||||
```
|
||||
Available KV cache memory: 5.44 GiB
|
||||
GPU KV cache size: 122,636 tokens
|
||||
Maximum concurrency for 114,688 tokens per request: 1.07x
|
||||
```
|
||||
|
||||
That is the number to quote for "how many concurrent conversations fit".
|
||||
Related trap: the arena/KV budget is **not monotonic in chunk size** — on one
|
||||
host, available KV was 5.44 GiB at `CHUNK=8192`, 4.22 GiB at 16384 and 4.16 GiB
|
||||
at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
|
||||
measure three points before treating a knob as a lever.
|
||||
|
||||
## 5. Soak before believing a tuning change
|
||||
|
||||
Run several families concurrently for 15-30 minutes and count failures, not just
|
||||
latency: a nightly soak returned 155/155 answers with zero failovers while only
|
||||
25% of turns hit the cache — which is how the capacity ceiling was found as a
|
||||
*latency* effect rather than an error. Report hit/miss split, medians, and the
|
||||
min/max, because the median of a bimodal distribution describes neither mode.
|
||||
|
||||
## 6. Verification that is not the benchmark
|
||||
|
||||
- Run the client the fleet actually uses (e.g. `omp --model <alias> -p
|
||||
--no-tools`) — a config file that parses is not a config that works, and a
|
||||
non-obvious API mode (raw `/v1/completions`) may still be fine because the
|
||||
client templates its own prompt.
|
||||
- A deployed inference unit is not proven until an **unattended reboot** brings
|
||||
it back: verify `is-active`, `/health`, and one real completion afterwards.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: live-llm-telemetry-no-bench
|
||||
description: "Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU."
|
||||
---
|
||||
|
||||
# Live LLM telemetry without benchmarking
|
||||
|
||||
Use when asked "how is the model performing?" on a host that SERVES production
|
||||
inference, where benchmarking policy forbids running `llama-bench`/`vllm bench`
|
||||
beside the live service. Instead of a synthetic bench, mine the runtime's own
|
||||
per-request telemetry — it measures the real workload and costs nothing.
|
||||
|
||||
## 1. Where the numbers live
|
||||
|
||||
| Runtime | Unit kind | Log command |
|
||||
|---|---|---|
|
||||
| llama.cpp (mainline, `-lv 3`) | root systemd unit | `sudo journalctl -u <unit> --since "-3h" --no-pager` |
|
||||
| halogen-flash-server | rootless Quadlet | `journalctl --user -u <unit> --no-pager` (**never** sudo — per-user storage) |
|
||||
|
||||
llama.cpp emits, per completed task:
|
||||
|
||||
```
|
||||
I slot print_timing: id 0 | task 82 | prompt eval time = 339.51 ms / 57 tokens ( 5.96 ms per token, 167.89 tokens per second)
|
||||
I slot print_timing: id 0 | task 82 | eval time = 428.49 ms / 36 tokens (12.24 ms per token, 81.68 tokens per second)
|
||||
I slot print_timing: id 0 | task 82 | total time = 768.00 ms / 93 tokens
|
||||
I slot print_timing: id 0 | task 82 | graphs reused = 84
|
||||
I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 ( 25 accepted / 33 generated), mean len = 3.27
|
||||
I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s
|
||||
```
|
||||
|
||||
halogen-flash-server emits one line per request:
|
||||
|
||||
```
|
||||
serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok
|
||||
```
|
||||
|
||||
Grep both with `grep -E "print_timing|release|serve_api: mtp"`. Long lines get
|
||||
truncated in a terminal — redirect to a file/artifact and read it whole.
|
||||
|
||||
## 2. Read the envelope, not one sample
|
||||
|
||||
- `prompt eval` on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use
|
||||
the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
|
||||
- `eval` (decode) and `draft acceptance` are content-dependent: measured
|
||||
acceptance 0.46-0.76 on one unchanged build. Never quote a single task as
|
||||
"the model's speed", and never compare a live sample to a deploy-time
|
||||
baseline without a masked `llama-bench`.
|
||||
- `graphs reused` > 0 and `selected slot by LCP similarity` prove prefix-cache
|
||||
reuse is working; `prompt 53 (48 cached), prefill 0.05s` vs `1.58s` cold is
|
||||
prefix caching caught in the act.
|
||||
- A runtime's own window metric (`tg_3s`) is a cross-check on your arithmetic.
|
||||
|
||||
## 3. Attribute PCIe without touching the model
|
||||
|
||||
Run the counter in the background and issue ONE ordinary request:
|
||||
|
||||
```bash
|
||||
ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
|
||||
curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
|
||||
http://127.0.0.1:7771/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
|
||||
wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'
|
||||
```
|
||||
|
||||
`-s t` = `rxpci`/`txpci` in MB/s. Measured on a fully-VRAM-resident 27B on a
|
||||
3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at
|
||||
idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane
|
||||
link the transport is a non-factor — say so with the measured number instead of
|
||||
hand-waving.
|
||||
|
||||
## 4. Traps
|
||||
|
||||
- **`wait` or you lose the telemetry.** If the remote command exits while the
|
||||
curl is backgrounded, ssh closes the session and the client dies: the journal
|
||||
shows `W srv stop: cancel task, id_task = N` and NO `print_timing` line, while
|
||||
curl still reports a plausible status. Always end the compound command with
|
||||
`wait`.
|
||||
- **Do not run a GPU workload to "get a reading".** One max_tokens=200 request
|
||||
is production traffic and is fine; anything that pins the card for a while is
|
||||
a bench and needs the stop-and-mask procedure first.
|
||||
- **Check which instance you are reading.** Quadlet units run `--replace --rm`
|
||||
and root units get restarted: `systemctl status <unit> | head -12` for the
|
||||
start time; `task N` IDs increment per request and jump across restarts.
|
||||
- **The engine's own ledger outranks your model of it** — quote `KV self size`,
|
||||
`n_slots`, `n_ctx_slot` and the VRAM figure from the log/NVML rather than
|
||||
arithmetic.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: llama-bench-fleet-sweep
|
||||
description: "Run a llama-bench sweep (batch/ubatch, ncmoe, split, or any dimension) on a Dank-fleet LLM host correctly: stop the service, run from a transient unit, one invocation per cell, sample VRAM, restore and verify — including the llama-bench vs llama-server traps that have already produced wrong numbers here."
|
||||
---
|
||||
|
||||
# llama-bench sweeps on a Dank-fleet LLM host
|
||||
|
||||
Use when tuning a llama.cpp service's serve-time parameters (batch/ubatch, `-ncmoe`, `-sm`/`-ts`, KV type) offline, i.e. any question of the form "how large may X be / what does X buy".
|
||||
|
||||
Reference implementation: `scripts/bench-batch-ubatch.sh` (repo) — read it before writing another harness.
|
||||
|
||||
## Preconditions
|
||||
|
||||
1. **Stop the service** — Benchmarking Policy in AGENTS.md is mandatory: a bench beside a live service starves the same GPU and corrupts results. Never bench against the live port.
|
||||
2. **Run from a transient unit, not a shell:**
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||||
```
|
||||
|
||||
The deploy's unit sets `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell gets 8 MiB (`ulimit -l`), which fails the load for a reason unrelated to what you are measuring. Passwordless sudo is available on both hosts.
|
||||
|
||||
## The traps (each one has produced a wrong number)
|
||||
|
||||
- **`/` vs `,`.** llama-bench splits `/` = devices *within one configuration*, `,` = *sweep values*. A legitimate llama-server `--tensor-split 36,29` becomes `-ts 36/29`; the comma form silently benches each card alone. Same for `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
|
||||
- **No `-c`.** `n_ctx = n_prompt + n_gen + n_depth`. The context is a consequence of the test shape, so a bench can NEVER prove that a configuration's activation buffers fit beside the *deployed* KV — that requires starting the real service at the real context and slot count.
|
||||
- **A test that cannot allocate calls `exit(1)`**, killing the whole process. One invocation per cell, or one infeasible value loses every later cell — and infeasibility is usually what you are looking for.
|
||||
- **No `--spec-*` and no `--parallel`.** Decode figures are the *drafterless* baseline, and slots are never exercised. `-d <n>` is what puts KV in front of the measurement: a `tg` instance runs with `n_prompt = 0`, i.e. at ~zero depth otherwise.
|
||||
- **A cell that looks fast because its allocation was smaller than intended is indistinguishable from a fast cell** unless you sample VRAM. Sample `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and report min-free per card.
|
||||
- **`local a=$1 b=$x` expands `$x` before the builtin assigns anything** — under `set -u` that is `b: unbound variable`, and it fires *after* your harness has already stopped the production service. Declare in two statements.
|
||||
- **Ranges exist:** `-ub 512-2048+256` == `512,768,...,2048`.
|
||||
|
||||
## Shape of a good harness
|
||||
|
||||
- One `llama-bench` invocation per cell; append one TSV row per cell as it completes, so progress is readable mid-run and an interrupted sweep resumes (skip cells whose `bench.json` is non-empty).
|
||||
- Write `bench.json` per cell (`-o json`), plus `bench.log` (`--progress`), plus the VRAM trace.
|
||||
- **Stop the service in the harness and restore it from a `trap ... EXIT`** — and still verify the service afterwards. A harness that crashes mid-run leaves production down.
|
||||
- Match the deployment's load flags exactly (`-ngl`, `-sm`, `-ts`, `-fa`, `-ctk/-ctv`, `-t`), or you are benching a different service. Note `-ts`/`-dev` take slashes here while the unit takes commas.
|
||||
|
||||
## Verify before burning hours
|
||||
|
||||
Dry-run the harness against a **stub** `llama-bench` (`BIN=/tmp/stubdir`, plus `SERVICE=<nonexistent-name>` so the real service is untouched) with a tiny grid, including one cell forced to `exit 3`: prove the TSV row, the failure capture, and the resume logic before the real sweep. Then validate the real flag set in ~1 s by pointing `-m` at a nonexistent file — argument errors surface *before* the model load.
|
||||
|
||||
## Interpreting results
|
||||
|
||||
- A prefill difference smaller than the run-to-run spread is not a result; check each cell's own `stddev_ts`.
|
||||
- Single-sequence decode is usually flat across batch/ubatch — that knob is prefill-side.
|
||||
- Batch ≥ ubatch always (llama.cpp clamps ubatch to batch, so smaller-batch cells silently duplicate).
|
||||
- Report what the harness cannot see (drafter, slots, real context) rather than implying the sweep settled it.
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: llama-bench-host-sweep
|
||||
description: "Run a llama-bench sweep (batch/ubatch, split, quant, depth) on a live fleet LLM host without fooling yourself: what llama-bench cannot express, the separator trap, per-cell isolation, the memlock/transient-unit requirement, and the dry-run that must precede a multi-hour run."
|
||||
---
|
||||
|
||||
# Running a llama-bench sweep on a fleet host
|
||||
|
||||
Use when a bench must answer a serving question on one of the LLM hosts (Dankclaw
|
||||
frank-large/frank-small, Dankserver frank-medium) — batch/ubatch sizing, split
|
||||
choice, KV type, depth behaviour.
|
||||
|
||||
Repo policy applies: **stop the production service first** (AGENTS.md Benchmarking
|
||||
Policy). The harness below does that for you and restores it.
|
||||
|
||||
## 1. What llama-bench cannot express — decide this BEFORE designing the sweep
|
||||
|
||||
| Not available | Consequence |
|
||||
|---|---|
|
||||
| `-c/--ctx-size` | `n_ctx = n_prompt + n_gen + n_depth` (llama-bench.cpp). The context is a consequence of the test shape, so **the deployment's real allocation is never tested** (e.g. a 16 GiB KV at 262144 over 2 slots). Only starting the real unit answers "does it fit". |
|
||||
| `--parallel` | single sequence only; no slot shape, no concurrency effect. |
|
||||
| `--spec-*` | no drafter. Any `tg` number is the **autoregressive baseline**, not the served rate. Useful as a control, never as the deployed figure. |
|
||||
| warmup per config | a warmup run is itself a full-size run; at 64k prompts `--no-warmup` buys a second sample for the same wall clock. |
|
||||
|
||||
`-d N` is a depth PREFIX: it puts N tokens of KV in place before the measured run and
|
||||
is cached across reps through a `llama_state_seq_*` save/restore, so the fill is paid
|
||||
once per cell and is not part of the measured number. Without `-d`, a `tg` test runs at
|
||||
~zero depth and the number is meaningless for long conversations.
|
||||
|
||||
## 2. Two traps that silently change the experiment
|
||||
|
||||
- **Separators.** `/` separates devices *within one configuration*; `,` separates
|
||||
configurations to sweep. `-ts` and `-dev` both follow this. `-ts 36,29` parses as the
|
||||
single value `36` → everything lands on one card and you have benched a different
|
||||
machine. llama-server takes the comma form for the same pair. List-valued args also
|
||||
accept ranges: `-ub 512-2048+256`.
|
||||
- **memlock.** The deploy units set `LimitMEMLOCK=infinity` because the ROCm backend
|
||||
pins staging buffers; a plain SSH shell on these hosts has soft=hard=8 MiB
|
||||
(`ulimit -l`), so a load can fail for a reason unrelated to the model. Run the sweep
|
||||
inside a transient unit:
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=fm-bench --collect --no-block \
|
||||
--property=Type=oneshot --property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
This also survives an ssh disconnect, unlike a foreground remote process.
|
||||
|
||||
## 3. One invocation per cell
|
||||
|
||||
A test that cannot run calls `exit(1)` immediately (failed depth/prompt/gen run), so a
|
||||
single cross-product invocation loses **every later cell** the moment one ubatch fails to
|
||||
allocate — and infeasibility is usually one of the questions being asked.
|
||||
|
||||
Drive the grid from a shell loop: one llama-bench call per (b, ub), continue on a
|
||||
non-zero exit, record why, and skip cells whose output file already exists and is
|
||||
non-empty (that makes an interrupted sweep resumable). Write rows to a TSV as the sweep
|
||||
proceeds so progress is readable mid-run.
|
||||
|
||||
## 4. Sample VRAM per cell, covering the load
|
||||
|
||||
Batch/ubatch sizes the activation buffers, so a cell that looks fast because its
|
||||
allocation was silently smaller must be distinguishable from one that is genuinely fast.
|
||||
Poll `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and keep the
|
||||
peak (minimum free) per card. Correlate by taking the window between the cell's start and
|
||||
end.
|
||||
|
||||
## 5. Always restore the service from a trap
|
||||
|
||||
`trap <restart> EXIT` in the harness, plus a manual check afterwards. A harness that
|
||||
stops a production unit and then dies leaves inference down; this fleet has had that
|
||||
happen.
|
||||
|
||||
## 6. Dry-run before a multi-hour run
|
||||
|
||||
Stub the binary so the whole path is exercised without loading a 29 GiB model:
|
||||
|
||||
```bash
|
||||
# /tmp/benchstub/llama-bench prints the -o json shape and exits non-zero for one ubatch
|
||||
SERVICE=frank-medium-dryrun BIN=/tmp/benchstub MODEL=/tmp/benchstub/m.gguf OUT=/tmp/bub-test \
|
||||
N_PROMPT=8 N_GEN=4 N_DEPTH=0 REPS=1 BATCHES="2048 4096" UBATCHES="512 1024" \
|
||||
bash scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
`SERVICE=<nonexistent>` means the trap and the stop are never reached, so the real
|
||||
service is untouched. Check that a FAILED cell still produces a row with the right column
|
||||
count before trusting the table.
|
||||
|
||||
## 7. Read the result sanely
|
||||
|
||||
- Bracket the sweep: run the reference cell (the deployed configuration) first and
|
||||
again last, so drift over hours is visible.
|
||||
- Sanity-check one number against the runbook's prior figures before believing the grid.
|
||||
- Remember what the grid does NOT prove: it did not run the drafter, the slots, or the
|
||||
deployment's context. The winner still has to be applied and verified live.
|
||||
|
||||
## Pitfall while writing such a harness
|
||||
|
||||
`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands **all** the builtin's arguments
|
||||
before the builtin assigns any of them, so under `set -u` this aborts with
|
||||
`b: unbound variable`. Declare in two statements:
|
||||
|
||||
```bash
|
||||
local b=$1 ub=$2
|
||||
local dir="$OUT/b${b}_ub${ub}" sampler start rc wall
|
||||
```
|
||||
|
||||
Ref: `scripts/bench-batch-ubatch.sh` in serverAppConfigs (the fleet's implementation of
|
||||
all of the above).
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: llama-cpp-spec-decode-ab
|
||||
description: "Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts."
|
||||
---
|
||||
|
||||
# Spec-decoding A/B on a live llama.cpp endpoint
|
||||
|
||||
Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter".
|
||||
Verdicts here are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.
|
||||
|
||||
## Step 0 — settle chaining in the source, before spending GPU time
|
||||
|
||||
`common/speculative.cpp` (`common_speculative_init_result`) builds `configs` and gates each type:
|
||||
|
||||
- **draftless** (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
|
||||
- **context-backed** (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) → require `params.draft.ctx_dft != nullptr`
|
||||
|
||||
Consequences: one context-backed drafter + any number of ngram types chains fine; **two context-backed types never do**
|
||||
(`draft-mtp,draft-dflash` → `context type MTP requested but model doesn't contain MTP layers`).
|
||||
Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.
|
||||
|
||||
## Step 1 — shapes (the part that decides the answer)
|
||||
|
||||
| Shape | Prompt | Who it can favour |
|
||||
|---|---|---|
|
||||
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
|
||||
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
|
||||
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
|
||||
| identity | short deterministic question | controls for lossless verification |
|
||||
|
||||
Design the echo passage so no long span repeats *inside* it — otherwise you measure the matcher against itself.
|
||||
For `structured`, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.
|
||||
|
||||
## Step 2 — protocol
|
||||
|
||||
1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait for the port.
|
||||
2. Per config: write an env file with overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health` until ok, then measure.
|
||||
3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
|
||||
4. Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).
|
||||
|
||||
## Step 3 — read the right counter
|
||||
|
||||
- Per request: `timings.draft_n` / `timings.draft_n_accepted`, `predicted_per_second`.
|
||||
- Cumulative: `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`.
|
||||
- **`draft_n == 0` while tokens flow = the proposer is not engaged**, even when every flag looks right.
|
||||
- A config with zero drafting is a useful *negative control*: it proves the other config's drafts came from the proposer under test.
|
||||
|
||||
## Traps that produce false verdicts
|
||||
|
||||
- **Thinking eats the answer.** Echo/structured prompts with thinking on put the copy in `reasoning_content`; `content` is empty, so identity checks compare `''` to `''`. Disable per request: `chat_template_kwargs: {"enable_thinking": false}`.
|
||||
- **No banner.** ngram-mod prints nothing at startup; the journal shows only the drafter's `adding speculative implementation`. Never conclude "chain not engaged" from the load log.
|
||||
- **Windows change the verdict.** ngram-mod `--spec-ngram-mod-n-min/--spec-ngram-mod-n-max` decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
|
||||
- **Unused-tensor warnings are normal**: with `draft-dflash` on an MTP-carrying GGUF, `blk.<n>.nextn.*` is `unused ... ignoring` — correct, not a truncated file.
|
||||
- **Harness hygiene**: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix), `pkill -x` never `-f`.
|
||||
|
||||
## Interpreting
|
||||
|
||||
Report `decode tok/s` per shape plus acceptance (`accepted/draft_n`) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: llamacpp-cross-backend-device-split
|
||||
description: "Decide whether two unlike devices (NVIDIA CUDA + AMD Vulkan, or any two backends) can serve one llama.cpp model, and measure the attempt honestly on a serving host — the two scheduler failure modes, what is actually expressible, the dual-backend build recipe, and the measurement traps that make results meaningless."
|
||||
---
|
||||
|
||||
# Cross-backend device splits in llama.cpp
|
||||
|
||||
Use when someone proposes serving ONE model across two devices of different
|
||||
backend families (NVIDIA CUDA + AMD Vulkan/HIP, or a dGPU + an APU), and you
|
||||
need to know whether it is expressible before spending a build and a service
|
||||
outage on it.
|
||||
|
||||
## Answer first: it is not expressible
|
||||
|
||||
A weight tensor in one *device* buffer cannot be consumed by an op the scheduler
|
||||
placed on another device. Two observed failure modes (mainline llama.cpp
|
||||
`37b53fd`, 2026-09-16):
|
||||
|
||||
| attempt | result |
|
||||
|---|---|
|
||||
| per-tensor: `-ot "exps=Vulkan1"` with `-sm none -mg 0 -dev CUDA0,Vulkan1` | `ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — scheduler abort |
|
||||
| per-layer: `-sm layer -ts a,b -dev CUDA0,Vulkan1`, or layer-scoped `-ot` | model load fails, message is only `failed to load model` |
|
||||
| device + **CPU**: `-ot "exps=CPU"` | **works** — the CPU backend is the universal fallback and host buffers can be copied |
|
||||
|
||||
The rule: **the CPU is the only legitimate partner in a heterogeneous split.**
|
||||
Everything else must be single-vendor. On a CUDA+AMD host that means a card can
|
||||
be the trunk *or* the offload host, never the complement of an AMD-resident
|
||||
peer — the same wall the R9700 pair hits from the RCCL/atomics direction.
|
||||
|
||||
If the answer is "not expressible", stop and consider the expressible
|
||||
alternatives in this order: (1) two independent endpoints, one model per device;
|
||||
(2) single-vendor layer split (`-sm layer`, no collective needed); (3)
|
||||
device + CPU per-tensor offload.
|
||||
|
||||
## Build the dual-backend binary
|
||||
|
||||
Needs both toolchains in one tree; Vulkan needs `glslc` + `libvulkan` headers
|
||||
(already present on this fleet's hosts).
|
||||
|
||||
```bash
|
||||
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
-DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=<sm> \
|
||||
-DGGML_CCACHE=ON -DLLAMA_CURL=OFF
|
||||
cmake --build build-dual --target llama-bench llama-cli llama-server -j $(nproc)
|
||||
```
|
||||
|
||||
**Device names are not what you expect.** `--list-devices` on an
|
||||
NVIDIA-plus-APU host lists the discrete card under BOTH backends (e.g. `CUDA0`
|
||||
and `Vulkan0`) and the APU last (`Vulkan1`). The iGPU is `Vulkan1`, not
|
||||
`Vulkan0`. Valid `-ot` buffer types are whatever that list prints; a wrong name
|
||||
makes llama.cpp print the valid set and exit, so the error is self-diagnosing.
|
||||
|
||||
## Measure it without fooling yourself
|
||||
|
||||
- **`tg` in llama-bench runs with `n_prompt = 0`.** A plain `-n 64` row is a
|
||||
~zero-depth decode: no KV traffic at all. Anything about context cost or KV
|
||||
placement must use `-d/--n-depth`, which fills the cache before timing.
|
||||
- `-p`/`-n` lists build separate instances (prefill instances with `n_gen = 0`,
|
||||
generation instances with `n_prompt = 0`), and `n_ctx = n_prompt + n_gen`.
|
||||
- Compare at **depth-matched** settings, and remember prefill into a deep cache
|
||||
is the slow case (e.g. 456 tok/s fresh vs 232 tok/s into 64k).
|
||||
- Sample the actual engines during the run (`/sys/class/drm/card*/device/gpu_busy_percent`,
|
||||
`nvidia-smi`) — an idle device or a 100 % APU is the difference between a real
|
||||
measurement and a silent CPU fallback.
|
||||
- Watch the **host memory** figure, not just tok/s. A Vulkan-resident model lives
|
||||
in device buffers (committed, not reclaimable), whereas an mmap'd CPU-placed
|
||||
tensor is page cache — the same model can commit 116 GiB one way and 95 GiB
|
||||
the other. On a host with co-tenants, that decides deployability.
|
||||
|
||||
## Serving-host discipline
|
||||
|
||||
- **Stop and mask every LLM service first** (repo benchmarking policy). For a
|
||||
rootless Quadlet, move the `.container` out of
|
||||
`~/.config/containers/systemd/` and `daemon-reload` — the generator re-creates
|
||||
the unit from that file on every reload, so `mask` is not the lever.
|
||||
- **`systemctl mask` is refused for a relinked root unit** whose
|
||||
`/etc/systemd/system/<svc>.service` is a symlink into the repo: it exits with
|
||||
`File ... already exists and is a symlink to ...`. Stop it and ensure nothing
|
||||
restarts it, or move the symlink aside.
|
||||
- Root units need `sudo systemctl start`; without it you get
|
||||
`Access denied as the requested operation requires interactive
|
||||
authentication` (polkit), which reads like a permissions bug.
|
||||
- Export the pinning guard: **mainline is `GGML_CUDA_NO_PINNED`** (ik_llama.cpp
|
||||
was `GGML_CUDA_NO_PINNED_WEIGHTS`). Without it a CPU-overridden tensor can land
|
||||
in a pinned `CUDA_Host` buffer — the failure mode that page-locked 108 GiB and
|
||||
starved a host's co-tenants until a power cycle.
|
||||
- Restore: move the quadlet back, `daemon-reload`, `reset-failed`, start, then
|
||||
verify `/health`, `/v1/models` **and the engine's own boot ledger** (e.g.
|
||||
halogen's `kv pool:` lines, which say whether the configured pool was clamped).
|
||||
|
||||
## Calibration point
|
||||
|
||||
On a 121 GiB Strix Halo host with a 3090: a 177B MoE (112 GiB GGUF, 12 of 48
|
||||
layers carrying KV) ran **32.6 tok/s shallow / 25.8 tok/s at 64k depth / 456 tok/s
|
||||
prefill** with everything on the iGPU via Vulkan, and **12.1 tok/s** with the
|
||||
same trunk but experts on the CPU. If a proposed split cannot beat the
|
||||
iGPU-only number at matched depth, it is not worth the failure modes above.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: llamacpp-device-split-feasibility
|
||||
description: "Decide which multi-device arrangements llama.cpp can actually express on a heterogeneous host (mixed vendors, iGPU+dGPU), and measure them without fooling yourself — covers the expressibility matrix, llama-bench's /-vs-, device/split syntax, independent placement proof, and the stop/restore path."
|
||||
---
|
||||
|
||||
# When to use
|
||||
|
||||
Asked "can we split this model across GPU A and GPU B?" on a host where the devices are mixed (NVIDIA + AMD, iGPU + dGPU) or where the obvious arrangement is "weights on device X, different weights on device Y". Also use before *believing* any earlier "it cannot be done" conclusion about a multi-device split — that verdict is easy to produce by accident.
|
||||
|
||||
# 1. Classify the arrangement before running anything
|
||||
|
||||
llama.cpp expresses exactly two things across devices, and they are not interchangeable:
|
||||
|
||||
| arrangement | how | expressible? |
|
||||
|---|---|---|
|
||||
| **Layer split** — layers 0..n on dev A, the rest on dev B | `-sm layer -ts <shareA>/<shareB>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured) |
|
||||
| **Per-tensor placement** — e.g. attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a required term of the scheduler's `pipeline_parallel` flag (the only read site is `llama-context.cpp`, in the `pipeline_parallel = ...` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp` (`pre-allocated tensor (...weight) in a buffer (...) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). `-ot x=CPU` works only because the CPU backend is last in the list. |
|
||||
|
||||
So: if the plan needs *tensors* on a specific device, stop — the answer is no, for any vendor mix, and pure-Vulkan vs CUDA+Vulkan changes nothing. If it needs *layers*, it is expressible and worth measuring.
|
||||
|
||||
Layer-split arithmetic to check before promising anything:
|
||||
|
||||
- A whole layer carries its experts, if the model is MoE. Compute GiB/layer, not GiB/param-group.
|
||||
- Only some layers carry KV (e.g. every 4th for hybrid attention). A share of layers is therefore only a share of KV — a contiguous range rarely holds the layers you care about.
|
||||
- Big non-repeating tensors (per-layer-token tables etc.) land on the main device and cannot be moved by `-ts`; they cap how small the primary device's footprint can get.
|
||||
|
||||
# 2. Get the harness syntax right, or every number is fiction
|
||||
|
||||
`llama-bench` writes `-dev <dev0/dev1/...>` and `-ts <ts0/ts1/..>`: **`/` separates values within one configuration, `,` separates configurations to sweep.**
|
||||
|
||||
- `-dev CUDA0,Vulkan1` = **two single-device runs**, never a two-device run.
|
||||
- `-ts 0.85,0.15` = two sweep values, each becoming `[0.85, 0, 0, …]` → device 0 takes everything.
|
||||
|
||||
Write `-dev A/B -ts x/y`. `llama-cli`/`llama-server` use the classic comma form — which is why a failure seen only under llama-bench is a harness bug, not an engine limit.
|
||||
|
||||
Cheap pre-flight: run with the intended flags and read the tool's own reported parameters (llama-bench's table has a `dev` column). If a row says one device, your intent was not parsed.
|
||||
|
||||
# 3. Prove placement — never trust the label
|
||||
|
||||
Sample the non-primary device *while the run is loaded*:
|
||||
|
||||
```bash
|
||||
( for i in $(seq 1 60); do nvidia-smi --query-gpu=memory.used --format=csv,noheader; sleep 10; done ) > /tmp/vram.txt &
|
||||
```
|
||||
|
||||
Expect roughly (layers × GiB/layer + that device's KV) resident. Tens of MiB means the split did not happen. Cross-check: several "different" configurations that all produce the same tok/s to within 0.2 % are probably all the same single-device run.
|
||||
|
||||
# 4. Measure at the depth that matters
|
||||
|
||||
- `llama-bench`'s `tg` runs with `n_prompt = 0`, i.e. ~zero depth. Use `-n 32 -d 65536` (or deeper) or the KV term — the entire reason for the split — is absent.
|
||||
- Compare against the **single-device baseline on the fast-memory device**, and against whatever production runtime already serves the model. A split that wins by a few percent at 64k depth may lose at 200k, because the KV term grows with context and prefill cost grows with the target depth.
|
||||
|
||||
# 5. Stop / restore on a host that serves
|
||||
|
||||
- Stop and mask the production units first (benchmarking policy). For rootless quadlets the lever is the `.container` file (move it aside + `daemon-reload`); `Restart=always` otherwise recreates the service mid-run. `systemctl mask` is refused for a unit that is a symlink into a repo checkout.
|
||||
- After restoring, verify `/health`, `/v1/models`, and the boot ledger numbers — and keep a memory-headroom note: the same model can come back with less contiguous host memory than before (a reboot re-fragments), which decides whether the next co-tenant fits.
|
||||
- Landmine seen on a rootless-podman host after a big GPU run: a dead rootless **pause process** takes *all* container management down (every `podman` command: `invalid internal status … could not find any running process`; container starts then fail because the rootless netns holding the bridges is gone: `failed to bind udp listener on 10.89.x.1:53`). `podman system migrate` panics (5.7.0, `teardownNetworkBackend`). Repair is a reboot. Before rebooting, check `loginctl show-user <u> -p Linger` and that units carry `[Install] WantedBy=default.target`, so everything returns by itself.
|
||||
- Do not publish a cause you did not check: `journalctl -k | grep -i oom` decides whether "memory pressure killed it" is a finding or a guess. An invented narrative is worse than an open question.
|
||||
|
||||
# 6. Report shape that survived review
|
||||
|
||||
State (a) which arrangement was tested and that the harness parsed it, (b) the placement evidence, (c) the depth-matched numbers next to the single-device baseline, (d) the memory cost on the borrowing device, and (e) whether the split can reach its own premise (KV on the fast device, long context) — not just that it runs.
|
||||
@@ -0,0 +1,168 @@
|
||||
---
|
||||
name: llamacpp-drafter-kv-split-sizing
|
||||
description: "Size a llama.cpp multi-GPU layer-split deployment that carries a speculative drafter: read the drafter's KV law from its GGUF header, then find the split balance by measurement (the batch asymmetry, the ctx/floor trade, and the OOM-vs-page-fault signatures)."
|
||||
---
|
||||
|
||||
# Sizing a llama.cpp layer-split with a speculative drafter
|
||||
|
||||
Use when a multi-GPU llama.cpp deployment carries a draft model (`--spec-draft-model`)
|
||||
and the question is *where do the layers go and how big can the context be*. The two
|
||||
effects below decide it, and neither is visible in the flags.
|
||||
|
||||
## 1. Read both GGUF headers first
|
||||
|
||||
The drafter's own properties set the memory bill. `llama-server` prints almost none of
|
||||
them, and the loader's ledger is not printed at default verbosity.
|
||||
|
||||
Minimal GGUF KV reader (no tooling needed) — write `ggufmeta.py`:
|
||||
|
||||
```python
|
||||
import struct, sys
|
||||
|
||||
def rd(path):
|
||||
f = open(path, 'rb')
|
||||
assert f.read(4) == b'GGUF'
|
||||
ver = struct.unpack('<I', f.read(4))[0]
|
||||
struct.unpack('<Q', f.read(8)) # tensor count
|
||||
nk = struct.unpack('<Q', f.read(8))[0] # kv count
|
||||
def s():
|
||||
n = struct.unpack('<Q', f.read(8))[0]
|
||||
return f.read(n).decode('utf-8', 'replace')
|
||||
# 0=UINT8 1=INT8 2=UINT16 3=INT16 4=UINT32 5=INT32 6=FLOAT32 7=BOOL
|
||||
# 8=STRING 9=ARRAY 10=UINT64 11=INT64 12=FLOAT64
|
||||
SCALAR = {0:1, 1:1, 2:2, 3:2, 4:4, 5:4, 6:4, 7:1, 10:8, 11:8, 12:8}
|
||||
FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
|
||||
keys = {}
|
||||
for _ in range(nk):
|
||||
k = s()
|
||||
t = struct.unpack('<I', f.read(4))[0]
|
||||
if t == 8:
|
||||
v = s()
|
||||
elif t in SCALAR:
|
||||
v = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
|
||||
else:
|
||||
et = struct.unpack('<I', f.read(4))[0] # element type FIRST
|
||||
n = struct.unpack('<Q', f.read(8))[0] # then the count
|
||||
if et == 8:
|
||||
v = ('arr_str', [s() for _ in range(n)])
|
||||
elif et in SCALAR:
|
||||
v = ('arr', et, n); f.read(n * SCALAR[et])
|
||||
else:
|
||||
raise Exception('unsupported arr elem type %d' % et)
|
||||
keys[k] = v
|
||||
return ver, keys
|
||||
|
||||
for p in sys.argv[1:]:
|
||||
ver, k = rd(p)
|
||||
arch = k.get('general.architecture')
|
||||
print('=====', p, 'v%d' % ver, 'arch:', arch)
|
||||
for key in sorted(k):
|
||||
if key.startswith('tokenizer.') or isinstance(k[key], tuple):
|
||||
continue
|
||||
if key.startswith(arch + '.') or not key.startswith(('general.', 'quantize.')):
|
||||
print(' ', key, '=', k[key])
|
||||
```
|
||||
|
||||
The three format traps, each of which cost a debugging session:
|
||||
|
||||
1. **The KV stream has NO alignment.** GGUF v3's alignment (from `general.alignment`,
|
||||
default 32) applies only to the *tensor data section* that follows. Do not pad
|
||||
between KV pairs.
|
||||
2. **Array layout is `type(9) -> element_type(u32) -> count(u64)`.** Count-then-element
|
||||
is wrong and yields a `MemoryError`/`UnicodeDecodeError` a few pairs later.
|
||||
3. **The type enum is not the order you expect**: `7=BOOL`, `9=ARRAY`, and the 64-bit
|
||||
types are `10/11/12`. Getting `3=INT16` (2 bytes) or `5=INT32` (4 bytes) wrong shifts
|
||||
the whole stream.
|
||||
|
||||
Fields that matter: `*.block_count`, `*.block_size`, `*.attention.head_count_kv`,
|
||||
`*.attention.key_length`/`value_length`, `*.attention.sliding_window`,
|
||||
`*.context_length`.
|
||||
|
||||
## 2. The drafter's KV is sized from the TARGET
|
||||
|
||||
`common/speculative.cpp` builds the draft context with:
|
||||
|
||||
```cpp
|
||||
// the draft context holds as many tokens per sequence as the target context
|
||||
cparams.n_ctx = llama_n_ctx(ctx_tgt);
|
||||
```
|
||||
|
||||
There is **no flag** to cap it (the `--spec-draft-*` surface has none). Therefore:
|
||||
|
||||
```
|
||||
draft KV bytes = ctx_tgt x block_count x 2 (K,V) x head_count_kv x key_length x dtype
|
||||
```
|
||||
|
||||
*unless* the drafter declares a sliding window, in which case the window bounds it.
|
||||
|
||||
Measured on the same 262144-token target: a sidecar with `sliding_window = 2048` cost
|
||||
~40 MiB of draft KV; a sidecar of the same 5-layer shape with **no** window cost
|
||||
**5.0 GiB** (20 KiB/position). Swapping between them invalidated an already-balanced
|
||||
layer split.
|
||||
|
||||
**Failure signature:** the engine loads the whole model, answers `/health`, and dies on
|
||||
the FIRST draft decode with an allocation error (`ROCm error: out of memory` /
|
||||
`cudaMalloc failed`) inside `common_speculative_impl_draft_dflash::process`. An
|
||||
allocation failure *after* a successful load is a sizing failure, not a broken drafter
|
||||
-- and under a self-healing unit it looks like a crash-loop instead.
|
||||
|
||||
## 3. A bigger batch is asymmetric across the split
|
||||
|
||||
`--ubatch-size` is the physical batch the graph is built at, so it sets the compute
|
||||
buffers. Those buffers are **~2x heavier on the card that holds the output layers and
|
||||
the drafter's graphs** than on the other(s). Measured, 2 GPUs at 262144 ctx:
|
||||
|
||||
| ubatch | output/drafter card | other card |
|
||||
|---|---|---|
|
||||
| 512 -> 1536 | +~3.3 GiB | +~1.7 GiB |
|
||||
|
||||
Consequence: **a split balanced at one batch is unbalanced at another.** A split at
|
||||
1:1 or a hand-measured skew must be re-measured after any change to `--ubatch-size`,
|
||||
`--ctx-size` or `--parallel`.
|
||||
|
||||
## 4. Find the balance by measurement, not arithmetic
|
||||
|
||||
The per-step cost is not derivable from the parameter count (KV lands only on the
|
||||
attention layers, and buffer placement is per-device). One `-ts` step is worth roughly
|
||||
1.5 GiB of load at ~6k batch / 262k ctx on 30 GiB-class cards — enough to bracket, not
|
||||
to solve. Procedure:
|
||||
|
||||
1. Copy the env file to a scratch path and run the serve script against it
|
||||
(`SERVICE_ENV_FILE=/tmp/test.env bash scripts/serve-<svc>.sh`) rather than editing
|
||||
the deployed file. Do it with the unit stopped — one GPU, one process.
|
||||
2. Bracket: try a step, read `rocm-smi --showmeminfo vram` for both cards after a real
|
||||
generation, and step again. Three points is usually enough to land within ~0.1 GiB.
|
||||
3. Prefer the balanced split, because the busier card's headroom is what bounds the
|
||||
context:
|
||||
|
||||
```
|
||||
ctx_extra ~= (headroom_target - measured_free) / (per-position cost on the busier card)
|
||||
```
|
||||
|
||||
where per-position cost = the card's share of the target's KV + the drafter's KV if
|
||||
that card carries it. A balanced split therefore MAXIMISES ctx for a given floor.
|
||||
4. Verify with the full battery before committing: `/health`, `/v1/models`, a real
|
||||
completion, a tool call, two concurrent long prompts (one per slot), `slots`
|
||||
reporting the expected `n_ctx` per slot, and the per-task `draft acceptance = ...`
|
||||
line per slot.
|
||||
|
||||
## 5. Keep the two failure signatures apart
|
||||
|
||||
| Signature | Meaning |
|
||||
|---|---|
|
||||
| loads, `/health` ok, first decode OOMs | sizing: drafter KV, batch buffers, or KV vs ctx (section 2) |
|
||||
| `SDMA0 ... page fault` + `illegal memory access` + `ggml_abort` | a GPU fault in the cross-card boundary copy — the layer split's hot path. **Not** an OOM; headroom does not prevent it |
|
||||
|
||||
The second one is why a rebalance should be validated by a request battery and not just
|
||||
by `rocm-smi` numbers.
|
||||
|
||||
## Ops traps
|
||||
|
||||
- A server started over ssh (including under a supervisor/`hub`-style runner) **outlives
|
||||
stopping that runner** — the remote process keeps the port, and the next start fails
|
||||
with `couldn't bind HTTP server socket`. `pkill -x <exact-binary-name>` first.
|
||||
- `systemctl start` of a root unit from a non-root ssh session fails with
|
||||
`Access denied ... requires interactive authentication`: use sudo.
|
||||
- An old crash's `ROCm error` lines remain in `journalctl -u <svc>` forever. Check the
|
||||
current instance's `ActiveEnterTimestamp`/`NRestarts` before treating a grep count as
|
||||
a live fault.
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
name: llamacpp-layer-split-rdna4
|
||||
description: "Serve one model across two (or more) AMD RDNA4/ROCm GPUs on a host where RCCL cannot dispatch collectives — mainline llama.cpp built for gfx1201 with a bundled ROCm runtime, --split-mode layer, DFlash2/MTP speculative decoding, and the verification that proves each layer works"
|
||||
---
|
||||
|
||||
# llama.cpp layer-split serving on RDNA4 (gfx1201)
|
||||
|
||||
Use when one model must use two AMD GPUs whose interconnect cannot carry a
|
||||
collective (no PCIe atomic routing on one card's path), or when a host has no
|
||||
usable RCCL. The shape: ONE llama-server process, `--split-mode layer` across
|
||||
both cards, spec decode from a drafter GGUF or the model's own MTP head.
|
||||
|
||||
## When to reach for this
|
||||
|
||||
- vLLM cannot help: TP needs a per-layer allreduce and **PP's stage handoff is a
|
||||
device-backend `send`/`recv`** (`vllm/distributed/parallel_state.py`,
|
||||
`send_tensor_dict` → tensor handles "on the device backend"), i.e. the same
|
||||
RCCL primitive that fails on an atomics-less card. Verify with
|
||||
`sudo lspci -vv -s <gpu> | grep AtomicOpsCtl` (want `ReqEn+`) and
|
||||
`lspci -tv` (a `Routing-` bridge is a hardware wall).
|
||||
- Confirming the wall: any RCCL kernel launch on that card returns
|
||||
`hipErrorIllegalState` with `AMD_LOG_LEVEL=3` printing
|
||||
`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`.
|
||||
A single-card test proves nothing (NCCL short-circuits at world_size=1).
|
||||
- **Never run a multi-GPU collective probe beside a live service** on these
|
||||
hosts: stop and mask it first, or use a host that serves nobody.
|
||||
|
||||
## Build (pin the ref, bundle the runtime)
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
||||
git checkout --detach <ref> # pin; do not float
|
||||
# ROCm toolchain: system ROCm or a TheRock nightly tarball, e.g.
|
||||
# https://rocm.nightlies.amd.com/tarball-multi-arch/therock-dist-linux-gfx120X-all-<ver>.tar.gz
|
||||
ROCM=$HOME/.local/share/rocm-<ver>
|
||||
cmake -S . -B build-rdna4 -G Ninja -DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_C_COMPILER=$ROCM/llvm/bin/clang -DCMAKE_CXX_COMPILER=$ROCM/llvm/bin/clang++ \
|
||||
-DCMAKE_CXX_FLAGS="-I$ROCM/include" -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||
-DGGML_HIP=ON -DGGML_HIP_RCCL=OFF -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
|
||||
-DCMAKE_HIP_ARCHITECTURES=gfx1201 -DGPU_TARGETS=gfx1201 \
|
||||
-DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF -DBUILD_SHARED_LIBS=OFF \
|
||||
-DCMAKE_PREFIX_PATH=$ROCM -DROCM_DIR=$ROCM
|
||||
cmake --build build-rdna4 -j $(nproc) --target llama-server llama-cli llama-bench
|
||||
patchelf --set-rpath '$ORIGIN/../lib' build-rdna4/bin/llama-*
|
||||
```
|
||||
|
||||
- `-DGGML_HIP_RCCL=OFF` is deliberate: with RCCL off, only `--split-mode tensor`
|
||||
degrades (butterfly fallback); layer split is unaffected.
|
||||
- **Bundle device libraries at the exact depth the runtime looks for.** Copy
|
||||
`$ROCM/lib/rocblas/library/` → `build/lib/rocblas/library/` and
|
||||
`$ROCM/lib/hipblaslt/library/` → `build/lib/hipblaslt/library/` with
|
||||
`mkdir -p DEST && cp -r SRC/. DEST/`. `cp -r SRC DEST/` with a missing DEST
|
||||
flattens one level and the failure appears at REQUEST time, not load time
|
||||
(`Cannot read .../lib/rocblas/library/TensileLibrary.dat`).
|
||||
- Build only the arch you own. A sibling-arch binary (gfx1200 on gfx1201)
|
||||
links, loads the model, then segfaults.
|
||||
- Verify: `build-rdna4/bin/llama-server --version` and
|
||||
`--list-devices` (expect both cards; an iGPU can enumerate as a third).
|
||||
|
||||
## Serve
|
||||
|
||||
```bash
|
||||
HIP_VISIBLE_DEVICES=0,1 llama-server \
|
||||
--model <target.gguf> --alias <fleet-name> --host 0.0.0.0 --port 7770 \
|
||||
--device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
|
||||
--gpu-layers 999 --fit off --flash-attn on \
|
||||
--ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
|
||||
--jinja --metrics --cache-prompt \
|
||||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||||
--spec-type draft-dflash --spec-draft-model <dflash2-q8_0.gguf> --spec-draft-n-max 7
|
||||
```
|
||||
|
||||
- `--fit off` + an explicit `--ctx-size`: auto-fit must not resize a context
|
||||
you set on purpose. `--parallel N` DIVIDES `--ctx-size`; it does not multiply.
|
||||
- Spec decode choices: `--spec-type draft-dflash` with a DFlash2 GGUF (the
|
||||
drafter's own metadata selects the DFlash2 path — the init line prints
|
||||
`block_size=8, n_extract=5, sample_from_anchor=true`); `n_max <= block_size-1`.
|
||||
Or `draft-mtp` for a model whose GGUF carries `blk.<n>.nextn.*` (no draft
|
||||
file; the head loads from the main weights). Types can be chained
|
||||
(`draft-mtp,draft-dflash`). With `draft-dflash`, the target's `nextn`
|
||||
tensors log as "unused tensor ... ignoring" — that is correct, not a broken
|
||||
download (check bytes with the fetch script's `--verify-only`).
|
||||
- Layer split puts the drafter on the card holding the target's last layers
|
||||
(`--spec-draft-device ROCm1`); under tensor mode an unpinned draft joins the
|
||||
meta-device KV allocation and aborts the KV build.
|
||||
|
||||
## Verify (each layer separately)
|
||||
|
||||
1. Load: `n_ctx_slot` in the log matches `--ctx-size`; `/health` answers.
|
||||
2. Both cards: `rocm-smi --showmeminfo vram` — each card holds roughly half the
|
||||
weights; the busier one also carries the drafter. Compare against
|
||||
`model_bytes/2 + drafter + ctx*64KiB/2 + ~2 GiB` (qwen35 hybrid: only its
|
||||
full-attention layers take KV, 64 KiB/token at f16).
|
||||
3. **Spec decode is live**: `curl /metrics | grep spec_decode_num_drafts_total`
|
||||
and `..._accepted_tokens_total` (their ratio is acceptance); a request's
|
||||
`timings.draft_n` / `draft_n_accepted` gives it per call. Drafts stuck at 0
|
||||
with tokens flowing means the drafter is not running.
|
||||
4. Real completion through the fleet alias, then a deep prompt (a 140k-token
|
||||
prompt proves a raised window is real, not just declared).
|
||||
5. Prefix cache: repeat the same prompt — `timings.prompt_n` should collapse to
|
||||
a handful of tokens.
|
||||
|
||||
## Pitfalls that cost real time
|
||||
|
||||
- A healthy-looking build that dies on the first request is almost always the
|
||||
bundled-library depth above.
|
||||
- Serve scripts that `source` an env file AFTER defaults cannot be overridden
|
||||
from the command line; override the env-file PATH instead.
|
||||
- Qwen3.8 chat templates RAISE on `reasoning_effort=high` (500). Probe
|
||||
low/medium/high/xhigh before declaring an endpoint ready, and map the
|
||||
harness's levels onto the supported set.
|
||||
- One slot at a large context is the concurrency limit: concurrent agent loops
|
||||
queue, and first-turn TTFT then includes the wait.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: llamacpp-layer-split-vram-rebalance
|
||||
description: "Re-measure and set the --tensor-split VRAM balance for a multi-GPU llama.cpp server when the batch, context, slots or spec drafter change, instead of interpolating between splits."
|
||||
---
|
||||
|
||||
Re-balancing is needed whenever a knob moves weight between cards: `--ubatch-size` (compute buffers), a spec drafter (weights + draft KV), `--ctx-size`/`--parallel` (KV), or a different drafter with a different window policy. A split balanced at one configuration is unbalanced at another — and the second card's asymmetry (output layers + drafter graphs) is what makes it non-obvious.
|
||||
|
||||
## 0. Stop the service first
|
||||
|
||||
Serving a user? Stop it, then free the port, before loading test instances. Do not bench beside a live server.
|
||||
|
||||
```bash
|
||||
sudo systemctl stop <svc> # root unit; for a rootless quadlet use systemctl --user
|
||||
pkill -x llama-server && sleep 3 # a manual/child run holds the port and the VRAM
|
||||
```
|
||||
|
||||
`pkill -f <pattern>` is forbidden here: it matches your own command line and kills the ssh session. `mask` FAILS on a unit that is a symlink into a repo checkout (`File ... already exists and is a symlink to ...`); a `stop` is enough because nothing else restarts it.
|
||||
|
||||
## 1. Measure one split per load
|
||||
|
||||
The serve script is env-driven, so point it at a copy of the env file with `TENSOR_SPLIT` patched and read the allocation back from `rocm-smi`. Load -> sample -> kill, with NO requests sent: this touches allocation only, not the compute path.
|
||||
|
||||
```bash
|
||||
WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
|
||||
export FRANK_MEDIUM_ENV_FILE=$WORK # the script's own override hook
|
||||
for ts in 39,26 38,27 37,28 36,29 35,30; do
|
||||
sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
|
||||
nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
|
||||
for _ in $(seq 1 150); do
|
||||
[ "$(curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:PORT/health)" = 200 ] && break
|
||||
pgrep -x llama-server >/dev/null || break
|
||||
sleep 2
|
||||
done
|
||||
python3 -c "
|
||||
import subprocess,re
|
||||
CAP=34208743424 # one card, as rocm-smi reports it
|
||||
o=subprocess.run(['rocm-smi','--showmeminfo','vram'],capture_output=True,text=True).stdout
|
||||
u=[int(x) for x in re.findall(r'VRAM Total Used Memory \\(B\\): (\\d+)',o)][:2]
|
||||
print('$ts', f'{(CAP-u[0])/2**30:.2f}', f'{(CAP-u[1])/2**30:.2f}')"
|
||||
pkill -x llama-server; sleep 6
|
||||
done
|
||||
```
|
||||
|
||||
Sample ~6 s after `/health` returns 200 — the last allocations land after listening. Run it under `nohup` and poll the log from a separate call: a load is 30-90 s, which exceeds a single command's patience.
|
||||
|
||||
## 2. Read the table, do not interpolate
|
||||
|
||||
Pick the split with the smallest inter-card gap, then confirm it clears any per-card floor (e.g. 1.5 GiB) with room to spare. **A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.** Measured on qwen35 (16 x (3 Gated DeltaNet + 1 Gated Attention)): 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one (weights + that layer's own f16 KV). So the gap can jump 2.9 GiB in a single step, and the best integer split is ~0.2 GiB from balanced while its neighbours are 1+ GiB off. Interpolating between two measured splits produces a split nobody should deploy.
|
||||
|
||||
Total free VRAM is invariant across splits (only its distribution moves) — use that to sanity-check a suspicious row.
|
||||
|
||||
## 3. Check the drafter's window before blaming the split
|
||||
|
||||
A draft context is built with `n_ctx = llama_n_ctx(ctx_tgt)` and there is no flag to cap it, so a drafter declaring NO sliding window caches the WHOLE target window on `--spec-draft-device`. DFlash2 (`sliding_window 2048`) = ~40 MiB; DSpark (no window) = 20 KiB/position = 5.0 GiB at 262144 — a four-layer swing in the balance point. Read `sliding_window` out of the drafter's GGUF header before sizing, and prefer a skewed `-ts` over cutting `--ctx-size`.
|
||||
|
||||
## 4. If the repo has a fitted VRAM model, refit it — with the same unit convention
|
||||
|
||||
Match the existing expression literally (e.g. `1442 * 1024 * ubatch` means ~1.4 MB per ubatch unit, not 1442 bytes). Fit so the deployed point reproduces `rocm-smi` to ~0.05 GiB, keep the physics (weights + proportional KV + a window-bounded draft KV + ubatch-scaled buffers), and treat a NEGATIVE fitted constant as a unit bug. State in the comment where the model is conservative: a uniform per-layer cost overstates gaps in a hybrid stack, and a fitted ubatch coefficient makes any larger-batch verdict an extrapolation, not a measurement.
|
||||
|
||||
## 5. Verify behaviourally after deploy, not just by flags
|
||||
|
||||
- `/slots` -> the expected `(id, n_ctx)` pairs.
|
||||
- The drafter's init lines from `journalctl -u <svc>`: implementation name, `n_max`, `block_size` — read the NUMBERS, since DFlash2 and DSpark log under the same `draft_dflash` symbol.
|
||||
- `rocm-smi` free VRAM at the deployed split (the fitted prediction and the real figure should agree to ~0.05 GiB).
|
||||
- One tool call and two CONCURRENT requests, one per slot, then `journalctl -o cat | grep 'slot print_timing' | grep -o 'slot print_timing: id *[01].*draft acceptance.*'` to prove drafting is live on BOTH slots (`journalctl` default output truncates these lines; `-o cat` plus `grep -o` gets them whole).
|
||||
- `grep -c 'ROCm error\|illegal memory\|ggml_abort'` on the current instance must be 0.
|
||||
|
||||
## Faults to watch, not to tune away
|
||||
|
||||
A cross-card boundary copy can fault (`amdgpu SDMA0 UTCL2 page fault` -> `ROCm error: an illegal memory access was encountered` -> `ggml_abort`) on a split that loads and answers `/health` perfectly. It is not an OOM — more headroom does not prevent it — so treat a split that has faulted as off-limits and re-measure rather than re-tuning. Also: a load that succeeds and then dies on the FIRST draft decode with an OOM is a sizing failure (the draft KV), not a broken drafter.
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
name: llamacpp-mtp-vision-endpoint-bringup
|
||||
description: "Bring up a new llama.cpp (mainline) OpenAI endpoint on a single CUDA GPU with MTP speculative decoding and vision — GGUF header preflight by HTTP range, hybrid-architecture KV arithmetic, the -mtp / --spec-type draft-mtp contract, --parallel vs ctx math, and the verification battery that proves MTP, tools and vision actually work."
|
||||
---
|
||||
|
||||
# llama.cpp MTP + vision endpoint bring-up (single CUDA GPU)
|
||||
|
||||
Use when standing up or swapping a mainline llama.cpp endpoint for a GGUF that
|
||||
may ship an MTP (NextN) head and a vision projector. Verified end to end on
|
||||
Dankclaw 2026-09-16 (RTX 3090 24 GiB, Qwen3.8-27B-GSQ-RCO IQ3_S `-mtp` +
|
||||
BF16 mmproj, 131072 ctx / 1 slot, port 7771).
|
||||
|
||||
## 1. Preflight the GGUF WITHOUT downloading it
|
||||
|
||||
GGUF metadata lives in the header, and HuggingFace honours ranged GETs, so a
|
||||
few MB answers every sizing question before a 13 GB transfer:
|
||||
|
||||
```python
|
||||
import urllib.request
|
||||
req = urllib.request.Request(url, headers={"Range": "bytes=0-8000000", "User-Agent": "curl/8"})
|
||||
buf = urllib.request.urlopen(req, timeout=180).read()
|
||||
```
|
||||
|
||||
Parse: magic `GGUF`, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs
|
||||
(string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte —
|
||||
the type hand-written parsers forget). Walk arrays by byte size even when you
|
||||
only want a summary: a naive parser desynchronises on `tokenizer.ggml.tokens`
|
||||
and lands gigabytes off.
|
||||
|
||||
Read these keys: `general.architecture`, `<arch>.block_count`,
|
||||
`<arch>.context_length`, `<arch>.nextn_predict_layers`, `<arch>.head_count_kv`,
|
||||
`<arch>.key_length` / `value_length`, `<arch>.full_attention_interval`,
|
||||
`<arch>.ssm.state_size` / `group_count` / `time_step_rank`, and for the
|
||||
projector file `clip.projector_type` (e.g. `qwen3vl_merger`) plus
|
||||
`clip.has_vision_encoder`.
|
||||
|
||||
## 2. Decide the file from `nextn_predict_layers`
|
||||
|
||||
**MTP is a property of the FILE, not of the flags.** If the release ships an
|
||||
`-mtp` build, that build carries the head as `blk.<n>.nextn.*` (a few hundred
|
||||
MB larger); the plain build has no MTP tensors at all.
|
||||
|
||||
With the `-mtp` file, llama.cpp needs only `--spec-type draft-mtp`: it sets
|
||||
`load_mtp` and the draft context runs on the MAIN model's weights
|
||||
(`shares_model`, `LLAMA_CONTEXT_TYPE_MTP`). Do NOT add `--model-draft` — that
|
||||
loads the whole file a second time (12 GiB twice on a 24 GiB card). A separate
|
||||
`-md` is only correct for standalone MTP-only sidecars.
|
||||
|
||||
Confirm the flag exists in the deployed build before writing the script:
|
||||
`grep -rn "common_speculative_all_types_str" common/speculative.cpp` and
|
||||
`llama-server --help | grep -A4 spec-type`.
|
||||
|
||||
## 3. KV arithmetic (do this before choosing ctx)
|
||||
|
||||
Only **full-attention** layers take KV. For a hybrid arch (Qwen3-Next/3.8 style
|
||||
`16 x (3 x Gated DeltaNet + 1 x Gated Attention)`):
|
||||
|
||||
```
|
||||
attn_layers = block_count / full_attention_interval
|
||||
bytes/token = attn_layers x n_head_kv x head_dim x 2 (K,V) x bytes_per_elem
|
||||
f16 = 2 B | q8_0 = 1.0625 B | q4_0 = 0.5625 B
|
||||
```
|
||||
|
||||
The linear-attention layers hold constant-size per-sequence state instead
|
||||
(~0.16 GiB per sequence at 48 layers / 128 d_state / 48 v-heads in f32), and
|
||||
the MTP draft context costs its own `n_ctx x (1 layer x n_head_kv x head_dim x
|
||||
2 x 2 B)` ≈ 4 KiB/token for a 1-layer head.
|
||||
|
||||
VRAM budget: weights + mmproj + MTP draft KV + DeltaNet state + ~1.5 GiB of
|
||||
CUDA context and compute buffers. On a 24 GiB card that leaves ~9.5 GiB for KV,
|
||||
which at 64 KiB/token is ~155k tokens total.
|
||||
|
||||
**`--parallel N` DIVIDES `--ctx-size` between slots** — it does not multiply.
|
||||
`-np 1 -c 131072` is one 131k conversation; `-np 4 -c 131072` is four 32k ones.
|
||||
|
||||
## 4. Flags that matter
|
||||
|
||||
- `-ngl 999`, `-fa on`, `-ctk/-ctv f16` for the usual shape.
|
||||
- Quantized KV (`q8_0`, `q4_0`) requires a build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`; without it CUDA prefill collapses ~12x. f16
|
||||
needs nothing, so prefer it unless the context demand is real.
|
||||
- `--jinja` — pin it explicitly even when the build defaults it on: the GGUF's
|
||||
own template supplies `reasoning_effort`, `preserve_thinking` and the
|
||||
tool-call grammar, and a future default flip must not silently remove tools.
|
||||
- `--metrics` if anything scrapes `/metrics`, `--alias` to set the served id,
|
||||
`--temp 0.0` for a greedy-by-default fleet contract,
|
||||
`--reasoning-format deepseek --reasoning-budget -1` to project thinking into
|
||||
`reasoning_content`.
|
||||
|
||||
## 5. Serve-script and unit invariants
|
||||
|
||||
Fail loudly before loading VRAM: check the binary, the model file and the
|
||||
mmproj exist, and `exit 1` with a named message otherwise — a missing weight
|
||||
must not become a `Restart=on-failure` crash loop that reads like a runtime
|
||||
fault. For a native CUDA binary the unit is a ROOT systemd unit
|
||||
(`[Install] WantedBy=multi-user.target`, `Restart=on-failure`,
|
||||
`RestartSec=10`, `LimitMEMLOCK=infinity`); a container would be a Quadlet.
|
||||
|
||||
## 6. Verification battery (prove each claim, do not infer)
|
||||
|
||||
1. `/health` 200, and `/v1/models` reports the intended alias.
|
||||
2. A chat completion returns `content` AND `reasoning_content`.
|
||||
3. A request carrying `tools` returns `tool_calls` — a RUNTIME property: the
|
||||
ik_llama.cpp runtime answered `500 tools param requires --jinja flag`.
|
||||
4. `GET /metrics` after a generation: `llamacpp:spec_decode_num_draft_tokens_total`
|
||||
and `..._accepted_tokens_total` both > 0, acceptance 0.6-0.95 ⇒ MTP engaged,
|
||||
not merely configured. The slot timing lines in the journal say the same.
|
||||
5. One image request (`data:` URL or bare base64) ⇒ an answer that reflects the
|
||||
image; confirm the journal printed a multimodal projector load.
|
||||
6. `nvidia-smi` memory used is close to the modelled total, and `n_slots` /
|
||||
`n_ctx_slot` in the load log match the intended shape.
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **A template can reject the client's thinking level.** Qwen3.8 accepts only
|
||||
`xhigh|medium|low` for `reasoning_effort` and raises otherwise — HTTP 500
|
||||
naming Jinja, not the model. Map every harness level onto the accepted set
|
||||
(e.g. `compat.reasoningEffortMap: {minimal: low, low: low, medium: medium,
|
||||
high: xhigh, max: xhigh}`) and probe each level with curl before routing
|
||||
roles at the endpoint.
|
||||
- **Thinking counts against `max_tokens`.** A budget that runs out mid-thought
|
||||
returns `finish_reason: "length"` with empty `content` — not a broken
|
||||
endpoint, not a vision failure. Raise the budget.
|
||||
- **Never point `--model-draft` at the `-mtp` file** (step 2).
|
||||
- **Any `-ot` / `-ncmoe`-style override can silently disable mmap** for the
|
||||
whole load, turning a mapped model into anonymous memory.
|
||||
- On a host that publishes ports through **pasta**, the replacement must not
|
||||
start until the predecessor released the port: poll
|
||||
`ss -tan "sport = :<port>"` (listener, accepted AND TIME_WAIT) or the
|
||||
container dies with `Address already in use`.
|
||||
- On a shared host, read the serving engine's own boot ledger for device
|
||||
budget vs need: a pool that silently clamped still answers a green `/health`.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: llamacpp-multigpu-layer-split-cutover
|
||||
description: "Cut a fleet LLM endpoint over to llama.cpp layer-split across multiple GPUs when cross-GPU collectives are impossible (RCCL/PCIe-atomics wall), including the build recipe, the bundling trap that fails on the first request, DFlash2/MTP wiring, and the measurement battery that proves the split and the drafter are real."
|
||||
---
|
||||
|
||||
# llama.cpp layer-split cutover (no-collective multi-GPU)
|
||||
|
||||
Use when one model must serve from TWO+ cards and the box cannot dispatch a
|
||||
cross-card collective (second card behind a chipset bridge with
|
||||
`AtomicOpsCap: Routing-`, so ROCr refuses RCCL's hostcall-buffer kernels).
|
||||
Decide with one line before designing anything:
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <card2> | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever
|
||||
```
|
||||
|
||||
`ReqEn-` kills vLLM entirely — tensor parallelism needs an allreduce **and
|
||||
pipeline parallelism needs an RCCL `send`/`recv` of the hidden state**
|
||||
(`vllm/distributed/parallel_state.py`: metadata on gloo, tensors on the device
|
||||
backend). llama.cpp's `--split-mode layer` needs no collective: layers are
|
||||
distributed in order, only the boundary activation crosses PCIe once per
|
||||
forward pass.
|
||||
|
||||
## 1. Pin the runtime by FEATURE, not by "latest"
|
||||
|
||||
Write down which features the deployment needs, then find the ref that has all
|
||||
of them and pin the 40-hex sha in the build script. On 2026-09-16 that was
|
||||
DFlash2 (`ggml-org/llama.cpp#27342`, detected from the draft GGUF:
|
||||
`selector_top_k > 0`, spec type stays `draft-dflash`) and qwen35 MTP. The
|
||||
ROCmFPX/r9700-toolbox images/fork tips predate both — verify by listing the
|
||||
built image:
|
||||
|
||||
```bash
|
||||
ls <venv>/lib/python*/site-packages/vllm/v1/worker/gpu/spec_decode/ # dflash/ vs dflash2/
|
||||
```
|
||||
|
||||
## 2. Build self-contained for one arch
|
||||
|
||||
- Private ROCm toolchain (TheRock nightly) with `ROCM_PATH`/`HIP_PATH` exported,
|
||||
`-DCMAKE_HIP_ARCHITECTURES=<gfx only>` (a sibling arch links, loads and then
|
||||
segfaults), `-DGGML_HIP_RCCL=OFF` when there is no atomics path, `-DGGML_NATIVE=OFF`.
|
||||
- Bundle the runtime into `build/lib` and set `--set-rpath '$ORIGIN/../lib'`.
|
||||
- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist.** rocBLAS and
|
||||
hipBLASLt look for device code at a FIXED depth (`../lib/rocblas/library`,
|
||||
`../lib/hipblaslt/library/<arch>`). Flattened, the engine loads every GiB of
|
||||
weights, answers `/health`, and dies on the **first request** with
|
||||
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch`.
|
||||
`mkdir -p DST && cp -r SRC/. DST/`.
|
||||
|
||||
## 3. Deploy as an env + serve script + unit trio
|
||||
|
||||
One env file holds every tunable with the sizing arithmetic in comments; the
|
||||
serve script defaults each var, refuses to start when the runtime or payload is
|
||||
missing, and builds an args array (spec flags assembled conditionally so the
|
||||
same script serves `draft-dflash` with a file and `draft-mtp` in-model). Root
|
||||
systemd unit: `User=<deploy user>`, `LimitMEMLOCK=infinity`,
|
||||
`Restart=on-failure`, `WantedBy=multi-user.target`, plus an
|
||||
`Environment=…_ENV_FILE=<repo path>` so the repo file is the single source.
|
||||
|
||||
Deploy handler must: refuse when runtime/payload is missing; retire the previous
|
||||
runtime's units (into trash, never `rm`); **wait for the port to be released**
|
||||
(a retired reverse proxy or container holds it); then enable/restart and gate on
|
||||
`/health`.
|
||||
|
||||
DRLaf: `--split-mode layer --device ROCm0,ROCm1 --tensor-split 1,1`,
|
||||
`--parallel 1`, f16 KV, `--fit off`. Pin the drafter with `--spec-draft-device`
|
||||
to the card holding the target's last layers. `--spec-draft-n-max` must be
|
||||
<= the draft's `block_size - 1` (DFlash2: 8 ⇒ 7).
|
||||
|
||||
## 4. Prove it, do not assert it
|
||||
|
||||
- **The split is real**: raise `--ctx-size` by N tokens and watch `rocm-smi` —
|
||||
a layer split adds ~N·(bytes/token)/2 to EACH card (both cards carry the KV
|
||||
for their own attention layers), and neither card's TOTAL can equal the model
|
||||
file, which is the arithmetic that rules out replication.
|
||||
- **The drafter is live**: `timings.draft_n`/`draft_n_accepted` per request, and
|
||||
the cumulative `llamacpp:spec_decode_num_drafts_total` — the flags can be
|
||||
right while the draft model silently failed to load. The loader's init line
|
||||
(`block_size=`, `n_extract=`, `mask_token_id=`) identifies the drafter family.
|
||||
- **Watch for a 500 on `reasoning_effort`**: Qwen3.8-family chat templates
|
||||
accept only low/medium/xhigh and RAISE on `high`; map every harness level or
|
||||
subagents 500 on their first request.
|
||||
- **Re-point the monitors in the same change**: a runtime swap changes the
|
||||
metric namespace (`vllm:` → `llamacpp:`), and rules/dashboard panels naming
|
||||
series that no longer exist draw nothing forever without erroring. Retire
|
||||
what cannot be expressed (llama.cpp exports no KV-fill gauge and no latency
|
||||
histogram) and name the consequence instead (queue depth, spec-decode-off).
|
||||
- One slot is a concurrency policy, not a bug: `--parallel` DIVIDES the context.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: llamacpp-vram-accounting
|
||||
description: "Produce and interpret a per-device VRAM/memory breakdown for a llama.cpp or halogen LLM deployment on the Dank fleet: we/weights vs KV vs compute pool, discrete vs UMA accounting, and the measurement traps that make the numbers lie."
|
||||
---
|
||||
|
||||
# llama.cpp / halogen VRAM accounting (Dank fleet)
|
||||
|
||||
Use when asked *where the memory went* for an LLM service on Dankclaw/Dankserver: "what's the VRAM breakdown", "why does it not fit", "how much headroom is left for the co-tenants", or before sizing a card for a new model.
|
||||
|
||||
## 0. Decide which kind of memory you are talking about
|
||||
|
||||
| device | kind | read it with |
|
||||
|---|---|---|
|
||||
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader` |
|
||||
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (real, 114 GiB) vs `mem_info_vram_{total,used}` (2 GiB carve-out window) |
|
||||
| R9700s (Dankserver) | discrete | rocm-smi figure is authoritative (64 GiB claimed, SVM-backed) |
|
||||
|
||||
Per-process truth on NVIDIA: `nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader`.
|
||||
|
||||
**Never** add the iGPU's GTT to host RAM as if they were separate pools, and never quote a UMA engine's total as "VRAM": halogens's 95.4 GiB is 68 GiB of **pinned host pages** (mlock + page cache) plus ~33.7 GiB inside the device budget. `free -g` is the arbiter for what the co-tenants have left.
|
||||
|
||||
## 1. Get the engine's own ledger first
|
||||
|
||||
```bash
|
||||
podman logs frank-large 2>&1 | grep -aiE "weights pinned|kv pool|left for everything|overlay|vision|budget"
|
||||
```
|
||||
|
||||
Halogen prints the numbers that matter: weights pinned (and how many contiguous 2 MiB blocks remain), the device budget (`host RAM − resident weights − reserve`), the KV pool size and pool-vs-working split, and the final "left for everything else" block count. Contiguous-block count is the honest fragmentation signal — 240 blocks / 12.3 GiB after a reboot vs 475 / 14.7 GiB before means the same model did not come back into the same hole.
|
||||
|
||||
## 2. For llama.cpp: the print sites exist, but no artifact here has them
|
||||
|
||||
- `src/llama-model.cpp:1827` — per-backend `model buffer size` (unconditional `LLAMA_LOG_INFO`)
|
||||
- `src/llama-context.cpp:692` — `compute buffer size`
|
||||
- plus a KV self-size line at context creation
|
||||
|
||||
Traps verified on 2026-09-16:
|
||||
|
||||
- **`llama-bench` suppresses the loader's INFO** — its logs contain only the device banner and the results table, so a bench matrix leaves *no* memory accounting behind.
|
||||
- **`llama-server` under systemd did not put them in the journal either** — `sudo journalctl -u frank-small | grep -c "model buffer size"` returned **0** for the unit's entire history. Check with `grep -c` before claiming a log has the breakdown.
|
||||
- Recipe that works: launch the model **once** through `llama-cli` (or `llama-server`) and keep the first ~200 lines; add `-v` if the level filters them.
|
||||
|
||||
## 3. Otherwise compose it — arithmetic on measured tensor sizes
|
||||
|
||||
Read the GGUF's tensor table once (names + shapes are enough) and group by role, e.g. Flash-Next (111 GiB):
|
||||
|
||||
| group | size | per layer |
|
||||
|---|---|---|
|
||||
| `ffn_{gate,up,down}_exps` | 75.0 GiB | **1.5625 GiB** (experts travel with their layer) |
|
||||
| `per_layer_token_embd` (PLE) | 32.8 GiB | one table |
|
||||
| everything else (attn/GDN/etc) | 3.2 GiB | 0.067 GiB |
|
||||
|
||||
And KV ≈ 2 KiB per position per f16 attention layer (12 attention layers on Flash-Next => ~24 KiB/position; halogen's ledger says ~28 KiB/position including block scratch). Multiply by the *derived* context (see §5).
|
||||
|
||||
Then take the measured per-device total as the constraint and let the remainder be compute/graph scratch. Vendor delta is real and sizeable: same 7 layers + KV on the 3090 came to **12.80 GiB under Vulkan** and **14.99 GiB under CUDA** — ~3.3 GiB vs ~1.1 GiB of graph/compute pool.
|
||||
|
||||
## 4. Interpretation traps
|
||||
|
||||
- **mmap PLE is reclaimable; `-ot` makes it not.** With everything on the iGPU the 32.8 GiB PLE sits in page cache (reclaimable). In a layer split, where `-ot` is disallowed, it becomes a pinned device buffer in GTT. "Move layers to the card" therefore converts reclaimable cache into pinned device memory and does not free room for co-tenants.
|
||||
- **A served model's footprint is not its weight size.** frank-small: 22,582–22,592 MiB of 24,576 for an 11.29 GiB model — weights + f16 KV at 131,072 × 1 slot + BF16 projector + CUDA compute pool.
|
||||
- **"Usable" is total minus driver/container overhead** (~24.1 GiB on a 24,576 MiB card).
|
||||
- Do the capacity subtraction *before* designing: 12 attention layers × 1.62 GiB = 19.4 GiB, plus 200k f16 KV (9.2 GiB) = 28.6 GiB > 24.1 — which is why a KV-on-VRAM design on this pair is only feasible to ~64k context.
|
||||
|
||||
## 5. Context lengths: derive, never remember
|
||||
|
||||
`llama-bench` never logs the context; it computes (`tools/llama-bench/llama-bench.cpp:1293`):
|
||||
|
||||
```c
|
||||
cparams.n_ctx = n_prompt + n_gen + n_depth;
|
||||
```
|
||||
|
||||
Defaults: `n_prompt 512`, `n_gen 128`, `n_depth 0`, `n_batch 2048`, `n_ubatch 512`, `type_k`/`type_v` **F16**, `flash_attn AUTO`, `split_mode LAYER`, `reps 5`.
|
||||
|
||||
So `-p 512,16384 -n 64` = contexts **576 / 16,448 / 64**, and `-n 32 -d 65536` = **66,080 / 65,568**. A `tg` row runs with `n_prompt = 0`, which is why a plain `tg` number is ~zero depth: `-d` is the only thing that puts KV in the picture. Cross-check by reading the table's own test column (`pp512 @ d65536`, `tg32 @ d65536`).
|
||||
|
||||
## 6. Reporting shape that survives review
|
||||
|
||||
Label every number: **measured** (nvidia-smi / sysfs / engine ledger line), **derived** (arithmetic on tensor sizes + derived context), or **inference** (e.g. depth extrapolation). State the context each memory figure was taken at, and whether KV was resident — 12.8 GiB at 65,568 tokens is not the same claim as 12.8 GiB at 200k.
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
name: llm-replica-routing-prefix-cache
|
||||
description: "Route requests across multiple LLM replicas by workload shape (pin agent-loop continuations for prefix-cache hits, spread batches for parallel prefill), with the measurement traps and the capacity wall that no routing policy fixes."
|
||||
---
|
||||
|
||||
# Prefix-cache-aware routing for multi-replica LLM serving
|
||||
|
||||
Use when several single-GPU inference replicas sit behind one endpoint and
|
||||
clients send long shared prefixes (agent loops, chat with a big system prompt,
|
||||
RAG with a fixed context).
|
||||
|
||||
## The two failure modes, both measurable
|
||||
|
||||
Measure with a **sequential-turn client**, not only a batch benchmark: a batch
|
||||
of distinct prompts parallelises prefill across replicas, so round-robin looks
|
||||
great there and hides the problem.
|
||||
|
||||
1. **Round-robin loses every other cache hit.** The loop resends the same
|
||||
prefix each turn; alternating replicas means half the turns prefill from
|
||||
scratch. Measured: 90k-token contexts, warm-turn TTFT 2676 ms round-robin
|
||||
vs 1096 ms pinned.
|
||||
2. **Pinning everything loses a card.** A cold batch pays one card's prefill
|
||||
throughput: 16 requests in 18.6 s spread vs 30.1 s pinned.
|
||||
|
||||
## The rule that gets both
|
||||
|
||||
Route by request shape:
|
||||
|
||||
- **Continuation** (request carries more than one message — the signature of an
|
||||
agent loop re-sending its context) -> pin to the replica that served that
|
||||
prefix before.
|
||||
- **Anything else** (one-shot question, batch member, parallel subagent) ->
|
||||
spread across replicas with a **counter**, not with a hash of the prefix.
|
||||
|
||||
Measured on the same host: 19.97 s burst *and* 1095 ms warm turns.
|
||||
|
||||
Also add **failover**: a replica that restarts is invisible to a health table
|
||||
refreshed every 10 s, so a request routed to it returns 502 even though the
|
||||
other replica is up. On connection failure, demote it immediately and retry the
|
||||
request on a live replica before writing anything to the client (the OpenAI
|
||||
endpoints are stateless, so re-sending a request that never reached an engine
|
||||
is safe; never retry after streaming started). Measured: six requests during a
|
||||
replica stop, six 200s, one logged failover, no client-visible error.
|
||||
|
||||
## The capacity wall that no policy fixes
|
||||
|
||||
Per-card KV is finite and **a running sequence and its cached prefix draw on
|
||||
the same pool**. With N families on M cards, one card carries two of them, and
|
||||
its budget must hold both cached prefixes plus whichever sequence is
|
||||
generating. Three concurrent 50-80k-token families against 122,636 tokens per
|
||||
card thrash regardless of how requests are placed:
|
||||
|
||||
| policy | result |
|
||||
|---|---|
|
||||
| counter placement | heavy 25.4 s median (23 % cached), mediums 34.7 / 45.2 s |
|
||||
| load-balanced placement (new families to the card carrying less context) | no improvement — reverted |
|
||||
| one large request admitted per card | no improvement — reverted |
|
||||
| smaller/larger prefill chunk | *costs* KV in both directions (a sweet spot exists; find it) |
|
||||
|
||||
So: measure placement and admission before shipping them, and when the working
|
||||
set exceeds aggregate KV, say so — the levers are capacity (another card),
|
||||
fewer concurrent large contexts (client-side context compaction), or a different
|
||||
platform. Routing cannot rescue arithmetic.
|
||||
|
||||
## Measurement traps
|
||||
|
||||
- **A lifetime engine counter is not a per-request verdict.** "Prefix cache hit
|
||||
rate: 21 %" is an average over every prompt the engine ever saw, including
|
||||
benchmark traffic that never repeats. Probe the question directly instead:
|
||||
send the same long prompt to an idle replica twice — 22.3 s cold then 1.4 s
|
||||
warm is what a working cache looks like (and it holds with a varying tail).
|
||||
- **Salt your corpora.** Two runs sharing a prefix measure a warm cache, not
|
||||
the thing under test; a unique leading line per run makes the prefix new.
|
||||
- **Record the sample size.** A 14-minute arm and an 8-minute arm are not a
|
||||
comparison, and medians from five turns say little.
|
||||
- **Check the tool's own requirements**: `vllm bench serve` needs
|
||||
`--tokenizer <local path>` (plus the model directory mounted) or it tries to
|
||||
resolve the served alias on HuggingFace and dies before sending a request —
|
||||
a run that produces no numbers and no obvious error.
|
||||
- **Verify metric names against the live endpoint** before writing dashboards
|
||||
or alerts: guessing one that does not exist is invisible until someone needs
|
||||
the graph.
|
||||
|
||||
## Test shape that catches it
|
||||
|
||||
Drive the routing function directly with synthetic request bodies (keep the
|
||||
policy in a stdlib-only module so tests need no HTTP framework):
|
||||
|
||||
- a batch of single-turn requests -> assert both replicas are used and
|
||||
consecutive picks alternate;
|
||||
- a growing conversation (same leading bytes, longer history) -> assert every
|
||||
turn agrees with the first turn's replica;
|
||||
- each mode with a replica marked down -> assert the live one is chosen, and
|
||||
-1 when none is;
|
||||
- the fixed comparison modes (`roundrobin`, `affinity`) kept behind an env var
|
||||
so the choice can be re-measured rather than argued.
|
||||
|
||||
One more: the identity of a conversation is its **first N KiB** of prompt. If
|
||||
the tested prefix is shorter than that window its varying tail falls inside it,
|
||||
turns hash differently, and pinning silently never happens.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: llm-serving-benchmark-methodology
|
||||
description: "Measure and A/B a self-hosted multi-replica LLM endpoint honestly: corpus construction, per-arm salting, repeated samples, engine-counter semantics, and the traps that make results meaningless."
|
||||
---
|
||||
|
||||
# Bench a multi-replica LLM deployment so the numbers mean something
|
||||
|
||||
Use when measuring or tuning a self-hosted LLM endpoint (vLLM, llama.cpp,
|
||||
radiance/MXFP4), especially behind a proxy in front of several replicas.
|
||||
|
||||
## Set up a client that can actually send requests
|
||||
|
||||
`vllm bench serve` needs `--tokenizer <local model path>` with the model
|
||||
directory mounted. Given only `--model <served-alias>` it tries to resolve that
|
||||
alias on HuggingFace and dies before the first request — a run that looks like
|
||||
it worked and produced nothing. Confirm with a single prompt before benching.
|
||||
|
||||
## Build the workload from real material
|
||||
|
||||
`--dataset-name random` generates token soup whose draft acceptance is nothing
|
||||
like production (measured 2.9-3.6 tok/update against 5.8 on code), so it
|
||||
mis-ranks speculative-decoding settings. Generate a JSONL corpus from the
|
||||
repository's own sources — `{"prompt": "<instruction>\n\n<file body>"}` — and
|
||||
fix output length with `--custom-output-len`. Keep two corpus shapes:
|
||||
|
||||
- **distinct prompts** (batch behaviour, prefill parallelism)
|
||||
- **a shared long prefix with a varying tail** (conversation turns, where
|
||||
prefix caching decides everything)
|
||||
|
||||
## Salt every arm
|
||||
|
||||
Put a unique marker in the first bytes of every prompt, different per A/B arm.
|
||||
Without it the second arm silently measures the first arm's warm cache and both
|
||||
arms look identical.
|
||||
|
||||
## Measure the same workload before and after
|
||||
|
||||
Keep the workload fixed and change one knob. Repeat each arm — run-to-run
|
||||
variance of 5-10 % is normal, and a single sample nearly made me ship the wrong
|
||||
setting. Report medians per family/class, not one aggregate number.
|
||||
|
||||
## Read engine counters correctly
|
||||
|
||||
- vLLM's `Prefix cache hit rate:` line is a **lifetime average** across every
|
||||
prompt that replica ever saw. It cannot answer "is this conversation served
|
||||
warm?". Send the same prompt twice to an idle engine and compare TTFTs
|
||||
instead (measured: 22.3 s cold → 1.4 s warm for a 50k-token prefix).
|
||||
- `GPU KV cache usage` tells you the pool is not full; it does not tell you a
|
||||
prefix survived. Capacity per card is the boot-time ledger line
|
||||
(`GPU KV cache size: N tokens`), not the card's VRAM.
|
||||
- Reserve memory is not allocation: a bigger reserve does not free anything for
|
||||
anyone else, it only removes cache.
|
||||
|
||||
## Route-shape traps
|
||||
|
||||
A cold batch of distinct prompts prefills in parallel across replicas; the same
|
||||
conversation's next turn only pays off if it lands where its prefix is cached.
|
||||
These pull in opposite directions, so measure both shapes separately and route
|
||||
by request shape (spread single-turn, pin continuations).
|
||||
|
||||
## When the answer is "no"
|
||||
|
||||
Write down rejected hypotheses with their numbers and the mechanism. Placement
|
||||
balance, concurrency admission and chunk-size tweaks all looked obvious and all
|
||||
measured flat or worse on this fleet; the wall turned out to be arithmetic
|
||||
(cached prefixes plus live sequences share one pool). A runbook that records
|
||||
that saves the next operator a night.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
name: niri-output-rotation-pinning
|
||||
description: "Rotate/re-arrange niri monitors (portrait transforms, positions, scale) persistently and verify — covers the output-block-resets-defaults trap and auto-layout reflow that flings unpinned outputs, plus the counter-clockwise transform convention."
|
||||
---
|
||||
|
||||
# Rotating and pinning outputs on niri
|
||||
|
||||
Verified on blankpad, niri 26.04, 2026-09-15 (two PM1601 externals + eDP-1 laptop panel).
|
||||
|
||||
## 0. Ground truth first
|
||||
|
||||
```bash
|
||||
niri msg outputs # connector names, current mode, scale, transform, logical pos/size
|
||||
niri --version
|
||||
```
|
||||
|
||||
Niri output **names are connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings. The EDID strings printed before them in `niri msg outputs` are decoration — use the parenthesized name.
|
||||
|
||||
On blankpad the live config is split by `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`, and `cfg/display.kdl` is a **symlink into the dotfiles repo** (`~/.omp/dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl`). Edit the repo path, not the symlink target path — they are the same file, but the repo path is the one that gets committed. Check with `ls -ld`.
|
||||
|
||||
## 1. Transform direction is counter-clockwise
|
||||
|
||||
```bash
|
||||
niri msg output <NAME> transform --help # "Transform to set, counter-clockwise"
|
||||
```
|
||||
|
||||
| value | effect |
|
||||
|---|---|
|
||||
| `90` | content rotated 90° CCW ⇒ the panel's **top bezel sits on the LEFT** |
|
||||
| `270` | top bezel on the RIGHT |
|
||||
| `180` | upside down |
|
||||
| `flipped-90` / `flipped-270` | mirrored variants |
|
||||
|
||||
`90` vs `270` is a physical-stand fact no tool can read (portable monitors expose no orientation in EDID/MCCS). Pick one, apply it, and give the user the one-liner to flip — don't stall on the question.
|
||||
|
||||
## 2. Two traps that bite when you edit the config
|
||||
|
||||
1. **An `output "X" { … }` block resets every property you don't restate to its default.** Pinning position on `eDP-1` without repeating `scale 1.25` silently drops the laptop panel to scale 1.0. Restate `scale` (and `mode` if you care) in every block you touch.
|
||||
2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning only the externals shoved `eDP-1` from `0,0` to `3936,0` (laptop became rightmost). If you want the pre-existing left-to-right order, pin every output's position.
|
||||
|
||||
Positions are logical pixels at each output's own scale. Original order preserved example:
|
||||
|
||||
```kdl
|
||||
output "DP-2" {
|
||||
scale 1
|
||||
transform "90"
|
||||
position x=1536 y=0
|
||||
}
|
||||
|
||||
output "HDMI-A-1" {
|
||||
scale 1
|
||||
transform "90"
|
||||
position x=2736 y=0
|
||||
}
|
||||
|
||||
output "eDP-1" {
|
||||
scale 1.25 // MUST restate: dashed-out by the block otherwise
|
||||
position x=0 y=0
|
||||
}
|
||||
```
|
||||
|
||||
x-offsets chain: `eDP-1` logical width 1536 → DP-2 at 1536 → +1200 (portrait width) → HDMI-A-1 at 2736.
|
||||
|
||||
## 3. Apply and verify
|
||||
|
||||
```bash
|
||||
niri validate -c ~/.config/niri/config.kdl # exit 0, "config is valid"
|
||||
niri msg action load-config-file # live reload; load-config-file, NOT reload-config
|
||||
sleep 1
|
||||
niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
|
||||
```
|
||||
|
||||
Success looks like `Logical size: 1200x1920` (swapped) plus `Transform: 90° counter-clockwise` on each rotated panel, with positions matching the intended chain.
|
||||
|
||||
One-off rotation without touching config (temporary, forgotten on next config change):
|
||||
|
||||
```bash
|
||||
niri msg output DP-2 transform 90
|
||||
```
|
||||
|
||||
## 4. Commit
|
||||
|
||||
Dotfiles repo is `dank/command-center` (`main`); push to the local Gitea is permitted.
|
||||
|
||||
```bash
|
||||
cd ~/.omp && git pull --ff-only
|
||||
git add dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl
|
||||
git commit -m "blankpad(niri): <what changed>"
|
||||
git push
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- `reload-config` is not a niri subcommand — it's `niri msg action load-config-file`.
|
||||
- Never SIGUSR1 a terminal to "reload"; niri config reloads do not touch ghostty themes.
|
||||
- No man page for niri on this host; `niri <sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
name: pcie-link-capability-probe
|
||||
description: "Measure a GPU-to-host PCIe link the way a weight-streaming engine experiences it, and convert the result into a tokens/s ceiling (streamed expert weights) versus the compute-in-RAM alternative — before recommending or rejecting an offload design."
|
||||
---
|
||||
|
||||
# PCIe link capability probe
|
||||
|
||||
Use when someone asks "is the link fast enough to run model X with engine Y (on one GPU)?"
|
||||
and Y keeps some weights in host RAM — MoE expert offload with an LRU GPU cache, layer
|
||||
paging, weight streaming. The answer is arithmetic once the link is measured, and the
|
||||
*shape* of the engine decides whether the link matters at all:
|
||||
|
||||
| engine shape | what crosses PCIe per token | link matters? |
|
||||
|---|---|---|
|
||||
| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
|
||||
| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
|
||||
| everything resident on the GPU | nothing | no |
|
||||
|
||||
## 1. Establish the hardware ceiling (never trust the card alone)
|
||||
|
||||
```bash
|
||||
lspci -tv | grep -iA2 vga # which bridge the GPU hangs off
|
||||
d=$(lspci -D | grep -i nvidia | awk '{print $1}' | head -1)
|
||||
sudo lspci -vv -s "$d" | grep -E 'LnkCap:|LnkSta:' # the CARD's capability + trained state
|
||||
sudo lspci -vv -s 00:03.1 | grep -E 'LnkCap:|LnkSta:' # the PLATFORM PORT (this is the real limit)
|
||||
```
|
||||
|
||||
- `LnkCap` of the **upstream port** = the electrical ceiling. A card with `Width x16` under
|
||||
an `x8` port cannot exceed x8; an x4 riser/cable caps both.
|
||||
- `LnkSta` at idle usually shows a downgraded speed (PCIe ASPM downshift, e.g.
|
||||
`2.5GT/s (downgraded)`) — that is NOT the loaded rate. Trust the measurement in step 2.
|
||||
- Gen4 usable payload: x4 **7.877 GB/s**, x8 15.75, x16 31.5 (128b/130b encoding).
|
||||
|
||||
## 2. Measure it (the program that did the job)
|
||||
|
||||
Stop **all** production services on the host first (fleet policy), then compile and run:
|
||||
|
||||
```cuda
|
||||
// pcie-link-test.cu : nvcc -O2 -o /tmp/pcie-link-test /tmp/pcie-link-test.cu
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <cuda_runtime.h>
|
||||
#define CK(x) do{ cudaError_t e=(x); if(e!=cudaSuccess){fprintf(stderr,"CUDA %s (%s:%d)\n",cudaGetErrorString(e),__FILE__,__LINE__); return 1;} }while(0)
|
||||
static double ms_between(cudaEvent_t a, cudaEvent_t b){float ms=0; cudaEventElapsedTime(&ms,a,b); return (double)ms;}
|
||||
static int bench_size(size_t bytes){
|
||||
const int iters = bytes>=(1UL<<30)?10:(bytes>=(256UL<<20)?20:100);
|
||||
void *dev=NULL,*host=NULL,*pageable=NULL; cudaStream_t s=NULL;
|
||||
CK(cudaMalloc(&dev,bytes)); CK(cudaHostAlloc(&host,bytes,cudaHostAllocDefault));
|
||||
memset(host,0x5a,bytes); CK(cudaStreamCreate(&s));
|
||||
cudaEvent_t a,b; CK(cudaEventCreate(&a)); CK(cudaEventCreate(&b));
|
||||
for(int i=0;i<3;i++) CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaDeviceSynchronize());
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double h2d=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpyAsync(dev,host,bytes,cudaMemcpyHostToDevice,s));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double h2da=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(host,dev,bytes,cudaMemcpyDeviceToHost));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double d2h=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(a)); CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice)); CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b));
|
||||
double one_us=ms_between(a,b)*1000.0;
|
||||
pageable=malloc(bytes); memset(pageable,0x5a,bytes);
|
||||
for(int i=0;i<2;i++) CK(cudaMemcpy(dev,pageable,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(dev,pageable,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double pag=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
printf("%8.1f MiB | %6.2f | %6.2f | %6.2f | %8.1f us | %6.2f\n", bytes/1048576.0,h2d,h2da,d2h,one_us,pag);
|
||||
free(pageable); cudaFreeHost(host); cudaFree(dev); cudaEventDestroy(a); cudaEventDestroy(b); cudaStreamDestroy(s); return 0;
|
||||
}
|
||||
int main(void){ cudaDeviceProp p; CK(cudaGetDeviceProperties(&p,0));
|
||||
printf("device %s | CUDA %d.%d\n", p.name, p.major, p.minor);
|
||||
printf(" MiB | H2D | async | D2H | 1x xfer | pageable\n");
|
||||
size_t sz[]={3UL<<20,16UL<<20,64UL<<20,256UL<<20,1UL<<30,2UL<<30};
|
||||
for(int i=0;i<6;i++) if(bench_size(sz[i])) return 1; return 0; }
|
||||
```
|
||||
|
||||
Size choice matters: **3 MiB** is the granularity an expert-streaming engine moves, so
|
||||
its single-transfer time is the number that matters; the big sizes give the ceiling.
|
||||
|
||||
## 3. Read it
|
||||
|
||||
- **Achieved GB/s is itself the proof of the trained link state** — 6.7 GB/s is only
|
||||
possible at Gen4 x4 (Gen1 x4 ≈ 1, Gen2 ≈ 2, Gen3 ≈ 3.94), so no separate `LnkSta`
|
||||
sample under load is needed.
|
||||
- **async ≈ sync and pageable ≈ pinned ⇒ link-limited, not staging-limited.** If async is
|
||||
much faster than sync, the engine's fill pipeline (batching copies on one stream) is
|
||||
where the win is; if pageable is much slower, pinning is a real lever. Measure, don't
|
||||
assume.
|
||||
- Per-slab launch overhead = single-xfer time − (slab bytes ÷ rate). Small next to
|
||||
bandwidth cost at 3 MiB.
|
||||
|
||||
## 4. Convert to a tokens/s ceiling
|
||||
|
||||
```
|
||||
active expert bytes/token = active_params × bytes/param (A3B @ Q4_K_M ≈ 1.70 GiB)
|
||||
link time/token = active expert bytes / measured H2D rate
|
||||
ceiling (tok/s) = 1 / link time/token × resident-fraction saved (LRU hits)
|
||||
```
|
||||
|
||||
Then compare against the alternatives on the same host — and against the
|
||||
compute-in-RAM path computed at host RAM bandwidth (256 GB/s LPDDR5X on this fleet's
|
||||
Strix Halo). If the link is >10× slower than the memory behind it (x4 is 31× here),
|
||||
streaming loses by construction and no tuning rescues it; doubling the link (x4 → x8)
|
||||
only doubles a losing number.
|
||||
|
||||
## Worked example (Dankclaw 2026-09-16)
|
||||
|
||||
Gen4 x4, measured H2D 6.72 / D2H 6.60 GB/s (85 % of spec), 3 MiB slab = 474 µs.
|
||||
A3B MoE, Q4_K_M (75 GiB experts, ~1.70 GiB active/token): 272 ms/token ⇒ **3.7 tok/s**
|
||||
(4.6 with 20 % resident). Same host, same model, measured: **20.46 tok/s** with the
|
||||
experts computed on the CPU, 25 (halogen), 26.5 (mainline on the iGPU). Verdict: the
|
||||
streaming design loses 3-5×, and the engine in question could not load the GGUF anyway.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Stop production services before running this (fleet policy) — and remember the 3090's
|
||||
VRAM may be almost fully occupied, so a 2 GiB test buffer needs the service stopped.
|
||||
- Don't quote a derived line from inside the program without checking it: a printf
|
||||
format/argument mismatch in this very probe printed a nonsense ceiling. Recompute by hand.
|
||||
- GiB vs GB: convert before dividing (1.70 GiB = 1.825 GB).
|
||||
- The idle `LnkSta` (2.5 GT/s, downgraded) is not the loaded link — quoting it makes a
|
||||
working x4 look like a broken x1.
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: pcie-link-degradation-vs-idle-triage
|
||||
description: "Decide whether a GPU/NVMe PCIe link is genuinely degraded (bad riser, bad slot, wrong connector) or just idle-downshifted, and whether moving the card to another slot could widen it — using kernel boot bandwidth lines, port LnkCap vs LnkSta, and NVML sampled under load."
|
||||
---
|
||||
|
||||
# PCIe link: degraded or just idle?
|
||||
|
||||
Answer "is my GPU running at full speed?" without fooling yourself. The trap: an
|
||||
idle NVIDIA GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` because the driver
|
||||
downshifts the link to Gen1 for power. That is not a fault, and it is not the
|
||||
slot's fault either.
|
||||
|
||||
## Step 1 — the path ceiling, from the kernel (cheapest, most decisive)
|
||||
|
||||
```bash
|
||||
sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"
|
||||
```
|
||||
|
||||
Reads out per endpoint, e.g.:
|
||||
|
||||
```
|
||||
pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s
|
||||
PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
|
||||
```
|
||||
|
||||
- `63.012 Gb/s` = Gen4 **x4** (~7.88 GB/s/direction after 128b/130b encoding).
|
||||
- "limited by ... link at <BDF>" names the **bottleneck device** (usually a root
|
||||
port) and its width — this is the number that decides whether a swap can help.
|
||||
- "capable of ... x16 link" is the **endpoint's own** capability, not the path's.
|
||||
A big gap between the two is the whole story.
|
||||
|
||||
`dmesg` is often rotated; `journalctl -k -b` keeps this boot.
|
||||
|
||||
## Step 2 — the two sides separately
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <endpoint> | grep -E "LnkCap:|LnkSta:|LnkCtl2:|LnkCap2:"
|
||||
sudo lspci -vv -s <root port> | grep -E "LnkCap:|LnkSta:|LnkCtl2:"
|
||||
```
|
||||
|
||||
- `LnkSta` on a *bridge* is the trained link; on an endpoint it is that
|
||||
endpoint's view. Widths/`(downgraded)` compare against the **own** `LnkCap`.
|
||||
- `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform *intends*
|
||||
Gen4, so a Gen1 current speed is power management, not misconfiguration.
|
||||
- A root port whose `LnkCap` width exceeds the trained width means the socket or
|
||||
adapter wires fewer lanes than the controller can drive. **M.2 connectors are
|
||||
4 lanes by pinout** — an M.2 riser can never deliver more, regardless of what
|
||||
the bridge advertises.
|
||||
|
||||
## Step 3 — prove the link under load (the step people skip)
|
||||
|
||||
Never conclude from an idle reading on NVIDIA. Sample NVML while the GPU works:
|
||||
|
||||
```bash
|
||||
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,\
|
||||
pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv
|
||||
```
|
||||
|
||||
Fire a small, legitimate request at whatever already serves on that GPU (e.g. a
|
||||
`max_tokens` completion against the local LLM endpoint) and poll the query while
|
||||
utilization is high:
|
||||
|
||||
- `gen.current` climbing 1 → `hostmax` = idle downshift, link healthy.
|
||||
- `gen.current` stuck below `hostmax` under load = real degradation.
|
||||
- `width.current < width.max` under load = **physical lane limit** (wiring,
|
||||
riser, or connector) — no software touches this.
|
||||
- On a busy production GPU, this is a usage smoke test, not a benchmark: never
|
||||
stop/bench a live service to read its link.
|
||||
|
||||
## Step 4 — do not plan hardware from `dmidecode -t slot`
|
||||
|
||||
On consumer/mini-PC firmware this table is routinely fiction — it has been seen
|
||||
labelling an Ethernet port and an SD-reader controller as "M.2 Socket 3" slots
|
||||
and advertising a free slot in a **PCI domain that does not exist** (e.g.
|
||||
`0200:00:03.2`). Cross-check with the live bridge list:
|
||||
|
||||
```bash
|
||||
lspci -nn | grep "PCI bridge" # every root port, whether populated or empty
|
||||
lspci -t # topology; which endpoints hang off which port
|
||||
```
|
||||
|
||||
Empty root ports still enumerate, so a genuinely absent port is genuinely
|
||||
absent. Trust live PCI, not SMBIOS.
|
||||
|
||||
## Step 5 — verdict on "would moving the card to another slot help?"
|
||||
|
||||
- Compare the **bottleneck device named in step 1** with the target slot's port
|
||||
`LnkCap` width. If the bottleneck is the *endpoint side* (riser/adapter) and
|
||||
the target port is wider, a move can help.
|
||||
- If both candidate connectors cap at the same width (two M.2 sockets = both
|
||||
x4), moving is a **no-op** — say so plainly instead of endorsing the swap.
|
||||
- Error counters are a useful tiebreaker: clean `DevSta`/`CESta` with a low
|
||||
trained speed means "wired narrow", not "failing".
|
||||
|
||||
## Reporting
|
||||
|
||||
Give the ceiling in Gb/s, the card's own capability, what the idle reading
|
||||
meant, and whether any physical change can move the number. If the answer is
|
||||
"the link is already at the hardware maximum", state that and quantify whether
|
||||
the workload actually cares (resident weights = PCIe is nearly idle per token;
|
||||
CPU layer offload = it hurts).
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: published-port-cutover-gate
|
||||
description: "When replacing or restarting a service that publishes a TCP port on this rootless-podman/pasta fleet, gate on the port actually being free first — otherwise pasta's bind fails at container create and the failure hides behind Restart=on-failure."
|
||||
---
|
||||
|
||||
# Published-port cutover gate (rootless podman / pasta)
|
||||
|
||||
On hosts where **rootless podman publishes ports through pasta**, the published
|
||||
port is bound in userspace at **container CREATE**, not when the app binds. A
|
||||
predecessor process that has just been stopped can still hold that port —
|
||||
listener, accepted connection, or TIME_WAIT residue from the connections it
|
||||
served — so a replacement started immediately after `systemctl stop` dies with:
|
||||
|
||||
```
|
||||
pasta: ... bind: Address already in use
|
||||
conmon ... exit status 126
|
||||
```
|
||||
|
||||
and because the unit is usually `Restart=on-failure`/`RestartSec=10`, it
|
||||
**self-heals seconds-to-minutes later**. That is the trap: the deploy looks
|
||||
like it failed, then looks fine, and the real defect (no port gate) stays
|
||||
invisible because the retry masks it.
|
||||
|
||||
Observed on Dankclaw 2026-09-16 cutting `frank-large` from a root unit
|
||||
(ik_llama.cpp) to a rootless Quadlet (halogen): `systemd` logged
|
||||
"Deactivated successfully", and pasta tried to bind 7770 about a second later
|
||||
and reported "Address already in use". Only the quadlet's own restart policy
|
||||
recovered it.
|
||||
|
||||
## Procedure
|
||||
|
||||
Before starting the replacement:
|
||||
|
||||
1. **Retire the old unit and wait for it to be genuinely inactive.**
|
||||
`systemctl stop <old>` (root) or `systemctl --user stop <old>`.
|
||||
2. **Stop the NEW unit too, if this is a redeploy.**
|
||||
`systemctl --user stop <new>.service || true`
|
||||
|
||||
Without this the gate deadlocks against itself: the currently-running
|
||||
replacement is the thing holding the port. This mistake was made and caught
|
||||
in the same session — a gate placed before the stop would have failed every
|
||||
subsequent deploy of an already-running service.
|
||||
3. **Gate on the port being empty, covering every socket state:**
|
||||
|
||||
```bash
|
||||
wait_port_free() {
|
||||
local port="$1" limit="${2:-90}" i
|
||||
for ((i = 1; i <= limit; i++)); do
|
||||
if [[ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "FAIL: port $port is still held after ${limit}s:" >&2
|
||||
ss -tan "sport = :$port" >&2 || true
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
`sport` matches both the listener and the server side of accepted
|
||||
connections, so this also catches TIME_WAIT residue that a plain
|
||||
`ss -ltn` check misses. Fail loudly with the socket table attached; a deploy
|
||||
that starts a doomed container has not deployed anything.
|
||||
4. **Then** install/pull and start, and wait on the service's health endpoint.
|
||||
|
||||
## Verifying the gate has teeth
|
||||
|
||||
- Re-run the same deploy: it must pass with the service already running (that
|
||||
is the idempotency case the self-deadlock broke).
|
||||
- Reproduce the failure once on purpose if the service is not user-facing:
|
||||
stop the container and start the replacement with the gate bypassed, and
|
||||
confirm `Address already in use` / exit 126.
|
||||
- Assert it in the repo's seam tests: e.g. `grep -q 'wait_port_free <port>'`
|
||||
and `grep -q '^wait_port_free() {'` in the deploy handler, so a later
|
||||
refactor cannot quietly drop the gate.
|
||||
|
||||
## Related traps
|
||||
|
||||
- `systemctl disable` on some of these hosts **deletes the unit file** rather
|
||||
than just the symlink, so a "move the unit to trash" step must tolerate a
|
||||
file that is already gone — and should say so, or "no message" carries no
|
||||
evidence.
|
||||
- Never `rm`; move retired units into trash (`$HOME/.AGENTTRASH/<date>-<topic>/`).
|
||||
- A rootless Quadlet's generated unit **cannot** be `systemctl --user enable`d;
|
||||
boot persistence comes from `[Install] WantedBy=default.target` in the
|
||||
`.container`. To keep a masked/failed quadlet from being regenerated, move
|
||||
the `.container` out of `~/.config/containers/systemd/` and `daemon-reload` —
|
||||
the generator's source is the lever, not `mask`.
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: rccl-pcie-atomics-collective-failure
|
||||
description: "Diagnose AMD multi-GPU RCCL/collective launch failures (hipErrorIllegalState, hostcall) by checking PCIe atomic routing on every upstream bridge, decide whether any software path exists, and rank the fix (CPU-attached M.2 riser vs x8/x8 board vs layer split); includes the bare 2-rank probe matrix and the stop-and-mask safety rule."
|
||||
---
|
||||
|
||||
# RCCL / PCIe atomics collective failure
|
||||
|
||||
Use when a multi-GPU AMD host fails to run tensor/pipeline parallelism, or when
|
||||
asked "can these GPUs work together?".
|
||||
|
||||
## 1. Identify the cause from the runtime, not the framework
|
||||
|
||||
The failure surfaces as `hipErrorIllegalState` from `hipLaunchKernel` for
|
||||
`ncclDevKernel_*`. Nothing in vLLM/RCCL/PyTorch names atomics. Get the real
|
||||
message with:
|
||||
|
||||
```bash
|
||||
AMD_LOG_LEVEL=3 <the failing process>
|
||||
# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
|
||||
```
|
||||
|
||||
## 2. Check the whole chain, not just the card
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap" # want ReqEn+
|
||||
lspci -tv # every bridge upstream
|
||||
sudo lspci -vv -s <bridge> | grep AtomicOpsCap # want Routing+
|
||||
```
|
||||
|
||||
A `Routing-` bridge is a **read-only hardware capability** — no software can
|
||||
enable it. `setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) does nothing
|
||||
because the value is consumed at driver probe time. Typical AM5 layout: one x16
|
||||
slot on CPU root ports (atomics fine), the rest behind the chipset (usually not).
|
||||
|
||||
The gate in source, for the record:
|
||||
|
||||
```cpp
|
||||
void Device::checkAtomicSupport() { // rocclr/.../rocdevice.cpp:181
|
||||
link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
|
||||
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
|
||||
if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else
|
||||
}
|
||||
```
|
||||
|
||||
Consumed at `rocvirtual.cpp` `case HiddenHostcallBuffer:` — the kernel *declares*
|
||||
a hostcall buffer and ROCr refuses to fill it without atomics. **No env var,
|
||||
module param, or sysfs toggle exists** (grep `getenv|HSA_|ROCR_|AMD_` around the
|
||||
gate). Do not promise a software override without checking this.
|
||||
|
||||
## 3. Run the probe matrix — one probe answers several design questions
|
||||
|
||||
Bare 2-rank `torch.distributed` (no vLLM, no model), one process per GPU,
|
||||
`init_process_group("nccl", timeout=timedelta(seconds=25))`:
|
||||
|
||||
| Op | CPU-lane card | chipset card | What it decides |
|
||||
|---|---|---|---|
|
||||
| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
|
||||
| `send`/`recv` (P2P) | hang | FAIL | **PP is dead too** — vLLM's PP handoff is this primitive |
|
||||
|
||||
Test `send`/`recv` **in isolation** (a prior failed op leaves the group broken
|
||||
and the result meaningless — it just hangs until the timeout). Set an explicit
|
||||
PG timeout so a blocked pair raises instead of hanging.
|
||||
|
||||
**SAFETY — non-negotiable.** Never run this on a host that is serving:
|
||||
`sudo systemctl stop <svc> && sudo systemctl mask <svc>` first, or use a host
|
||||
that serves nobody. A probe run against a live GPU service preceded a hard host
|
||||
down (no ARP, no ICMP from two subnets, power cycle required). See the
|
||||
benchmarking policy: it covers ANY GPU experiment, not just benchmarks.
|
||||
|
||||
Telling "off" from "wedged" afterwards: `ip neigh show <ip>` from a peer —
|
||||
`FAILED`/`INCOMPLETE` = off or NIC dead; a merely wedged kernel still answers ARP.
|
||||
|
||||
## 4. Rank the fixes
|
||||
|
||||
1. **Move the card to a CPU-attached M.2 slot** (M.2→PCIe x4 riser). Cheap and
|
||||
reversible. Get the board's *own spec page*: it lists which M.2 slots come
|
||||
from the CPU vs the chipset, and the chipset x16 slot's real width (often
|
||||
x1 mode). M.2 slots can be empty even when the x16 slots are full.
|
||||
2. **Board whose CPU x16 bifurcates x8/x8** — both cards on the CPU root port.
|
||||
3. **llama.cpp / ik_llama.cpp `--split-mode layer`** — no collectives at all, so
|
||||
it sidesteps the wall entirely; costs the GGUF conversion plus paged
|
||||
attention, continuous batching, and spec decode.
|
||||
4. Two TP=1 replicas + a routing proxy — what you do when you cannot fix the
|
||||
topology; keeps throughput but the cards never share one model.
|
||||
|
||||
**Verify before benchmarking anything:** `AtomicOpsCtl: ReqEn+` on the moved
|
||||
card. X4 is usually enough for TP (≈10 MB per token-step for a 27B at batch 4
|
||||
≈1.4 ms at Gen4 x4) — compute-bound, not interconnect-bound.
|
||||
|
||||
## 5. Closed software routes (don't re-litigate)
|
||||
|
||||
- vLLM's IPC custom allreduce is gated by a **CUDA-sm-version-keyed table**
|
||||
(`{9.0, 10.0, 10.3, 10.7}`); ROCm reports `12.0` on gfx1201 → `False`. The ops
|
||||
are compiled, the gate is not. Patching the table is still not enough —
|
||||
`broadcast`/`barrier`/TP-logits `all_gather` have no non-RCCL path.
|
||||
- A fork that wraps `.all_reduce` (e.g. radiance's `RADIANCE_USE_R4D_AR` one-shot
|
||||
P2P allreduce, gloo for IPC-handle exchange) does not intercept
|
||||
`PyNcclCommunicator.__init__`'s warmup allreduce or `pynccl_comm.all_gather`.
|
||||
- NCCL symmetric memory: needs an RCCL exposing `ncclDevComm*`; many builds ship
|
||||
the NCCL version string without the device API.
|
||||
- P/D disaggregation: needs a KV connector *and* coordination collectives.
|
||||
- gloo: CPU/TCP, unreachable for per-layer TP.
|
||||
|
||||
**Open, unverified**: recompiling RCCL so its kernels stop declaring a hostcall
|
||||
buffer (`-mprintf-kind=buffered` is the usual claim). Counter-evidence: RCCL's
|
||||
device source has no live `printf`/`assert`/`__assertfail`, so the trigger is
|
||||
unexplained. Test: rebuild, then dump a kernel's declared args
|
||||
(`roc-obj-extract` + `llvm-readelf --notes`) and look for
|
||||
`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: spec-decode-ab-single-gpu-host
|
||||
description: "A/B a speculative-decoding variant (or any serve-flag challenger) against an incumbent LLM endpoint on a single-GPU host that serves users: stop the unit, run the challenger from a patched env file, prove losslessness by identical greedy text, and keep the incumbent unless the challenger wins."
|
||||
---
|
||||
|
||||
# A/B a serve-flag challenger against a live LLM endpoint
|
||||
|
||||
Use when asked "try it without X / with Y" for an inference endpoint that is
|
||||
serving users from a GPU you cannot share (one model copy per host). The
|
||||
deliverable is a measurement plus a decision, and the endpoint must end the
|
||||
experiment on the configuration you started it with.
|
||||
|
||||
Ready-made harness for llama.cpp endpoints: `scripts/bench-spec-ab.py`
|
||||
(`<env-file> <serve-script> [--base-url]`) — it stops the unit, runs each
|
||||
config, compares greedy text and restores the unit. Below is what it encodes
|
||||
and the traps it hit.
|
||||
|
||||
## Protocol
|
||||
|
||||
1. **Fixed measurement shapes, one prompt per phase.** A large prompt with a
|
||||
tiny budget isolates prefill; a small prompt with a long budget isolates
|
||||
decode. `temperature 0` throughout. Never compare across sessions — run the
|
||||
*incumbent* again in the same session, on the same box, or you are comparing
|
||||
two machines' worth of noise.
|
||||
2. **Prove losslessness, do not claim it.** Send one fixed prompt at temp 0 to
|
||||
every config and diff the greedy text. Speculation is only allowed to buy
|
||||
speed; if the texts differ, stop and treat it as a correctness bug.
|
||||
3. **Stop the unit for the duration** (benchmarking policy on these hosts: a
|
||||
bench beside a live service starves the GPU). `sudo systemctl stop <unit> &&
|
||||
passive`, run the challenger by hand with `setsid`, then `sudo systemctl start
|
||||
<unit>` at the end *whatever happens* — wrap the restore so an exception
|
||||
cannot leave the fleet dark.
|
||||
4. **Override tunables through a patched env file**, never through the shell:
|
||||
serve scripts that `source` their env file with `set -a` win over environment
|
||||
variables, so `CTX=... bash serve.sh` silently does nothing. `sed` the repo env
|
||||
into `/tmp/challenger.env` and point the service's env-file variable at it.
|
||||
5. **Read the acceptance counters, not the flags.** `/metrics`
|
||||
(`llamacpp:spec_decode_num_drafts_total` vs `..._accepted_tokens_total`) or the
|
||||
per-request `timings.draft_n` / `draft_n_accepted`. A draft rate of 0 while
|
||||
tokens are produced means the drafter is not running even though the flags look
|
||||
right.
|
||||
6. **Sweep the depth, not just the mode.** Draft depth is model-specific and
|
||||
single-valued: measure at least two (e.g. n_max 3 and 7). Deeper often loses —
|
||||
at n=7 a third of proposals survived, at n=3 more than half.
|
||||
7. **Watch per-card VRAM** (`rocm-smi --showmeminfo vram`). A proposer is not
|
||||
free: a drafter file cost 2.06 GB on its card while loading the target's own
|
||||
in-model MTP head cost ~3.3 GB.
|
||||
8. **Decide and land it.** If the challenger loses, say so and change nothing;
|
||||
if it wins, put the value in the repo env file, run the suite, deploy, and
|
||||
verify with a real completion. Record the losing numbers in the env file with
|
||||
"do not switch without re-measuring" so nobody re-litigates it.
|
||||
|
||||
## Traps that cost time here
|
||||
|
||||
- **`pgrep -f <script>` and `pkill -f <pattern>` match your own command line**
|
||||
(and the ssh wrapper's). Use `pgrep -x llama-server` / `pkill -x` and confirm
|
||||
with `ps -eo pid,args | grep "[s]cript"`.
|
||||
- **PID 1 is not the app** inside a container; for units, `systemctl restart`
|
||||
is the reload.
|
||||
- **A retired predecessor's port lingers**: a published-port proxy leaves the
|
||||
endpoint port in TIME_WAIT. Wait for the port to be free (and for the manual
|
||||
instance to be killed) before starting the challenger, or the bind fails with
|
||||
address-in-use long after the unit stopped.
|
||||
- **Nested f-strings with backslash escapes are a SyntaxError on the fleet's
|
||||
Python** (<3.12 semantics). Build log lines with string concatenation.
|
||||
- **If a config dies at load, read the error literally before "fixing" it.**
|
||||
`context type MTP requested but model doesn't contain MTP layers` was not a
|
||||
bad flag: a chain has ONE draft context, so MTP met the *drafter's* file. Two
|
||||
proposers that each own the draft slot cannot be combined, and pinning that
|
||||
in a seam prevents someone "fixing" the flag later.
|
||||
|
||||
## Reporting
|
||||
|
||||
One table: config, decode tok/s, acceptance, VRAM per card. Then one line per
|
||||
finding, each with its number. Then the decision, plus whether the deployed
|
||||
state changed. "The thing you asked me to try lost to what was already running"
|
||||
is a complete and useful answer.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: spec-decode-worth-it-triage
|
||||
description: "Decide whether speculative decoding is worth deploying for a target/drafter pair: compute break-even acceptance from measured step times, classify the drafter by per-round cost, and check the real attachment surfaces (closed engines vs llama.cpp --spec-draft-device)."
|
||||
---
|
||||
|
||||
# Speculative decoding: is it worth deploying?
|
||||
|
||||
Use when someone proposes a drafter for an existing endpoint ("use the 3090 as a drafter", "hook a DFlash drafter to X", "wouldn't speculation make Y faster?"). Answer with arithmetic against the INCUMBENT, not against zero.
|
||||
|
||||
## 0. Check whether the target already speculates
|
||||
|
||||
Many engines ship a drafter of their own; if it is on, your baseline is `A_incumbent / t_s`, not `1 / t_s`.
|
||||
|
||||
- halogen-flash-server (Dankclaw frank-large): MTP head **on by default**. Read it from its own accounting, e.g. `serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
|
||||
- llama.cpp: `/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`; per-request lines say `draft acceptance = 0.76 (125 accepted / 165 generated)`.
|
||||
|
||||
## 1. Measure the two step times (never guess)
|
||||
|
||||
- target step `t_s` = 1 / (client-visible tok/s ÷ commit-per-round). Ask the operator for the tok/s they actually SEE: engine-internal decode-only numbers (short generations, prefill and delivery excluded) run ~1.7–2× optimistic.
|
||||
- drafter step `d_s`: llama.cpp logs `eval time = 2117.32 ms / 178 tokens` per task; that is the drafter's per-token price when it is a plain model on the same box.
|
||||
|
||||
## 2. Arithmetic
|
||||
|
||||
```
|
||||
round_time = K * d_s + 1.1 * t_s # K draft tokens proposed, K+1 verified in ONE target pass
|
||||
A = (1 - p^(K+1)) / (1 - p) # expected accepted tokens, p = per-position acceptance
|
||||
rate = A / round_time # compare to the incumbent
|
||||
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
|
||||
```
|
||||
|
||||
Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67 commit/round), `d_s = 12 ms` (27B IQ3_S on a 3090) → break-even `p ≈ 0.70` at K=3, `≈0.68` at K=5; realistic cross-finetune `p = 0.4–0.6` gives 17–20 tok/s, i.e. a LOSS. Drafters that are only ~3× faster per token cannot pay for K sequential forwards.
|
||||
|
||||
## 3. Classify the drafter by per-round cost (this decides more than acceptance)
|
||||
|
||||
| drafter | cost per round | needs |
|
||||
|---|---|---|
|
||||
| plain model via `-md`/`draft-simple` | **K sequential forwards** | 10–50× faster than the target |
|
||||
| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
|
||||
| trained multi-token head (`draft-mtp`, MTP/EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
|
||||
|
||||
Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden states, so a drafter built for one model cannot be re-pointed at another (a dense-27B DFlash2 is useless for a 512-expert MoE trunk).
|
||||
|
||||
## 4. Acceptance reality checks
|
||||
|
||||
- Fleet datum: a TRAINED z-lab DFlash2 at depth 7 accepted ~41% on its matched target (`draft-dflash 7 → 43.5 tok/s, 41% accepted`). Treat 40–60% as the honest band for a matched pair; <85% rarely pays unless the drafter is nearly free.
|
||||
- Cheapest proxy for `p` at temp 0, when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights moved.
|
||||
- llama.cpp can compute the projection without any draft artifact: `--spec-synth-rates P0,P1,...` / `--spec-synth-len L`.
|
||||
- Speculation pays IN PROPORTION TO HOW SLOW THE TARGET IS. The same 41% gave 2.3× on an 18.8 tok/s layer-split target and a loss on a 25 tok/s one. Fast targets are usually sparse MoE (few active experts → little per-token weight traffic) — the worst case for drafting.
|
||||
|
||||
## 5. Check the attachment surface BEFORE the arithmetic
|
||||
|
||||
- **Closed engines refuse external drafters by design.** halogen-flash-server: `DRAFTERS = {serial:0, mtp:1, dflash2:2}`, `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint, and an unknown/unavailable name is a deliberate 400 (a silent downgrade would make comparisons lie). "Drafter" is a field in its internal loopback token protocol (`drafter`, `spec_verify`, `spec_commit`) — there is no device routing, so a drafter can never sit on another card. Probe it live: POST `"drafter":"dflash2"` and read the error.
|
||||
- **pure llama.cpp DOES route the draft**: `--spec-draft-device/-devd <devs>` (default: follows `--device`), plus `-ngld/--gpu-layers-draft`, `-otd`, `-ctkd/-ctvd`, `-td/-tbd`, `-ncmoed`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`. In production on the fleet as `SPEC_DRAFT_DEVICE=ROCm1` with a layer-split target.
|
||||
- **Devices**: one llama.cpp process sees devices only from backends it was built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot coexist); a CUDA-only build has no `Vulkan0` to draft with.
|
||||
- **Chaining is not universal**: `--spec-type` takes a comma list, but `draft-mtp,draft-dflash` was measured to DIE AT LOAD while `draft-dflash,ngram-mod` is structurally allowed. Test at load, never assume composition.
|
||||
|
||||
## 6. Cross-model pairs need a byte-identical tokenizer
|
||||
|
||||
Speculation compares token IDs, so the draft and target tokenizers must map id → the same string. Verify cheaply from the GGUF headers (no full download; a range read of the first ~8 MB over HTTP is enough for a remote file): parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and a sha256 over the first 2000 tokens.
|
||||
|
||||
```
|
||||
vocab=248320 merges=247587 model=gpt2 pre=qwen35
|
||||
sha256(first 2000 tokens)=8619c68fe4466d3a # identical => ids are interchangeable
|
||||
```
|
||||
|
||||
Identity is necessary, NOT sufficient — the drafter still has to be cheap and accepted.
|
||||
|
||||
## 7. Verdict shape
|
||||
|
||||
Report, in this order: (a) does the target already draft, and at what rate; (b) break-even `p` at K=1,3,7 versus the incumbent; (c) the realistic `p` band and the resulting rate; (d) whether the drafter even fits (VRAM) and whether the attachment surface exists; (e) the one measurement that would settle it, and what you expect it to show.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: stale-compose-container-quadlet-migration
|
||||
description: "Diagnose a container that crash-loops after a reboot because a stale, pre-config-change compose container keeps being restarted, confirm it with the container's real CreateCommand, and migrate the service to a repo-tracked Quadlet unit."
|
||||
---
|
||||
|
||||
# Diagnose a crash-looping container after a reboot: stale compose container
|
||||
|
||||
## When to use
|
||||
|
||||
A service that worked before a reboot now crash-loops (or is `Exited`), its
|
||||
config file looks correct, and the failure message blames something the config
|
||||
does declare — e.g. `RuntimeError: No HIP GPUs are available` for a container
|
||||
whose compose service passes `/dev/kfd`.
|
||||
|
||||
## Root-cause pattern
|
||||
|
||||
The running container was created by an EARLIER revision of the service
|
||||
(before the missing lines were added) and `restart: unless-stopped` keeps
|
||||
resurrecting it:
|
||||
|
||||
- `podman compose up -d` does not recreate an existing container because its
|
||||
*declaration* changed — it only reconciles what compose tracks.
|
||||
- So the container keeps its original flags forever, and every restart mode
|
||||
(`unless-stopped`, `always`, podman-compose unit, quadlet) faithfully keeps
|
||||
restarting a container that can never work.
|
||||
- Symptom shape: high `RestartCount`, exit code 0 or 1, a config that looks
|
||||
right, and a failure that contradicts the config.
|
||||
|
||||
## Confirm in three commands
|
||||
|
||||
```bash
|
||||
# 1. How long has it been looping, and what actually started it?
|
||||
podman inspect <c> --format 'restarts={{.RestartCount}} created={{.Created}} exit={{.State.ExitCode}} unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'
|
||||
|
||||
# 2. The flags it REALLY has — this is ground truth. Do NOT trust
|
||||
# {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}}: both report 0/[] for
|
||||
# containers that were started with --device and --group-add.
|
||||
podman inspect <c> --format '{{.Config.CreateCommand}}' | tr ' ' '\n' | grep -A1 -E 'device|group-add'
|
||||
|
||||
# 3. Behavioural truth inside it
|
||||
podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))'
|
||||
```
|
||||
|
||||
Compare the container's `created` timestamp with when the config gained the
|
||||
missing lines (`git log -p --follow <config>`). Created-before-the-edit is the
|
||||
signature.
|
||||
|
||||
## Fix
|
||||
|
||||
Recreating the compose container works, but on a repo-governed fleet the fix is
|
||||
to give the service a **repo-tracked, boot-persistent Quadlet unit** (every
|
||||
container should already be one), then retire the compose path:
|
||||
|
||||
1. `docker/quadlet/<name>.container` with the full run posture: image (use
|
||||
`Image=localhost/<tag>` for locally built images + `Pull=never`),
|
||||
`AddDevice=`, `GroupAdd=keep-groups`, `Volume=`, `PublishPort=127.0.0.1:...`
|
||||
for an unguarded service, `HealthCmd=`, `Restart=always`, and
|
||||
`[Install] WantedBy=default.target`.
|
||||
2. Healthcheck as a **mounted file**, not inline: the podman 5.x v1 generator
|
||||
splits `HealthCmd` on spaces into an argv array, so
|
||||
`python3 -c "..."` becomes a silent quoting puzzle.
|
||||
3. Delete the compose service (one owner per container name) and make the
|
||||
deployer build → `install_quadlet` → retire any legacy hand-written unit
|
||||
into `~/.AGENTTRASH/` → `podman rm -f <name>` (so the unit's container can
|
||||
take the name) → restart → `wait_for_health`.
|
||||
4. Wire the planner (`<target>/<svc>/**` → service) and the dispatcher
|
||||
allowlist (`<target>:<svc>`), or the deploy refuses the service.
|
||||
5. Update the seams to read the unit rather than compose for devices, command
|
||||
and groups — and resolve build inputs from the deployer's declaration, never
|
||||
from a `find | head -1` directory listing.
|
||||
|
||||
## Verify behaviourally, not structurally
|
||||
|
||||
`systemctl --user is-enabled` → `generated`, container `healthy`,
|
||||
`RestartCount` stops climbing, the port answers (`curl -o /dev/null -w
|
||||
'%{http_code}'`), and the in-container probe reports the device
|
||||
(`hip available: True | Radeon 8060S Graphics`).
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: stale-signal-and-hollow-gate-audit
|
||||
description: "Audit a repo for signals that cannot be correct — vacuous test assertions after a file is retired, permanently-red health probes, and prose claims invalidated by your own changes; includes the linter-validation-before-acting discipline and Obsidian link-resolution rules."
|
||||
---
|
||||
|
||||
# Stale-signal and hollow-gate audit
|
||||
|
||||
Use when you want assurance rather than a feature: "is anything in this repo
|
||||
quietly lying?" The productive question is **which signal here cannot possibly
|
||||
be correct?** Every pass on serverAppConfig (2026-09-16) found real defects this
|
||||
way — including one in the rule file agents read first.
|
||||
|
||||
## 1. Vacuous assertions (highest severity — the gate is already dead)
|
||||
|
||||
A seam that greps a file passes *by absence* once the file is retired, because
|
||||
`grep` exits 2 on a missing path and `!` inverts it.
|
||||
|
||||
```bash
|
||||
# Find assertions anchored on files that may not exist
|
||||
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
|
||||
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
|
||||
```
|
||||
|
||||
Fix shape — one place asserts the invariant, each local site keeps a live check:
|
||||
|
||||
```bats
|
||||
@test "no <host> compose file competes with the quadlet units" {
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
|
||||
}
|
||||
|
||||
# at the per-service site: still fails if the entry returns
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
|
||||
```
|
||||
|
||||
**Always teeth-test**: recreate the offending file (`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams, confirm they go RED, then move the file to trash. A re-armed check that was never seen failing is still unverified.
|
||||
|
||||
## 2. Permanently-red probes
|
||||
|
||||
For each monitored target ask: *can this probe succeed from the monitor's
|
||||
vantage point?* Failure classes seen: container-internal ports probed from
|
||||
another host (`192.168.0.235:3306`, `127.0.0.1:6379` on a published-port-only
|
||||
netavark host), and "probe" jobs that scrape `/` (HTML → `unsupported
|
||||
Content-Type`) or a `/metrics` that is disabled (Gitea 404).
|
||||
|
||||
- A card/job that can never be up must be **fixed or explicitly marked
|
||||
not-probed** (`wip: true` on homepage cards) — never left red.
|
||||
- The right probe for "is this HTTP service answering?" is the **blackbox
|
||||
exporter** (`/probe?target=…&module=…`), which also gives guarded vhosts a real
|
||||
uptime signal since Prometheus cannot solve an Anubis challenge.
|
||||
- Cheap check on a homepage board: read the deployed `status.json` and triage
|
||||
every `state != up` entry against reality (container running? port published?).
|
||||
|
||||
## 3. Prose claims invalidated by your own change
|
||||
|
||||
After retiring a file or adding a feature, grep for **every** reference, not
|
||||
just code:
|
||||
|
||||
```bash
|
||||
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
|
||||
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
|
||||
```
|
||||
|
||||
This is what catches the ones that bite: a rule file (AGENTS.md) told agents
|
||||
ComfyUI runs CPU-only long after it was restored with GPU access; a unit said
|
||||
"there are no alerting rules" hours after five were added; a docs page
|
||||
documented three scrape jobs as working when they could never report up.
|
||||
**Clean cutover includes prose.**
|
||||
|
||||
## 4. Validate the linter before acting on it
|
||||
|
||||
A checker's output is a hypothesis about another tool's semantics. Modelling
|
||||
Obsidian with naive text matching produced 45 "broken links" of which 3 were
|
||||
real — the rest were case-sensitivity (`[[qBittorrent]]` → `qbittorrent.md`),
|
||||
path-qualified resolution (`[[folder/Note]]` resolves by path suffix, not stem),
|
||||
and `%20` URL-encoding in markdown links. Fix the checker first; re-derive the
|
||||
report; only then edit documents. In this repo the result lives in
|
||||
`scripts/check-vault-links.py` with those three rules in its docstring, gated by
|
||||
the `no unresolved links in the docs vault` seam.
|
||||
|
||||
## Reporting shape
|
||||
|
||||
Report per finding: the surface, why the signal *could not* have been correct,
|
||||
the fix, and how you proved the fix has teeth (red/green evidence). Flag any
|
||||
regression you introduced yourself in the same breath — one was found this way
|
||||
(a tightened bind killed a reverse-proxied vhost while every gate stayed green,
|
||||
because the spec covering that hostname did not exist).
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
name: stale-signal-audit
|
||||
description: "Audit homelab/fleet surfaces (status boards, metrics targets, test gates, docs, rule files) for signals that are already wrong — impossible-to-pass probes, assertions that pass by absence, linters that produce false positives, and prose invalidated by your own change. Use after a change lands, when a gate looks green but you want to know what it actually proves, or when asked what is broken."
|
||||
---
|
||||
|
||||
# Stale-signal audit
|
||||
|
||||
Signals rot without failing. This is the battery that found real defects on 4 of 4
|
||||
passes in serverAppConfig; run it after any change and before trusting a green gate.
|
||||
|
||||
## The five checks
|
||||
|
||||
1. **Probe reachability.** For every monitor/probe/alert target, ask: *can this
|
||||
checker physically reach that endpoint?* A probe that cannot succeed reports a
|
||||
false outage forever, and an always-red signal teaches the operator to ignore
|
||||
the surface.
|
||||
- Homepage (`shared/homepage/services.json` + `homepage-status` on Dankclaw): the
|
||||
monitor sees Dankclaw's loopback, container names, and **published** LAN ports
|
||||
only. A card probing another host's container-internal port can never be up →
|
||||
mark `wip` with the reason and let the app's card carry the status.
|
||||
- Prometheus: a *scrape* cannot express "is this HTTP service answering?" —
|
||||
scraping an HTML page gives `unsupported Content-Type`; a `/metrics` that 404s
|
||||
stays down. Remedy: blackbox exporter in probe mode. Never "fix" a probe by
|
||||
publishing a database or adding a wildcard bind.
|
||||
|
||||
2. **Assertions that pass by absence.** Grep-based tests against a file that was
|
||||
deleted or retired pass vacuously: a missing path makes `grep` exit 2 and a
|
||||
negated `! grep` succeeds. Look for tests referencing paths that no longer exist
|
||||
(`git ls-files` the target, or just `[ -f ]` it).
|
||||
- Fix: assert the invariant directly ("no <file> competes with the unit"), keep a
|
||||
conditional live check for the file returning, and drop dead variables that
|
||||
only referenced it.
|
||||
- **Prove teeth**: recreate the file/condition and watch the test go red. A test
|
||||
you have never seen fail is not a test.
|
||||
|
||||
3. **Linter validity.** Before acting on any checker's output, validate the checker
|
||||
against the resolver/contract it models. Obsidian example: wikilinks are
|
||||
case-insensitive, `[[folder/Note]]` resolves by **path suffix** not stem, and
|
||||
`%20` in markdown links is URL-encoding — a naive checker reported 45 broken
|
||||
links when 2 were real. Same class: a `find … | head -1` selector that reads the
|
||||
wrong file; a truncated listing that makes you "fix" a file that exists.
|
||||
|
||||
4. **Gates that never run.** Playwright specs in `tests/e2e/` are matched by
|
||||
**number prefix** in `playwright.config.ts`; an unregistered spec silently never
|
||||
runs and the suite count stays the same. Verify by counting tests before/after
|
||||
(`tests/e2e_scaffold.bats` now fails on an unregistered spec).
|
||||
|
||||
5. **Prose and comments invalidated by your own change.** Clean cutover includes
|
||||
docs, unit docstrings, and the repo's rule file (`AGENTS.md`) — which is what
|
||||
every future agent reads first, so a stale claim there is the most expensive kind.
|
||||
Check: runbooks/rows naming a retired service, "unconfigured" claims after you
|
||||
configured something, upgrade instructions pointing at a deleted file, and
|
||||
lessons that the deployment has since contradicted.
|
||||
|
||||
## Order
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test gates,
|
||||
docs/rule files, dashboard panels).
|
||||
2. For each, ask what the signal *can* express, not what it says.
|
||||
3. Validate any checker before editing the things it flags.
|
||||
4. Fix at the root; never weaken an assertion to make it pass.
|
||||
5. Gate the class so it cannot return (a seam), then prove the seam fails on the
|
||||
defect.
|
||||
|
||||
## Verification
|
||||
|
||||
- Vault links: `./scripts/check-vault-links.py` (exit 1 on breaks).
|
||||
- Prometheus panels/rules: query each panel expression against the API — 0 series on
|
||||
a panel that should show data is a defect, 0 `ALERTS` when nothing fires is not.
|
||||
- Suite: `./tests/run.sh`; declares `ALL TESTS PASSED`.
|
||||
- A gate you changed: recreate the defect and confirm red.
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: status-surface-dead-signal-audit
|
||||
description: "Audit a Dank-fleet status surface (homepage cards, Prometheus targets, tests) for entries that can never be green because the producer cannot reach the target, classify them as outage vs config defect, and fix honestly (wip, repoint, blackbox) instead of silencing them."
|
||||
---
|
||||
|
||||
# Dead-signal audit on the Dank fleet's status surfaces
|
||||
|
||||
Use when a status board, Prometheus target list, alert set or test suite has
|
||||
permanently non-green entries, or before trusting one of those surfaces. The
|
||||
failure mode is always the same shape: a signal that **cannot** succeed by
|
||||
construction, which teaches the operator to ignore the surface it lives on.
|
||||
|
||||
## The surfaces, and what can actually be probed
|
||||
|
||||
| Surface | Producer's vantage point | Can reach |
|
||||
|---|---|---|
|
||||
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
||||
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, loopback of that host only |
|
||||
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
||||
|
||||
Anything else — another host's loopback, or a container-internal port that was
|
||||
never published — is unreachable no matter how healthy the service is.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **List the non-green entries.** Homepage: `python3 - boardcheck.py
|
||||
/srv/homepage/status.json` on Dankclaw (`status.json` holds one dict per card
|
||||
with `state` up/down/wip). Prometheus: query
|
||||
`/api/v1/targets?state=active` and print `health != "up"` with `lastError`.
|
||||
2. **Classify each one by producer reach**, not by the service. Ask: can the
|
||||
thing that probes it get there at all?
|
||||
3. **Fix in the direction of honesty, never of silence:**
|
||||
- unreachable target → mark the card `wip` with the reason in its
|
||||
`description` (the checker skips `wip`), or repoint the job at something
|
||||
reachable. Do **not** open a database port on the LAN to make a badge
|
||||
green, and do **not** delete a scrape job — an absent scrape and a healthy
|
||||
one are indistinguishable.
|
||||
- wrong path / wrong content type → fix the path (e.g. Gitea's `/metrics`
|
||||
needs metrics enabled), or move page-only targets to a blackbox exporter in
|
||||
probe mode.
|
||||
- failing test → fix the selector. A chronic red test is a defect with an
|
||||
owner, not a baseline (`find … | head -1` picked another service's file).
|
||||
4. **Verify on the surface itself**, not in the config: re-read `status.json`
|
||||
(expect 0 `down`), query `up == 0` targets, run the suite to green.
|
||||
5. Write the rule where the next person will read it — the Homepage and
|
||||
Observability pages both state it now.
|
||||
|
||||
## Traps
|
||||
|
||||
- The producer's vantage point is the whole question; a card or job can be
|
||||
"down" for a service that answers perfectly from the host it runs on.
|
||||
- `wip` is the existing convention for "real service, not probeable" (Gluetun,
|
||||
WireGuard/UDP), not a suppression: keep the reason in the description.
|
||||
- Alert rules must not depend on signals that are always red; scope them to
|
||||
the services that can actually be healthy (`LlmEndpointDown` is scoped to
|
||||
`frank-*` for exactly that reason).
|
||||
Reference in new issue
Block a user