agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
Frank
+6
-6
@@ -24,16 +24,16 @@ providers:
|
||||
- mojeek
|
||||
- public
|
||||
modelRoles:
|
||||
smol: llamacpp (dankserver)/frank-small:off
|
||||
slow: llamacpp (dankclaw)/frank-large:medium
|
||||
vision: llamacpp (dankclaw)/frank-large:auto
|
||||
vision: llamacpp (dankclaw-3090)/frank-small:xhigh
|
||||
plan: llamacpp (dankclaw)/frank-large:medium
|
||||
designer: llamacpp (dankclaw)/frank-large:medium
|
||||
commit: llamacpp (dankserver)/frank-small:medium
|
||||
tiny: llamacpp (dankserver)/frank-small:off
|
||||
task: llamacpp (dankclaw)/frank-large:medium
|
||||
commit: llamacpp (dankserver)/frank-medium:medium
|
||||
tiny: llamacpp (dankclaw-3090)/frank-small:off
|
||||
task: llamacpp (dankclaw-3090)/frank-small:off
|
||||
advisor: llamacpp (dankclaw)/frank-large:medium
|
||||
default: llamacpp (dankserver)/frank-small:medium
|
||||
default: llamacpp (dankserver)/frank-medium:medium
|
||||
smol: llamacpp (dankclaw-3090)/frank-small:xhigh
|
||||
symbolPreset: nerd
|
||||
composer:
|
||||
shape: claude
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
name: amd-multigpu-cooperation-triage
|
||||
description: "Decide whether two (or more) AMD GPUs on a host can work together on one model, or must run as independent single-GPU replicas, by checking PCIe atomic-op routing on the full path, the runtime's collective needs, and the non-collective alternatives."
|
||||
---
|
||||
|
||||
# AMD Multi-GPU Cooperation Triage
|
||||
|
||||
When someone wants N GPUs "working together" on one model (TP/PP), decide *feasibility first* — the answer on consumer AMD platforms is often "no, and here is the $30 fix".
|
||||
|
||||
## 1. Check atomic-op routing on the whole path
|
||||
|
||||
RCCL collective and P2P kernels carry a hidden **hostcall buffer** parameter. ROCr only supplies it when the device reports PCIe atomics, which is read **once at driver probe** from a *queried hardware property*:
|
||||
|
||||
```cpp
|
||||
void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp
|
||||
link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
|
||||
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
|
||||
if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else, no override
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
# Endpoint capability + negotiated link, and EVERY upstream bridge
|
||||
for d in <gpu-btfs>; do sudo lspci -vv -s $d | grep -E 'LnkCap:|LnkSta:|AtomicOpsCap|AtomicOpsCtl'; done
|
||||
lspci -tv # is the card on a CPU root port or behind the chipset?
|
||||
```
|
||||
|
||||
- Endpoint must show `AtomicOpsCtl: ReqEn+`.
|
||||
- Every hop to the CPU must show `AtomicOpsCap: Routing+`. `Routing-` is a **read-only capability** — no software can enable it, and `setpci ... CAP_EXP+28.w=0440` does not help because the driver already consumed the value at probe.
|
||||
|
||||
Failure signature (never names atomics): `AMD_LOG_LEVEL=3` → `Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!` → `hipLaunchKernel: hipErrorIllegalState` on `ncclDevKernel_Generic_4`.
|
||||
|
||||
## 2. Map the board before recommending anything
|
||||
|
||||
Read the **vendor's techspec page**, not a review: which slots are CPU-attached vs chipset, each slot's *electrical* width (a "x16 slot" can be x1 mode), and which M.2 slots the CPU owns. Free CPU-attached M.2 slots are gold: an M.2→PCIe x4 riser puts a card on the CPU root complex for ~$30 instead of a ~$300 x8/x8 board swap. Bandwidth needed for TP allreduce is small: hidden_states ≈ 10 KB per token at batch 1, ~10 MB per token-step for a 27B at batch 4 → ~1.4 ms at Gen4 x4.
|
||||
|
||||
## 3. Prove what the runtime actually needs (do it OFF the production host)
|
||||
|
||||
Bare 2-rank probe, one device per rank, with a short PG timeout so a walled op raises instead of hanging:
|
||||
|
||||
```python
|
||||
torch.distributed.init_process_group("nccl", init_method="tcp://127.0.0.1:PORT",
|
||||
world_size=2, rank=r, timeout=timedelta(seconds=25))
|
||||
# probe: broadcast, all_reduce, barrier, send/recv, all_gather
|
||||
```
|
||||
|
||||
- Expect the chipset-attached rank to fail **every** op — including `send/recv`, which is what pipeline parallelism uses, so PP=2 is not an escape from a TP wall.
|
||||
- `world_size=1` always succeeds; one-card success proves nothing.
|
||||
- Run the probe with a *real file* (`torch.multiprocessing.spawn` re-executes it — piping a script into `python3 -` fails with `FileNotFoundError: '/<stdin>'`).
|
||||
- **Stop and mask the serving unit first** (`systemctl stop X && systemctl mask X`; mask, never disable). A hung dispatch can wedge the GPU driver and take the host down — this has happened.
|
||||
|
||||
## 4. Don't be fooled by "collective-free" claims
|
||||
|
||||
Check the gate, not the feature list. vLLM's IPC custom allreduce (`custom_ar=True` when the ops are compiled) is gated by a **CUDA-sm-keyed table** (`CUSTOM_ALL_REDUCE_MAX_SIZES = {'9.0','10.0','10.3','10.7'}`) while ROCm reports e.g. `DeviceCapability(major=12, minor=0)` — so it is off on RDNA4 by a *code* limit, and `current_platform.is_rocm()` exclusions appear nearby. Even patching it is insufficient: TP init's `broadcast`/`barrier`/`all_gather` have no non-RCCL path and die the same way. **No RCCL kernel involving the atomics-less card can launch** ⇒ anything built on RCCL (TP, PP, P/D KV transfer) is out.
|
||||
|
||||
## 5. The option set, ranked
|
||||
|
||||
| Option | Works when | Cost | Trade |
|
||||
|---|---|---|---|
|
||||
| CPU-attached M.2 (x4) riser | a CPU-lane M.2 slot is free | ~$30 | TP=2 with x4 bandwidth — sufficient |
|
||||
| CPU x8/x8 board | board supports CPU bifurcation | ~$300+ | best bandwidth, no riser |
|
||||
| llama.cpp/ik_llama.cpp `--split-mode layer` | always (no collectives) | $0 | GGUF model; loses paged attention, continuous batching, spec decode |
|
||||
| Independent TP=1 replicas + router | always | $0 | cards never share one model; best throughput today |
|
||||
| Patch the custom-AR table | never | — | other collectives still die |
|
||||
| P/D disaggregation | ROCm KV-transfer connector + collectives | — | out on this class of host |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- A one-card collective test proves nothing (NCCL short-circuits at `world_size=1`).
|
||||
- Validation must come from the peer/host doing the work: probe reachability with `ip neigh show <ip>` — `FAILED`/`INCOMPLETE` means off or NIC dead, whereas a wedged kernel usually still answers ARP.
|
||||
- Validate your own probe's naming before reporting a service outage (e.g. `systemctl --user is-active gitea` vs the real unit `gitea-server`).
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: container-config-drift-crashloop
|
||||
description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order."
|
||||
---
|
||||
|
||||
# A container that restarts forever while its definition is correct
|
||||
|
||||
## When to use
|
||||
|
||||
A container is crash-looping or perpetually "unhealthy", but the file that
|
||||
defines it (compose service, unit, deploy script) looks right. Also when a
|
||||
service has been quietly down for days and nobody noticed, because a restart
|
||||
policy kept something running.
|
||||
|
||||
The cause is usually **config drift**: the running container was created from an
|
||||
older definition, and the restart policy restarts it instead of recreating it.
|
||||
A compose/unit edit does not touch an existing container.
|
||||
|
||||
## Diagnose (5 minutes, in this order)
|
||||
|
||||
1. **Who started it?** The label names the owner, not your guesswork:
|
||||
|
||||
```
|
||||
podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
|
||||
```
|
||||
|
||||
`podman-compose@…` means compose owns it, whatever unit files exist.
|
||||
Compare `created` with when the definition last changed. A container older
|
||||
than the edit that should have replaced it is the bug.
|
||||
|
||||
2. **What is it ACTUALLY running?** Read the argv, not the summary fields:
|
||||
|
||||
```
|
||||
podman inspect <name> --format '{{.Config.CreateCommand}}'
|
||||
```
|
||||
|
||||
Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device`
|
||||
flags are present — a false negative that can send you hunting the wrong
|
||||
root cause. `CreateCommand` is the truth; grep it for the flag you care
|
||||
about (`--device`, `--group-add`, `-e`, mounts).
|
||||
|
||||
3. **What does the process say?** `podman logs <name> | grep -iE 'error|cannot|Runtime'`.
|
||||
A capability error (`No HIP GPUs are available`) alongside a missing device
|
||||
flag in step 2 confirms drift rather than a code or image fault. Count the
|
||||
damage: `podman inspect --format '{{.RestartCount}}'`.
|
||||
|
||||
## Fix
|
||||
|
||||
Quickest unblock: recreate from the definition (`podman compose up -d
|
||||
--force-recreate <svc>`), which is what should have happened automatically.
|
||||
|
||||
Durable fix, when the host is governed by a repo that requires boot-persistent
|
||||
units (this one does): migrate the service to a Quadlet. Order that worked:
|
||||
|
||||
1. build the image and `install` the unit file to `~/.config/containers/systemd/`
|
||||
2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a
|
||||
leftover `~/.config/systemd/user/<svc>.service` collides by name with the
|
||||
unit the Quadlet generator produces
|
||||
3. `podman rm -f <container>` — the stale container holds the name the unit's
|
||||
container needs, and it carries the old config
|
||||
4. `systemctl --user daemon-reload && systemctl --user restart <svc>.service`
|
||||
5. delete the compose service so only one owner remains per container name
|
||||
|
||||
## Verify the capability, not the status
|
||||
|
||||
`Up` and even `healthy` can be true of a half-configured container. Check the
|
||||
thing that was broken, from inside:
|
||||
|
||||
```
|
||||
podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
|
||||
ss -ltn | grep <port> # confirm the bind addresses, not the intent
|
||||
```
|
||||
|
||||
And confirm boot persistence the way the generator exposes it:
|
||||
`systemctl --user is-enabled <svc>.service` → `generated` is healthy; an
|
||||
explicit `enable` fails on generated units.
|
||||
|
||||
## Related reading in this repo
|
||||
|
||||
- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
|
||||
- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident
|
||||
- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: fleet-container-ownership-audit
|
||||
description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them."
|
||||
---
|
||||
|
||||
# Finding the containers that will not survive a reboot
|
||||
|
||||
Use when auditing a Podman fleet for boot persistence, when a service is
|
||||
"up" but stopped working at some past reboot, or when a compose-managed stack is
|
||||
being converted to Quadlet units. This is the *detection* half; the conversion
|
||||
checklist is separate.
|
||||
|
||||
## The one-line test
|
||||
|
||||
Every running container on a boot-persistent host should carry a
|
||||
`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and
|
||||
rootless Podman does not restart bare containers after the explicit stop that
|
||||
shutdown performs — so it will be missing tomorrow morning.
|
||||
|
||||
```bash
|
||||
for c in $(podman ps --format '{{.Names}}'); do
|
||||
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
|
||||
done
|
||||
```
|
||||
|
||||
Compare the result against the host's installed units
|
||||
(`ls ~/.config/containers/systemd/*.container`) and the repo's
|
||||
`<host>/<service>/quadlet/`. Three outcomes, three meanings:
|
||||
|
||||
- **label present** → the unit owns it; boot persistence comes from `[Install]
|
||||
WantedBy=default.target` plus `loginctl show-user <user> -p Linger` = yes.
|
||||
- **label absent, compose service exists** → the classic gap: it dies at reboot.
|
||||
Also check the publish address while you are there; compose forms often bind
|
||||
the wildcard, which puts an unauthenticated UI/API on every interface.
|
||||
- **label present AND a compose service with the same name** → a second owner.
|
||||
The runtime is fine, but a stray `podman compose up` can create a duplicate
|
||||
that fights the unit for the container name and the port. Remove the compose
|
||||
service and leave an explanatory comment; that is the repo's convention.
|
||||
|
||||
## Traps that make this hard to see
|
||||
|
||||
- **Editing a compose service does not replace a running container.**
|
||||
`restart: unless-stopped` will happily keep an old container — old image, old
|
||||
device list, old mounts — alive forever. One instance ran 9,146 restart
|
||||
attempts on stale config before anyone looked. If the container predates the
|
||||
edit, recreate it explicitly; do not debug the service.
|
||||
- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.**
|
||||
It can read empty on a container that demonstrably has the devices. Read the
|
||||
real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside:
|
||||
`podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available())'`.
|
||||
- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape
|
||||
that flips from `connection refused` to `unsupported Content-Type "text/html"`
|
||||
means the service came up and only the scrape is wrong. Read the error, not
|
||||
just up/down — and never "fix" a red probe by deleting the job, because an
|
||||
absent scrape and a healthy one look identical.
|
||||
- **A hand-written `~/.config/systemd/user/<svc>.service` that shells out to
|
||||
`podman compose up` collides by name with the Quadlet the generator produces
|
||||
for `<svc>.container`.** Retire it to a dated trash directory before
|
||||
installing the unit.
|
||||
- **`is-enabled` saying `generated` is the healthy state** for Quadlet units;
|
||||
`systemctl --user enable` fails on them.
|
||||
|
||||
## Migration shape that works
|
||||
|
||||
1. Unit file in the repo at `<host>/<service>/quadlet/<service>.container`:
|
||||
`Image=` with the `localhost/` prefix for locally built images, `Pull=never`,
|
||||
explicit `PublishPort` (loopback unless the service is the documented LAN
|
||||
exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed,
|
||||
`Restart=always`, `[Install] WantedBy=default.target`.
|
||||
2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator
|
||||
splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle
|
||||
that fails silently.
|
||||
3. `deploy_<svc>()` in the remote handler: install the unit, retire any
|
||||
compose-era unit into `~/.AGENTTRASH/`, `podman rm -f <container>` so the
|
||||
unit's own can take the name, daemon-reload, restart, wait for health.
|
||||
4. Delete the compose service (one owner per container name) and leave a comment
|
||||
naming the unit, so the next reader does not re-add it.
|
||||
5. Planner: add a path rule (`<host>/<dir>/**` -> service) **and** a
|
||||
`service_is_allowed` entry, or the shared dispatcher rejects the service
|
||||
before anything runs.
|
||||
6. Migrate the seams: tests that read the compose file for devices, command
|
||||
flags or groups must read the unit instead — and resolve file paths from the
|
||||
tooling's own declarations, never from `find ... | head -1`.
|
||||
|
||||
## When the last service leaves a compose file
|
||||
|
||||
The repo's compose-validity seam requires at least one service, so a file whose
|
||||
services have all migrated fails its own test. Either keep the file as
|
||||
comments and relax that seam deliberately, or delete the file and fix the
|
||||
planner's compose-diff path — decide with the owner, because the deploy tooling
|
||||
reads it.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
name: fleet-llm-payload-purge-refresh
|
||||
description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap."
|
||||
---
|
||||
|
||||
# Purge and refresh LLM model payloads (Dank fleet)
|
||||
|
||||
Use when asked to clear / reclaim / refresh model caches on a host that is
|
||||
*serving* users, or to prove that a payload is up to date.
|
||||
|
||||
## 1. Inventory before touching anything
|
||||
|
||||
```bash
|
||||
du -sh /home/dank/models/* | sort -h # what is actually there
|
||||
```
|
||||
|
||||
Then find which of those the live services mount — never delete a payload a unit
|
||||
mounts without an explicit, in-this-conversation instruction:
|
||||
|
||||
```bash
|
||||
grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \
|
||||
/home/dank/.config/containers/systemd/*.container \
|
||||
/etc/systemd/system/frank-*.service
|
||||
```
|
||||
|
||||
On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn
|
||||
checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF +
|
||||
mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not**
|
||||
`models/checkpoints` — verify, because `models/checkpoints` looks like cache but
|
||||
has held Flux weights whose upstream is licence-gated.
|
||||
|
||||
## 2. Confirm each payload is re-fetchable BEFORE deleting it
|
||||
|
||||
Query the Hub anonymously; a gated repo 403s even with the host token:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
|
||||
```
|
||||
|
||||
An error/empty result means it is NOT safe to delete. Also filter by **quant
|
||||
name**, never by `endswith('.gguf')` — mradermacher ships large quants as
|
||||
`.partNofM` fragments which that test hides.
|
||||
|
||||
## 3. Fingerprint, then delete
|
||||
|
||||
Record byte sizes (and sha256 when the file matters) **before** removal: that is
|
||||
the only way to answer "did the refresh actually change anything" afterwards.
|
||||
|
||||
Per this repo's policy `rm` needs explicit operator permission; otherwise stage
|
||||
into `~/.AGENTTRASH/<date>-<topic>/`. Note that a same-filesystem trash reclaims
|
||||
nothing — say so rather than implying space was freed. Root-owned leftovers (a
|
||||
trashed swapfile) need `sudo`, and check `swapon --show` first.
|
||||
|
||||
## 4. Refresh, per service
|
||||
|
||||
**frank-large (halogen)** — the engine fetches for you. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit:
|
||||
|
||||
```bash
|
||||
rm -rf /home/dank/models/halogen/* # keep the directory itself
|
||||
systemctl --user start frank-large.service
|
||||
journalctl --user -u frank-large.service -f | grep -a halogen:
|
||||
```
|
||||
|
||||
The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD`
|
||||
(`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`,
|
||||
resuming if interrupted. **Trap:** podman refuses a bind mount whose source
|
||||
directory is missing — `Error: statfs /home/dank/models/halogen: no such file or
|
||||
directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty
|
||||
directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load.
|
||||
|
||||
**frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start
|
||||
with weights missing, so pull them yourself and verify:
|
||||
|
||||
```bash
|
||||
DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
|
||||
# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf
|
||||
curl -fL -C - --retry 10 -o "$DEST/<file>" \
|
||||
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
|
||||
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to the LFS oid + GGUF magic
|
||||
```
|
||||
|
||||
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always
|
||||
`-f`, always compare byte counts to the Hub's declared size, always check the LFS
|
||||
sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from
|
||||
argv entirely (it is world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
## 5. Verify, then record
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # frank-large
|
||||
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health # frank-small
|
||||
curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models
|
||||
# a REAL completion on both, then halogen's boot ledger:
|
||||
journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything"
|
||||
```
|
||||
|
||||
Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`,
|
||||
`gitea`, `loki` — `curl -sk --resolve <host>:443:192.168.0.143`) because a purge
|
||||
touches state the whole host shares.
|
||||
|
||||
## 6. Things that bite
|
||||
|
||||
- **Sidecars are versioned in place.** Equal file sizes are not proof the repo is
|
||||
unchanged, and a stale local sidecar is invisible from its name. Compare local
|
||||
sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB
|
||||
overlay update was caught.
|
||||
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The
|
||||
A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding
|
||||
cache churn there risks the OOM/state failures this fleet has already seen.
|
||||
- **Reboot is the repair of last resort** if rootless podman's pause process dies
|
||||
during the work (`invalid internal status … could not find any running
|
||||
process`); `podman system migrate` panics in 5.7.0.
|
||||
- Because a peer session may be committing to `serverAppConfig` concurrently,
|
||||
re-read a doc region immediately before editing it, and check that the edit
|
||||
response's neighbouring lines are the ones you expect.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
name: fleet-model-cache-purge-refresh
|
||||
description: "Clear and re-fetch model payloads on a Dank-fleet host that serves production LLM endpoints — inventory, protect live payloads, verify every fetch, restore the service, and use the refresh to detect upstream updates."
|
||||
---
|
||||
|
||||
Clearing a model cache on a serving host is a two-sided operation: the operator wants no stale
|
||||
payload, and the host must keep serving. Do it in this order, on the target host.
|
||||
|
||||
## 1. Inventory before touching anything
|
||||
|
||||
```bash
|
||||
du -sh /home/dank/models/* | sort -h # payload dirs, biggest last
|
||||
grep -rlE "models/|<model-name>" /home/dank/.config/containers/systemd/ /etc/systemd/system/*.service
|
||||
```
|
||||
|
||||
Classify every dir: **live service payload** (referenced by a unit/env), **retired runtime
|
||||
payload**, **unreferenced model dir**, **tool/script dir**, **logs** (evidence — keep). Check the
|
||||
service env files (`scripts/*.env`) for `MODEL_DIR`/`MODEL` paths, not just unit `Volume=` lines:
|
||||
a payload can be mounted by path and named by env. `models/logs` and small tool dirs are not cache.
|
||||
|
||||
## 2. Record fingerprints first
|
||||
|
||||
`ls -l --block-size=1 <dir>` gives sizes; for the LLM payloads compare each to the Hub's declared
|
||||
size (`/api/models/<repo>/tree/main?recursive=true&blobs=true`). They can differ — see §6. If a
|
||||
file is irreplaceable upstream (delisted revision, gated repo) say so before deleting it.
|
||||
|
||||
## 3. Protect live payloads by delete-then-refetch, never delete-only
|
||||
|
||||
Deleting a live payload leaves the endpoint unable to start. Know the recovery path *before*
|
||||
deleting: some engines fetch their own weights (halogen: `HALOGEN_DOWNLOAD`, triggered when the
|
||||
checkpoint is absent), others need a manual fetch (llama.cpp GGUF services: the serve script
|
||||
refuses to start without the weights).
|
||||
|
||||
**Bind-mount trap:** podman refuses a mount whose source directory is missing —
|
||||
`Error: statfs /home/dank/models/<dir>: no such file or directory`, with the unit looping on
|
||||
`Restart=on-failure`. Recreate the directory **empty** (`mkdir -p`), and the engine's first-start
|
||||
fetch runs.
|
||||
|
||||
## 4. Fetch with verification (never trust an exit code)
|
||||
|
||||
`curl -o FILE URL` exits 0 on 401/403 and writes the error body — a gated repo produced three
|
||||
153-byte shards with identical sha256s and a cheerful DONE line. Always: `-fL -C -`, compare the
|
||||
byte count against the Hub's declared size, and compare `sha256sum` against the LFS oid. Public
|
||||
repos means **no token in argv** (it is world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
**Parts trap:** mradermacher-style repos ship large quants as `.part1ofN … .partNofN` — a filter on
|
||||
`endswith('.gguf')` makes those quants look absent. Fetch the parts, verify each, then
|
||||
`cat part1 part2 part3 > name.gguf` in numeric order; the assembled file's size equals the sum
|
||||
(concatenation is deterministic, so per-part hashes verify the whole).
|
||||
|
||||
## 5. Restore, then verify behaviour — not health alone
|
||||
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), `reset-failed`, `start`;
|
||||
root units need `sudo`. Watch the engine's own fetch/load lines in the journal, then check
|
||||
`/health`, `/v1/models`, and a **real completion** on every restored endpoint. Read the engine's
|
||||
memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits.
|
||||
|
||||
## 6. A refresh is also an update check
|
||||
|
||||
Clearing and re-fetching only guarantees freshness *at fetch time* — if upstream has not changed,
|
||||
the bytes come back identical, which is itself the finding. Compare recorded fingerprints against
|
||||
the new local sizes and the Hub's declared sizes: sidecars are often versioned **in place**, so a
|
||||
stale copy is invisible from its file name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560
|
||||
while the main checkpoint returned byte-identical).
|
||||
|
||||
## 7. Close out
|
||||
|
||||
- Clear test payloads once the benchmark using them is done, and keep the payload the *next*
|
||||
experiment needs.
|
||||
- Leave the operator's decision points explicit: unreferenced-but-licensed dirs, other services'
|
||||
caches, and whether they want the payloads absent at their next run or fresh as of now.
|
||||
- Record sizes/verdicts in the vault (runbook + `log.md`); never `rm` without explicit permission,
|
||||
and remember a same-filesystem trash reclaims no space.
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: fleet-monitoring-gap-audit
|
||||
description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses."
|
||||
---
|
||||
|
||||
# Auditing a homelab fleet's monitoring for what *should* be there
|
||||
|
||||
Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose
|
||||
services are declared in a git repo. The two failure modes this catches are
|
||||
different, and only the first announces itself.
|
||||
|
||||
## 1. A service that should exist and doesn't is invisible
|
||||
|
||||
A missing exporter shows up as **nothing**: no failing target if the job was
|
||||
configured for a host that no longer serves, no alert because the series is
|
||||
simply absent. So check existence, not health:
|
||||
|
||||
```bash
|
||||
# Every container the host actually runs, including dead ones
|
||||
podman ps -a --format '{{.Names}} {{.Status}}'
|
||||
# Every container that is supposed to be boot-persistent
|
||||
ls ~/.config/containers/systemd/*.container
|
||||
systemctl --user is-enabled <name>.service # expect: generated
|
||||
```
|
||||
|
||||
A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic
|
||||
case: rootless Podman does not restart bare containers after the stop that
|
||||
shutdown performs, so a compose-only service dies at the first reboot and stays
|
||||
dead. Check the compose file for the name — if it is there and has no unit, the
|
||||
repo's own rule is being violated. Fix by writing the unit (mirror the same
|
||||
service on the other host), deleting the compose entry (one owner per container
|
||||
name), and making sure the deploy path can reach it:
|
||||
|
||||
- planner: service in the target's service set + a path rule for its directory
|
||||
- dispatcher allowlist: `<target>:<service>`
|
||||
- handler: install from `"$TARGET/..."`, never a hardcoded peer's path
|
||||
|
||||
## 2. Classify every down target — do not report a count
|
||||
|
||||
`targets?state=active` gives health **and** the last error. Two down targets can
|
||||
mean opposite things:
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) |
|
||||
| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine |
|
||||
| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter |
|
||||
| target absent entirely | see §1 |
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
if t['health'] != 'up':
|
||||
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
||||
# and probe each by hand before believing the scraper
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics
|
||||
```
|
||||
|
||||
Say which are down and why. "7 of 11 up" without the four reasons is how a real
|
||||
outage hides behind three config defects — that is exactly what one fleet did:
|
||||
its crash-looping GPU service was one of four failures reported as a number.
|
||||
|
||||
## 3. Check the exposure posture while you are in there
|
||||
|
||||
An exporter that binds the wildcard publishes an unauthenticated API on every
|
||||
interface. Expect exactly two listeners per host-exporter — loopback for
|
||||
host-local probes, the host's own LAN address for the Prometheus container:
|
||||
|
||||
```bash
|
||||
ss -ltn | grep <port> # must show the two explicit addresses, never 0.0.0.0
|
||||
```
|
||||
|
||||
## 4. Leave the decisions that are not yours
|
||||
|
||||
Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**:
|
||||
disable a probe to make a dashboard green, or "fix" a crash-looping service whose
|
||||
correct posture depends on what the owner wants — capture the evidence (restart
|
||||
count, exit code, last error line) and hand it over intact.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: fleet-second-llm-endpoint
|
||||
description: "Add a second LLM endpoint to a host that already serves one on the Dank fleet (new port, unit shape, co-tenancy budget, harness provider/roles, monitoring, cutover, verification) — with the port-bind race, the reasoning_effort template trap, and the tools/--jinja trap that silently break subagents."
|
||||
---
|
||||
|
||||
# Adding a second LLM endpoint to a Dank fleet host
|
||||
|
||||
Use when a host must serve two models at once (e.g. an iGPU runtime plus a
|
||||
second card), or when one endpoint is moving and a new one takes the freed
|
||||
hardware. Rebuilt from the 2026-09-16 Dankclaw split (halogen `frank-large` on
|
||||
:7770 + llama.cpp `frank-small` on :7771).
|
||||
|
||||
## 0. Decide the shape before touching hardware
|
||||
|
||||
- **One provider per HOST:PORT** in `~/.omp/agent/models.yml` (on the box where
|
||||
the harness runs, not on the servers). A second endpoint is a new provider
|
||||
block, not a second model in the existing one.
|
||||
- **Read the budget as an inequality, not a wish.** For a unified-memory runtime
|
||||
the engine's own fit check is the arbiter (`budget = host − pinned weights −
|
||||
reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV by
|
||||
counting only the full-attention layers — and remember `--parallel N` DIVIDES
|
||||
`--ctx-size`, it does not multiply.
|
||||
- **A co-tenant is a budget line, not a footnote.** Free host RAM by changing
|
||||
the POOL or the ARENA (the reserve is a fit-check term, never an allocation),
|
||||
and drop optional charges (a vision tower is ~2 GiB) rather than hoping.
|
||||
|
||||
## 1. Repo first, and retire in the same commit
|
||||
|
||||
1. New service files: `scripts/<svc>.env` (documented tunables), `.service`
|
||||
(root unit for a native binary; a Quadlet only for a container),
|
||||
`serve-<svc>.sh` (defaults + hard guards for missing artifacts).
|
||||
2. Planner: add the service to `SERVICE_ORDER` AND the target's service set in
|
||||
`scripts/deploy-plan.py`, map its paths (including "the unit/env/serve trio
|
||||
selects only me"), add it to the inventories in `deploy.sh` and
|
||||
`deploy-remote.sh`, and the dispatch case.
|
||||
3. Retire what the cutover obsoletes into `.AGENTTRASH/` with a suffix — and
|
||||
**check what pointed at it**: bench harnesses, a chat template only the old
|
||||
runtime read, and `scripts/test-changed.sh` mappings are the usual victims
|
||||
(a mapping to a test file retired earlier fails the pre-commit wrapper
|
||||
silently for weeks).
|
||||
4. Seams: one bats file per service asserting the *invariants* (`pool == 2 ×
|
||||
ctx`, KV bytes ≤ card budget, port separation), not just constant values.
|
||||
A negative grep must strip comments (`grep -vE '^\s*#' file | grep -q ...`)
|
||||
or it will match the prose that explains the very thing you removed.
|
||||
|
||||
## 2. The port-bind race (rootless podman)
|
||||
|
||||
pasta binds the published port at container CREATE. A predecessor that has just
|
||||
stopped still holds it, so a naive handler starts a doomed container:
|
||||
|
||||
```
|
||||
Deactivated successfully <- predecessor gone
|
||||
pasta: Listen failed for HOST TCP port */PORT: Address already in use
|
||||
exit 126 <- only Restart=on-failure saves you
|
||||
```
|
||||
|
||||
Order the handler: **retire the old manager's unit → stop THIS service's own
|
||||
unit (no-op on a first cutover) → wait for every socket on the port to clear
|
||||
(listener, accepted, TIME_WAIT) → install → start.** Without the second step a
|
||||
redeploy deadlocks against its own gate.
|
||||
|
||||
## 3. Wire the surfaces, and refuse impossible signals
|
||||
|
||||
- Harness roles: point the small/fast + image roles at the new endpoint, and
|
||||
move the vision role with the vision tower (only one endpoint should own
|
||||
each capability).
|
||||
- `shared/homepage/services.json`: a card per endpoint, probed from the
|
||||
monitor's own host (loopback is legal there; the LAN address otherwise).
|
||||
- Prometheus: only add a job if the runtime actually exposes Prometheus text.
|
||||
Halogen publishes `/health`, `/cache` and `/v1/*` and nothing else — scraping
|
||||
it would be red by construction, so the job is REMOVED with the reason written
|
||||
down rather than left as a permanent red.
|
||||
|
||||
## 4. Cutover, then verify by driving it
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh <target> --service <large> --service <small> # order matters
|
||||
```
|
||||
|
||||
Then, per endpoint: `/health`; `/v1/models` reports the alias; a chat completion
|
||||
returns content AND `reasoning_content`; **a request carrying `tools` returns
|
||||
`tool_calls`**; an image request is answered (or refused with a named reason);
|
||||
the runtime's own speculation counters move; and the unit is `enabled` /
|
||||
`generated` with linger on. Read the engine's boot ledger for a silent clamp.
|
||||
|
||||
Finally spend one real request through the harness per changed role — routing
|
||||
labels lie, the servers' own journals do not: `journalctl -u <svc>` shows which
|
||||
endpoint actually served it.
|
||||
|
||||
## Traps that look like model faults
|
||||
|
||||
| symptom | cause |
|
||||
|---|---|
|
||||
| `500 tools param requires --jinja flag` | the endpoint's runtime lacks the Jinja chat-template path, so every tool-using agent on it is broken. mainline llama.cpp defaults it on — pin it |
|
||||
| `500 Jinja Exception: Unexpected reasoning effort <level>` | the model's own template has a narrower accepted set (Qwen3.8: xhigh/medium/low only) than the harness's levels; fold them in `compat.reasoningEffortMap` |
|
||||
| the same request works by hand but a subagent dies | subagents run at a different thinking level — check both |
|
||||
| the container dies instantly although the predecessor "stopped" | the port race above |
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: fleet-service-precommit-iteration
|
||||
description: "Iterate on a service that lives in the serverAppConfigs git checkout on a fleet host (dankclaw/dankserver) before committing: ship files for live testing, keep the checkout pullable, then commit and deploy — covers the untracked-files pull trap that recurs every session."
|
||||
---
|
||||
|
||||
# Iterating on a fleet service before you commit
|
||||
|
||||
This repo's contract is git → `deploy.sh` → host. But live testing needs the new
|
||||
file *on the host* before it is committed, and the naive way (copy it into the
|
||||
host checkout) blocks the very deploy that would install it. This is the loop
|
||||
that works, plus the traps that cost real time.
|
||||
|
||||
## The loop
|
||||
|
||||
1. Edit in the local repo. Run the seams: `./tests/run.sh`.
|
||||
2. Ship the working files to the host **for testing**:
|
||||
`cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
|
||||
(then `chmod +x` if it is executed — scp/`cat >` drops the exec bit, and a
|
||||
container entrypoint mounted from that path fails with status 126, "command
|
||||
found but not executable").
|
||||
3. Test against the live service; iterate until it works.
|
||||
4. **Before committing**, prove the host copy is byte-identical to what you are
|
||||
about to commit: `sha256sum` locally and on the host. If they differ, you are
|
||||
about to lose work in step 5.
|
||||
5. Commit and push.
|
||||
6. Clean the host checkout so the deploy can pull:
|
||||
- untracked hand copies → move them out of the checkout
|
||||
(`mkdir -p /tmp/handcopy && mv <paths> /tmp/handcopy/`);
|
||||
- modified tracked files → `git checkout -- <paths>` (this *discards* the
|
||||
hand copy, which is safe only because of step 4);
|
||||
- confirm with `git status --short` printing nothing.
|
||||
7. `./scripts/deploy.sh <target> --service <svc>` — it pushes, pulls ff-only,
|
||||
installs units/config, restarts, waits for health, then runs the Playwright
|
||||
e2e gate.
|
||||
|
||||
## Traps, in the order they bite
|
||||
|
||||
- **The pull aborts on hand copies.** `deploy.sh` pulls with ff-only and git
|
||||
refuses: *"The following untracked working tree files would be overwritten by
|
||||
merge"*. Any untracked file in the checkout whose path the pull wants to
|
||||
create stops the deploy — which is exactly the file you just shipped. Always
|
||||
do step 6.
|
||||
- **`git checkout --` silently eats uncommitted work.** Without step 4 you can
|
||||
destroy the only copy of an edit that exists nowhere else.
|
||||
- **A runtime mounts files *straight out of the checkout*.** Here
|
||||
`dankserver/frank-medium/**` and `shared/grafana/provisioning/**` are mounted
|
||||
read-only into containers, so a hand copy takes effect at the next container
|
||||
restart with no deploy. Test that way, then let the deploy make it official.
|
||||
- **Root-owned files from a container write** (e.g. a patched `.so` produced
|
||||
inside a build container) can land in the checkout as `root:root`; `chown` them
|
||||
back to the deploy user before git operations.
|
||||
- **A restart is not a verification.** Health endpoints answer from a stale
|
||||
process, a proxied service can be served by a peer, and a provisioned
|
||||
dashboard may live in unified storage rather than the SQLite table you grep.
|
||||
Verify the *specific* claim: the new code path's own output, the file's
|
||||
checksum on the host, the API that serves the resource.
|
||||
|
||||
## Why not keep a scratch copy on the host?
|
||||
|
||||
Because the deployed artifact must be the committed one. Iterating outside git
|
||||
(the `/tmp` loop is fine for *probes*) is for measurement scripts; anything the
|
||||
service actually runs has to land in the checkout, and the checkout has to stay
|
||||
pullable. Two directories, one rule: probes may live anywhere, runtime state
|
||||
lives in git.
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: frank-medium-batch-split-retune
|
||||
description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify."
|
||||
---
|
||||
|
||||
# frank-medium batch geometry + split rebalance
|
||||
|
||||
Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the
|
||||
model on frank-medium, or when someone reports that prefill "does not respond" to
|
||||
those knobs. The batch and the tensor split are **coupled**: the compute buffers are
|
||||
not symmetric across the two cards, so any batch change moves the balance point.
|
||||
|
||||
## Order of work (do not skip the second half)
|
||||
|
||||
1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold
|
||||
a shape constant across dozens of configs:
|
||||
`scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation
|
||||
per cell, `-r 3 --no-warmup`.
|
||||
2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by
|
||||
loading the real service with a per-config env override and reading `rocm-smi`. A
|
||||
llama-bench number cannot decide the split: it has no `--parallel`, sizes its own
|
||||
context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
|
||||
3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed
|
||||
point, then **deploy** and verify.
|
||||
|
||||
## 1. The sweep
|
||||
|
||||
```bash
|
||||
# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is
|
||||
# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here
|
||||
sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
--setenv=OUT=/home/dank/models/logs/bench-bub \
|
||||
--setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \
|
||||
/home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv`
|
||||
(appended as it goes, so progress is readable), samples VRAM per cell, and restarts
|
||||
the service from an EXIT trap. A re-run skips cells that already have a `bench.json`.
|
||||
|
||||
Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not
|
||||
track the request, nothing else in the table means anything. The VRAM columns scaling
|
||||
monotonically with ubatch is the independent confirmation.
|
||||
|
||||
Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth):
|
||||
an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046,
|
||||
384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral
|
||||
(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every
|
||||
cell). Test **both** ends of the range: sampling only upward makes a shallow basin look
|
||||
like a plateau.
|
||||
|
||||
Sanity rule: the same ubatch measured under different logical batches should agree to
|
||||
under 1 tok/s. If it does not, the table is noise and the first suspect is a stray
|
||||
GPU user, not the knob.
|
||||
|
||||
## 2. The split, at the deployed shape
|
||||
|
||||
`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env`
|
||||
to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with
|
||||
`FRANK_MEDIUM_ENV_FILE=<temp>`, polls `/health` for `"status":"ok"`, reads
|
||||
`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under
|
||||
`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`.
|
||||
|
||||
**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH
|
||||
cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the
|
||||
process dies — a 180 s sleep fails, and even 600 s has failed while the process is still
|
||||
alive inside the unit. Skipping the gate produces
|
||||
`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from
|
||||
"this split does not fit" and will send you chasing a phantom OOM.
|
||||
|
||||
Free bytes to expect, and the arithmetic behind them:
|
||||
`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs
|
||||
the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB
|
||||
against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one
|
||||
layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the
|
||||
reachable minimum gap is about one DeltaNet layer, never zero.
|
||||
|
||||
2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32;
|
||||
**35,30 @512 -> 6.44 / 6.90 (deployed)**.
|
||||
|
||||
## 3. Seam, deploy, verify
|
||||
|
||||
- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured
|
||||
1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the
|
||||
model reproduces the new point. Check that neighbouring splits still FAIL the balance
|
||||
gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the
|
||||
slack at one DeltaNet layer.
|
||||
- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares
|
||||
summing to the model's 65 blocks, ROCm1 the smaller).
|
||||
- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`.
|
||||
- Verify live: the flags in `/proc/<pid>/cmdline`, `/slots` = 2 x 131072, VRAM matching
|
||||
the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance`
|
||||
lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB
|
||||
lower once the service has served traffic than straight after load.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: grafana-prometheus-provisioning-as-code
|
||||
description: "Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives."
|
||||
---
|
||||
|
||||
# Observability as code (Grafana + Prometheus, Dankclaw fleet)
|
||||
|
||||
Use when adding a scrape target, an alert rule, a datasource, or a dashboard to
|
||||
this homelab. Everything lives in git; Grafana and Prometheus only ever read
|
||||
what the deployer installs.
|
||||
|
||||
## The triad — every change touches three places
|
||||
|
||||
1. **Config in the repo**
|
||||
- `shared/prometheus/prometheus.yml` — `scrape_configs`, and `rule_files:`
|
||||
pointing at `/etc/prometheus/rules/*.yml`
|
||||
- `shared/prometheus/rules/<name>.yml` — alert/recording rules
|
||||
- `shared/grafana/provisioning/datasources/prometheus.yml` — datasource
|
||||
- `shared/grafana/provisioning/dashboards/{provider.yml,<name>.json}`
|
||||
2. **The copier in `scripts/deploy-remote.sh`** — `copy_prometheus_config`
|
||||
installs config **and** rules into `/opt/prometheus/config/` (the container
|
||||
mounts that to `/etc/prometheus`); `copy_grafana_provisioning` installs into
|
||||
`/opt/grafana/provisioning/` (the grafana unit already mounts it read-only).
|
||||
Paths are absolute and `run_privileged`.
|
||||
3. **The planner in `scripts/deploy-plan.py`** — a `shared/<group>/**` change
|
||||
must select the service that reads it (`shared/prometheus` → prometheus,
|
||||
`shared/grafana` → grafana). Missing mapping = the file deploys nowhere and
|
||||
nothing reports an error.
|
||||
|
||||
Add a seam in `tests/observability_quadlet.bats` for each change: the file
|
||||
exists, the config references it, the deployer copies it, and — where two files
|
||||
must agree (target vs published port) — that they do.
|
||||
|
||||
## Reach targets by container name, never the loopback publish
|
||||
|
||||
Datasources use `http://prometheus:9090` and scrape jobs for co-located
|
||||
services use `prometheus:9090` / `loki:3100`. Published ports are
|
||||
`127.0.0.1`-only by design (see the exposure rules), so a browser-facing URL or
|
||||
a published port in a datasource is wrong from inside the network. LLM
|
||||
endpoints are the exception: they are on other hosts, so use their LAN address.
|
||||
|
||||
## Three failure modes that produce no error message
|
||||
|
||||
- **A rule file nothing loads evaluates nothing.** Prometheus reports no
|
||||
problem for a `rule_files` entry that matches nothing, or for a rules file
|
||||
nobody references. Ship both together and assert the reference in a seam.
|
||||
- **A provisioned dashboard is not in the legacy `dashboard` table** on the
|
||||
current Grafana. Reading `grafana.db` for it returns empty and looks like a
|
||||
failed provision. It lives in unified storage behind
|
||||
`dashboard.grafana.app`; query the API instead.
|
||||
- **Lifetime averages are not per-request evidence.** e.g. an engine's
|
||||
"Prefix cache hit rate: 29 %" is every prompt it ever saw, including
|
||||
benchmarks. Measure the behaviour you care about with a targeted probe.
|
||||
|
||||
## Verify through the channel the consumer uses
|
||||
|
||||
```bash
|
||||
# Prometheus: config and rules parse/load
|
||||
ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/<name>.yml'
|
||||
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \
|
||||
| python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"'
|
||||
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...' # every target health=up
|
||||
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...' # 0 firing when healthy
|
||||
|
||||
# Grafana: provisioned objects and a live query (admin/admin on this install)
|
||||
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"'
|
||||
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/<uid>"'
|
||||
ssh dankclaw 'curl -s -u admin:admin \
|
||||
"http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"'
|
||||
```
|
||||
|
||||
A target that is `up` proves reachability, not data: also run one query for a
|
||||
series you expect and confirm it returns a sample.
|
||||
|
||||
## Grafana facts for this install
|
||||
|
||||
- No UI-configured state: `editable: false` on the datasource, fixed
|
||||
`uid: prometheus` so dashboards reference it without a lookup.
|
||||
- Dashboards re-read from disk every 30 s (`updateIntervalSeconds`), so a JSON
|
||||
edit appears without a restart — but the file must still be *installed*, which
|
||||
means a deploy.
|
||||
- There is **no Alertmanager and no contact point**: rules evaluate, alerts show
|
||||
on `/alerts`, in the `ALERTS` metric and on a dashboard panel, and reach
|
||||
nobody. Say that plainly in docs rather than implying observability.
|
||||
|
||||
## Naming metrics
|
||||
|
||||
Never write a series name from memory into a config, a rule, or a dashboard —
|
||||
one shipped comment here named a metric that does not exist. Dump the live
|
||||
names first and paste them:
|
||||
|
||||
```bash
|
||||
ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u'
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: grafana-provisioning-as-code
|
||||
description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative."
|
||||
---
|
||||
|
||||
# Grafana provisioning as code (Dankclaw fleet)
|
||||
|
||||
Use when adding or changing a Grafana datasource or dashboard on this fleet, or
|
||||
when a dashboard "does not appear" after a deploy.
|
||||
|
||||
## Shape of the thing
|
||||
|
||||
Grafana on Dankclaw is an **empty install configured entirely from git**. The
|
||||
quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a
|
||||
provisioned artefact exists in exactly three places:
|
||||
|
||||
```
|
||||
shared/grafana/provisioning/datasources/<name>.yml # repo: source of truth
|
||||
shared/grafana/provisioning/dashboards/provider.yml # dashboard loader config
|
||||
shared/grafana/provisioning/dashboards/<name>.json # dashboards
|
||||
| deploy-remote.sh: copy_grafana_provisioning()
|
||||
v
|
||||
/opt/grafana/provisioning/... # host: installed state
|
||||
| read-only mount
|
||||
v
|
||||
/etc/grafana/provisioning/... # generated: inside container
|
||||
```
|
||||
|
||||
Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be
|
||||
invisible to review, unknown to anyone reading the repo, and lost with the data
|
||||
volume.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Write the files under `shared/grafana/provisioning/`.
|
||||
- Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by
|
||||
uid, so a random one breaks them), `access: proxy`, and the URL by
|
||||
**container name** (`http://prometheus:9090`) — the published port is
|
||||
loopback-only and unreachable from inside the network.
|
||||
- Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab;
|
||||
failing an operator who wants a different rate window is the wrong trade)
|
||||
and a JSON dashboard under a stable `uid`.
|
||||
2. Add the file(s) to `copy_grafana_provisioning()` in
|
||||
`scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each
|
||||
file — follow `copy_prometheus_config` for style).
|
||||
3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify
|
||||
with `python3 scripts/deploy-plan.py --target dankclaw --paths <file>`.
|
||||
4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse,
|
||||
the datasource URL must be the container name, the unit must mount
|
||||
`/opt/grafana/provisioning`, and the deployer must call the copier. Then
|
||||
`./tests/run.sh`.
|
||||
5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the
|
||||
provisioning tree, restarts Grafana, then runs the e2e gate).
|
||||
|
||||
## Verify — and the trap that makes the obvious check lie
|
||||
|
||||
A **provisioned dashboard does not appear in the legacy `dashboard` table** of
|
||||
`grafana.db` on the current Grafana build; dashboards live in unified storage
|
||||
behind the `dashboard.grafana.app` API group. The datasource *does* land in
|
||||
`data_source`, and that asymmetry is easy to misread as "datasource worked,
|
||||
dashboard failed". A `sqlite3`/python query against `dashboard` is a **false
|
||||
negative**, not evidence.
|
||||
|
||||
Check in this order instead:
|
||||
|
||||
```bash
|
||||
# 1. provisioning actually ran (these loggers name themselves)
|
||||
podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"
|
||||
|
||||
# 2. the dashboard exists (uid is what you set in the JSON)
|
||||
curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"
|
||||
|
||||
# 3. the datasource really reaches Prometheus (proves the whole chain)
|
||||
curl -s -u admin:admin \
|
||||
"http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"
|
||||
```
|
||||
|
||||
Grafana publishes on loopback `:3002` only; the off-host path is the guarded
|
||||
vhost. Its admin login is still the default `admin:admin` — which is why every
|
||||
assertion above needs credentials, and which is worth changing.
|
||||
|
||||
## Before writing any panel or config comment
|
||||
|
||||
Grep the metric names out of the live endpoint; naming a series that does not
|
||||
exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the
|
||||
real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`,
|
||||
whose *rates* give the hit rate). A comment naming an unqueryable metric, or a
|
||||
panel that silently renders "No data", is worse than no panel:
|
||||
|
||||
```bash
|
||||
curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm
|
||||
```
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: heterogeneous-accelerator-split-triage
|
||||
description: "Decide whether splitting one model across two unequal accelerators on a single host (fast small-VRAM GPU + big slow unified-memory device, or a GPU over a narrow PCIe link) is worth building — with the KV/prefill arithmetic, the probes that settle it, and the doors that are already closed."
|
||||
---
|
||||
|
||||
# Heterogeneous accelerator split — triage before building
|
||||
|
||||
Use when someone asks "could we split model X across device A and device B" on one host, when the devices differ in memory size/bandwidth (e.g. 24 GB GDDR6X vs an APU sharing LPDDR), or when a link is narrow (PCIe x4). Produces a go/no-go with arithmetic instead of enthusiasm.
|
||||
|
||||
## 1. Characterise the link (bytes vs latency)
|
||||
|
||||
```bash
|
||||
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:" # capability vs actual
|
||||
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
|
||||
```
|
||||
|
||||
Expect **idle-downshift**: a Gen4 x4 dock can idle at Gen1 x4 (~1 GB/s) and only upshift under load. Measure while the device is busy, or you will size the design against the wrong number.
|
||||
|
||||
Compute the per-hop cost against the hidden state (`n_embd x 2 B` for f16): a 5120-dim model moves 10 KiB/hop = ~1.3 µs at Gen4 x4. Then:
|
||||
|
||||
- bytes per token = hops/token x hidden state. Typically ~1 MiB/token for 2 hops/layer x 64 layers -> ~0.3 % of a 40 ms token. **Bytes are almost never the constraint.**
|
||||
- the real budget line is **hop COUNT**: every device boundary is a host-side sync. A runtime that splits per *op* can hit 100 splits/token and pin the GPU at ~5 % SM. A runtime that batches per-backend graphs is far cheaper — measure `graph splits` in the loader log, do not assume.
|
||||
|
||||
**Design rule: partition by where the weights live, never by where the activations flow.** Resident weights, activations crossing.
|
||||
|
||||
## 2. Get the workload's real bottleneck (KV is usually it)
|
||||
|
||||
KV per position x context = bytes read per token, because attention reads the whole cached context every token:
|
||||
|
||||
```
|
||||
KV/token = kv_bytes_per_position * context
|
||||
```
|
||||
|
||||
Compare that against (a) expert/weight traffic per token (read once per token per layer) and (b) total token time. At long context or on a hybrid model with few attention layers but many positions, **KV dominates**. Then:
|
||||
|
||||
- the highest-leverage placement is **KV cache on the fastest-memory device, together with its attention compute** — never KV alone (separating them puts `KV/token` over the link, which is fatal: GBs across a 7.88 GB/s link = hundreds of ms/token);
|
||||
- check the KV + attention weights actually fit: `kv_positions_per_slot * slots * bytes_per_position + attention_trunk <= VRAM`.
|
||||
|
||||
## 3. What placement never buys
|
||||
|
||||
Prefill is compute-bound and expert weights are read **once** per batch, so moving them to faster memory buys nothing while adding a serial stage. Expect a regression, and size it from calibration: on Strix Halo + RTX 3090 (Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the GPU gave **decode +12 %, prefill 192 -> 32 tok/s**.
|
||||
|
||||
A `1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that decode gain to within ~2 points, so it is usable for decode projections — and it assumes the stages are sequential (true at batch 1: no pipelining).
|
||||
|
||||
## 4. Is the second device even a separate pool?
|
||||
|
||||
On a UMA APU, check before assuming additivity:
|
||||
|
||||
```bash
|
||||
cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small)
|
||||
cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
|
||||
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
|
||||
```
|
||||
|
||||
A large GTT window means the "GPU memory" is the *same* LPDDR under one memory controller: adding a stage there adds a compute engine, not capacity or bandwidth. Only the discrete card's VRAM is additive. Fencing more (`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
|
||||
|
||||
## 5. Speculative decoding as the "cheap" split
|
||||
|
||||
A drafter on the *other* device is bandwidth-free (token ids per round, tens of bytes), so it is tempting. Before proposing it:
|
||||
|
||||
- the drafter needs the target's **exact tokenizer** for a plain draft model (verify by comparing GGUF `tokenizer.ggml.tokens`, not by trusting the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between models);
|
||||
- a plain drafter costs **K sequential forwards per round**, so it must be 10-50x faster than the target. Compute the break-even: with `r = t_draft/t_target`, `rate_ratio = A/(K*r + 1)`, `A = (1-p^(K+1))/(1-p)`. At `r ≈ 0.4-0.5` (a drafter only ~2x faster) break-even needs `p ≈ 0.7+` at K=3;
|
||||
- always compare against the **incumbent**, not the no-draft baseline: a target already running its own MTP head (e.g. ~1.7 tokens committed/round) is a high bar;
|
||||
- check the runtime can even accept an external drafter. Closed engines may enumerate a fixed drafter set and 400 anything else (`DRAFTER_CAP`-style capability gates keyed on the loaded checkpoint) — that is a design boundary, not a config gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd` (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd-ctvd`.
|
||||
|
||||
## 6. Expressing a split (llama.cpp, the usual vehicle)
|
||||
|
||||
```bash
|
||||
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest)
|
||||
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (what KV co-location needs)
|
||||
```
|
||||
|
||||
- buffer types are matched by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values;
|
||||
- `-cmoe` / `-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
|
||||
- several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist), and `GPU_BACKEND_A=OFF` in the CMakeCache of the existing build tells you what needs adding;
|
||||
- contiguous ranges are the *wrong* split when KV bandwidth dominates, because they drag half the attention layers (and their KV) back to the slow pool.
|
||||
|
||||
## 7. Hazards checklist
|
||||
|
||||
- paged/sidecar tables (huge n-gram/embedding tables) must stay where they are read from; moving them is pure loss;
|
||||
- keep `mmap` on for models with huge embedding tables (anonymous memory gets the process cgroup-killed);
|
||||
- CUDA graph capture can abort on unusual graph shapes (hyper-connections etc.) — check before assuming optimisations apply;
|
||||
- the second device's brand-new backend may not implement the arch's ops at all: make the first measurement a **baseline of the runtime on that device alone**, because a split's arithmetic is worthless if the baseline is worse than the incumbent engine;
|
||||
- a split consumes the fast device entirely — check what else it was serving before trading it away;
|
||||
- benchmark/experiment discipline on production hosts: stop and mask first.
|
||||
|
||||
## 8. Verdict shape
|
||||
|
||||
Rank and report, and prefer the boring answer when it wins:
|
||||
|
||||
1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link;
|
||||
2. **One process, weights-resident, KV co-located with attention on the fast pool** — promising, quantifiable (+% decode at long context), needs a dual-backend build and a baseline measurement;
|
||||
3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check `misses x bytes x tok/s` against link bandwidth) and overlapped, and it injects PCIe latency into the critical path; no mainstream runtime implements a dynamic expert cache;
|
||||
4. list explicitly what does *not* help: TP/collectives across unequal devices, KV on the far device, model loading over the link, prefill-motivated placements.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
name: hf-gguf-payload-fetch-verify
|
||||
description: "Inventory, fetch, hash-verify and assemble a Hugging Face GGUF payload (including mradermacher's .partNofM fragments and gated mirrors) before pointing an LLM service at it — on the Dank fleet or anywhere else."
|
||||
---
|
||||
|
||||
# Fetch and verify an HF GGUF payload
|
||||
|
||||
Use before any model swap where the weights come from Hugging Face, especially
|
||||
large (>32 GiB) quantised models. The failure modes below all look like success.
|
||||
|
||||
## 1. Inventory — query by quant name, never by extension
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<org>/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
for e in json.load(sys.stdin):
|
||||
p=e.get('path','')
|
||||
if 'Q4_K_S' in p: # the QUANT, not '.gguf'
|
||||
print(p, e.get('size',0), (e.get('lfs') or {}).get('oid','-'))
|
||||
"
|
||||
```
|
||||
|
||||
**Large quants ship as `.partNofM` fragments.** `mradermacher/*-i1-GGUF` repos
|
||||
hold e.g. Q4_K_M as `part1of3/2of3/3of3` (37.00+37.00+36.97 = 110.97 GiB) and
|
||||
Q4_K_S the same way, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are
|
||||
single `.gguf` files. Filtering on `endswith('.gguf')` therefore makes Q4_K_S
|
||||
look like it does not exist — do not conclude a quant is missing from an
|
||||
extension-filtered listing.
|
||||
|
||||
Check both mradermacher repos: `…-i1-GGUF` (imatrix) and `…-GGUF` (plain), and
|
||||
the model's own repo. Gated mirrors exist (`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`
|
||||
carries the same ladder) and return **403 even with a valid HF token** until
|
||||
that account accepts the gate — prefer the ungated copy rather than chasing access.
|
||||
|
||||
## 2. Fetch — `-f` is mandatory, secrets stay out of argv
|
||||
|
||||
```bash
|
||||
# token out of band: mode-600 config, never -H on the command line
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
|
||||
chmod 600 /tmp/hf.curlrc
|
||||
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc \
|
||||
-o "$DEST/$name" "$BASE/$name"
|
||||
```
|
||||
|
||||
- `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes the error body as the
|
||||
artifact. A gated repo yields identical small files (153 bytes) that hash the
|
||||
same and a `DOWNLOAD_DONE` in seconds for a 104 GiB payload.
|
||||
- A token in `-H "Authorization: Bearer …"` is visible in `/proc/<pid>/cmdline`
|
||||
to any `ps`/`pgrep -a` on the host. Never `pgrep -a` a running fetch.
|
||||
|
||||
## 3. Verify, then assemble
|
||||
|
||||
```bash
|
||||
# per part: byte count must equal the Hub's declared size, sha256 must equal lfs.oid
|
||||
sha256sum "$PARTS"/*.part* | sort
|
||||
|
||||
# concatenate in NUMERIC order, then verify the total and the container magic
|
||||
cat "$PARTS"/X.gguf.part1of3 "$PARTS"/X.gguf.part2of3 "$PARTS"/X.gguf.part3of3 > "$OUT"
|
||||
stat -c%s "$OUT" # must equal the sum of the part sizes
|
||||
head -c 4 "$OUT" # must print GGUF
|
||||
```
|
||||
|
||||
The assembled file has no published hash (only the parts do), so the per-part
|
||||
LFS oids plus an exact total are the verification chain. If a size check fails,
|
||||
suspect your own arithmetic before the payload: the sum of the parts is the
|
||||
expected value, and it is easy to mis-add by hand (`python3 -c "print(a+b+c)"`).
|
||||
|
||||
## 4. Housekeeping
|
||||
|
||||
- Failure artifacts (error pages) and stray directories: move to
|
||||
`~/.AGENTTRASH/YYYYMMDD-topic`, never `rm`; list them and wait for approval to
|
||||
purge.
|
||||
- Keep the `.partNofM` fragments after assembly when disk allows: they are the
|
||||
only copies that can be re-verified against the Hub. Note the total they cost
|
||||
and ask before purging.
|
||||
- Never delete the payload being replaced as a side effect of a swap: upstreams
|
||||
delist quants (both mradermacher and orcarouter have delisted Q4_K_M for
|
||||
Qwen3.8-Flash-Next-Uncensored), and the local file may be the last copy.
|
||||
|
||||
## 5. Waiting for a long fetch
|
||||
|
||||
Do not `pgrep -f <script>` where the pattern appears in your own command line —
|
||||
the loop matches itself and spins its full iteration count while the download is
|
||||
fine. Watch the artifact instead:
|
||||
|
||||
```bash
|
||||
while [ "$(stat -c%s "$DEST/$name" 2>/dev/null || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done
|
||||
```
|
||||
|
||||
## 6. Before choosing a smaller quant
|
||||
|
||||
A smaller quant buys memory and ~3–5 % decode, and costs quality. It does **not**
|
||||
change what the runtime can express: on llama.cpp, per-tensor cross-device
|
||||
placement is refused whenever `--override-tensor` is set
|
||||
(`has_tensor_overrides()` gates `pipeline_parallel`, which sets `n_copies`), and a
|
||||
layer split can only move contiguous layer ranges. State which of the two you are
|
||||
buying — headroom or a design change — before spending 100 GiB of bandwidth.
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: ik-llama-hybrid-moe-performance
|
||||
description: Diagnose and tune ik_llama.cpp hybrid CPU/GPU MoE decode/prefill on the dankclaw 3090+Strix Halo frank-large deployment; includes the barrier-split root-cause method and the systemd-unit test-interference trap
|
||||
---
|
||||
|
||||
# ik_llama.cpp hybrid MoE performance campaign (dankclaw frank-large)
|
||||
|
||||
Use when tuning or debugging decode/prefill speed of the ik_llama.cpp hybrid deployment (RTX 3090 + Strix Halo LPDDR, CPU-resident MoE experts) on dankclaw, or when a config experiment gives surprising numbers.
|
||||
|
||||
## Measured envelope (2026-09-16)
|
||||
|
||||
- Decode 3.8-4.3 tok/s, flat across thread counts (T1 ≈ T16). Prefill ~192 tok/s at 1.8k-32k prompts = the CPU AVX512-VNNI compute ceiling for CPU experts (batch amortizes expert reads; compute-bound, NOT memory-bound).
|
||||
- Decode root cause: ~100 CPU↔GPU backend splits/token; GPU at 5% SM during decode (launch/sync-bound over the PCIe x4 dock); T16 profile shows ~90% of cycles in `pause` (ggml_barrier spin) because threads wait on the serial critical path. Memory placement (mmap/anon/THP/deferred) does NOT matter — all variants land 3.8-4.6.
|
||||
- CUDA-graph capture aborts: qwen4exp's hc broadcasts use per-layer `ggml_repeat_4d` whose CUDA path does a capture-illegal op. Until ik gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs stay dead. Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels 4x slower than ik iqk) — do not switch for speed.
|
||||
- MTP/speculative impossible for mradermacher i1 GGUFs: the NextN head is stripped (no blk.48 tensors).
|
||||
|
||||
## Tuning levers, ranked
|
||||
|
||||
1. `-ncmoe 42` puts 6 layers' experts on the 3090: decode 4.54, prefill (small prompt) 32 — but VRAM 22/24.6 GiB at 262k ctx; needs an OOM-margin decision from the operator.
|
||||
2. `--prefetch-experts` is a no-op when the page cache is resident.
|
||||
3. `-thp` huge-mmap usually fails (`Cannot allocate memory`) and falls back silently.
|
||||
|
||||
## Traps
|
||||
|
||||
- The frank-large root systemd unit has Restart=always: `pkill`ing llama-server respawns it in ~4 s. For manual experiments: `sudo systemctl stop frank-large.service` first (root unit — NOT `systemctl --user`, that is stale halogen-era docs); verify `pgrep -xc llama-server` == 1 and 'HTTP server listening' in the current log before trusting any measurement.
|
||||
- Any `-ot`/`-ncmoe` override disables mmap for the whole model load in ik (`create_tensors` returns false).
|
||||
- `-ncmoe N` without `GGML_CUDA_NO_PINNED_WEIGHTS=1` page-locks ~108 GiB (co-tenant killer — the 2026-09-15 incident).
|
||||
- `--defer-experts` defers only PLE; under `-ngl 999` the experts still target CUDA0 → cudaMalloc OOM (~79.7 GiB on a 24 GiB card).
|
||||
- Never run `-j16` builds concurrently with decode measurements — both results are garbage.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: incident-window-reconstruction
|
||||
description: "Reconstruct which command/config was running at the instant an incident symptom first appeared, from on-disk artifacts (log mtimes + per-run durations + journal first-failure time) when memory and /tmp are gone."
|
||||
---
|
||||
|
||||
# Incident window reconstruction
|
||||
|
||||
Use when the question is **"what was running when it broke?"** — an incident,
|
||||
outage, resource blow-up, or unexplained service death — and the answer must
|
||||
come from artifacts, not recollection. Typical triggers: a host went down during
|
||||
an experiment, a service died mid-run, someone asks for the parameters in effect
|
||||
at the moment of failure.
|
||||
|
||||
## Principle
|
||||
|
||||
Three independent clocks, cross-checked — never recollection:
|
||||
|
||||
1. **Per-run duration** — every script wraps commands in `time`, so the log
|
||||
holds `real 4m19.129s`.
|
||||
2. **Log mtime** — the last write of that log.
|
||||
3. **Journal first-failure instant** — the first error line from the affected
|
||||
service.
|
||||
|
||||
`start ≈ mtime − real` for the *last* run in a log, and each run nests inside
|
||||
the previous one's window. The run whose `[start, end]` contains the journal's
|
||||
first-failure instant is the answer.
|
||||
|
||||
## Collect
|
||||
|
||||
```bash
|
||||
# 1. ordering and clocks of every artifact (log mtimes)
|
||||
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
|
||||
|
||||
# 2. durations + per-run section markers + the exact command lines
|
||||
ssh <host> 'grep -ah "^real" <logdir>/*.log'
|
||||
ssh <host> 'for f in <names>; do echo "## $f"; \
|
||||
grep -aE "^####|^\+ /path/to/binary" <logdir>/$f.log \
|
||||
| sed "s/\(.\{130\}\)/\1\n /g"; \
|
||||
grep -aE "^real|_EXIT=" <logdir>/$f.log; done'
|
||||
|
||||
# 3. first failure of the affected service, and what preceded it
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -aiE "Started|Created container" | tail -6'
|
||||
```
|
||||
|
||||
## Traps (all hit for real)
|
||||
|
||||
- **`/tmp` does not survive a reboot.** Launch experiment scripts from a
|
||||
persistent path (e.g. `~/models/...`) and log beside them; if the repair path
|
||||
is a reboot (common), `/tmp` scripts *and* any sampler output written there are
|
||||
gone. On the Dank fleet: `/home/dank/models/logs/<tag>.log` survived while
|
||||
`/tmp/x4*.sh` did not.
|
||||
- **A log's mtime is not proof the run finished.** A log truncated (or rotated)
|
||||
while a script still holds the fd keeps growing from that point; a 6-line log
|
||||
with an `ALL_DONE` marker means the earlier bulk was removed, not that nothing
|
||||
ran.
|
||||
- **Long command lines are display-truncated by agent tools.** Wrap before
|
||||
reading: `sed 's/\(.\{130\}\)/\1\n /g'`. Otherwise you will report
|
||||
parameters that were never used.
|
||||
- **Off-by-one file names silently skip evidence.** `x4b.log` vs
|
||||
`x4b-experiment.log` in a `for f in …; do [ -f $f.log ]` loop made half the
|
||||
artifacts invisible. List the directory and match exactly.
|
||||
- **Overlap is a hypothesis, not an observation.** Two resource-heavy
|
||||
invocations in the same minute may or may not have been resident at once;
|
||||
say so explicitly unless a sampler or cgroup counter proves it.
|
||||
- **Do not invent the cause.** Reconstruct the *window* (evidence) and keep the
|
||||
*mechanism* separate: grep the proof that would establish it (`journalctl -k |
|
||||
grep -i oom` for a killed process, driver logs for a GPU fault). If it is not
|
||||
there, publish "trigger not established" — an invented cause is worse than an
|
||||
open question because the next agent plans against it.
|
||||
|
||||
## Fleet-specific sensitivity
|
||||
|
||||
On this fleet, container healthchecks shell out to the **`podman` CLI**, so the
|
||||
first `invalid internal status … could not find any running process` in the user
|
||||
journal is an *early, per-minute* indicator of rootless-podman state death — and
|
||||
it timestamps the incident to within one healthcheck interval. That is how the
|
||||
18:58:30 Dankclaw break was pinned to the bench still loading at the time.
|
||||
|
||||
## Worked example (2026-09-16, Dankclaw)
|
||||
|
||||
| evidence | value |
|
||||
|---|---|
|
||||
| log mtimes | x4e-run 18:58:29, x4f-run 18:59:56 |
|
||||
| durations | E1 `real 4m53.946s` (llama-cli spam), F `real 5m24.785s` (exit 143 = killed) |
|
||||
| journal | first `invalid internal status` 18:58:30 |
|
||||
| ⇒ in flight | **F**, launched 18:54:31 (inside its 18:54:31–18:59:56 window) |
|
||||
|
||||
Answer delivered: the exact flags of F, plus the note that the preceding
|
||||
`llama-cli` had written 3.19 GB of spinner lines and that the `/tmp` VRAM sampler
|
||||
was gone — so the overlap of the two 111 GiB-class invocations stayed marked as
|
||||
[INFERENCE] rather than becoming a claim.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: live-llm-benchmark-hygiene
|
||||
description: "Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification."
|
||||
---
|
||||
|
||||
# Benchmarking a live LLM server honestly
|
||||
|
||||
Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance,
|
||||
any OpenAI-compatible endpoint). The point is to avoid the four confounds that
|
||||
turned a night's numbers into fiction before each one was fixed.
|
||||
|
||||
## 1. Match the corpus to the workload
|
||||
|
||||
Synthetic token soup (`vllm bench serve --dataset-name random`) under-reports
|
||||
speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
|
||||
code, and acceptance — not step time — is what moves throughput. For a coding
|
||||
fleet, build the corpus from the repo's own sources: file body plus an
|
||||
instruction that requires reading it. Then `tok/update` and TPOT mean something.
|
||||
|
||||
## 2. Salt every run so the prefix cache cannot help
|
||||
|
||||
vLLM runs with `--enable-prefix-caching`, and a repeated corpus silently
|
||||
measures a warm cache. Put a unique per-run token in the first bytes (it is both
|
||||
the engine's cache key and, if a proxy routes on it, the routing key). Without
|
||||
this, the second run of any comparison is not comparable to the first.
|
||||
|
||||
## 3. Measure turns, not only batches
|
||||
|
||||
A batch of distinct prompts parallelises prefill and hides the real cost of a
|
||||
conversation: an agent loop re-sends its context every turn. Two workloads need
|
||||
two clients:
|
||||
|
||||
- **batch/throughput** — `vllm bench serve --dataset-name custom --dataset-path
|
||||
<jsonl>` with `--custom-output-len`, several concurrency levels.
|
||||
- **sequential turns** — a small client that sends turn after turn with growing
|
||||
history and a large shared prefix, timing **TTFT of the first streamed chunk**
|
||||
per turn. This is the only measurement that exposes prefix-cache behaviour.
|
||||
|
||||
## 4. Read the engine's own ledger for capacity
|
||||
|
||||
Do not infer memory from the card's size. The server prints the truth at boot:
|
||||
|
||||
```
|
||||
Available KV cache memory: 5.44 GiB
|
||||
GPU KV cache size: 122,636 tokens
|
||||
Maximum concurrency for 114,688 tokens per request: 1.07x
|
||||
```
|
||||
|
||||
That is the number to quote for "how many concurrent conversations fit".
|
||||
Related trap: the arena/KV budget is **not monotonic in chunk size** — on one
|
||||
host, available KV was 5.44 GiB at `CHUNK=8192`, 4.22 GiB at 16384 and 4.16 GiB
|
||||
at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
|
||||
measure three points before treating a knob as a lever.
|
||||
|
||||
## 5. Soak before believing a tuning change
|
||||
|
||||
Run several families concurrently for 15-30 minutes and count failures, not just
|
||||
latency: a nightly soak returned 155/155 answers with zero failovers while only
|
||||
25% of turns hit the cache — which is how the capacity ceiling was found as a
|
||||
*latency* effect rather than an error. Report hit/miss split, medians, and the
|
||||
min/max, because the median of a bimodal distribution describes neither mode.
|
||||
|
||||
## 6. Verification that is not the benchmark
|
||||
|
||||
- Run the client the fleet actually uses (e.g. `omp --model <alias> -p
|
||||
--no-tools`) — a config file that parses is not a config that works, and a
|
||||
non-obvious API mode (raw `/v1/completions`) may still be fine because the
|
||||
client templates its own prompt.
|
||||
- A deployed inference unit is not proven until an **unattended reboot** brings
|
||||
it back: verify `is-active`, `/health`, and one real completion afterwards.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: live-llm-telemetry-no-bench
|
||||
description: "Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU."
|
||||
---
|
||||
|
||||
# Live LLM telemetry without benchmarking
|
||||
|
||||
Use when asked "how is the model performing?" on a host that SERVES production
|
||||
inference, where benchmarking policy forbids running `llama-bench`/`vllm bench`
|
||||
beside the live service. Instead of a synthetic bench, mine the runtime's own
|
||||
per-request telemetry — it measures the real workload and costs nothing.
|
||||
|
||||
## 1. Where the numbers live
|
||||
|
||||
| Runtime | Unit kind | Log command |
|
||||
|---|---|---|
|
||||
| llama.cpp (mainline, `-lv 3`) | root systemd unit | `sudo journalctl -u <unit> --since "-3h" --no-pager` |
|
||||
| halogen-flash-server | rootless Quadlet | `journalctl --user -u <unit> --no-pager` (**never** sudo — per-user storage) |
|
||||
|
||||
llama.cpp emits, per completed task:
|
||||
|
||||
```
|
||||
I slot print_timing: id 0 | task 82 | prompt eval time = 339.51 ms / 57 tokens ( 5.96 ms per token, 167.89 tokens per second)
|
||||
I slot print_timing: id 0 | task 82 | eval time = 428.49 ms / 36 tokens (12.24 ms per token, 81.68 tokens per second)
|
||||
I slot print_timing: id 0 | task 82 | total time = 768.00 ms / 93 tokens
|
||||
I slot print_timing: id 0 | task 82 | graphs reused = 84
|
||||
I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 ( 25 accepted / 33 generated), mean len = 3.27
|
||||
I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s
|
||||
```
|
||||
|
||||
halogen-flash-server emits one line per request:
|
||||
|
||||
```
|
||||
serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok
|
||||
```
|
||||
|
||||
Grep both with `grep -E "print_timing|release|serve_api: mtp"`. Long lines get
|
||||
truncated in a terminal — redirect to a file/artifact and read it whole.
|
||||
|
||||
## 2. Read the envelope, not one sample
|
||||
|
||||
- `prompt eval` on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use
|
||||
the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
|
||||
- `eval` (decode) and `draft acceptance` are content-dependent: measured
|
||||
acceptance 0.46-0.76 on one unchanged build. Never quote a single task as
|
||||
"the model's speed", and never compare a live sample to a deploy-time
|
||||
baseline without a masked `llama-bench`.
|
||||
- `graphs reused` > 0 and `selected slot by LCP similarity` prove prefix-cache
|
||||
reuse is working; `prompt 53 (48 cached), prefill 0.05s` vs `1.58s` cold is
|
||||
prefix caching caught in the act.
|
||||
- A runtime's own window metric (`tg_3s`) is a cross-check on your arithmetic.
|
||||
|
||||
## 3. Attribute PCIe without touching the model
|
||||
|
||||
Run the counter in the background and issue ONE ordinary request:
|
||||
|
||||
```bash
|
||||
ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
|
||||
curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
|
||||
http://127.0.0.1:7771/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
|
||||
wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'
|
||||
```
|
||||
|
||||
`-s t` = `rxpci`/`txpci` in MB/s. Measured on a fully-VRAM-resident 27B on a
|
||||
3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at
|
||||
idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane
|
||||
link the transport is a non-factor — say so with the measured number instead of
|
||||
hand-waving.
|
||||
|
||||
## 4. Traps
|
||||
|
||||
- **`wait` or you lose the telemetry.** If the remote command exits while the
|
||||
curl is backgrounded, ssh closes the session and the client dies: the journal
|
||||
shows `W srv stop: cancel task, id_task = N` and NO `print_timing` line, while
|
||||
curl still reports a plausible status. Always end the compound command with
|
||||
`wait`.
|
||||
- **Do not run a GPU workload to "get a reading".** One max_tokens=200 request
|
||||
is production traffic and is fine; anything that pins the card for a while is
|
||||
a bench and needs the stop-and-mask procedure first.
|
||||
- **Check which instance you are reading.** Quadlet units run `--replace --rm`
|
||||
and root units get restarted: `systemctl status <unit> | head -12` for the
|
||||
start time; `task N` IDs increment per request and jump across restarts.
|
||||
- **The engine's own ledger outranks your model of it** — quote `KV self size`,
|
||||
`n_slots`, `n_ctx_slot` and the VRAM figure from the log/NVML rather than
|
||||
arithmetic.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: llama-bench-fleet-sweep
|
||||
description: "Run a llama-bench sweep (batch/ubatch, ncmoe, split, or any dimension) on a Dank-fleet LLM host correctly: stop the service, run from a transient unit, one invocation per cell, sample VRAM, restore and verify — including the llama-bench vs llama-server traps that have already produced wrong numbers here."
|
||||
---
|
||||
|
||||
# llama-bench sweeps on a Dank-fleet LLM host
|
||||
|
||||
Use when tuning a llama.cpp service's serve-time parameters (batch/ubatch, `-ncmoe`, `-sm`/`-ts`, KV type) offline, i.e. any question of the form "how large may X be / what does X buy".
|
||||
|
||||
Reference implementation: `scripts/bench-batch-ubatch.sh` (repo) — read it before writing another harness.
|
||||
|
||||
## Preconditions
|
||||
|
||||
1. **Stop the service** — Benchmarking Policy in AGENTS.md is mandatory: a bench beside a live service starves the same GPU and corrupts results. Never bench against the live port.
|
||||
2. **Run from a transient unit, not a shell:**
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||||
```
|
||||
|
||||
The deploy's unit sets `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell gets 8 MiB (`ulimit -l`), which fails the load for a reason unrelated to what you are measuring. Passwordless sudo is available on both hosts.
|
||||
|
||||
## The traps (each one has produced a wrong number)
|
||||
|
||||
- **`/` vs `,`.** llama-bench splits `/` = devices *within one configuration*, `,` = *sweep values*. A legitimate llama-server `--tensor-split 36,29` becomes `-ts 36/29`; the comma form silently benches each card alone. Same for `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
|
||||
- **No `-c`.** `n_ctx = n_prompt + n_gen + n_depth`. The context is a consequence of the test shape, so a bench can NEVER prove that a configuration's activation buffers fit beside the *deployed* KV — that requires starting the real service at the real context and slot count.
|
||||
- **A test that cannot allocate calls `exit(1)`**, killing the whole process. One invocation per cell, or one infeasible value loses every later cell — and infeasibility is usually what you are looking for.
|
||||
- **No `--spec-*` and no `--parallel`.** Decode figures are the *drafterless* baseline, and slots are never exercised. `-d <n>` is what puts KV in front of the measurement: a `tg` instance runs with `n_prompt = 0`, i.e. at ~zero depth otherwise.
|
||||
- **A cell that looks fast because its allocation was smaller than intended is indistinguishable from a fast cell** unless you sample VRAM. Sample `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and report min-free per card.
|
||||
- **`local a=$1 b=$x` expands `$x` before the builtin assigns anything** — under `set -u` that is `b: unbound variable`, and it fires *after* your harness has already stopped the production service. Declare in two statements.
|
||||
- **Ranges exist:** `-ub 512-2048+256` == `512,768,...,2048`.
|
||||
|
||||
## Shape of a good harness
|
||||
|
||||
- One `llama-bench` invocation per cell; append one TSV row per cell as it completes, so progress is readable mid-run and an interrupted sweep resumes (skip cells whose `bench.json` is non-empty).
|
||||
- Write `bench.json` per cell (`-o json`), plus `bench.log` (`--progress`), plus the VRAM trace.
|
||||
- **Stop the service in the harness and restore it from a `trap ... EXIT`** — and still verify the service afterwards. A harness that crashes mid-run leaves production down.
|
||||
- Match the deployment's load flags exactly (`-ngl`, `-sm`, `-ts`, `-fa`, `-ctk/-ctv`, `-t`), or you are benching a different service. Note `-ts`/`-dev` take slashes here while the unit takes commas.
|
||||
|
||||
## Verify before burning hours
|
||||
|
||||
Dry-run the harness against a **stub** `llama-bench` (`BIN=/tmp/stubdir`, plus `SERVICE=<nonexistent-name>` so the real service is untouched) with a tiny grid, including one cell forced to `exit 3`: prove the TSV row, the failure capture, and the resume logic before the real sweep. Then validate the real flag set in ~1 s by pointing `-m` at a nonexistent file — argument errors surface *before* the model load.
|
||||
|
||||
## Interpreting results
|
||||
|
||||
- A prefill difference smaller than the run-to-run spread is not a result; check each cell's own `stddev_ts`.
|
||||
- Single-sequence decode is usually flat across batch/ubatch — that knob is prefill-side.
|
||||
- Batch ≥ ubatch always (llama.cpp clamps ubatch to batch, so smaller-batch cells silently duplicate).
|
||||
- Report what the harness cannot see (drafter, slots, real context) rather than implying the sweep settled it.
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: llama-bench-host-sweep
|
||||
description: "Run a llama-bench sweep (batch/ubatch, split, quant, depth) on a live fleet LLM host without fooling yourself: what llama-bench cannot express, the separator trap, per-cell isolation, the memlock/transient-unit requirement, and the dry-run that must precede a multi-hour run."
|
||||
---
|
||||
|
||||
# Running a llama-bench sweep on a fleet host
|
||||
|
||||
Use when a bench must answer a serving question on one of the LLM hosts (Dankclaw
|
||||
frank-large/frank-small, Dankserver frank-medium) — batch/ubatch sizing, split
|
||||
choice, KV type, depth behaviour.
|
||||
|
||||
Repo policy applies: **stop the production service first** (AGENTS.md Benchmarking
|
||||
Policy). The harness below does that for you and restores it.
|
||||
|
||||
## 1. What llama-bench cannot express — decide this BEFORE designing the sweep
|
||||
|
||||
| Not available | Consequence |
|
||||
|---|---|
|
||||
| `-c/--ctx-size` | `n_ctx = n_prompt + n_gen + n_depth` (llama-bench.cpp). The context is a consequence of the test shape, so **the deployment's real allocation is never tested** (e.g. a 16 GiB KV at 262144 over 2 slots). Only starting the real unit answers "does it fit". |
|
||||
| `--parallel` | single sequence only; no slot shape, no concurrency effect. |
|
||||
| `--spec-*` | no drafter. Any `tg` number is the **autoregressive baseline**, not the served rate. Useful as a control, never as the deployed figure. |
|
||||
| warmup per config | a warmup run is itself a full-size run; at 64k prompts `--no-warmup` buys a second sample for the same wall clock. |
|
||||
|
||||
`-d N` is a depth PREFIX: it puts N tokens of KV in place before the measured run and
|
||||
is cached across reps through a `llama_state_seq_*` save/restore, so the fill is paid
|
||||
once per cell and is not part of the measured number. Without `-d`, a `tg` test runs at
|
||||
~zero depth and the number is meaningless for long conversations.
|
||||
|
||||
## 2. Two traps that silently change the experiment
|
||||
|
||||
- **Separators.** `/` separates devices *within one configuration*; `,` separates
|
||||
configurations to sweep. `-ts` and `-dev` both follow this. `-ts 36,29` parses as the
|
||||
single value `36` → everything lands on one card and you have benched a different
|
||||
machine. llama-server takes the comma form for the same pair. List-valued args also
|
||||
accept ranges: `-ub 512-2048+256`.
|
||||
- **memlock.** The deploy units set `LimitMEMLOCK=infinity` because the ROCm backend
|
||||
pins staging buffers; a plain SSH shell on these hosts has soft=hard=8 MiB
|
||||
(`ulimit -l`), so a load can fail for a reason unrelated to the model. Run the sweep
|
||||
inside a transient unit:
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=fm-bench --collect --no-block \
|
||||
--property=Type=oneshot --property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
This also survives an ssh disconnect, unlike a foreground remote process.
|
||||
|
||||
## 3. One invocation per cell
|
||||
|
||||
A test that cannot run calls `exit(1)` immediately (failed depth/prompt/gen run), so a
|
||||
single cross-product invocation loses **every later cell** the moment one ubatch fails to
|
||||
allocate — and infeasibility is usually one of the questions being asked.
|
||||
|
||||
Drive the grid from a shell loop: one llama-bench call per (b, ub), continue on a
|
||||
non-zero exit, record why, and skip cells whose output file already exists and is
|
||||
non-empty (that makes an interrupted sweep resumable). Write rows to a TSV as the sweep
|
||||
proceeds so progress is readable mid-run.
|
||||
|
||||
## 4. Sample VRAM per cell, covering the load
|
||||
|
||||
Batch/ubatch sizes the activation buffers, so a cell that looks fast because its
|
||||
allocation was silently smaller must be distinguishable from one that is genuinely fast.
|
||||
Poll `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and keep the
|
||||
peak (minimum free) per card. Correlate by taking the window between the cell's start and
|
||||
end.
|
||||
|
||||
## 5. Always restore the service from a trap
|
||||
|
||||
`trap <restart> EXIT` in the harness, plus a manual check afterwards. A harness that
|
||||
stops a production unit and then dies leaves inference down; this fleet has had that
|
||||
happen.
|
||||
|
||||
## 6. Dry-run before a multi-hour run
|
||||
|
||||
Stub the binary so the whole path is exercised without loading a 29 GiB model:
|
||||
|
||||
```bash
|
||||
# /tmp/benchstub/llama-bench prints the -o json shape and exits non-zero for one ubatch
|
||||
SERVICE=frank-medium-dryrun BIN=/tmp/benchstub MODEL=/tmp/benchstub/m.gguf OUT=/tmp/bub-test \
|
||||
N_PROMPT=8 N_GEN=4 N_DEPTH=0 REPS=1 BATCHES="2048 4096" UBATCHES="512 1024" \
|
||||
bash scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
`SERVICE=<nonexistent>` means the trap and the stop are never reached, so the real
|
||||
service is untouched. Check that a FAILED cell still produces a row with the right column
|
||||
count before trusting the table.
|
||||
|
||||
## 7. Read the result sanely
|
||||
|
||||
- Bracket the sweep: run the reference cell (the deployed configuration) first and
|
||||
again last, so drift over hours is visible.
|
||||
- Sanity-check one number against the runbook's prior figures before believing the grid.
|
||||
- Remember what the grid does NOT prove: it did not run the drafter, the slots, or the
|
||||
deployment's context. The winner still has to be applied and verified live.
|
||||
|
||||
## Pitfall while writing such a harness
|
||||
|
||||
`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands **all** the builtin's arguments
|
||||
before the builtin assigns any of them, so under `set -u` this aborts with
|
||||
`b: unbound variable`. Declare in two statements:
|
||||
|
||||
```bash
|
||||
local b=$1 ub=$2
|
||||
local dir="$OUT/b${b}_ub${ub}" sampler start rc wall
|
||||
```
|
||||
|
||||
Ref: `scripts/bench-batch-ubatch.sh` in serverAppConfigs (the fleet's implementation of
|
||||
all of the above).
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: llama-cpp-spec-decode-ab
|
||||
description: "Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts."
|
||||
---
|
||||
|
||||
# Spec-decoding A/B on a live llama.cpp endpoint
|
||||
|
||||
Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter".
|
||||
Verdicts here are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.
|
||||
|
||||
## Step 0 — settle chaining in the source, before spending GPU time
|
||||
|
||||
`common/speculative.cpp` (`common_speculative_init_result`) builds `configs` and gates each type:
|
||||
|
||||
- **draftless** (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
|
||||
- **context-backed** (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) → require `params.draft.ctx_dft != nullptr`
|
||||
|
||||
Consequences: one context-backed drafter + any number of ngram types chains fine; **two context-backed types never do**
|
||||
(`draft-mtp,draft-dflash` → `context type MTP requested but model doesn't contain MTP layers`).
|
||||
Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.
|
||||
|
||||
## Step 1 — shapes (the part that decides the answer)
|
||||
|
||||
| Shape | Prompt | Who it can favour |
|
||||
|---|---|---|
|
||||
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
|
||||
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
|
||||
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
|
||||
| identity | short deterministic question | controls for lossless verification |
|
||||
|
||||
Design the echo passage so no long span repeats *inside* it — otherwise you measure the matcher against itself.
|
||||
For `structured`, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.
|
||||
|
||||
## Step 2 — protocol
|
||||
|
||||
1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait for the port.
|
||||
2. Per config: write an env file with overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health` until ok, then measure.
|
||||
3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
|
||||
4. Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).
|
||||
|
||||
## Step 3 — read the right counter
|
||||
|
||||
- Per request: `timings.draft_n` / `timings.draft_n_accepted`, `predicted_per_second`.
|
||||
- Cumulative: `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`.
|
||||
- **`draft_n == 0` while tokens flow = the proposer is not engaged**, even when every flag looks right.
|
||||
- A config with zero drafting is a useful *negative control*: it proves the other config's drafts came from the proposer under test.
|
||||
|
||||
## Traps that produce false verdicts
|
||||
|
||||
- **Thinking eats the answer.** Echo/structured prompts with thinking on put the copy in `reasoning_content`; `content` is empty, so identity checks compare `''` to `''`. Disable per request: `chat_template_kwargs: {"enable_thinking": false}`.
|
||||
- **No banner.** ngram-mod prints nothing at startup; the journal shows only the drafter's `adding speculative implementation`. Never conclude "chain not engaged" from the load log.
|
||||
- **Windows change the verdict.** ngram-mod `--spec-ngram-mod-n-min/--spec-ngram-mod-n-max` decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
|
||||
- **Unused-tensor warnings are normal**: with `draft-dflash` on an MTP-carrying GGUF, `blk.<n>.nextn.*` is `unused ... ignoring` — correct, not a truncated file.
|
||||
- **Harness hygiene**: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix), `pkill -x` never `-f`.
|
||||
|
||||
## Interpreting
|
||||
|
||||
Report `decode tok/s` per shape plus acceptance (`accepted/draft_n`) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: llamacpp-cross-backend-device-split
|
||||
description: "Decide whether two unlike devices (NVIDIA CUDA + AMD Vulkan, or any two backends) can serve one llama.cpp model, and measure the attempt honestly on a serving host — the two scheduler failure modes, what is actually expressible, the dual-backend build recipe, and the measurement traps that make results meaningless."
|
||||
---
|
||||
|
||||
# Cross-backend device splits in llama.cpp
|
||||
|
||||
Use when someone proposes serving ONE model across two devices of different
|
||||
backend families (NVIDIA CUDA + AMD Vulkan/HIP, or a dGPU + an APU), and you
|
||||
need to know whether it is expressible before spending a build and a service
|
||||
outage on it.
|
||||
|
||||
## Answer first: it is not expressible
|
||||
|
||||
A weight tensor in one *device* buffer cannot be consumed by an op the scheduler
|
||||
placed on another device. Two observed failure modes (mainline llama.cpp
|
||||
`37b53fd`, 2026-09-16):
|
||||
|
||||
| attempt | result |
|
||||
|---|---|
|
||||
| per-tensor: `-ot "exps=Vulkan1"` with `-sm none -mg 0 -dev CUDA0,Vulkan1` | `ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — scheduler abort |
|
||||
| per-layer: `-sm layer -ts a,b -dev CUDA0,Vulkan1`, or layer-scoped `-ot` | model load fails, message is only `failed to load model` |
|
||||
| device + **CPU**: `-ot "exps=CPU"` | **works** — the CPU backend is the universal fallback and host buffers can be copied |
|
||||
|
||||
The rule: **the CPU is the only legitimate partner in a heterogeneous split.**
|
||||
Everything else must be single-vendor. On a CUDA+AMD host that means a card can
|
||||
be the trunk *or* the offload host, never the complement of an AMD-resident
|
||||
peer — the same wall the R9700 pair hits from the RCCL/atomics direction.
|
||||
|
||||
If the answer is "not expressible", stop and consider the expressible
|
||||
alternatives in this order: (1) two independent endpoints, one model per device;
|
||||
(2) single-vendor layer split (`-sm layer`, no collective needed); (3)
|
||||
device + CPU per-tensor offload.
|
||||
|
||||
## Build the dual-backend binary
|
||||
|
||||
Needs both toolchains in one tree; Vulkan needs `glslc` + `libvulkan` headers
|
||||
(already present on this fleet's hosts).
|
||||
|
||||
```bash
|
||||
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
-DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=<sm> \
|
||||
-DGGML_CCACHE=ON -DLLAMA_CURL=OFF
|
||||
cmake --build build-dual --target llama-bench llama-cli llama-server -j $(nproc)
|
||||
```
|
||||
|
||||
**Device names are not what you expect.** `--list-devices` on an
|
||||
NVIDIA-plus-APU host lists the discrete card under BOTH backends (e.g. `CUDA0`
|
||||
and `Vulkan0`) and the APU last (`Vulkan1`). The iGPU is `Vulkan1`, not
|
||||
`Vulkan0`. Valid `-ot` buffer types are whatever that list prints; a wrong name
|
||||
makes llama.cpp print the valid set and exit, so the error is self-diagnosing.
|
||||
|
||||
## Measure it without fooling yourself
|
||||
|
||||
- **`tg` in llama-bench runs with `n_prompt = 0`.** A plain `-n 64` row is a
|
||||
~zero-depth decode: no KV traffic at all. Anything about context cost or KV
|
||||
placement must use `-d/--n-depth`, which fills the cache before timing.
|
||||
- `-p`/`-n` lists build separate instances (prefill instances with `n_gen = 0`,
|
||||
generation instances with `n_prompt = 0`), and `n_ctx = n_prompt + n_gen`.
|
||||
- Compare at **depth-matched** settings, and remember prefill into a deep cache
|
||||
is the slow case (e.g. 456 tok/s fresh vs 232 tok/s into 64k).
|
||||
- Sample the actual engines during the run (`/sys/class/drm/card*/device/gpu_busy_percent`,
|
||||
`nvidia-smi`) — an idle device or a 100 % APU is the difference between a real
|
||||
measurement and a silent CPU fallback.
|
||||
- Watch the **host memory** figure, not just tok/s. A Vulkan-resident model lives
|
||||
in device buffers (committed, not reclaimable), whereas an mmap'd CPU-placed
|
||||
tensor is page cache — the same model can commit 116 GiB one way and 95 GiB
|
||||
the other. On a host with co-tenants, that decides deployability.
|
||||
|
||||
## Serving-host discipline
|
||||
|
||||
- **Stop and mask every LLM service first** (repo benchmarking policy). For a
|
||||
rootless Quadlet, move the `.container` out of
|
||||
`~/.config/containers/systemd/` and `daemon-reload` — the generator re-creates
|
||||
the unit from that file on every reload, so `mask` is not the lever.
|
||||
- **`systemctl mask` is refused for a relinked root unit** whose
|
||||
`/etc/systemd/system/<svc>.service` is a symlink into the repo: it exits with
|
||||
`File ... already exists and is a symlink to ...`. Stop it and ensure nothing
|
||||
restarts it, or move the symlink aside.
|
||||
- Root units need `sudo systemctl start`; without it you get
|
||||
`Access denied as the requested operation requires interactive
|
||||
authentication` (polkit), which reads like a permissions bug.
|
||||
- Export the pinning guard: **mainline is `GGML_CUDA_NO_PINNED`** (ik_llama.cpp
|
||||
was `GGML_CUDA_NO_PINNED_WEIGHTS`). Without it a CPU-overridden tensor can land
|
||||
in a pinned `CUDA_Host` buffer — the failure mode that page-locked 108 GiB and
|
||||
starved a host's co-tenants until a power cycle.
|
||||
- Restore: move the quadlet back, `daemon-reload`, `reset-failed`, start, then
|
||||
verify `/health`, `/v1/models` **and the engine's own boot ledger** (e.g.
|
||||
halogen's `kv pool:` lines, which say whether the configured pool was clamped).
|
||||
|
||||
## Calibration point
|
||||
|
||||
On a 121 GiB Strix Halo host with a 3090: a 177B MoE (112 GiB GGUF, 12 of 48
|
||||
layers carrying KV) ran **32.6 tok/s shallow / 25.8 tok/s at 64k depth / 456 tok/s
|
||||
prefill** with everything on the iGPU via Vulkan, and **12.1 tok/s** with the
|
||||
same trunk but experts on the CPU. If a proposed split cannot beat the
|
||||
iGPU-only number at matched depth, it is not worth the failure modes above.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: llamacpp-device-split-feasibility
|
||||
description: "Decide which multi-device arrangements llama.cpp can actually express on a heterogeneous host (mixed vendors, iGPU+dGPU), and measure them without fooling yourself — covers the expressibility matrix, llama-bench's /-vs-, device/split syntax, independent placement proof, and the stop/restore path."
|
||||
---
|
||||
|
||||
# When to use
|
||||
|
||||
Asked "can we split this model across GPU A and GPU B?" on a host where the devices are mixed (NVIDIA + AMD, iGPU + dGPU) or where the obvious arrangement is "weights on device X, different weights on device Y". Also use before *believing* any earlier "it cannot be done" conclusion about a multi-device split — that verdict is easy to produce by accident.
|
||||
|
||||
# 1. Classify the arrangement before running anything
|
||||
|
||||
llama.cpp expresses exactly two things across devices, and they are not interchangeable:
|
||||
|
||||
| arrangement | how | expressible? |
|
||||
|---|---|---|
|
||||
| **Layer split** — layers 0..n on dev A, the rest on dev B | `-sm layer -ts <shareA>/<shareB>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured) |
|
||||
| **Per-tensor placement** — e.g. attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a required term of the scheduler's `pipeline_parallel` flag (the only read site is `llama-context.cpp`, in the `pipeline_parallel = ...` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp` (`pre-allocated tensor (...weight) in a buffer (...) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). `-ot x=CPU` works only because the CPU backend is last in the list. |
|
||||
|
||||
So: if the plan needs *tensors* on a specific device, stop — the answer is no, for any vendor mix, and pure-Vulkan vs CUDA+Vulkan changes nothing. If it needs *layers*, it is expressible and worth measuring.
|
||||
|
||||
Layer-split arithmetic to check before promising anything:
|
||||
|
||||
- A whole layer carries its experts, if the model is MoE. Compute GiB/layer, not GiB/param-group.
|
||||
- Only some layers carry KV (e.g. every 4th for hybrid attention). A share of layers is therefore only a share of KV — a contiguous range rarely holds the layers you care about.
|
||||
- Big non-repeating tensors (per-layer-token tables etc.) land on the main device and cannot be moved by `-ts`; they cap how small the primary device's footprint can get.
|
||||
|
||||
# 2. Get the harness syntax right, or every number is fiction
|
||||
|
||||
`llama-bench` writes `-dev <dev0/dev1/...>` and `-ts <ts0/ts1/..>`: **`/` separates values within one configuration, `,` separates configurations to sweep.**
|
||||
|
||||
- `-dev CUDA0,Vulkan1` = **two single-device runs**, never a two-device run.
|
||||
- `-ts 0.85,0.15` = two sweep values, each becoming `[0.85, 0, 0, …]` → device 0 takes everything.
|
||||
|
||||
Write `-dev A/B -ts x/y`. `llama-cli`/`llama-server` use the classic comma form — which is why a failure seen only under llama-bench is a harness bug, not an engine limit.
|
||||
|
||||
Cheap pre-flight: run with the intended flags and read the tool's own reported parameters (llama-bench's table has a `dev` column). If a row says one device, your intent was not parsed.
|
||||
|
||||
# 3. Prove placement — never trust the label
|
||||
|
||||
Sample the non-primary device *while the run is loaded*:
|
||||
|
||||
```bash
|
||||
( for i in $(seq 1 60); do nvidia-smi --query-gpu=memory.used --format=csv,noheader; sleep 10; done ) > /tmp/vram.txt &
|
||||
```
|
||||
|
||||
Expect roughly (layers × GiB/layer + that device's KV) resident. Tens of MiB means the split did not happen. Cross-check: several "different" configurations that all produce the same tok/s to within 0.2 % are probably all the same single-device run.
|
||||
|
||||
# 4. Measure at the depth that matters
|
||||
|
||||
- `llama-bench`'s `tg` runs with `n_prompt = 0`, i.e. ~zero depth. Use `-n 32 -d 65536` (or deeper) or the KV term — the entire reason for the split — is absent.
|
||||
- Compare against the **single-device baseline on the fast-memory device**, and against whatever production runtime already serves the model. A split that wins by a few percent at 64k depth may lose at 200k, because the KV term grows with context and prefill cost grows with the target depth.
|
||||
|
||||
# 5. Stop / restore on a host that serves
|
||||
|
||||
- Stop and mask the production units first (benchmarking policy). For rootless quadlets the lever is the `.container` file (move it aside + `daemon-reload`); `Restart=always` otherwise recreates the service mid-run. `systemctl mask` is refused for a unit that is a symlink into a repo checkout.
|
||||
- After restoring, verify `/health`, `/v1/models`, and the boot ledger numbers — and keep a memory-headroom note: the same model can come back with less contiguous host memory than before (a reboot re-fragments), which decides whether the next co-tenant fits.
|
||||
- Landmine seen on a rootless-podman host after a big GPU run: a dead rootless **pause process** takes *all* container management down (every `podman` command: `invalid internal status … could not find any running process`; container starts then fail because the rootless netns holding the bridges is gone: `failed to bind udp listener on 10.89.x.1:53`). `podman system migrate` panics (5.7.0, `teardownNetworkBackend`). Repair is a reboot. Before rebooting, check `loginctl show-user <u> -p Linger` and that units carry `[Install] WantedBy=default.target`, so everything returns by itself.
|
||||
- Do not publish a cause you did not check: `journalctl -k | grep -i oom` decides whether "memory pressure killed it" is a finding or a guess. An invented narrative is worse than an open question.
|
||||
|
||||
# 6. Report shape that survived review
|
||||
|
||||
State (a) which arrangement was tested and that the harness parsed it, (b) the placement evidence, (c) the depth-matched numbers next to the single-device baseline, (d) the memory cost on the borrowing device, and (e) whether the split can reach its own premise (KV on the fast device, long context) — not just that it runs.
|
||||
@@ -0,0 +1,168 @@
|
||||
---
|
||||
name: llamacpp-drafter-kv-split-sizing
|
||||
description: "Size a llama.cpp multi-GPU layer-split deployment that carries a speculative drafter: read the drafter's KV law from its GGUF header, then find the split balance by measurement (the batch asymmetry, the ctx/floor trade, and the OOM-vs-page-fault signatures)."
|
||||
---
|
||||
|
||||
# Sizing a llama.cpp layer-split with a speculative drafter
|
||||
|
||||
Use when a multi-GPU llama.cpp deployment carries a draft model (`--spec-draft-model`)
|
||||
and the question is *where do the layers go and how big can the context be*. The two
|
||||
effects below decide it, and neither is visible in the flags.
|
||||
|
||||
## 1. Read both GGUF headers first
|
||||
|
||||
The drafter's own properties set the memory bill. `llama-server` prints almost none of
|
||||
them, and the loader's ledger is not printed at default verbosity.
|
||||
|
||||
Minimal GGUF KV reader (no tooling needed) — write `ggufmeta.py`:
|
||||
|
||||
```python
|
||||
import struct, sys
|
||||
|
||||
def rd(path):
|
||||
f = open(path, 'rb')
|
||||
assert f.read(4) == b'GGUF'
|
||||
ver = struct.unpack('<I', f.read(4))[0]
|
||||
struct.unpack('<Q', f.read(8)) # tensor count
|
||||
nk = struct.unpack('<Q', f.read(8))[0] # kv count
|
||||
def s():
|
||||
n = struct.unpack('<Q', f.read(8))[0]
|
||||
return f.read(n).decode('utf-8', 'replace')
|
||||
# 0=UINT8 1=INT8 2=UINT16 3=INT16 4=UINT32 5=INT32 6=FLOAT32 7=BOOL
|
||||
# 8=STRING 9=ARRAY 10=UINT64 11=INT64 12=FLOAT64
|
||||
SCALAR = {0:1, 1:1, 2:2, 3:2, 4:4, 5:4, 6:4, 7:1, 10:8, 11:8, 12:8}
|
||||
FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
|
||||
keys = {}
|
||||
for _ in range(nk):
|
||||
k = s()
|
||||
t = struct.unpack('<I', f.read(4))[0]
|
||||
if t == 8:
|
||||
v = s()
|
||||
elif t in SCALAR:
|
||||
v = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
|
||||
else:
|
||||
et = struct.unpack('<I', f.read(4))[0] # element type FIRST
|
||||
n = struct.unpack('<Q', f.read(8))[0] # then the count
|
||||
if et == 8:
|
||||
v = ('arr_str', [s() for _ in range(n)])
|
||||
elif et in SCALAR:
|
||||
v = ('arr', et, n); f.read(n * SCALAR[et])
|
||||
else:
|
||||
raise Exception('unsupported arr elem type %d' % et)
|
||||
keys[k] = v
|
||||
return ver, keys
|
||||
|
||||
for p in sys.argv[1:]:
|
||||
ver, k = rd(p)
|
||||
arch = k.get('general.architecture')
|
||||
print('=====', p, 'v%d' % ver, 'arch:', arch)
|
||||
for key in sorted(k):
|
||||
if key.startswith('tokenizer.') or isinstance(k[key], tuple):
|
||||
continue
|
||||
if key.startswith(arch + '.') or not key.startswith(('general.', 'quantize.')):
|
||||
print(' ', key, '=', k[key])
|
||||
```
|
||||
|
||||
The three format traps, each of which cost a debugging session:
|
||||
|
||||
1. **The KV stream has NO alignment.** GGUF v3's alignment (from `general.alignment`,
|
||||
default 32) applies only to the *tensor data section* that follows. Do not pad
|
||||
between KV pairs.
|
||||
2. **Array layout is `type(9) -> element_type(u32) -> count(u64)`.** Count-then-element
|
||||
is wrong and yields a `MemoryError`/`UnicodeDecodeError` a few pairs later.
|
||||
3. **The type enum is not the order you expect**: `7=BOOL`, `9=ARRAY`, and the 64-bit
|
||||
types are `10/11/12`. Getting `3=INT16` (2 bytes) or `5=INT32` (4 bytes) wrong shifts
|
||||
the whole stream.
|
||||
|
||||
Fields that matter: `*.block_count`, `*.block_size`, `*.attention.head_count_kv`,
|
||||
`*.attention.key_length`/`value_length`, `*.attention.sliding_window`,
|
||||
`*.context_length`.
|
||||
|
||||
## 2. The drafter's KV is sized from the TARGET
|
||||
|
||||
`common/speculative.cpp` builds the draft context with:
|
||||
|
||||
```cpp
|
||||
// the draft context holds as many tokens per sequence as the target context
|
||||
cparams.n_ctx = llama_n_ctx(ctx_tgt);
|
||||
```
|
||||
|
||||
There is **no flag** to cap it (the `--spec-draft-*` surface has none). Therefore:
|
||||
|
||||
```
|
||||
draft KV bytes = ctx_tgt x block_count x 2 (K,V) x head_count_kv x key_length x dtype
|
||||
```
|
||||
|
||||
*unless* the drafter declares a sliding window, in which case the window bounds it.
|
||||
|
||||
Measured on the same 262144-token target: a sidecar with `sliding_window = 2048` cost
|
||||
~40 MiB of draft KV; a sidecar of the same 5-layer shape with **no** window cost
|
||||
**5.0 GiB** (20 KiB/position). Swapping between them invalidated an already-balanced
|
||||
layer split.
|
||||
|
||||
**Failure signature:** the engine loads the whole model, answers `/health`, and dies on
|
||||
the FIRST draft decode with an allocation error (`ROCm error: out of memory` /
|
||||
`cudaMalloc failed`) inside `common_speculative_impl_draft_dflash::process`. An
|
||||
allocation failure *after* a successful load is a sizing failure, not a broken drafter
|
||||
-- and under a self-healing unit it looks like a crash-loop instead.
|
||||
|
||||
## 3. A bigger batch is asymmetric across the split
|
||||
|
||||
`--ubatch-size` is the physical batch the graph is built at, so it sets the compute
|
||||
buffers. Those buffers are **~2x heavier on the card that holds the output layers and
|
||||
the drafter's graphs** than on the other(s). Measured, 2 GPUs at 262144 ctx:
|
||||
|
||||
| ubatch | output/drafter card | other card |
|
||||
|---|---|---|
|
||||
| 512 -> 1536 | +~3.3 GiB | +~1.7 GiB |
|
||||
|
||||
Consequence: **a split balanced at one batch is unbalanced at another.** A split at
|
||||
1:1 or a hand-measured skew must be re-measured after any change to `--ubatch-size`,
|
||||
`--ctx-size` or `--parallel`.
|
||||
|
||||
## 4. Find the balance by measurement, not arithmetic
|
||||
|
||||
The per-step cost is not derivable from the parameter count (KV lands only on the
|
||||
attention layers, and buffer placement is per-device). One `-ts` step is worth roughly
|
||||
1.5 GiB of load at ~6k batch / 262k ctx on 30 GiB-class cards — enough to bracket, not
|
||||
to solve. Procedure:
|
||||
|
||||
1. Copy the env file to a scratch path and run the serve script against it
|
||||
(`SERVICE_ENV_FILE=/tmp/test.env bash scripts/serve-<svc>.sh`) rather than editing
|
||||
the deployed file. Do it with the unit stopped — one GPU, one process.
|
||||
2. Bracket: try a step, read `rocm-smi --showmeminfo vram` for both cards after a real
|
||||
generation, and step again. Three points is usually enough to land within ~0.1 GiB.
|
||||
3. Prefer the balanced split, because the busier card's headroom is what bounds the
|
||||
context:
|
||||
|
||||
```
|
||||
ctx_extra ~= (headroom_target - measured_free) / (per-position cost on the busier card)
|
||||
```
|
||||
|
||||
where per-position cost = the card's share of the target's KV + the drafter's KV if
|
||||
that card carries it. A balanced split therefore MAXIMISES ctx for a given floor.
|
||||
4. Verify with the full battery before committing: `/health`, `/v1/models`, a real
|
||||
completion, a tool call, two concurrent long prompts (one per slot), `slots`
|
||||
reporting the expected `n_ctx` per slot, and the per-task `draft acceptance = ...`
|
||||
line per slot.
|
||||
|
||||
## 5. Keep the two failure signatures apart
|
||||
|
||||
| Signature | Meaning |
|
||||
|---|---|
|
||||
| loads, `/health` ok, first decode OOMs | sizing: drafter KV, batch buffers, or KV vs ctx (section 2) |
|
||||
| `SDMA0 ... page fault` + `illegal memory access` + `ggml_abort` | a GPU fault in the cross-card boundary copy — the layer split's hot path. **Not** an OOM; headroom does not prevent it |
|
||||
|
||||
The second one is why a rebalance should be validated by a request battery and not just
|
||||
by `rocm-smi` numbers.
|
||||
|
||||
## Ops traps
|
||||
|
||||
- A server started over ssh (including under a supervisor/`hub`-style runner) **outlives
|
||||
stopping that runner** — the remote process keeps the port, and the next start fails
|
||||
with `couldn't bind HTTP server socket`. `pkill -x <exact-binary-name>` first.
|
||||
- `systemctl start` of a root unit from a non-root ssh session fails with
|
||||
`Access denied ... requires interactive authentication`: use sudo.
|
||||
- An old crash's `ROCm error` lines remain in `journalctl -u <svc>` forever. Check the
|
||||
current instance's `ActiveEnterTimestamp`/`NRestarts` before treating a grep count as
|
||||
a live fault.
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
name: llamacpp-layer-split-rdna4
|
||||
description: "Serve one model across two (or more) AMD RDNA4/ROCm GPUs on a host where RCCL cannot dispatch collectives — mainline llama.cpp built for gfx1201 with a bundled ROCm runtime, --split-mode layer, DFlash2/MTP speculative decoding, and the verification that proves each layer works"
|
||||
---
|
||||
|
||||
# llama.cpp layer-split serving on RDNA4 (gfx1201)
|
||||
|
||||
Use when one model must use two AMD GPUs whose interconnect cannot carry a
|
||||
collective (no PCIe atomic routing on one card's path), or when a host has no
|
||||
usable RCCL. The shape: ONE llama-server process, `--split-mode layer` across
|
||||
both cards, spec decode from a drafter GGUF or the model's own MTP head.
|
||||
|
||||
## When to reach for this
|
||||
|
||||
- vLLM cannot help: TP needs a per-layer allreduce and **PP's stage handoff is a
|
||||
device-backend `send`/`recv`** (`vllm/distributed/parallel_state.py`,
|
||||
`send_tensor_dict` → tensor handles "on the device backend"), i.e. the same
|
||||
RCCL primitive that fails on an atomics-less card. Verify with
|
||||
`sudo lspci -vv -s <gpu> | grep AtomicOpsCtl` (want `ReqEn+`) and
|
||||
`lspci -tv` (a `Routing-` bridge is a hardware wall).
|
||||
- Confirming the wall: any RCCL kernel launch on that card returns
|
||||
`hipErrorIllegalState` with `AMD_LOG_LEVEL=3` printing
|
||||
`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`.
|
||||
A single-card test proves nothing (NCCL short-circuits at world_size=1).
|
||||
- **Never run a multi-GPU collective probe beside a live service** on these
|
||||
hosts: stop and mask it first, or use a host that serves nobody.
|
||||
|
||||
## Build (pin the ref, bundle the runtime)
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
||||
git checkout --detach <ref> # pin; do not float
|
||||
# ROCm toolchain: system ROCm or a TheRock nightly tarball, e.g.
|
||||
# https://rocm.nightlies.amd.com/tarball-multi-arch/therock-dist-linux-gfx120X-all-<ver>.tar.gz
|
||||
ROCM=$HOME/.local/share/rocm-<ver>
|
||||
cmake -S . -B build-rdna4 -G Ninja -DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_C_COMPILER=$ROCM/llvm/bin/clang -DCMAKE_CXX_COMPILER=$ROCM/llvm/bin/clang++ \
|
||||
-DCMAKE_CXX_FLAGS="-I$ROCM/include" -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
|
||||
-DGGML_HIP=ON -DGGML_HIP_RCCL=OFF -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
|
||||
-DCMAKE_HIP_ARCHITECTURES=gfx1201 -DGPU_TARGETS=gfx1201 \
|
||||
-DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF -DBUILD_SHARED_LIBS=OFF \
|
||||
-DCMAKE_PREFIX_PATH=$ROCM -DROCM_DIR=$ROCM
|
||||
cmake --build build-rdna4 -j $(nproc) --target llama-server llama-cli llama-bench
|
||||
patchelf --set-rpath '$ORIGIN/../lib' build-rdna4/bin/llama-*
|
||||
```
|
||||
|
||||
- `-DGGML_HIP_RCCL=OFF` is deliberate: with RCCL off, only `--split-mode tensor`
|
||||
degrades (butterfly fallback); layer split is unaffected.
|
||||
- **Bundle device libraries at the exact depth the runtime looks for.** Copy
|
||||
`$ROCM/lib/rocblas/library/` → `build/lib/rocblas/library/` and
|
||||
`$ROCM/lib/hipblaslt/library/` → `build/lib/hipblaslt/library/` with
|
||||
`mkdir -p DEST && cp -r SRC/. DEST/`. `cp -r SRC DEST/` with a missing DEST
|
||||
flattens one level and the failure appears at REQUEST time, not load time
|
||||
(`Cannot read .../lib/rocblas/library/TensileLibrary.dat`).
|
||||
- Build only the arch you own. A sibling-arch binary (gfx1200 on gfx1201)
|
||||
links, loads the model, then segfaults.
|
||||
- Verify: `build-rdna4/bin/llama-server --version` and
|
||||
`--list-devices` (expect both cards; an iGPU can enumerate as a third).
|
||||
|
||||
## Serve
|
||||
|
||||
```bash
|
||||
HIP_VISIBLE_DEVICES=0,1 llama-server \
|
||||
--model <target.gguf> --alias <fleet-name> --host 0.0.0.0 --port 7770 \
|
||||
--device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
|
||||
--gpu-layers 999 --fit off --flash-attn on \
|
||||
--ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
|
||||
--jinja --metrics --cache-prompt \
|
||||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||||
--spec-type draft-dflash --spec-draft-model <dflash2-q8_0.gguf> --spec-draft-n-max 7
|
||||
```
|
||||
|
||||
- `--fit off` + an explicit `--ctx-size`: auto-fit must not resize a context
|
||||
you set on purpose. `--parallel N` DIVIDES `--ctx-size`; it does not multiply.
|
||||
- Spec decode choices: `--spec-type draft-dflash` with a DFlash2 GGUF (the
|
||||
drafter's own metadata selects the DFlash2 path — the init line prints
|
||||
`block_size=8, n_extract=5, sample_from_anchor=true`); `n_max <= block_size-1`.
|
||||
Or `draft-mtp` for a model whose GGUF carries `blk.<n>.nextn.*` (no draft
|
||||
file; the head loads from the main weights). Types can be chained
|
||||
(`draft-mtp,draft-dflash`). With `draft-dflash`, the target's `nextn`
|
||||
tensors log as "unused tensor ... ignoring" — that is correct, not a broken
|
||||
download (check bytes with the fetch script's `--verify-only`).
|
||||
- Layer split puts the drafter on the card holding the target's last layers
|
||||
(`--spec-draft-device ROCm1`); under tensor mode an unpinned draft joins the
|
||||
meta-device KV allocation and aborts the KV build.
|
||||
|
||||
## Verify (each layer separately)
|
||||
|
||||
1. Load: `n_ctx_slot` in the log matches `--ctx-size`; `/health` answers.
|
||||
2. Both cards: `rocm-smi --showmeminfo vram` — each card holds roughly half the
|
||||
weights; the busier one also carries the drafter. Compare against
|
||||
`model_bytes/2 + drafter + ctx*64KiB/2 + ~2 GiB` (qwen35 hybrid: only its
|
||||
full-attention layers take KV, 64 KiB/token at f16).
|
||||
3. **Spec decode is live**: `curl /metrics | grep spec_decode_num_drafts_total`
|
||||
and `..._accepted_tokens_total` (their ratio is acceptance); a request's
|
||||
`timings.draft_n` / `draft_n_accepted` gives it per call. Drafts stuck at 0
|
||||
with tokens flowing means the drafter is not running.
|
||||
4. Real completion through the fleet alias, then a deep prompt (a 140k-token
|
||||
prompt proves a raised window is real, not just declared).
|
||||
5. Prefix cache: repeat the same prompt — `timings.prompt_n` should collapse to
|
||||
a handful of tokens.
|
||||
|
||||
## Pitfalls that cost real time
|
||||
|
||||
- A healthy-looking build that dies on the first request is almost always the
|
||||
bundled-library depth above.
|
||||
- Serve scripts that `source` an env file AFTER defaults cannot be overridden
|
||||
from the command line; override the env-file PATH instead.
|
||||
- Qwen3.8 chat templates RAISE on `reasoning_effort=high` (500). Probe
|
||||
low/medium/high/xhigh before declaring an endpoint ready, and map the
|
||||
harness's levels onto the supported set.
|
||||
- One slot at a large context is the concurrency limit: concurrent agent loops
|
||||
queue, and first-turn TTFT then includes the wait.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: llamacpp-layer-split-vram-rebalance
|
||||
description: "Re-measure and set the --tensor-split VRAM balance for a multi-GPU llama.cpp server when the batch, context, slots or spec drafter change, instead of interpolating between splits."
|
||||
---
|
||||
|
||||
Re-balancing is needed whenever a knob moves weight between cards: `--ubatch-size` (compute buffers), a spec drafter (weights + draft KV), `--ctx-size`/`--parallel` (KV), or a different drafter with a different window policy. A split balanced at one configuration is unbalanced at another — and the second card's asymmetry (output layers + drafter graphs) is what makes it non-obvious.
|
||||
|
||||
## 0. Stop the service first
|
||||
|
||||
Serving a user? Stop it, then free the port, before loading test instances. Do not bench beside a live server.
|
||||
|
||||
```bash
|
||||
sudo systemctl stop <svc> # root unit; for a rootless quadlet use systemctl --user
|
||||
pkill -x llama-server && sleep 3 # a manual/child run holds the port and the VRAM
|
||||
```
|
||||
|
||||
`pkill -f <pattern>` is forbidden here: it matches your own command line and kills the ssh session. `mask` FAILS on a unit that is a symlink into a repo checkout (`File ... already exists and is a symlink to ...`); a `stop` is enough because nothing else restarts it.
|
||||
|
||||
## 1. Measure one split per load
|
||||
|
||||
The serve script is env-driven, so point it at a copy of the env file with `TENSOR_SPLIT` patched and read the allocation back from `rocm-smi`. Load -> sample -> kill, with NO requests sent: this touches allocation only, not the compute path.
|
||||
|
||||
```bash
|
||||
WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
|
||||
export FRANK_MEDIUM_ENV_FILE=$WORK # the script's own override hook
|
||||
for ts in 39,26 38,27 37,28 36,29 35,30; do
|
||||
sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
|
||||
nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
|
||||
for _ in $(seq 1 150); do
|
||||
[ "$(curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:PORT/health)" = 200 ] && break
|
||||
pgrep -x llama-server >/dev/null || break
|
||||
sleep 2
|
||||
done
|
||||
python3 -c "
|
||||
import subprocess,re
|
||||
CAP=34208743424 # one card, as rocm-smi reports it
|
||||
o=subprocess.run(['rocm-smi','--showmeminfo','vram'],capture_output=True,text=True).stdout
|
||||
u=[int(x) for x in re.findall(r'VRAM Total Used Memory \\(B\\): (\\d+)',o)][:2]
|
||||
print('$ts', f'{(CAP-u[0])/2**30:.2f}', f'{(CAP-u[1])/2**30:.2f}')"
|
||||
pkill -x llama-server; sleep 6
|
||||
done
|
||||
```
|
||||
|
||||
Sample ~6 s after `/health` returns 200 — the last allocations land after listening. Run it under `nohup` and poll the log from a separate call: a load is 30-90 s, which exceeds a single command's patience.
|
||||
|
||||
## 2. Read the table, do not interpolate
|
||||
|
||||
Pick the split with the smallest inter-card gap, then confirm it clears any per-card floor (e.g. 1.5 GiB) with room to spare. **A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.** Measured on qwen35 (16 x (3 Gated DeltaNet + 1 Gated Attention)): 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one (weights + that layer's own f16 KV). So the gap can jump 2.9 GiB in a single step, and the best integer split is ~0.2 GiB from balanced while its neighbours are 1+ GiB off. Interpolating between two measured splits produces a split nobody should deploy.
|
||||
|
||||
Total free VRAM is invariant across splits (only its distribution moves) — use that to sanity-check a suspicious row.
|
||||
|
||||
## 3. Check the drafter's window before blaming the split
|
||||
|
||||
A draft context is built with `n_ctx = llama_n_ctx(ctx_tgt)` and there is no flag to cap it, so a drafter declaring NO sliding window caches the WHOLE target window on `--spec-draft-device`. DFlash2 (`sliding_window 2048`) = ~40 MiB; DSpark (no window) = 20 KiB/position = 5.0 GiB at 262144 — a four-layer swing in the balance point. Read `sliding_window` out of the drafter's GGUF header before sizing, and prefer a skewed `-ts` over cutting `--ctx-size`.
|
||||
|
||||
## 4. If the repo has a fitted VRAM model, refit it — with the same unit convention
|
||||
|
||||
Match the existing expression literally (e.g. `1442 * 1024 * ubatch` means ~1.4 MB per ubatch unit, not 1442 bytes). Fit so the deployed point reproduces `rocm-smi` to ~0.05 GiB, keep the physics (weights + proportional KV + a window-bounded draft KV + ubatch-scaled buffers), and treat a NEGATIVE fitted constant as a unit bug. State in the comment where the model is conservative: a uniform per-layer cost overstates gaps in a hybrid stack, and a fitted ubatch coefficient makes any larger-batch verdict an extrapolation, not a measurement.
|
||||
|
||||
## 5. Verify behaviourally after deploy, not just by flags
|
||||
|
||||
- `/slots` -> the expected `(id, n_ctx)` pairs.
|
||||
- The drafter's init lines from `journalctl -u <svc>`: implementation name, `n_max`, `block_size` — read the NUMBERS, since DFlash2 and DSpark log under the same `draft_dflash` symbol.
|
||||
- `rocm-smi` free VRAM at the deployed split (the fitted prediction and the real figure should agree to ~0.05 GiB).
|
||||
- One tool call and two CONCURRENT requests, one per slot, then `journalctl -o cat | grep 'slot print_timing' | grep -o 'slot print_timing: id *[01].*draft acceptance.*'` to prove drafting is live on BOTH slots (`journalctl` default output truncates these lines; `-o cat` plus `grep -o` gets them whole).
|
||||
- `grep -c 'ROCm error\|illegal memory\|ggml_abort'` on the current instance must be 0.
|
||||
|
||||
## Faults to watch, not to tune away
|
||||
|
||||
A cross-card boundary copy can fault (`amdgpu SDMA0 UTCL2 page fault` -> `ROCm error: an illegal memory access was encountered` -> `ggml_abort`) on a split that loads and answers `/health` perfectly. It is not an OOM — more headroom does not prevent it — so treat a split that has faulted as off-limits and re-measure rather than re-tuning. Also: a load that succeeds and then dies on the FIRST draft decode with an OOM is a sizing failure (the draft KV), not a broken drafter.
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
name: llamacpp-mtp-vision-endpoint-bringup
|
||||
description: "Bring up a new llama.cpp (mainline) OpenAI endpoint on a single CUDA GPU with MTP speculative decoding and vision — GGUF header preflight by HTTP range, hybrid-architecture KV arithmetic, the -mtp / --spec-type draft-mtp contract, --parallel vs ctx math, and the verification battery that proves MTP, tools and vision actually work."
|
||||
---
|
||||
|
||||
# llama.cpp MTP + vision endpoint bring-up (single CUDA GPU)
|
||||
|
||||
Use when standing up or swapping a mainline llama.cpp endpoint for a GGUF that
|
||||
may ship an MTP (NextN) head and a vision projector. Verified end to end on
|
||||
Dankclaw 2026-09-16 (RTX 3090 24 GiB, Qwen3.8-27B-GSQ-RCO IQ3_S `-mtp` +
|
||||
BF16 mmproj, 131072 ctx / 1 slot, port 7771).
|
||||
|
||||
## 1. Preflight the GGUF WITHOUT downloading it
|
||||
|
||||
GGUF metadata lives in the header, and HuggingFace honours ranged GETs, so a
|
||||
few MB answers every sizing question before a 13 GB transfer:
|
||||
|
||||
```python
|
||||
import urllib.request
|
||||
req = urllib.request.Request(url, headers={"Range": "bytes=0-8000000", "User-Agent": "curl/8"})
|
||||
buf = urllib.request.urlopen(req, timeout=180).read()
|
||||
```
|
||||
|
||||
Parse: magic `GGUF`, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs
|
||||
(string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte —
|
||||
the type hand-written parsers forget). Walk arrays by byte size even when you
|
||||
only want a summary: a naive parser desynchronises on `tokenizer.ggml.tokens`
|
||||
and lands gigabytes off.
|
||||
|
||||
Read these keys: `general.architecture`, `<arch>.block_count`,
|
||||
`<arch>.context_length`, `<arch>.nextn_predict_layers`, `<arch>.head_count_kv`,
|
||||
`<arch>.key_length` / `value_length`, `<arch>.full_attention_interval`,
|
||||
`<arch>.ssm.state_size` / `group_count` / `time_step_rank`, and for the
|
||||
projector file `clip.projector_type` (e.g. `qwen3vl_merger`) plus
|
||||
`clip.has_vision_encoder`.
|
||||
|
||||
## 2. Decide the file from `nextn_predict_layers`
|
||||
|
||||
**MTP is a property of the FILE, not of the flags.** If the release ships an
|
||||
`-mtp` build, that build carries the head as `blk.<n>.nextn.*` (a few hundred
|
||||
MB larger); the plain build has no MTP tensors at all.
|
||||
|
||||
With the `-mtp` file, llama.cpp needs only `--spec-type draft-mtp`: it sets
|
||||
`load_mtp` and the draft context runs on the MAIN model's weights
|
||||
(`shares_model`, `LLAMA_CONTEXT_TYPE_MTP`). Do NOT add `--model-draft` — that
|
||||
loads the whole file a second time (12 GiB twice on a 24 GiB card). A separate
|
||||
`-md` is only correct for standalone MTP-only sidecars.
|
||||
|
||||
Confirm the flag exists in the deployed build before writing the script:
|
||||
`grep -rn "common_speculative_all_types_str" common/speculative.cpp` and
|
||||
`llama-server --help | grep -A4 spec-type`.
|
||||
|
||||
## 3. KV arithmetic (do this before choosing ctx)
|
||||
|
||||
Only **full-attention** layers take KV. For a hybrid arch (Qwen3-Next/3.8 style
|
||||
`16 x (3 x Gated DeltaNet + 1 x Gated Attention)`):
|
||||
|
||||
```
|
||||
attn_layers = block_count / full_attention_interval
|
||||
bytes/token = attn_layers x n_head_kv x head_dim x 2 (K,V) x bytes_per_elem
|
||||
f16 = 2 B | q8_0 = 1.0625 B | q4_0 = 0.5625 B
|
||||
```
|
||||
|
||||
The linear-attention layers hold constant-size per-sequence state instead
|
||||
(~0.16 GiB per sequence at 48 layers / 128 d_state / 48 v-heads in f32), and
|
||||
the MTP draft context costs its own `n_ctx x (1 layer x n_head_kv x head_dim x
|
||||
2 x 2 B)` ≈ 4 KiB/token for a 1-layer head.
|
||||
|
||||
VRAM budget: weights + mmproj + MTP draft KV + DeltaNet state + ~1.5 GiB of
|
||||
CUDA context and compute buffers. On a 24 GiB card that leaves ~9.5 GiB for KV,
|
||||
which at 64 KiB/token is ~155k tokens total.
|
||||
|
||||
**`--parallel N` DIVIDES `--ctx-size` between slots** — it does not multiply.
|
||||
`-np 1 -c 131072` is one 131k conversation; `-np 4 -c 131072` is four 32k ones.
|
||||
|
||||
## 4. Flags that matter
|
||||
|
||||
- `-ngl 999`, `-fa on`, `-ctk/-ctv f16` for the usual shape.
|
||||
- Quantized KV (`q8_0`, `q4_0`) requires a build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`; without it CUDA prefill collapses ~12x. f16
|
||||
needs nothing, so prefer it unless the context demand is real.
|
||||
- `--jinja` — pin it explicitly even when the build defaults it on: the GGUF's
|
||||
own template supplies `reasoning_effort`, `preserve_thinking` and the
|
||||
tool-call grammar, and a future default flip must not silently remove tools.
|
||||
- `--metrics` if anything scrapes `/metrics`, `--alias` to set the served id,
|
||||
`--temp 0.0` for a greedy-by-default fleet contract,
|
||||
`--reasoning-format deepseek --reasoning-budget -1` to project thinking into
|
||||
`reasoning_content`.
|
||||
|
||||
## 5. Serve-script and unit invariants
|
||||
|
||||
Fail loudly before loading VRAM: check the binary, the model file and the
|
||||
mmproj exist, and `exit 1` with a named message otherwise — a missing weight
|
||||
must not become a `Restart=on-failure` crash loop that reads like a runtime
|
||||
fault. For a native CUDA binary the unit is a ROOT systemd unit
|
||||
(`[Install] WantedBy=multi-user.target`, `Restart=on-failure`,
|
||||
`RestartSec=10`, `LimitMEMLOCK=infinity`); a container would be a Quadlet.
|
||||
|
||||
## 6. Verification battery (prove each claim, do not infer)
|
||||
|
||||
1. `/health` 200, and `/v1/models` reports the intended alias.
|
||||
2. A chat completion returns `content` AND `reasoning_content`.
|
||||
3. A request carrying `tools` returns `tool_calls` — a RUNTIME property: the
|
||||
ik_llama.cpp runtime answered `500 tools param requires --jinja flag`.
|
||||
4. `GET /metrics` after a generation: `llamacpp:spec_decode_num_draft_tokens_total`
|
||||
and `..._accepted_tokens_total` both > 0, acceptance 0.6-0.95 ⇒ MTP engaged,
|
||||
not merely configured. The slot timing lines in the journal say the same.
|
||||
5. One image request (`data:` URL or bare base64) ⇒ an answer that reflects the
|
||||
image; confirm the journal printed a multimodal projector load.
|
||||
6. `nvidia-smi` memory used is close to the modelled total, and `n_slots` /
|
||||
`n_ctx_slot` in the load log match the intended shape.
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **A template can reject the client's thinking level.** Qwen3.8 accepts only
|
||||
`xhigh|medium|low` for `reasoning_effort` and raises otherwise — HTTP 500
|
||||
naming Jinja, not the model. Map every harness level onto the accepted set
|
||||
(e.g. `compat.reasoningEffortMap: {minimal: low, low: low, medium: medium,
|
||||
high: xhigh, max: xhigh}`) and probe each level with curl before routing
|
||||
roles at the endpoint.
|
||||
- **Thinking counts against `max_tokens`.** A budget that runs out mid-thought
|
||||
returns `finish_reason: "length"` with empty `content` — not a broken
|
||||
endpoint, not a vision failure. Raise the budget.
|
||||
- **Never point `--model-draft` at the `-mtp` file** (step 2).
|
||||
- **Any `-ot` / `-ncmoe`-style override can silently disable mmap** for the
|
||||
whole load, turning a mapped model into anonymous memory.
|
||||
- On a host that publishes ports through **pasta**, the replacement must not
|
||||
start until the predecessor released the port: poll
|
||||
`ss -tan "sport = :<port>"` (listener, accepted AND TIME_WAIT) or the
|
||||
container dies with `Address already in use`.
|
||||
- On a shared host, read the serving engine's own boot ledger for device
|
||||
budget vs need: a pool that silently clamped still answers a green `/health`.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: llamacpp-multigpu-layer-split-cutover
|
||||
description: "Cut a fleet LLM endpoint over to llama.cpp layer-split across multiple GPUs when cross-GPU collectives are impossible (RCCL/PCIe-atomics wall), including the build recipe, the bundling trap that fails on the first request, DFlash2/MTP wiring, and the measurement battery that proves the split and the drafter are real."
|
||||
---
|
||||
|
||||
# llama.cpp layer-split cutover (no-collective multi-GPU)
|
||||
|
||||
Use when one model must serve from TWO+ cards and the box cannot dispatch a
|
||||
cross-card collective (second card behind a chipset bridge with
|
||||
`AtomicOpsCap: Routing-`, so ROCr refuses RCCL's hostcall-buffer kernels).
|
||||
Decide with one line before designing anything:
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <card2> | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever
|
||||
```
|
||||
|
||||
`ReqEn-` kills vLLM entirely — tensor parallelism needs an allreduce **and
|
||||
pipeline parallelism needs an RCCL `send`/`recv` of the hidden state**
|
||||
(`vllm/distributed/parallel_state.py`: metadata on gloo, tensors on the device
|
||||
backend). llama.cpp's `--split-mode layer` needs no collective: layers are
|
||||
distributed in order, only the boundary activation crosses PCIe once per
|
||||
forward pass.
|
||||
|
||||
## 1. Pin the runtime by FEATURE, not by "latest"
|
||||
|
||||
Write down which features the deployment needs, then find the ref that has all
|
||||
of them and pin the 40-hex sha in the build script. On 2026-09-16 that was
|
||||
DFlash2 (`ggml-org/llama.cpp#27342`, detected from the draft GGUF:
|
||||
`selector_top_k > 0`, spec type stays `draft-dflash`) and qwen35 MTP. The
|
||||
ROCmFPX/r9700-toolbox images/fork tips predate both — verify by listing the
|
||||
built image:
|
||||
|
||||
```bash
|
||||
ls <venv>/lib/python*/site-packages/vllm/v1/worker/gpu/spec_decode/ # dflash/ vs dflash2/
|
||||
```
|
||||
|
||||
## 2. Build self-contained for one arch
|
||||
|
||||
- Private ROCm toolchain (TheRock nightly) with `ROCM_PATH`/`HIP_PATH` exported,
|
||||
`-DCMAKE_HIP_ARCHITECTURES=<gfx only>` (a sibling arch links, loads and then
|
||||
segfaults), `-DGGML_HIP_RCCL=OFF` when there is no atomics path, `-DGGML_NATIVE=OFF`.
|
||||
- Bundle the runtime into `build/lib` and set `--set-rpath '$ORIGIN/../lib'`.
|
||||
- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist.** rocBLAS and
|
||||
hipBLASLt look for device code at a FIXED depth (`../lib/rocblas/library`,
|
||||
`../lib/hipblaslt/library/<arch>`). Flattened, the engine loads every GiB of
|
||||
weights, answers `/health`, and dies on the **first request** with
|
||||
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch`.
|
||||
`mkdir -p DST && cp -r SRC/. DST/`.
|
||||
|
||||
## 3. Deploy as an env + serve script + unit trio
|
||||
|
||||
One env file holds every tunable with the sizing arithmetic in comments; the
|
||||
serve script defaults each var, refuses to start when the runtime or payload is
|
||||
missing, and builds an args array (spec flags assembled conditionally so the
|
||||
same script serves `draft-dflash` with a file and `draft-mtp` in-model). Root
|
||||
systemd unit: `User=<deploy user>`, `LimitMEMLOCK=infinity`,
|
||||
`Restart=on-failure`, `WantedBy=multi-user.target`, plus an
|
||||
`Environment=…_ENV_FILE=<repo path>` so the repo file is the single source.
|
||||
|
||||
Deploy handler must: refuse when runtime/payload is missing; retire the previous
|
||||
runtime's units (into trash, never `rm`); **wait for the port to be released**
|
||||
(a retired reverse proxy or container holds it); then enable/restart and gate on
|
||||
`/health`.
|
||||
|
||||
DRLaf: `--split-mode layer --device ROCm0,ROCm1 --tensor-split 1,1`,
|
||||
`--parallel 1`, f16 KV, `--fit off`. Pin the drafter with `--spec-draft-device`
|
||||
to the card holding the target's last layers. `--spec-draft-n-max` must be
|
||||
<= the draft's `block_size - 1` (DFlash2: 8 ⇒ 7).
|
||||
|
||||
## 4. Prove it, do not assert it
|
||||
|
||||
- **The split is real**: raise `--ctx-size` by N tokens and watch `rocm-smi` —
|
||||
a layer split adds ~N·(bytes/token)/2 to EACH card (both cards carry the KV
|
||||
for their own attention layers), and neither card's TOTAL can equal the model
|
||||
file, which is the arithmetic that rules out replication.
|
||||
- **The drafter is live**: `timings.draft_n`/`draft_n_accepted` per request, and
|
||||
the cumulative `llamacpp:spec_decode_num_drafts_total` — the flags can be
|
||||
right while the draft model silently failed to load. The loader's init line
|
||||
(`block_size=`, `n_extract=`, `mask_token_id=`) identifies the drafter family.
|
||||
- **Watch for a 500 on `reasoning_effort`**: Qwen3.8-family chat templates
|
||||
accept only low/medium/xhigh and RAISE on `high`; map every harness level or
|
||||
subagents 500 on their first request.
|
||||
- **Re-point the monitors in the same change**: a runtime swap changes the
|
||||
metric namespace (`vllm:` → `llamacpp:`), and rules/dashboard panels naming
|
||||
series that no longer exist draw nothing forever without erroring. Retire
|
||||
what cannot be expressed (llama.cpp exports no KV-fill gauge and no latency
|
||||
histogram) and name the consequence instead (queue depth, spec-decode-off).
|
||||
- One slot is a concurrency policy, not a bug: `--parallel` DIVIDES the context.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: llamacpp-vram-accounting
|
||||
description: "Produce and interpret a per-device VRAM/memory breakdown for a llama.cpp or halogen LLM deployment on the Dank fleet: we/weights vs KV vs compute pool, discrete vs UMA accounting, and the measurement traps that make the numbers lie."
|
||||
---
|
||||
|
||||
# llama.cpp / halogen VRAM accounting (Dank fleet)
|
||||
|
||||
Use when asked *where the memory went* for an LLM service on Dankclaw/Dankserver: "what's the VRAM breakdown", "why does it not fit", "how much headroom is left for the co-tenants", or before sizing a card for a new model.
|
||||
|
||||
## 0. Decide which kind of memory you are talking about
|
||||
|
||||
| device | kind | read it with |
|
||||
|---|---|---|
|
||||
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader` |
|
||||
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (real, 114 GiB) vs `mem_info_vram_{total,used}` (2 GiB carve-out window) |
|
||||
| R9700s (Dankserver) | discrete | rocm-smi figure is authoritative (64 GiB claimed, SVM-backed) |
|
||||
|
||||
Per-process truth on NVIDIA: `nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader`.
|
||||
|
||||
**Never** add the iGPU's GTT to host RAM as if they were separate pools, and never quote a UMA engine's total as "VRAM": halogens's 95.4 GiB is 68 GiB of **pinned host pages** (mlock + page cache) plus ~33.7 GiB inside the device budget. `free -g` is the arbiter for what the co-tenants have left.
|
||||
|
||||
## 1. Get the engine's own ledger first
|
||||
|
||||
```bash
|
||||
podman logs frank-large 2>&1 | grep -aiE "weights pinned|kv pool|left for everything|overlay|vision|budget"
|
||||
```
|
||||
|
||||
Halogen prints the numbers that matter: weights pinned (and how many contiguous 2 MiB blocks remain), the device budget (`host RAM − resident weights − reserve`), the KV pool size and pool-vs-working split, and the final "left for everything else" block count. Contiguous-block count is the honest fragmentation signal — 240 blocks / 12.3 GiB after a reboot vs 475 / 14.7 GiB before means the same model did not come back into the same hole.
|
||||
|
||||
## 2. For llama.cpp: the print sites exist, but no artifact here has them
|
||||
|
||||
- `src/llama-model.cpp:1827` — per-backend `model buffer size` (unconditional `LLAMA_LOG_INFO`)
|
||||
- `src/llama-context.cpp:692` — `compute buffer size`
|
||||
- plus a KV self-size line at context creation
|
||||
|
||||
Traps verified on 2026-09-16:
|
||||
|
||||
- **`llama-bench` suppresses the loader's INFO** — its logs contain only the device banner and the results table, so a bench matrix leaves *no* memory accounting behind.
|
||||
- **`llama-server` under systemd did not put them in the journal either** — `sudo journalctl -u frank-small | grep -c "model buffer size"` returned **0** for the unit's entire history. Check with `grep -c` before claiming a log has the breakdown.
|
||||
- Recipe that works: launch the model **once** through `llama-cli` (or `llama-server`) and keep the first ~200 lines; add `-v` if the level filters them.
|
||||
|
||||
## 3. Otherwise compose it — arithmetic on measured tensor sizes
|
||||
|
||||
Read the GGUF's tensor table once (names + shapes are enough) and group by role, e.g. Flash-Next (111 GiB):
|
||||
|
||||
| group | size | per layer |
|
||||
|---|---|---|
|
||||
| `ffn_{gate,up,down}_exps` | 75.0 GiB | **1.5625 GiB** (experts travel with their layer) |
|
||||
| `per_layer_token_embd` (PLE) | 32.8 GiB | one table |
|
||||
| everything else (attn/GDN/etc) | 3.2 GiB | 0.067 GiB |
|
||||
|
||||
And KV ≈ 2 KiB per position per f16 attention layer (12 attention layers on Flash-Next => ~24 KiB/position; halogen's ledger says ~28 KiB/position including block scratch). Multiply by the *derived* context (see §5).
|
||||
|
||||
Then take the measured per-device total as the constraint and let the remainder be compute/graph scratch. Vendor delta is real and sizeable: same 7 layers + KV on the 3090 came to **12.80 GiB under Vulkan** and **14.99 GiB under CUDA** — ~3.3 GiB vs ~1.1 GiB of graph/compute pool.
|
||||
|
||||
## 4. Interpretation traps
|
||||
|
||||
- **mmap PLE is reclaimable; `-ot` makes it not.** With everything on the iGPU the 32.8 GiB PLE sits in page cache (reclaimable). In a layer split, where `-ot` is disallowed, it becomes a pinned device buffer in GTT. "Move layers to the card" therefore converts reclaimable cache into pinned device memory and does not free room for co-tenants.
|
||||
- **A served model's footprint is not its weight size.** frank-small: 22,582–22,592 MiB of 24,576 for an 11.29 GiB model — weights + f16 KV at 131,072 × 1 slot + BF16 projector + CUDA compute pool.
|
||||
- **"Usable" is total minus driver/container overhead** (~24.1 GiB on a 24,576 MiB card).
|
||||
- Do the capacity subtraction *before* designing: 12 attention layers × 1.62 GiB = 19.4 GiB, plus 200k f16 KV (9.2 GiB) = 28.6 GiB > 24.1 — which is why a KV-on-VRAM design on this pair is only feasible to ~64k context.
|
||||
|
||||
## 5. Context lengths: derive, never remember
|
||||
|
||||
`llama-bench` never logs the context; it computes (`tools/llama-bench/llama-bench.cpp:1293`):
|
||||
|
||||
```c
|
||||
cparams.n_ctx = n_prompt + n_gen + n_depth;
|
||||
```
|
||||
|
||||
Defaults: `n_prompt 512`, `n_gen 128`, `n_depth 0`, `n_batch 2048`, `n_ubatch 512`, `type_k`/`type_v` **F16**, `flash_attn AUTO`, `split_mode LAYER`, `reps 5`.
|
||||
|
||||
So `-p 512,16384 -n 64` = contexts **576 / 16,448 / 64**, and `-n 32 -d 65536` = **66,080 / 65,568**. A `tg` row runs with `n_prompt = 0`, which is why a plain `tg` number is ~zero depth: `-d` is the only thing that puts KV in the picture. Cross-check by reading the table's own test column (`pp512 @ d65536`, `tg32 @ d65536`).
|
||||
|
||||
## 6. Reporting shape that survives review
|
||||
|
||||
Label every number: **measured** (nvidia-smi / sysfs / engine ledger line), **derived** (arithmetic on tensor sizes + derived context), or **inference** (e.g. depth extrapolation). State the context each memory figure was taken at, and whether KV was resident — 12.8 GiB at 65,568 tokens is not the same claim as 12.8 GiB at 200k.
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
name: llm-replica-routing-prefix-cache
|
||||
description: "Route requests across multiple LLM replicas by workload shape (pin agent-loop continuations for prefix-cache hits, spread batches for parallel prefill), with the measurement traps and the capacity wall that no routing policy fixes."
|
||||
---
|
||||
|
||||
# Prefix-cache-aware routing for multi-replica LLM serving
|
||||
|
||||
Use when several single-GPU inference replicas sit behind one endpoint and
|
||||
clients send long shared prefixes (agent loops, chat with a big system prompt,
|
||||
RAG with a fixed context).
|
||||
|
||||
## The two failure modes, both measurable
|
||||
|
||||
Measure with a **sequential-turn client**, not only a batch benchmark: a batch
|
||||
of distinct prompts parallelises prefill across replicas, so round-robin looks
|
||||
great there and hides the problem.
|
||||
|
||||
1. **Round-robin loses every other cache hit.** The loop resends the same
|
||||
prefix each turn; alternating replicas means half the turns prefill from
|
||||
scratch. Measured: 90k-token contexts, warm-turn TTFT 2676 ms round-robin
|
||||
vs 1096 ms pinned.
|
||||
2. **Pinning everything loses a card.** A cold batch pays one card's prefill
|
||||
throughput: 16 requests in 18.6 s spread vs 30.1 s pinned.
|
||||
|
||||
## The rule that gets both
|
||||
|
||||
Route by request shape:
|
||||
|
||||
- **Continuation** (request carries more than one message — the signature of an
|
||||
agent loop re-sending its context) -> pin to the replica that served that
|
||||
prefix before.
|
||||
- **Anything else** (one-shot question, batch member, parallel subagent) ->
|
||||
spread across replicas with a **counter**, not with a hash of the prefix.
|
||||
|
||||
Measured on the same host: 19.97 s burst *and* 1095 ms warm turns.
|
||||
|
||||
Also add **failover**: a replica that restarts is invisible to a health table
|
||||
refreshed every 10 s, so a request routed to it returns 502 even though the
|
||||
other replica is up. On connection failure, demote it immediately and retry the
|
||||
request on a live replica before writing anything to the client (the OpenAI
|
||||
endpoints are stateless, so re-sending a request that never reached an engine
|
||||
is safe; never retry after streaming started). Measured: six requests during a
|
||||
replica stop, six 200s, one logged failover, no client-visible error.
|
||||
|
||||
## The capacity wall that no policy fixes
|
||||
|
||||
Per-card KV is finite and **a running sequence and its cached prefix draw on
|
||||
the same pool**. With N families on M cards, one card carries two of them, and
|
||||
its budget must hold both cached prefixes plus whichever sequence is
|
||||
generating. Three concurrent 50-80k-token families against 122,636 tokens per
|
||||
card thrash regardless of how requests are placed:
|
||||
|
||||
| policy | result |
|
||||
|---|---|
|
||||
| counter placement | heavy 25.4 s median (23 % cached), mediums 34.7 / 45.2 s |
|
||||
| load-balanced placement (new families to the card carrying less context) | no improvement — reverted |
|
||||
| one large request admitted per card | no improvement — reverted |
|
||||
| smaller/larger prefill chunk | *costs* KV in both directions (a sweet spot exists; find it) |
|
||||
|
||||
So: measure placement and admission before shipping them, and when the working
|
||||
set exceeds aggregate KV, say so — the levers are capacity (another card),
|
||||
fewer concurrent large contexts (client-side context compaction), or a different
|
||||
platform. Routing cannot rescue arithmetic.
|
||||
|
||||
## Measurement traps
|
||||
|
||||
- **A lifetime engine counter is not a per-request verdict.** "Prefix cache hit
|
||||
rate: 21 %" is an average over every prompt the engine ever saw, including
|
||||
benchmark traffic that never repeats. Probe the question directly instead:
|
||||
send the same long prompt to an idle replica twice — 22.3 s cold then 1.4 s
|
||||
warm is what a working cache looks like (and it holds with a varying tail).
|
||||
- **Salt your corpora.** Two runs sharing a prefix measure a warm cache, not
|
||||
the thing under test; a unique leading line per run makes the prefix new.
|
||||
- **Record the sample size.** A 14-minute arm and an 8-minute arm are not a
|
||||
comparison, and medians from five turns say little.
|
||||
- **Check the tool's own requirements**: `vllm bench serve` needs
|
||||
`--tokenizer <local path>` (plus the model directory mounted) or it tries to
|
||||
resolve the served alias on HuggingFace and dies before sending a request —
|
||||
a run that produces no numbers and no obvious error.
|
||||
- **Verify metric names against the live endpoint** before writing dashboards
|
||||
or alerts: guessing one that does not exist is invisible until someone needs
|
||||
the graph.
|
||||
|
||||
## Test shape that catches it
|
||||
|
||||
Drive the routing function directly with synthetic request bodies (keep the
|
||||
policy in a stdlib-only module so tests need no HTTP framework):
|
||||
|
||||
- a batch of single-turn requests -> assert both replicas are used and
|
||||
consecutive picks alternate;
|
||||
- a growing conversation (same leading bytes, longer history) -> assert every
|
||||
turn agrees with the first turn's replica;
|
||||
- each mode with a replica marked down -> assert the live one is chosen, and
|
||||
-1 when none is;
|
||||
- the fixed comparison modes (`roundrobin`, `affinity`) kept behind an env var
|
||||
so the choice can be re-measured rather than argued.
|
||||
|
||||
One more: the identity of a conversation is its **first N KiB** of prompt. If
|
||||
the tested prefix is shorter than that window its varying tail falls inside it,
|
||||
turns hash differently, and pinning silently never happens.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: llm-serving-benchmark-methodology
|
||||
description: "Measure and A/B a self-hosted multi-replica LLM endpoint honestly: corpus construction, per-arm salting, repeated samples, engine-counter semantics, and the traps that make results meaningless."
|
||||
---
|
||||
|
||||
# Bench a multi-replica LLM deployment so the numbers mean something
|
||||
|
||||
Use when measuring or tuning a self-hosted LLM endpoint (vLLM, llama.cpp,
|
||||
radiance/MXFP4), especially behind a proxy in front of several replicas.
|
||||
|
||||
## Set up a client that can actually send requests
|
||||
|
||||
`vllm bench serve` needs `--tokenizer <local model path>` with the model
|
||||
directory mounted. Given only `--model <served-alias>` it tries to resolve that
|
||||
alias on HuggingFace and dies before the first request — a run that looks like
|
||||
it worked and produced nothing. Confirm with a single prompt before benching.
|
||||
|
||||
## Build the workload from real material
|
||||
|
||||
`--dataset-name random` generates token soup whose draft acceptance is nothing
|
||||
like production (measured 2.9-3.6 tok/update against 5.8 on code), so it
|
||||
mis-ranks speculative-decoding settings. Generate a JSONL corpus from the
|
||||
repository's own sources — `{"prompt": "<instruction>\n\n<file body>"}` — and
|
||||
fix output length with `--custom-output-len`. Keep two corpus shapes:
|
||||
|
||||
- **distinct prompts** (batch behaviour, prefill parallelism)
|
||||
- **a shared long prefix with a varying tail** (conversation turns, where
|
||||
prefix caching decides everything)
|
||||
|
||||
## Salt every arm
|
||||
|
||||
Put a unique marker in the first bytes of every prompt, different per A/B arm.
|
||||
Without it the second arm silently measures the first arm's warm cache and both
|
||||
arms look identical.
|
||||
|
||||
## Measure the same workload before and after
|
||||
|
||||
Keep the workload fixed and change one knob. Repeat each arm — run-to-run
|
||||
variance of 5-10 % is normal, and a single sample nearly made me ship the wrong
|
||||
setting. Report medians per family/class, not one aggregate number.
|
||||
|
||||
## Read engine counters correctly
|
||||
|
||||
- vLLM's `Prefix cache hit rate:` line is a **lifetime average** across every
|
||||
prompt that replica ever saw. It cannot answer "is this conversation served
|
||||
warm?". Send the same prompt twice to an idle engine and compare TTFTs
|
||||
instead (measured: 22.3 s cold → 1.4 s warm for a 50k-token prefix).
|
||||
- `GPU KV cache usage` tells you the pool is not full; it does not tell you a
|
||||
prefix survived. Capacity per card is the boot-time ledger line
|
||||
(`GPU KV cache size: N tokens`), not the card's VRAM.
|
||||
- Reserve memory is not allocation: a bigger reserve does not free anything for
|
||||
anyone else, it only removes cache.
|
||||
|
||||
## Route-shape traps
|
||||
|
||||
A cold batch of distinct prompts prefills in parallel across replicas; the same
|
||||
conversation's next turn only pays off if it lands where its prefix is cached.
|
||||
These pull in opposite directions, so measure both shapes separately and route
|
||||
by request shape (spread single-turn, pin continuations).
|
||||
|
||||
## When the answer is "no"
|
||||
|
||||
Write down rejected hypotheses with their numbers and the mechanism. Placement
|
||||
balance, concurrency admission and chunk-size tweaks all looked obvious and all
|
||||
measured flat or worse on this fleet; the wall turned out to be arithmetic
|
||||
(cached prefixes plus live sequences share one pool). A runbook that records
|
||||
that saves the next operator a night.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
name: niri-output-rotation-pinning
|
||||
description: "Rotate/re-arrange niri monitors (portrait transforms, positions, scale) persistently and verify — covers the output-block-resets-defaults trap and auto-layout reflow that flings unpinned outputs, plus the counter-clockwise transform convention."
|
||||
---
|
||||
|
||||
# Rotating and pinning outputs on niri
|
||||
|
||||
Verified on blankpad, niri 26.04, 2026-09-15 (two PM1601 externals + eDP-1 laptop panel).
|
||||
|
||||
## 0. Ground truth first
|
||||
|
||||
```bash
|
||||
niri msg outputs # connector names, current mode, scale, transform, logical pos/size
|
||||
niri --version
|
||||
```
|
||||
|
||||
Niri output **names are connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings. The EDID strings printed before them in `niri msg outputs` are decoration — use the parenthesized name.
|
||||
|
||||
On blankpad the live config is split by `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`, and `cfg/display.kdl` is a **symlink into the dotfiles repo** (`~/.omp/dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl`). Edit the repo path, not the symlink target path — they are the same file, but the repo path is the one that gets committed. Check with `ls -ld`.
|
||||
|
||||
## 1. Transform direction is counter-clockwise
|
||||
|
||||
```bash
|
||||
niri msg output <NAME> transform --help # "Transform to set, counter-clockwise"
|
||||
```
|
||||
|
||||
| value | effect |
|
||||
|---|---|
|
||||
| `90` | content rotated 90° CCW ⇒ the panel's **top bezel sits on the LEFT** |
|
||||
| `270` | top bezel on the RIGHT |
|
||||
| `180` | upside down |
|
||||
| `flipped-90` / `flipped-270` | mirrored variants |
|
||||
|
||||
`90` vs `270` is a physical-stand fact no tool can read (portable monitors expose no orientation in EDID/MCCS). Pick one, apply it, and give the user the one-liner to flip — don't stall on the question.
|
||||
|
||||
## 2. Two traps that bite when you edit the config
|
||||
|
||||
1. **An `output "X" { … }` block resets every property you don't restate to its default.** Pinning position on `eDP-1` without repeating `scale 1.25` silently drops the laptop panel to scale 1.0. Restate `scale` (and `mode` if you care) in every block you touch.
|
||||
2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning only the externals shoved `eDP-1` from `0,0` to `3936,0` (laptop became rightmost). If you want the pre-existing left-to-right order, pin every output's position.
|
||||
|
||||
Positions are logical pixels at each output's own scale. Original order preserved example:
|
||||
|
||||
```kdl
|
||||
output "DP-2" {
|
||||
scale 1
|
||||
transform "90"
|
||||
position x=1536 y=0
|
||||
}
|
||||
|
||||
output "HDMI-A-1" {
|
||||
scale 1
|
||||
transform "90"
|
||||
position x=2736 y=0
|
||||
}
|
||||
|
||||
output "eDP-1" {
|
||||
scale 1.25 // MUST restate: dashed-out by the block otherwise
|
||||
position x=0 y=0
|
||||
}
|
||||
```
|
||||
|
||||
x-offsets chain: `eDP-1` logical width 1536 → DP-2 at 1536 → +1200 (portrait width) → HDMI-A-1 at 2736.
|
||||
|
||||
## 3. Apply and verify
|
||||
|
||||
```bash
|
||||
niri validate -c ~/.config/niri/config.kdl # exit 0, "config is valid"
|
||||
niri msg action load-config-file # live reload; load-config-file, NOT reload-config
|
||||
sleep 1
|
||||
niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
|
||||
```
|
||||
|
||||
Success looks like `Logical size: 1200x1920` (swapped) plus `Transform: 90° counter-clockwise` on each rotated panel, with positions matching the intended chain.
|
||||
|
||||
One-off rotation without touching config (temporary, forgotten on next config change):
|
||||
|
||||
```bash
|
||||
niri msg output DP-2 transform 90
|
||||
```
|
||||
|
||||
## 4. Commit
|
||||
|
||||
Dotfiles repo is `dank/command-center` (`main`); push to the local Gitea is permitted.
|
||||
|
||||
```bash
|
||||
cd ~/.omp && git pull --ff-only
|
||||
git add dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl
|
||||
git commit -m "blankpad(niri): <what changed>"
|
||||
git push
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- `reload-config` is not a niri subcommand — it's `niri msg action load-config-file`.
|
||||
- Never SIGUSR1 a terminal to "reload"; niri config reloads do not touch ghostty themes.
|
||||
- No man page for niri on this host; `niri <sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
name: pcie-link-capability-probe
|
||||
description: "Measure a GPU-to-host PCIe link the way a weight-streaming engine experiences it, and convert the result into a tokens/s ceiling (streamed expert weights) versus the compute-in-RAM alternative — before recommending or rejecting an offload design."
|
||||
---
|
||||
|
||||
# PCIe link capability probe
|
||||
|
||||
Use when someone asks "is the link fast enough to run model X with engine Y (on one GPU)?"
|
||||
and Y keeps some weights in host RAM — MoE expert offload with an LRU GPU cache, layer
|
||||
paging, weight streaming. The answer is arithmetic once the link is measured, and the
|
||||
*shape* of the engine decides whether the link matters at all:
|
||||
|
||||
| engine shape | what crosses PCIe per token | link matters? |
|
||||
|---|---|---|
|
||||
| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
|
||||
| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
|
||||
| everything resident on the GPU | nothing | no |
|
||||
|
||||
## 1. Establish the hardware ceiling (never trust the card alone)
|
||||
|
||||
```bash
|
||||
lspci -tv | grep -iA2 vga # which bridge the GPU hangs off
|
||||
d=$(lspci -D | grep -i nvidia | awk '{print $1}' | head -1)
|
||||
sudo lspci -vv -s "$d" | grep -E 'LnkCap:|LnkSta:' # the CARD's capability + trained state
|
||||
sudo lspci -vv -s 00:03.1 | grep -E 'LnkCap:|LnkSta:' # the PLATFORM PORT (this is the real limit)
|
||||
```
|
||||
|
||||
- `LnkCap` of the **upstream port** = the electrical ceiling. A card with `Width x16` under
|
||||
an `x8` port cannot exceed x8; an x4 riser/cable caps both.
|
||||
- `LnkSta` at idle usually shows a downgraded speed (PCIe ASPM downshift, e.g.
|
||||
`2.5GT/s (downgraded)`) — that is NOT the loaded rate. Trust the measurement in step 2.
|
||||
- Gen4 usable payload: x4 **7.877 GB/s**, x8 15.75, x16 31.5 (128b/130b encoding).
|
||||
|
||||
## 2. Measure it (the program that did the job)
|
||||
|
||||
Stop **all** production services on the host first (fleet policy), then compile and run:
|
||||
|
||||
```cuda
|
||||
// pcie-link-test.cu : nvcc -O2 -o /tmp/pcie-link-test /tmp/pcie-link-test.cu
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <cuda_runtime.h>
|
||||
#define CK(x) do{ cudaError_t e=(x); if(e!=cudaSuccess){fprintf(stderr,"CUDA %s (%s:%d)\n",cudaGetErrorString(e),__FILE__,__LINE__); return 1;} }while(0)
|
||||
static double ms_between(cudaEvent_t a, cudaEvent_t b){float ms=0; cudaEventElapsedTime(&ms,a,b); return (double)ms;}
|
||||
static int bench_size(size_t bytes){
|
||||
const int iters = bytes>=(1UL<<30)?10:(bytes>=(256UL<<20)?20:100);
|
||||
void *dev=NULL,*host=NULL,*pageable=NULL; cudaStream_t s=NULL;
|
||||
CK(cudaMalloc(&dev,bytes)); CK(cudaHostAlloc(&host,bytes,cudaHostAllocDefault));
|
||||
memset(host,0x5a,bytes); CK(cudaStreamCreate(&s));
|
||||
cudaEvent_t a,b; CK(cudaEventCreate(&a)); CK(cudaEventCreate(&b));
|
||||
for(int i=0;i<3;i++) CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaDeviceSynchronize());
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double h2d=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpyAsync(dev,host,bytes,cudaMemcpyHostToDevice,s));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double h2da=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(host,dev,bytes,cudaMemcpyDeviceToHost));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double d2h=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(a)); CK(cudaMemcpy(dev,host,bytes,cudaMemcpyHostToDevice)); CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b));
|
||||
double one_us=ms_between(a,b)*1000.0;
|
||||
pageable=malloc(bytes); memset(pageable,0x5a,bytes);
|
||||
for(int i=0;i<2;i++) CK(cudaMemcpy(dev,pageable,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(a)); for(int i=0;i<iters;i++) CK(cudaMemcpy(dev,pageable,bytes,cudaMemcpyHostToDevice));
|
||||
CK(cudaEventRecord(b)); CK(cudaEventSynchronize(b)); double pag=(double)bytes*iters/(ms_between(a,b)/1000.0)/1e9;
|
||||
printf("%8.1f MiB | %6.2f | %6.2f | %6.2f | %8.1f us | %6.2f\n", bytes/1048576.0,h2d,h2da,d2h,one_us,pag);
|
||||
free(pageable); cudaFreeHost(host); cudaFree(dev); cudaEventDestroy(a); cudaEventDestroy(b); cudaStreamDestroy(s); return 0;
|
||||
}
|
||||
int main(void){ cudaDeviceProp p; CK(cudaGetDeviceProperties(&p,0));
|
||||
printf("device %s | CUDA %d.%d\n", p.name, p.major, p.minor);
|
||||
printf(" MiB | H2D | async | D2H | 1x xfer | pageable\n");
|
||||
size_t sz[]={3UL<<20,16UL<<20,64UL<<20,256UL<<20,1UL<<30,2UL<<30};
|
||||
for(int i=0;i<6;i++) if(bench_size(sz[i])) return 1; return 0; }
|
||||
```
|
||||
|
||||
Size choice matters: **3 MiB** is the granularity an expert-streaming engine moves, so
|
||||
its single-transfer time is the number that matters; the big sizes give the ceiling.
|
||||
|
||||
## 3. Read it
|
||||
|
||||
- **Achieved GB/s is itself the proof of the trained link state** — 6.7 GB/s is only
|
||||
possible at Gen4 x4 (Gen1 x4 ≈ 1, Gen2 ≈ 2, Gen3 ≈ 3.94), so no separate `LnkSta`
|
||||
sample under load is needed.
|
||||
- **async ≈ sync and pageable ≈ pinned ⇒ link-limited, not staging-limited.** If async is
|
||||
much faster than sync, the engine's fill pipeline (batching copies on one stream) is
|
||||
where the win is; if pageable is much slower, pinning is a real lever. Measure, don't
|
||||
assume.
|
||||
- Per-slab launch overhead = single-xfer time − (slab bytes ÷ rate). Small next to
|
||||
bandwidth cost at 3 MiB.
|
||||
|
||||
## 4. Convert to a tokens/s ceiling
|
||||
|
||||
```
|
||||
active expert bytes/token = active_params × bytes/param (A3B @ Q4_K_M ≈ 1.70 GiB)
|
||||
link time/token = active expert bytes / measured H2D rate
|
||||
ceiling (tok/s) = 1 / link time/token × resident-fraction saved (LRU hits)
|
||||
```
|
||||
|
||||
Then compare against the alternatives on the same host — and against the
|
||||
compute-in-RAM path computed at host RAM bandwidth (256 GB/s LPDDR5X on this fleet's
|
||||
Strix Halo). If the link is >10× slower than the memory behind it (x4 is 31× here),
|
||||
streaming loses by construction and no tuning rescues it; doubling the link (x4 → x8)
|
||||
only doubles a losing number.
|
||||
|
||||
## Worked example (Dankclaw 2026-09-16)
|
||||
|
||||
Gen4 x4, measured H2D 6.72 / D2H 6.60 GB/s (85 % of spec), 3 MiB slab = 474 µs.
|
||||
A3B MoE, Q4_K_M (75 GiB experts, ~1.70 GiB active/token): 272 ms/token ⇒ **3.7 tok/s**
|
||||
(4.6 with 20 % resident). Same host, same model, measured: **20.46 tok/s** with the
|
||||
experts computed on the CPU, 25 (halogen), 26.5 (mainline on the iGPU). Verdict: the
|
||||
streaming design loses 3-5×, and the engine in question could not load the GGUF anyway.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Stop production services before running this (fleet policy) — and remember the 3090's
|
||||
VRAM may be almost fully occupied, so a 2 GiB test buffer needs the service stopped.
|
||||
- Don't quote a derived line from inside the program without checking it: a printf
|
||||
format/argument mismatch in this very probe printed a nonsense ceiling. Recompute by hand.
|
||||
- GiB vs GB: convert before dividing (1.70 GiB = 1.825 GB).
|
||||
- The idle `LnkSta` (2.5 GT/s, downgraded) is not the loaded link — quoting it makes a
|
||||
working x4 look like a broken x1.
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: pcie-link-degradation-vs-idle-triage
|
||||
description: "Decide whether a GPU/NVMe PCIe link is genuinely degraded (bad riser, bad slot, wrong connector) or just idle-downshifted, and whether moving the card to another slot could widen it — using kernel boot bandwidth lines, port LnkCap vs LnkSta, and NVML sampled under load."
|
||||
---
|
||||
|
||||
# PCIe link: degraded or just idle?
|
||||
|
||||
Answer "is my GPU running at full speed?" without fooling yourself. The trap: an
|
||||
idle NVIDIA GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` because the driver
|
||||
downshifts the link to Gen1 for power. That is not a fault, and it is not the
|
||||
slot's fault either.
|
||||
|
||||
## Step 1 — the path ceiling, from the kernel (cheapest, most decisive)
|
||||
|
||||
```bash
|
||||
sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"
|
||||
```
|
||||
|
||||
Reads out per endpoint, e.g.:
|
||||
|
||||
```
|
||||
pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s
|
||||
PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
|
||||
```
|
||||
|
||||
- `63.012 Gb/s` = Gen4 **x4** (~7.88 GB/s/direction after 128b/130b encoding).
|
||||
- "limited by ... link at <BDF>" names the **bottleneck device** (usually a root
|
||||
port) and its width — this is the number that decides whether a swap can help.
|
||||
- "capable of ... x16 link" is the **endpoint's own** capability, not the path's.
|
||||
A big gap between the two is the whole story.
|
||||
|
||||
`dmesg` is often rotated; `journalctl -k -b` keeps this boot.
|
||||
|
||||
## Step 2 — the two sides separately
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <endpoint> | grep -E "LnkCap:|LnkSta:|LnkCtl2:|LnkCap2:"
|
||||
sudo lspci -vv -s <root port> | grep -E "LnkCap:|LnkSta:|LnkCtl2:"
|
||||
```
|
||||
|
||||
- `LnkSta` on a *bridge* is the trained link; on an endpoint it is that
|
||||
endpoint's view. Widths/`(downgraded)` compare against the **own** `LnkCap`.
|
||||
- `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform *intends*
|
||||
Gen4, so a Gen1 current speed is power management, not misconfiguration.
|
||||
- A root port whose `LnkCap` width exceeds the trained width means the socket or
|
||||
adapter wires fewer lanes than the controller can drive. **M.2 connectors are
|
||||
4 lanes by pinout** — an M.2 riser can never deliver more, regardless of what
|
||||
the bridge advertises.
|
||||
|
||||
## Step 3 — prove the link under load (the step people skip)
|
||||
|
||||
Never conclude from an idle reading on NVIDIA. Sample NVML while the GPU works:
|
||||
|
||||
```bash
|
||||
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,\
|
||||
pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv
|
||||
```
|
||||
|
||||
Fire a small, legitimate request at whatever already serves on that GPU (e.g. a
|
||||
`max_tokens` completion against the local LLM endpoint) and poll the query while
|
||||
utilization is high:
|
||||
|
||||
- `gen.current` climbing 1 → `hostmax` = idle downshift, link healthy.
|
||||
- `gen.current` stuck below `hostmax` under load = real degradation.
|
||||
- `width.current < width.max` under load = **physical lane limit** (wiring,
|
||||
riser, or connector) — no software touches this.
|
||||
- On a busy production GPU, this is a usage smoke test, not a benchmark: never
|
||||
stop/bench a live service to read its link.
|
||||
|
||||
## Step 4 — do not plan hardware from `dmidecode -t slot`
|
||||
|
||||
On consumer/mini-PC firmware this table is routinely fiction — it has been seen
|
||||
labelling an Ethernet port and an SD-reader controller as "M.2 Socket 3" slots
|
||||
and advertising a free slot in a **PCI domain that does not exist** (e.g.
|
||||
`0200:00:03.2`). Cross-check with the live bridge list:
|
||||
|
||||
```bash
|
||||
lspci -nn | grep "PCI bridge" # every root port, whether populated or empty
|
||||
lspci -t # topology; which endpoints hang off which port
|
||||
```
|
||||
|
||||
Empty root ports still enumerate, so a genuinely absent port is genuinely
|
||||
absent. Trust live PCI, not SMBIOS.
|
||||
|
||||
## Step 5 — verdict on "would moving the card to another slot help?"
|
||||
|
||||
- Compare the **bottleneck device named in step 1** with the target slot's port
|
||||
`LnkCap` width. If the bottleneck is the *endpoint side* (riser/adapter) and
|
||||
the target port is wider, a move can help.
|
||||
- If both candidate connectors cap at the same width (two M.2 sockets = both
|
||||
x4), moving is a **no-op** — say so plainly instead of endorsing the swap.
|
||||
- Error counters are a useful tiebreaker: clean `DevSta`/`CESta` with a low
|
||||
trained speed means "wired narrow", not "failing".
|
||||
|
||||
## Reporting
|
||||
|
||||
Give the ceiling in Gb/s, the card's own capability, what the idle reading
|
||||
meant, and whether any physical change can move the number. If the answer is
|
||||
"the link is already at the hardware maximum", state that and quantify whether
|
||||
the workload actually cares (resident weights = PCIe is nearly idle per token;
|
||||
CPU layer offload = it hurts).
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: published-port-cutover-gate
|
||||
description: "When replacing or restarting a service that publishes a TCP port on this rootless-podman/pasta fleet, gate on the port actually being free first — otherwise pasta's bind fails at container create and the failure hides behind Restart=on-failure."
|
||||
---
|
||||
|
||||
# Published-port cutover gate (rootless podman / pasta)
|
||||
|
||||
On hosts where **rootless podman publishes ports through pasta**, the published
|
||||
port is bound in userspace at **container CREATE**, not when the app binds. A
|
||||
predecessor process that has just been stopped can still hold that port —
|
||||
listener, accepted connection, or TIME_WAIT residue from the connections it
|
||||
served — so a replacement started immediately after `systemctl stop` dies with:
|
||||
|
||||
```
|
||||
pasta: ... bind: Address already in use
|
||||
conmon ... exit status 126
|
||||
```
|
||||
|
||||
and because the unit is usually `Restart=on-failure`/`RestartSec=10`, it
|
||||
**self-heals seconds-to-minutes later**. That is the trap: the deploy looks
|
||||
like it failed, then looks fine, and the real defect (no port gate) stays
|
||||
invisible because the retry masks it.
|
||||
|
||||
Observed on Dankclaw 2026-09-16 cutting `frank-large` from a root unit
|
||||
(ik_llama.cpp) to a rootless Quadlet (halogen): `systemd` logged
|
||||
"Deactivated successfully", and pasta tried to bind 7770 about a second later
|
||||
and reported "Address already in use". Only the quadlet's own restart policy
|
||||
recovered it.
|
||||
|
||||
## Procedure
|
||||
|
||||
Before starting the replacement:
|
||||
|
||||
1. **Retire the old unit and wait for it to be genuinely inactive.**
|
||||
`systemctl stop <old>` (root) or `systemctl --user stop <old>`.
|
||||
2. **Stop the NEW unit too, if this is a redeploy.**
|
||||
`systemctl --user stop <new>.service || true`
|
||||
|
||||
Without this the gate deadlocks against itself: the currently-running
|
||||
replacement is the thing holding the port. This mistake was made and caught
|
||||
in the same session — a gate placed before the stop would have failed every
|
||||
subsequent deploy of an already-running service.
|
||||
3. **Gate on the port being empty, covering every socket state:**
|
||||
|
||||
```bash
|
||||
wait_port_free() {
|
||||
local port="$1" limit="${2:-90}" i
|
||||
for ((i = 1; i <= limit; i++)); do
|
||||
if [[ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]]; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "FAIL: port $port is still held after ${limit}s:" >&2
|
||||
ss -tan "sport = :$port" >&2 || true
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
`sport` matches both the listener and the server side of accepted
|
||||
connections, so this also catches TIME_WAIT residue that a plain
|
||||
`ss -ltn` check misses. Fail loudly with the socket table attached; a deploy
|
||||
that starts a doomed container has not deployed anything.
|
||||
4. **Then** install/pull and start, and wait on the service's health endpoint.
|
||||
|
||||
## Verifying the gate has teeth
|
||||
|
||||
- Re-run the same deploy: it must pass with the service already running (that
|
||||
is the idempotency case the self-deadlock broke).
|
||||
- Reproduce the failure once on purpose if the service is not user-facing:
|
||||
stop the container and start the replacement with the gate bypassed, and
|
||||
confirm `Address already in use` / exit 126.
|
||||
- Assert it in the repo's seam tests: e.g. `grep -q 'wait_port_free <port>'`
|
||||
and `grep -q '^wait_port_free() {'` in the deploy handler, so a later
|
||||
refactor cannot quietly drop the gate.
|
||||
|
||||
## Related traps
|
||||
|
||||
- `systemctl disable` on some of these hosts **deletes the unit file** rather
|
||||
than just the symlink, so a "move the unit to trash" step must tolerate a
|
||||
file that is already gone — and should say so, or "no message" carries no
|
||||
evidence.
|
||||
- Never `rm`; move retired units into trash (`$HOME/.AGENTTRASH/<date>-<topic>/`).
|
||||
- A rootless Quadlet's generated unit **cannot** be `systemctl --user enable`d;
|
||||
boot persistence comes from `[Install] WantedBy=default.target` in the
|
||||
`.container`. To keep a masked/failed quadlet from being regenerated, move
|
||||
the `.container` out of `~/.config/containers/systemd/` and `daemon-reload` —
|
||||
the generator's source is the lever, not `mask`.
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: rccl-pcie-atomics-collective-failure
|
||||
description: "Diagnose AMD multi-GPU RCCL/collective launch failures (hipErrorIllegalState, hostcall) by checking PCIe atomic routing on every upstream bridge, decide whether any software path exists, and rank the fix (CPU-attached M.2 riser vs x8/x8 board vs layer split); includes the bare 2-rank probe matrix and the stop-and-mask safety rule."
|
||||
---
|
||||
|
||||
# RCCL / PCIe atomics collective failure
|
||||
|
||||
Use when a multi-GPU AMD host fails to run tensor/pipeline parallelism, or when
|
||||
asked "can these GPUs work together?".
|
||||
|
||||
## 1. Identify the cause from the runtime, not the framework
|
||||
|
||||
The failure surfaces as `hipErrorIllegalState` from `hipLaunchKernel` for
|
||||
`ncclDevKernel_*`. Nothing in vLLM/RCCL/PyTorch names atomics. Get the real
|
||||
message with:
|
||||
|
||||
```bash
|
||||
AMD_LOG_LEVEL=3 <the failing process>
|
||||
# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
|
||||
```
|
||||
|
||||
## 2. Check the whole chain, not just the card
|
||||
|
||||
```bash
|
||||
sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap" # want ReqEn+
|
||||
lspci -tv # every bridge upstream
|
||||
sudo lspci -vv -s <bridge> | grep AtomicOpsCap # want Routing+
|
||||
```
|
||||
|
||||
A `Routing-` bridge is a **read-only hardware capability** — no software can
|
||||
enable it. `setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) does nothing
|
||||
because the value is consumed at driver probe time. Typical AM5 layout: one x16
|
||||
slot on CPU root ports (atomics fine), the rest behind the chipset (usually not).
|
||||
|
||||
The gate in source, for the record:
|
||||
|
||||
```cpp
|
||||
void Device::checkAtomicSupport() { // rocclr/.../rocdevice.cpp:181
|
||||
link_attrs.push_back({LinkAttribute::kLinkAtomicSupport, 0});
|
||||
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs))
|
||||
if (link_attrs[0].second == 1) info_.pcie_atomics_ = true; // no else
|
||||
}
|
||||
```
|
||||
|
||||
Consumed at `rocvirtual.cpp` `case HiddenHostcallBuffer:` — the kernel *declares*
|
||||
a hostcall buffer and ROCr refuses to fill it without atomics. **No env var,
|
||||
module param, or sysfs toggle exists** (grep `getenv|HSA_|ROCR_|AMD_` around the
|
||||
gate). Do not promise a software override without checking this.
|
||||
|
||||
## 3. Run the probe matrix — one probe answers several design questions
|
||||
|
||||
Bare 2-rank `torch.distributed` (no vLLM, no model), one process per GPU,
|
||||
`init_process_group("nccl", timeout=timedelta(seconds=25))`:
|
||||
|
||||
| Op | CPU-lane card | chipset card | What it decides |
|
||||
|---|---|---|---|
|
||||
| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
|
||||
| `send`/`recv` (P2P) | hang | FAIL | **PP is dead too** — vLLM's PP handoff is this primitive |
|
||||
|
||||
Test `send`/`recv` **in isolation** (a prior failed op leaves the group broken
|
||||
and the result meaningless — it just hangs until the timeout). Set an explicit
|
||||
PG timeout so a blocked pair raises instead of hanging.
|
||||
|
||||
**SAFETY — non-negotiable.** Never run this on a host that is serving:
|
||||
`sudo systemctl stop <svc> && sudo systemctl mask <svc>` first, or use a host
|
||||
that serves nobody. A probe run against a live GPU service preceded a hard host
|
||||
down (no ARP, no ICMP from two subnets, power cycle required). See the
|
||||
benchmarking policy: it covers ANY GPU experiment, not just benchmarks.
|
||||
|
||||
Telling "off" from "wedged" afterwards: `ip neigh show <ip>` from a peer —
|
||||
`FAILED`/`INCOMPLETE` = off or NIC dead; a merely wedged kernel still answers ARP.
|
||||
|
||||
## 4. Rank the fixes
|
||||
|
||||
1. **Move the card to a CPU-attached M.2 slot** (M.2→PCIe x4 riser). Cheap and
|
||||
reversible. Get the board's *own spec page*: it lists which M.2 slots come
|
||||
from the CPU vs the chipset, and the chipset x16 slot's real width (often
|
||||
x1 mode). M.2 slots can be empty even when the x16 slots are full.
|
||||
2. **Board whose CPU x16 bifurcates x8/x8** — both cards on the CPU root port.
|
||||
3. **llama.cpp / ik_llama.cpp `--split-mode layer`** — no collectives at all, so
|
||||
it sidesteps the wall entirely; costs the GGUF conversion plus paged
|
||||
attention, continuous batching, and spec decode.
|
||||
4. Two TP=1 replicas + a routing proxy — what you do when you cannot fix the
|
||||
topology; keeps throughput but the cards never share one model.
|
||||
|
||||
**Verify before benchmarking anything:** `AtomicOpsCtl: ReqEn+` on the moved
|
||||
card. X4 is usually enough for TP (≈10 MB per token-step for a 27B at batch 4
|
||||
≈1.4 ms at Gen4 x4) — compute-bound, not interconnect-bound.
|
||||
|
||||
## 5. Closed software routes (don't re-litigate)
|
||||
|
||||
- vLLM's IPC custom allreduce is gated by a **CUDA-sm-version-keyed table**
|
||||
(`{9.0, 10.0, 10.3, 10.7}`); ROCm reports `12.0` on gfx1201 → `False`. The ops
|
||||
are compiled, the gate is not. Patching the table is still not enough —
|
||||
`broadcast`/`barrier`/TP-logits `all_gather` have no non-RCCL path.
|
||||
- A fork that wraps `.all_reduce` (e.g. radiance's `RADIANCE_USE_R4D_AR` one-shot
|
||||
P2P allreduce, gloo for IPC-handle exchange) does not intercept
|
||||
`PyNcclCommunicator.__init__`'s warmup allreduce or `pynccl_comm.all_gather`.
|
||||
- NCCL symmetric memory: needs an RCCL exposing `ncclDevComm*`; many builds ship
|
||||
the NCCL version string without the device API.
|
||||
- P/D disaggregation: needs a KV connector *and* coordination collectives.
|
||||
- gloo: CPU/TCP, unreachable for per-layer TP.
|
||||
|
||||
**Open, unverified**: recompiling RCCL so its kernels stop declaring a hostcall
|
||||
buffer (`-mprintf-kind=buffered` is the usual claim). Counter-evidence: RCCL's
|
||||
device source has no live `printf`/`assert`/`__assertfail`, so the trigger is
|
||||
unexplained. Test: rebuild, then dump a kernel's declared args
|
||||
(`roc-obj-extract` + `llvm-readelf --notes`) and look for
|
||||
`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: spec-decode-ab-single-gpu-host
|
||||
description: "A/B a speculative-decoding variant (or any serve-flag challenger) against an incumbent LLM endpoint on a single-GPU host that serves users: stop the unit, run the challenger from a patched env file, prove losslessness by identical greedy text, and keep the incumbent unless the challenger wins."
|
||||
---
|
||||
|
||||
# A/B a serve-flag challenger against a live LLM endpoint
|
||||
|
||||
Use when asked "try it without X / with Y" for an inference endpoint that is
|
||||
serving users from a GPU you cannot share (one model copy per host). The
|
||||
deliverable is a measurement plus a decision, and the endpoint must end the
|
||||
experiment on the configuration you started it with.
|
||||
|
||||
Ready-made harness for llama.cpp endpoints: `scripts/bench-spec-ab.py`
|
||||
(`<env-file> <serve-script> [--base-url]`) — it stops the unit, runs each
|
||||
config, compares greedy text and restores the unit. Below is what it encodes
|
||||
and the traps it hit.
|
||||
|
||||
## Protocol
|
||||
|
||||
1. **Fixed measurement shapes, one prompt per phase.** A large prompt with a
|
||||
tiny budget isolates prefill; a small prompt with a long budget isolates
|
||||
decode. `temperature 0` throughout. Never compare across sessions — run the
|
||||
*incumbent* again in the same session, on the same box, or you are comparing
|
||||
two machines' worth of noise.
|
||||
2. **Prove losslessness, do not claim it.** Send one fixed prompt at temp 0 to
|
||||
every config and diff the greedy text. Speculation is only allowed to buy
|
||||
speed; if the texts differ, stop and treat it as a correctness bug.
|
||||
3. **Stop the unit for the duration** (benchmarking policy on these hosts: a
|
||||
bench beside a live service starves the GPU). `sudo systemctl stop <unit> &&
|
||||
passive`, run the challenger by hand with `setsid`, then `sudo systemctl start
|
||||
<unit>` at the end *whatever happens* — wrap the restore so an exception
|
||||
cannot leave the fleet dark.
|
||||
4. **Override tunables through a patched env file**, never through the shell:
|
||||
serve scripts that `source` their env file with `set -a` win over environment
|
||||
variables, so `CTX=... bash serve.sh` silently does nothing. `sed` the repo env
|
||||
into `/tmp/challenger.env` and point the service's env-file variable at it.
|
||||
5. **Read the acceptance counters, not the flags.** `/metrics`
|
||||
(`llamacpp:spec_decode_num_drafts_total` vs `..._accepted_tokens_total`) or the
|
||||
per-request `timings.draft_n` / `draft_n_accepted`. A draft rate of 0 while
|
||||
tokens are produced means the drafter is not running even though the flags look
|
||||
right.
|
||||
6. **Sweep the depth, not just the mode.** Draft depth is model-specific and
|
||||
single-valued: measure at least two (e.g. n_max 3 and 7). Deeper often loses —
|
||||
at n=7 a third of proposals survived, at n=3 more than half.
|
||||
7. **Watch per-card VRAM** (`rocm-smi --showmeminfo vram`). A proposer is not
|
||||
free: a drafter file cost 2.06 GB on its card while loading the target's own
|
||||
in-model MTP head cost ~3.3 GB.
|
||||
8. **Decide and land it.** If the challenger loses, say so and change nothing;
|
||||
if it wins, put the value in the repo env file, run the suite, deploy, and
|
||||
verify with a real completion. Record the losing numbers in the env file with
|
||||
"do not switch without re-measuring" so nobody re-litigates it.
|
||||
|
||||
## Traps that cost time here
|
||||
|
||||
- **`pgrep -f <script>` and `pkill -f <pattern>` match your own command line**
|
||||
(and the ssh wrapper's). Use `pgrep -x llama-server` / `pkill -x` and confirm
|
||||
with `ps -eo pid,args | grep "[s]cript"`.
|
||||
- **PID 1 is not the app** inside a container; for units, `systemctl restart`
|
||||
is the reload.
|
||||
- **A retired predecessor's port lingers**: a published-port proxy leaves the
|
||||
endpoint port in TIME_WAIT. Wait for the port to be free (and for the manual
|
||||
instance to be killed) before starting the challenger, or the bind fails with
|
||||
address-in-use long after the unit stopped.
|
||||
- **Nested f-strings with backslash escapes are a SyntaxError on the fleet's
|
||||
Python** (<3.12 semantics). Build log lines with string concatenation.
|
||||
- **If a config dies at load, read the error literally before "fixing" it.**
|
||||
`context type MTP requested but model doesn't contain MTP layers` was not a
|
||||
bad flag: a chain has ONE draft context, so MTP met the *drafter's* file. Two
|
||||
proposers that each own the draft slot cannot be combined, and pinning that
|
||||
in a seam prevents someone "fixing" the flag later.
|
||||
|
||||
## Reporting
|
||||
|
||||
One table: config, decode tok/s, acceptance, VRAM per card. Then one line per
|
||||
finding, each with its number. Then the decision, plus whether the deployed
|
||||
state changed. "The thing you asked me to try lost to what was already running"
|
||||
is a complete and useful answer.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: spec-decode-worth-it-triage
|
||||
description: "Decide whether speculative decoding is worth deploying for a target/drafter pair: compute break-even acceptance from measured step times, classify the drafter by per-round cost, and check the real attachment surfaces (closed engines vs llama.cpp --spec-draft-device)."
|
||||
---
|
||||
|
||||
# Speculative decoding: is it worth deploying?
|
||||
|
||||
Use when someone proposes a drafter for an existing endpoint ("use the 3090 as a drafter", "hook a DFlash drafter to X", "wouldn't speculation make Y faster?"). Answer with arithmetic against the INCUMBENT, not against zero.
|
||||
|
||||
## 0. Check whether the target already speculates
|
||||
|
||||
Many engines ship a drafter of their own; if it is on, your baseline is `A_incumbent / t_s`, not `1 / t_s`.
|
||||
|
||||
- halogen-flash-server (Dankclaw frank-large): MTP head **on by default**. Read it from its own accounting, e.g. `serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
|
||||
- llama.cpp: `/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`; per-request lines say `draft acceptance = 0.76 (125 accepted / 165 generated)`.
|
||||
|
||||
## 1. Measure the two step times (never guess)
|
||||
|
||||
- target step `t_s` = 1 / (client-visible tok/s ÷ commit-per-round). Ask the operator for the tok/s they actually SEE: engine-internal decode-only numbers (short generations, prefill and delivery excluded) run ~1.7–2× optimistic.
|
||||
- drafter step `d_s`: llama.cpp logs `eval time = 2117.32 ms / 178 tokens` per task; that is the drafter's per-token price when it is a plain model on the same box.
|
||||
|
||||
## 2. Arithmetic
|
||||
|
||||
```
|
||||
round_time = K * d_s + 1.1 * t_s # K draft tokens proposed, K+1 verified in ONE target pass
|
||||
A = (1 - p^(K+1)) / (1 - p) # expected accepted tokens, p = per-position acceptance
|
||||
rate = A / round_time # compare to the incumbent
|
||||
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
|
||||
```
|
||||
|
||||
Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67 commit/round), `d_s = 12 ms` (27B IQ3_S on a 3090) → break-even `p ≈ 0.70` at K=3, `≈0.68` at K=5; realistic cross-finetune `p = 0.4–0.6` gives 17–20 tok/s, i.e. a LOSS. Drafters that are only ~3× faster per token cannot pay for K sequential forwards.
|
||||
|
||||
## 3. Classify the drafter by per-round cost (this decides more than acceptance)
|
||||
|
||||
| drafter | cost per round | needs |
|
||||
|---|---|---|
|
||||
| plain model via `-md`/`draft-simple` | **K sequential forwards** | 10–50× faster than the target |
|
||||
| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
|
||||
| trained multi-token head (`draft-mtp`, MTP/EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
|
||||
|
||||
Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden states, so a drafter built for one model cannot be re-pointed at another (a dense-27B DFlash2 is useless for a 512-expert MoE trunk).
|
||||
|
||||
## 4. Acceptance reality checks
|
||||
|
||||
- Fleet datum: a TRAINED z-lab DFlash2 at depth 7 accepted ~41% on its matched target (`draft-dflash 7 → 43.5 tok/s, 41% accepted`). Treat 40–60% as the honest band for a matched pair; <85% rarely pays unless the drafter is nearly free.
|
||||
- Cheapest proxy for `p` at temp 0, when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights moved.
|
||||
- llama.cpp can compute the projection without any draft artifact: `--spec-synth-rates P0,P1,...` / `--spec-synth-len L`.
|
||||
- Speculation pays IN PROPORTION TO HOW SLOW THE TARGET IS. The same 41% gave 2.3× on an 18.8 tok/s layer-split target and a loss on a 25 tok/s one. Fast targets are usually sparse MoE (few active experts → little per-token weight traffic) — the worst case for drafting.
|
||||
|
||||
## 5. Check the attachment surface BEFORE the arithmetic
|
||||
|
||||
- **Closed engines refuse external drafters by design.** halogen-flash-server: `DRAFTERS = {serial:0, mtp:1, dflash2:2}`, `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint, and an unknown/unavailable name is a deliberate 400 (a silent downgrade would make comparisons lie). "Drafter" is a field in its internal loopback token protocol (`drafter`, `spec_verify`, `spec_commit`) — there is no device routing, so a drafter can never sit on another card. Probe it live: POST `"drafter":"dflash2"` and read the error.
|
||||
- **pure llama.cpp DOES route the draft**: `--spec-draft-device/-devd <devs>` (default: follows `--device`), plus `-ngld/--gpu-layers-draft`, `-otd`, `-ctkd/-ctvd`, `-td/-tbd`, `-ncmoed`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`. In production on the fleet as `SPEC_DRAFT_DEVICE=ROCm1` with a layer-split target.
|
||||
- **Devices**: one llama.cpp process sees devices only from backends it was built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot coexist); a CUDA-only build has no `Vulkan0` to draft with.
|
||||
- **Chaining is not universal**: `--spec-type` takes a comma list, but `draft-mtp,draft-dflash` was measured to DIE AT LOAD while `draft-dflash,ngram-mod` is structurally allowed. Test at load, never assume composition.
|
||||
|
||||
## 6. Cross-model pairs need a byte-identical tokenizer
|
||||
|
||||
Speculation compares token IDs, so the draft and target tokenizers must map id → the same string. Verify cheaply from the GGUF headers (no full download; a range read of the first ~8 MB over HTTP is enough for a remote file): parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and a sha256 over the first 2000 tokens.
|
||||
|
||||
```
|
||||
vocab=248320 merges=247587 model=gpt2 pre=qwen35
|
||||
sha256(first 2000 tokens)=8619c68fe4466d3a # identical => ids are interchangeable
|
||||
```
|
||||
|
||||
Identity is necessary, NOT sufficient — the drafter still has to be cheap and accepted.
|
||||
|
||||
## 7. Verdict shape
|
||||
|
||||
Report, in this order: (a) does the target already draft, and at what rate; (b) break-even `p` at K=1,3,7 versus the incumbent; (c) the realistic `p` band and the resulting rate; (d) whether the drafter even fits (VRAM) and whether the attachment surface exists; (e) the one measurement that would settle it, and what you expect it to show.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: stale-compose-container-quadlet-migration
|
||||
description: "Diagnose a container that crash-loops after a reboot because a stale, pre-config-change compose container keeps being restarted, confirm it with the container's real CreateCommand, and migrate the service to a repo-tracked Quadlet unit."
|
||||
---
|
||||
|
||||
# Diagnose a crash-looping container after a reboot: stale compose container
|
||||
|
||||
## When to use
|
||||
|
||||
A service that worked before a reboot now crash-loops (or is `Exited`), its
|
||||
config file looks correct, and the failure message blames something the config
|
||||
does declare — e.g. `RuntimeError: No HIP GPUs are available` for a container
|
||||
whose compose service passes `/dev/kfd`.
|
||||
|
||||
## Root-cause pattern
|
||||
|
||||
The running container was created by an EARLIER revision of the service
|
||||
(before the missing lines were added) and `restart: unless-stopped` keeps
|
||||
resurrecting it:
|
||||
|
||||
- `podman compose up -d` does not recreate an existing container because its
|
||||
*declaration* changed — it only reconciles what compose tracks.
|
||||
- So the container keeps its original flags forever, and every restart mode
|
||||
(`unless-stopped`, `always`, podman-compose unit, quadlet) faithfully keeps
|
||||
restarting a container that can never work.
|
||||
- Symptom shape: high `RestartCount`, exit code 0 or 1, a config that looks
|
||||
right, and a failure that contradicts the config.
|
||||
|
||||
## Confirm in three commands
|
||||
|
||||
```bash
|
||||
# 1. How long has it been looping, and what actually started it?
|
||||
podman inspect <c> --format 'restarts={{.RestartCount}} created={{.Created}} exit={{.State.ExitCode}} unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'
|
||||
|
||||
# 2. The flags it REALLY has — this is ground truth. Do NOT trust
|
||||
# {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}}: both report 0/[] for
|
||||
# containers that were started with --device and --group-add.
|
||||
podman inspect <c> --format '{{.Config.CreateCommand}}' | tr ' ' '\n' | grep -A1 -E 'device|group-add'
|
||||
|
||||
# 3. Behavioural truth inside it
|
||||
podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))'
|
||||
```
|
||||
|
||||
Compare the container's `created` timestamp with when the config gained the
|
||||
missing lines (`git log -p --follow <config>`). Created-before-the-edit is the
|
||||
signature.
|
||||
|
||||
## Fix
|
||||
|
||||
Recreating the compose container works, but on a repo-governed fleet the fix is
|
||||
to give the service a **repo-tracked, boot-persistent Quadlet unit** (every
|
||||
container should already be one), then retire the compose path:
|
||||
|
||||
1. `docker/quadlet/<name>.container` with the full run posture: image (use
|
||||
`Image=localhost/<tag>` for locally built images + `Pull=never`),
|
||||
`AddDevice=`, `GroupAdd=keep-groups`, `Volume=`, `PublishPort=127.0.0.1:...`
|
||||
for an unguarded service, `HealthCmd=`, `Restart=always`, and
|
||||
`[Install] WantedBy=default.target`.
|
||||
2. Healthcheck as a **mounted file**, not inline: the podman 5.x v1 generator
|
||||
splits `HealthCmd` on spaces into an argv array, so
|
||||
`python3 -c "..."` becomes a silent quoting puzzle.
|
||||
3. Delete the compose service (one owner per container name) and make the
|
||||
deployer build → `install_quadlet` → retire any legacy hand-written unit
|
||||
into `~/.AGENTTRASH/` → `podman rm -f <name>` (so the unit's container can
|
||||
take the name) → restart → `wait_for_health`.
|
||||
4. Wire the planner (`<target>/<svc>/**` → service) and the dispatcher
|
||||
allowlist (`<target>:<svc>`), or the deploy refuses the service.
|
||||
5. Update the seams to read the unit rather than compose for devices, command
|
||||
and groups — and resolve build inputs from the deployer's declaration, never
|
||||
from a `find | head -1` directory listing.
|
||||
|
||||
## Verify behaviourally, not structurally
|
||||
|
||||
`systemctl --user is-enabled` → `generated`, container `healthy`,
|
||||
`RestartCount` stops climbing, the port answers (`curl -o /dev/null -w
|
||||
'%{http_code}'`), and the in-container probe reports the device
|
||||
(`hip available: True | Radeon 8060S Graphics`).
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: stale-signal-and-hollow-gate-audit
|
||||
description: "Audit a repo for signals that cannot be correct — vacuous test assertions after a file is retired, permanently-red health probes, and prose claims invalidated by your own changes; includes the linter-validation-before-acting discipline and Obsidian link-resolution rules."
|
||||
---
|
||||
|
||||
# Stale-signal and hollow-gate audit
|
||||
|
||||
Use when you want assurance rather than a feature: "is anything in this repo
|
||||
quietly lying?" The productive question is **which signal here cannot possibly
|
||||
be correct?** Every pass on serverAppConfig (2026-09-16) found real defects this
|
||||
way — including one in the rule file agents read first.
|
||||
|
||||
## 1. Vacuous assertions (highest severity — the gate is already dead)
|
||||
|
||||
A seam that greps a file passes *by absence* once the file is retired, because
|
||||
`grep` exits 2 on a missing path and `!` inverts it.
|
||||
|
||||
```bash
|
||||
# Find assertions anchored on files that may not exist
|
||||
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
|
||||
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
|
||||
```
|
||||
|
||||
Fix shape — one place asserts the invariant, each local site keeps a live check:
|
||||
|
||||
```bats
|
||||
@test "no <host> compose file competes with the quadlet units" {
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
|
||||
}
|
||||
|
||||
# at the per-service site: still fails if the entry returns
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
|
||||
```
|
||||
|
||||
**Always teeth-test**: recreate the offending file (`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams, confirm they go RED, then move the file to trash. A re-armed check that was never seen failing is still unverified.
|
||||
|
||||
## 2. Permanently-red probes
|
||||
|
||||
For each monitored target ask: *can this probe succeed from the monitor's
|
||||
vantage point?* Failure classes seen: container-internal ports probed from
|
||||
another host (`192.168.0.235:3306`, `127.0.0.1:6379` on a published-port-only
|
||||
netavark host), and "probe" jobs that scrape `/` (HTML → `unsupported
|
||||
Content-Type`) or a `/metrics` that is disabled (Gitea 404).
|
||||
|
||||
- A card/job that can never be up must be **fixed or explicitly marked
|
||||
not-probed** (`wip: true` on homepage cards) — never left red.
|
||||
- The right probe for "is this HTTP service answering?" is the **blackbox
|
||||
exporter** (`/probe?target=…&module=…`), which also gives guarded vhosts a real
|
||||
uptime signal since Prometheus cannot solve an Anubis challenge.
|
||||
- Cheap check on a homepage board: read the deployed `status.json` and triage
|
||||
every `state != up` entry against reality (container running? port published?).
|
||||
|
||||
## 3. Prose claims invalidated by your own change
|
||||
|
||||
After retiring a file or adding a feature, grep for **every** reference, not
|
||||
just code:
|
||||
|
||||
```bash
|
||||
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
|
||||
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
|
||||
```
|
||||
|
||||
This is what catches the ones that bite: a rule file (AGENTS.md) told agents
|
||||
ComfyUI runs CPU-only long after it was restored with GPU access; a unit said
|
||||
"there are no alerting rules" hours after five were added; a docs page
|
||||
documented three scrape jobs as working when they could never report up.
|
||||
**Clean cutover includes prose.**
|
||||
|
||||
## 4. Validate the linter before acting on it
|
||||
|
||||
A checker's output is a hypothesis about another tool's semantics. Modelling
|
||||
Obsidian with naive text matching produced 45 "broken links" of which 3 were
|
||||
real — the rest were case-sensitivity (`[[qBittorrent]]` → `qbittorrent.md`),
|
||||
path-qualified resolution (`[[folder/Note]]` resolves by path suffix, not stem),
|
||||
and `%20` URL-encoding in markdown links. Fix the checker first; re-derive the
|
||||
report; only then edit documents. In this repo the result lives in
|
||||
`scripts/check-vault-links.py` with those three rules in its docstring, gated by
|
||||
the `no unresolved links in the docs vault` seam.
|
||||
|
||||
## Reporting shape
|
||||
|
||||
Report per finding: the surface, why the signal *could not* have been correct,
|
||||
the fix, and how you proved the fix has teeth (red/green evidence). Flag any
|
||||
regression you introduced yourself in the same breath — one was found this way
|
||||
(a tightened bind killed a reverse-proxied vhost while every gate stayed green,
|
||||
because the spec covering that hostname did not exist).
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
name: stale-signal-audit
|
||||
description: "Audit homelab/fleet surfaces (status boards, metrics targets, test gates, docs, rule files) for signals that are already wrong — impossible-to-pass probes, assertions that pass by absence, linters that produce false positives, and prose invalidated by your own change. Use after a change lands, when a gate looks green but you want to know what it actually proves, or when asked what is broken."
|
||||
---
|
||||
|
||||
# Stale-signal audit
|
||||
|
||||
Signals rot without failing. This is the battery that found real defects on 4 of 4
|
||||
passes in serverAppConfig; run it after any change and before trusting a green gate.
|
||||
|
||||
## The five checks
|
||||
|
||||
1. **Probe reachability.** For every monitor/probe/alert target, ask: *can this
|
||||
checker physically reach that endpoint?* A probe that cannot succeed reports a
|
||||
false outage forever, and an always-red signal teaches the operator to ignore
|
||||
the surface.
|
||||
- Homepage (`shared/homepage/services.json` + `homepage-status` on Dankclaw): the
|
||||
monitor sees Dankclaw's loopback, container names, and **published** LAN ports
|
||||
only. A card probing another host's container-internal port can never be up →
|
||||
mark `wip` with the reason and let the app's card carry the status.
|
||||
- Prometheus: a *scrape* cannot express "is this HTTP service answering?" —
|
||||
scraping an HTML page gives `unsupported Content-Type`; a `/metrics` that 404s
|
||||
stays down. Remedy: blackbox exporter in probe mode. Never "fix" a probe by
|
||||
publishing a database or adding a wildcard bind.
|
||||
|
||||
2. **Assertions that pass by absence.** Grep-based tests against a file that was
|
||||
deleted or retired pass vacuously: a missing path makes `grep` exit 2 and a
|
||||
negated `! grep` succeeds. Look for tests referencing paths that no longer exist
|
||||
(`git ls-files` the target, or just `[ -f ]` it).
|
||||
- Fix: assert the invariant directly ("no <file> competes with the unit"), keep a
|
||||
conditional live check for the file returning, and drop dead variables that
|
||||
only referenced it.
|
||||
- **Prove teeth**: recreate the file/condition and watch the test go red. A test
|
||||
you have never seen fail is not a test.
|
||||
|
||||
3. **Linter validity.** Before acting on any checker's output, validate the checker
|
||||
against the resolver/contract it models. Obsidian example: wikilinks are
|
||||
case-insensitive, `[[folder/Note]]` resolves by **path suffix** not stem, and
|
||||
`%20` in markdown links is URL-encoding — a naive checker reported 45 broken
|
||||
links when 2 were real. Same class: a `find … | head -1` selector that reads the
|
||||
wrong file; a truncated listing that makes you "fix" a file that exists.
|
||||
|
||||
4. **Gates that never run.** Playwright specs in `tests/e2e/` are matched by
|
||||
**number prefix** in `playwright.config.ts`; an unregistered spec silently never
|
||||
runs and the suite count stays the same. Verify by counting tests before/after
|
||||
(`tests/e2e_scaffold.bats` now fails on an unregistered spec).
|
||||
|
||||
5. **Prose and comments invalidated by your own change.** Clean cutover includes
|
||||
docs, unit docstrings, and the repo's rule file (`AGENTS.md`) — which is what
|
||||
every future agent reads first, so a stale claim there is the most expensive kind.
|
||||
Check: runbooks/rows naming a retired service, "unconfigured" claims after you
|
||||
configured something, upgrade instructions pointing at a deleted file, and
|
||||
lessons that the deployment has since contradicted.
|
||||
|
||||
## Order
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test gates,
|
||||
docs/rule files, dashboard panels).
|
||||
2. For each, ask what the signal *can* express, not what it says.
|
||||
3. Validate any checker before editing the things it flags.
|
||||
4. Fix at the root; never weaken an assertion to make it pass.
|
||||
5. Gate the class so it cannot return (a seam), then prove the seam fails on the
|
||||
defect.
|
||||
|
||||
## Verification
|
||||
|
||||
- Vault links: `./scripts/check-vault-links.py` (exit 1 on breaks).
|
||||
- Prometheus panels/rules: query each panel expression against the API — 0 series on
|
||||
a panel that should show data is a defect, 0 `ALERTS` when nothing fires is not.
|
||||
- Suite: `./tests/run.sh`; declares `ALL TESTS PASSED`.
|
||||
- A gate you changed: recreate the defect and confirm red.
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: status-surface-dead-signal-audit
|
||||
description: "Audit a Dank-fleet status surface (homepage cards, Prometheus targets, tests) for entries that can never be green because the producer cannot reach the target, classify them as outage vs config defect, and fix honestly (wip, repoint, blackbox) instead of silencing them."
|
||||
---
|
||||
|
||||
# Dead-signal audit on the Dank fleet's status surfaces
|
||||
|
||||
Use when a status board, Prometheus target list, alert set or test suite has
|
||||
permanently non-green entries, or before trusting one of those surfaces. The
|
||||
failure mode is always the same shape: a signal that **cannot** succeed by
|
||||
construction, which teaches the operator to ignore the surface it lives on.
|
||||
|
||||
## The surfaces, and what can actually be probed
|
||||
|
||||
| Surface | Producer's vantage point | Can reach |
|
||||
|---|---|---|
|
||||
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
||||
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, loopback of that host only |
|
||||
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
||||
|
||||
Anything else — another host's loopback, or a container-internal port that was
|
||||
never published — is unreachable no matter how healthy the service is.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **List the non-green entries.** Homepage: `python3 - boardcheck.py
|
||||
/srv/homepage/status.json` on Dankclaw (`status.json` holds one dict per card
|
||||
with `state` up/down/wip). Prometheus: query
|
||||
`/api/v1/targets?state=active` and print `health != "up"` with `lastError`.
|
||||
2. **Classify each one by producer reach**, not by the service. Ask: can the
|
||||
thing that probes it get there at all?
|
||||
3. **Fix in the direction of honesty, never of silence:**
|
||||
- unreachable target → mark the card `wip` with the reason in its
|
||||
`description` (the checker skips `wip`), or repoint the job at something
|
||||
reachable. Do **not** open a database port on the LAN to make a badge
|
||||
green, and do **not** delete a scrape job — an absent scrape and a healthy
|
||||
one are indistinguishable.
|
||||
- wrong path / wrong content type → fix the path (e.g. Gitea's `/metrics`
|
||||
needs metrics enabled), or move page-only targets to a blackbox exporter in
|
||||
probe mode.
|
||||
- failing test → fix the selector. A chronic red test is a defect with an
|
||||
owner, not a baseline (`find … | head -1` picked another service's file).
|
||||
4. **Verify on the surface itself**, not in the config: re-read `status.json`
|
||||
(expect 0 `down`), query `up == 0` targets, run the suite to green.
|
||||
5. Write the rule where the next person will read it — the Homepage and
|
||||
Observability pages both state it now.
|
||||
|
||||
## Traps
|
||||
|
||||
- The producer's vantage point is the whole question; a card or job can be
|
||||
"down" for a service that answers perfectly from the host it runs on.
|
||||
- `wip` is the existing convention for "real service, not probeable" (Gluetun,
|
||||
WireGuard/UDP), not a suppression: keep the reason in the description.
|
||||
- Alert rules must not depend on signals that are always red; scope them to
|
||||
the services that can actually be healthy (`LlmEndpointDown` is scoped to
|
||||
`frank-*` for exactly that reason).
|
||||
@@ -1,3 +1,4 @@
|
||||
- niri output config on blankpad (niri 26.04): transform values are COUNTER-CLOCKWISE, so "90" means the panel's top bezel must physically sit on the LEFT ("270" = top bezel right). Two traps when pinning outputs in ~/.config/niri/cfg/display.kdl (symlink into ~/.omp/dotfiles/zz-blankpad/): (1) an output "X" {} block RESETS every property you don't restate to its default — eDP-1 needed scale 1.25 repeated or it would silently drop to 1.0; (2) niri re-runs its auto-layout for any output without a pinned position, so pinning only the externals shoved eDP-1 to the far right (3936,0) — pin eDP-1 at x=0 too if you want the original left-to-right order. Apply with niri validate -c ~/.config/niri/config.kdl then niri msg action load-config-file; confirm with niri msg outputs (shows Logical size swapped to 1200x1920 and "Transform: 90° counter-clockwise"). Live one-off rotation without config: niri msg output NAME transform 90. _(context: Rotated both external PM1601 panels (DP-2, HDMI-A-1) to portrait on blankpad 2026-09-15; verified via niri msg outputs, committed 2baf0c0 to dank/command-center.)_
|
||||
- Frank's frank-large ceiling is now 200,000 context in ~/.omp/agent/models.yml (provider llamacpp (dankclaw), committed 60dc0bb to dank/command-center 2026-09-13), while halogen on 192.168.0.143:7770 still serves context=262144 — a deliberate under-serve, not drift. Convention when changing any contextWindow there: read the served value from curl baseUrl/health (it reports context, model, vision.enabled) and check KV-pool admission before/after — admission reserves prompt + max_tokens against the pool (401,408 positions since 2026-09-13), so contextWindow + maxTokens must stay under it or the request hangs the full 3600 s queue timeout before a 503. maxTokens stays pinned at 65536 because HALOGEN_MAX_TOKENS_CAP 400s anything larger. Also: OMP reads models.yml only at startup, so a running session keeps the old ceiling; tell the user new omp invocations pick up the change.
|
||||
- Frank's terminal font stack on Dankware: package ttf-terminus-nerd (Arch, group nerd-fonts), which provides the TERMINESS outline derivative — families "Terminess Nerd Font" / " Mono" / " Propo". The bitmap original terminus-font is NOT installed, so classic-Terminus crispness depends entirely on 1-bit hinted rendering. Crisp/bitmap look = freetype-load-flags = hinting,monochrome (monochrome = 1-bit, disables AA; the light flag stays in the resolved value but is inert once monochrome is on, per ghostty.5.md). Sizing rule that matters: pixel cell = pt × (96/72) × monitor_scale, and the crisp sweet spot is when that lands on Terminus' native bitmap heights (8x16, 9x18, 10x20, 11x22, 12x24). On Frank's eDP-1 (1920x1080 @ scale 1.25) font-size 12 → 20px → exactly the 10x20 grid; 14pt → 23.3px, fractional and blurry regardless of flags. Check scale with niri msg outputs. Also: ghostty +show-config prints only values differing from default, and Ghostty's default font-size IS 12 — so a 12pt setting vanishes from the diff output; confirm against +show-config --default before concluding the key was ignored.
|
||||
- NEVER send SIGUSR1 to ghostty on Frank's workstations — it does not reload the config, it kills/respawns the terminal (observed 2026-09-13: PID churn, Frank said "stop killing the terminal"). The old recipe in agent/memories/--home-blank-.omp--/MEMORY.md and skills/dotfiles_git/scripts/reload_ghostty.sh ("pgrep ghostty | while read pid; do kill -USR1 $pid; done") is WRONG for ghostty 1.3.1. Correct way to apply a changed ghostty theme/config without losing shells: press the default keybind ctrl+shift+, (action reload_config) in each window — confirmed as the default in ghostty +show-config --default. For applying theme changes desktop-wide without touching terminals: noctalia msg templates-apply regenerates rendered theme files, and niri msg action load-config-file reloads the compositor (reload-config is NOT a valid niri subcommand). Custom noctalia palettes live at ~/.config/noctalia/palettes/Name.json and are selected with noctalia msg color-scheme-set custom "Name"; that source is immune to the community-palette catalog/md5 sync, unlike editing the cache in ~/.local/state/noctalia/community-palettes/. Also: noctalia msg screenshot-fullscreen saves to ~/Pictures (no need to scrape the PNG out of the clipboard with wl-paste).
|
||||
|
||||
@@ -1,5 +1,92 @@
|
||||
# Raw Memories
|
||||
|
||||
## 01a0a7ff-dae2-71fd-9bd0-a8aa7c9d3676
|
||||
updated_at: 1789533306
|
||||
## llama.cpp Multi-GPU Parallelism (verified against docs/multi-gpu.md on master)
|
||||
|
||||
### Split modes (--split-mode / -sm)
|
||||
- **layer** (default): Pipeline parallelism. Each GPU holds a contiguous slice of layers. Token flows through GPUs sequentially. Minimizes inter-GPU data movement; tolerates slow interconnects (PCIe fine). Best for prefill and fitting large models. Weakness: needs many tokens to keep pipeline full, poor low-batch latency.
|
||||
- **tensor** (experimental): True tensor parallelism. Splits each layer's weights AND KV cache across all GPUs via a 'meta device', doing cross-GPU all-reduce per layer (NCCL). Minimizes latency, parallelizes any workload size. Heavily bottlenecked by interconnect speed (wants NVLink + NCCL). Best for fast token generation.
|
||||
- **row** (deprecated): Row-split dense weights only. Predecessor to tensor; comparatively poor performance.
|
||||
- **none**: Single GPU, selected via --main-gpu.
|
||||
|
||||
### Key constraints on tensor mode
|
||||
- Requires flash attention ON (-fa on)
|
||||
- KV cache must be non-quantized: f32/f16/bf16 only
|
||||
- NOT implemented for: MoE/hybrid, Mamba/SSM, and several other architectures (hard-fails with SPLIT_MODE_TENSOR not implemented)
|
||||
- --fit is disabled; must size --ctx-size / --parallel manually
|
||||
|
||||
### Separate axis (NOT model parallelism)
|
||||
- **--parallel / -np**: Number of concurrent server slots (each with own KV cache). Serving-level concurrency/batch parallelism. On single GPU this is the main throughput knob.
|
||||
- **CPU+GPU offload** (--n-gpu-layers, --n-cpu-moe): Hybrid inference when model exceeds VRAM. Device-distribution strategy, not 'parallelism' in the strict sense.
|
||||
|
||||
### What does NOT exist in llama.cpp
|
||||
- No first-class context/sequence parallelism mode (unlike vLLM/TensorRT-LLM). If you see that term applied to llama.cpp, it's loose third-party language.
|
||||
|
||||
### User hardware context
|
||||
- Single RTX 4070 Max-Q (~12 GB VRAM). Multi-GPU split modes (layer/tensor) do not apply. Relevant knobs: --parallel for concurrency, CPU+GPU offloading for models >12 GB.
|
||||
|
||||
## 01a0a817-7382-7525-92f1-14cd7fade072
|
||||
updated_at: 1789529008
|
||||
## Blankpad niri layout: externals left of eDP-1
|
||||
|
||||
Final committed layout (83c9c92, 2026-09-16):
|
||||
- DP-2: x=0, portrait 1200x1920, transform 90 CCW, scale 1
|
||||
- HDMI-A-1: x=1200, portrait 1200x1920, transform 270 CCW, scale 1
|
||||
- eDP-1: x=2400, 1536x864, scale 1.25 (restated per block-reset trap)
|
||||
|
||||
X-offset chain: 0 → 1200 (DP-2 portrait width) → 2400 (HDMI-A-1 portrait width) → eDP-1.
|
||||
|
||||
Config path: ~/.omp/dotfiles/zz-blankpad/niri/.config/niri/cfg/display.kdl
|
||||
Repo: dank/command-center (main), local Gitea at 192.168.0.143:3000.
|
||||
|
||||
The skill niri-output-rotation-pinning was followed correctly end-to-end. No new pitfalls discovered beyond what the skill already documents.
|
||||
|
||||
## 01a0a798-f61e-71d3-8fb6-55879410c3fc
|
||||
updated_at: 1789518997
|
||||
## Niri Display Rotation (Portrait 180° Flip)
|
||||
|
||||
### Hardware
|
||||
- eDP-1: Lenovo laptop panel, 1920x1080, scale 1.25, normal transform
|
||||
- DP-2: PNP(XYM) PM1601, 1920x1200, scale 1, portrait (was 270° CCW → now 90° CCW)
|
||||
- HDMI-A-1: PNP(XYM) PM1601 demoset-1, 1920x1200, scale 1, portrait (was 90° CCW → now 270° CCW)
|
||||
|
||||
### Key Technical Facts
|
||||
- Niri config split into `~/.config/niri/cfg/*.kdl` (Noctalia-style includes)
|
||||
- Display config: `~/.config/niri/cfg/display.kdl` — per-output `transform` field in KDL
|
||||
- `niri msg outputs` — shows current transform, logical size, scale, position
|
||||
- `niri validate -c <path>` — dry-run validation before applying
|
||||
- `niri msg action load-config-file` — hot-reload config at runtime (no restart needed)
|
||||
- After rotation in portrait, logical size swaps (1920x1200 → 1200x1920) but logical position X may shift; verify with `niri msg outputs` after reload
|
||||
|
||||
### Critical Pitfall: Portrait 180° Rotation
|
||||
- In portrait, "flip 180°" = swap 90° ↔ 270° transform values
|
||||
- Setting `transform "180"` on a portrait monitor kicks it to landscape-upside-down
|
||||
- Niri transform values are absolute: `normal`, `90° counter-clockwise`, `180°`, `270° counter-clockwise` — no modular arithmetic
|
||||
|
||||
### Workflow (Reusable)
|
||||
1. `niri msg outputs` — capture current state
|
||||
2. Read `~/.config/niri/cfg/display.kdl`
|
||||
3. Edit transform values (90↔270 swap for 180° portrait flip)
|
||||
4. `niri validate -c ~/.config/niri/config.kdl` — confirm valid
|
||||
5. `niri msg action load-config-file` — apply
|
||||
6. `niri msg outputs | grep -E '^Output|Transform|Logical|Scale'` — verify
|
||||
7. `git add && git commit && git push` in dotfiles repo (`/home/blank/.omp`)
|
||||
|
||||
## 01a09e49-b17b-74c3-83e9-2eb95ed25621
|
||||
updated_at: 1789368866
|
||||
Task: user requested changing Noctalia theme yellow to red and terminal text color to same red; rollout ended before applying edits after an ask about crispness scope was interrupted.
|
||||
|
||||
Noctalia facts: Niri-based DE at /usr/bin/noctalia. Main config ~/.config/noctalia/config.toml. Niri config includes noctalia.kdl for display settings. IPC uses direct `noctalia msg <command>` instead of qs wrapper; key commands panel-toggle, session, lock, launcher. App templates in /usr/share/noctalia/assets/templates/ for gtk, ghostty, alacritty, hyprland, sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango, umbriel, foot, emacs, kitty. Palette is generated from an image with `noctalia theme [image]`. Migration notes: Ghostty should use `theme = noctalia`; Alacritty can import noctalia.toml theme; GTK 3/4 accent palette default mentioned as #b267e4 purple; Niri keybinds should map to noctalia msg commands.
|
||||
|
||||
Theme-edit workflow: for changing a Noctalia accent such as yellow to red, prefer locating and editing the generated Noctalia palette/theme source before hardcoding per-app colors. If Ghostty is set to `theme = noctalia`, direct terminal color overrides may be less durable than updating the theme itself. Terminal text color should match the chosen red in both the Noctalia palette and any terminal-specific override if used.
|
||||
|
||||
Config discovery in this environment: expected ~/.config/noctalia/config.toml and ~/.config/ghostty/config were not successfully read due shaken/pruned tool output. Candidate locations included ~/.config/noctalia, ~/.config/ghostty, and /home/blank/.omp/dotfiles/noctalia/.local/share/noctalia under cwd /home/blank/.omp. Use ls/glob/read on specific files rather than assuming a single canonical path.
|
||||
|
||||
Niri crispness facts: in niri 26.04, true compositor AA is limited to corner rounding via niri_rounding_alpha() smoothstep and unused gradient/hue interpolation. Corner AA is driven by window-rule geometry-corner-radius with clip-to-geometry; setting radius 0 short-circuits the shader. Background blur/noise are not AA but cause softness: background-effect { blur true } plus top-level blur { passes 2, offset 3.0, noise 0.03 } creates frosted glass and film grain. Shadows were already off. Wayland clients rasterize text; niri does not alter glyph AA. Ghostty font AA is controlled by freetype-load-flags, while other apps use fontconfig or app-specific settings. No nearest-neighbor resample knob was found in this build.
|
||||
|
||||
Decision guidance: when asked to turn off all compositor AA for a crisp look, explain there is no single switch; offer scoped options: hard corners only; hard corners plus disable backdrop blur/noise; full maximum crispness with terminal opacity 1.0/blur 0 and global fontconfig antialias false; or terminal-only crispness. Warn that removing blur/noise changes Noctalia's glass aesthetic.
|
||||
|
||||
## 01a0993b-f83f-7331-9cad-fff3fc7f3070
|
||||
updated_at: 1789286338
|
||||
Target: agent/models.yml (frank-large entry). Sampler defaults overridden from halogen/greedy to temp=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0. Added preserve_thinking=true to retain thinking blocks across turns. Deployed to dank/command-center on dankclaw. Live effect triggers on next config read; no container restart required. Git LFS locking notice is advisory, does not block pushes. Repo path: ~/.omp.
|
||||
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a09e49-b17b-74c3-83e9-2eb95ed25621
|
||||
updated_at: 1789368866
|
||||
|
||||
Noctalia theme change request (yellow to red, terminal text same red) led to locating Noctalia/Niri config surfaces and crispness controls. No final edit was applied before abort. Durable: Noctalia is Niri-based; palette generated via noctalia theme image; Ghostty uses theme=noctalia; niri compositor AA only affects corner rounding, while blur/noise cause soft glass look and text AA is client-side.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0a798-f61e-71d3-8fb6-55879410c3fc
|
||||
updated_at: 1789518997
|
||||
|
||||
Rotated two external PNP(XYM) PM1601 portable monitors 180° in portrait via niri display config edit. Key insight: in portrait mode, a 180° flip is the 90°↔270° transform swap, not adding 180°. Workflow: niri msg outputs → edit cfg/display.kdl → niri validate → niri msg action load-config-file → verify → git commit/push.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0a7ff-dae2-71fd-9bd0-a8aa7c9d3676
|
||||
updated_at: 1789533306
|
||||
|
||||
llama.cpp parallelism types: two real model-parallelism modes (pipeline/layer split = default, tensor = experimental), plus deprecated 'row' and single-GPU 'none'. --parallel is a separate serving-level concurrency axis, not model splitting. No first-class context/sequence parallelism. Tensor mode requires flash attention, non-quantized KV, and excludes MoE/Mamba/SSM architectures.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0a817-7382-7525-92f1-14cd7fade072
|
||||
updated_at: 1789529008
|
||||
|
||||
Re-arranged blankpad niri display: moved two rotated PM1601 externals (DP-2, HDMI-A-1) from right of eDP-1 to left. New x-chain: DP-2@0 → HDMI-A-1@1200 → eDP-1@2400. Committed as 83c9c92 to dank/command-center.
|
||||
@@ -1,22 +1,34 @@
|
||||
# dankclaw Deployment & Model Services
|
||||
## System & Hardware Topology
|
||||
- **dankserver**: Dual AMD R9700 GPUs.
|
||||
- *Issue*: RCCL fails on chipset-attached PCIe (PCIEX16_2) because lanes lack required atomic-op routing.
|
||||
- *Fix*: Use a $30 M.2→PCIe x4 riser to move card #2 to CPU-attached M.2_2 slot.
|
||||
- **dankclaw**: RTX 3090 (PCIe 4.0 x4 dock) + AMD Strix Halo (gfx1151, Vulkan).
|
||||
- *Kernel*: Ubuntu 24.04, `amd_iommu=off`.
|
||||
|
||||
## frank-large (Quadlet Service)
|
||||
Deployed as a rootless Quadlet service (`frank-large.service`) on dankclaw. Hosts ~68 GiB weights and a 524,288-position KV pool (~14.4 GiB). HALOGEN_KV_SLOTS reduced to 3, stabilizing total memory at ~103.4 GiB within the 121 GiB budget. Admission is pool-bound, capping at 3 simultaneous full-length requests.
|
||||
## LLM & Software Stack
|
||||
- **vLLM / ROCm**: Version 0.27.1 / torch 2.11.0 / ROCm 7.14.
|
||||
- *Constraint*: HIP runtime reads bare `TEMP` and `HOST` env vars; naming these explicitly can cause collisions.
|
||||
- *Constraint*: `GGML_CUDA_FA_ALL_QUANTS` required for flash-attention with quantized KV.
|
||||
- **llama.cpp (ik_llama.cpp)**:
|
||||
- *Model*: Qwen3.8-Flash-Next-Uncensored-i1-GGUF (i1-Q4_K_M).
|
||||
- *Performance*: ~3.8-4.0 t/s decode (bottlenecked by PCIe x4 sync/graph splits), ~192 t/s prefill.
|
||||
- *Note*: 30 t/s decode is unreachable via current PCIe x4/split-mode setup; requires mainline PR #28901 for capture-safe graphs.
|
||||
|
||||
### Ports & Logging
|
||||
- Engine: :8730 (loopback), Front-end: :8731, Published: :7770
|
||||
- Logging: `podman logs frank-large` (full/filtered), `journalctl --user -u frank-large`
|
||||
- Env Vars: Avoid `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE` (HIP runtime reads bare `TEMP`).
|
||||
## Service Management (Quadlet & Podman)
|
||||
- **Persistence**: `restart: unless-stopped` survives crashes, not explicit stops. Every container must have a Quadlet `.container` file for boot-persistence.
|
||||
- **frank-large (HALOGEN)**: Rootless Quadlet. Weights pinned (~68 GiB). KV pool: 524288 positions (~14.4 GiB).
|
||||
- *Stall Behavior*: Multi-minute full-CPU stalls (no disk/output) are documented behavior under co-tenancy with `HALOGEN_FLASH_PIN_TRUNK=1`. Do not restart.
|
||||
- *First-touch*: 45-60s silence after restart as engine first-touches fresh KV pool.
|
||||
- *Constraints*: `maxTokens` <= 65536. Prompt + max_tokens must fit KV pool.
|
||||
- **frank-small**: Defaults to sampling (temp 1.0, top_k 20, top_p 0.95).
|
||||
|
||||
### Sampling & API Quirks
|
||||
- `frank-large` defaults to greedy; `frank-small` defaults to sampling (temp=1.0, top_k=20, top_p=0.95). Force small determinism via `temperature: 0` or `--sampling-defaults none`.
|
||||
- `preserve_thinking` is per-request (client-side `chat_template_kwargs`), not controlled by HALOGEN_* env vars.
|
||||
- `max_tokens: 200` with empty content indicates thinking consumed budget. frank-small returns 200 for unknown fields. File-cache shortage causes uninterruptible sleep, not OOM. `host_reserve_gib=24` protects n-gram table.
|
||||
- Tests: 285/286 pass (test 73 pre-existing comfyui selector bug). `pyyaml` missing on system python3; `uv` available.
|
||||
## Networking & Auth
|
||||
- **Marreta Auth Chain**: Caddy TLS → Anubis (:8923) → Authentik Outpost → Marreta (:8017).
|
||||
- *Fix*: Populated `invalidation_flow`, `authentication_flow`, and `authorization_flow` to prevent Rust outpost serde crashes.
|
||||
- *Probing*: Host-to-listener returns HTTP 000 (Caddy netns); probe inside netns or grep config for internal status.
|
||||
|
||||
## Marreta & Authentik Guard Chain
|
||||
Deployed via podman compose, sharing gluetun egress with searxng. Caddy TLS edge routes through Anubis (:8923) → Authentik outpost → internal :8017 → marreta upstream.
|
||||
- Auth Flow: Provider-first creation → route flip (vhost `reverse_proxy anubis:8923`) → deploy.
|
||||
- Resolved Login-Wall: Null `ProxyProvider.application` and missing auth/authz/invalidation flows crashed the Rust outpost. Fix: populated flows, linked `Application(slug=marreta)`, restarted services.
|
||||
- Pitfalls: Host probe (`localhost:8017`) returns HTTP 000 due to Caddy netns; probe internally or grep config. Non-browser clients fail JS challenges. Audit service-to-service traffic for backend URLs hitting guarded hostnames. IdP, Gitea (git smart-HTTP), and self-auth services are exempt.
|
||||
- Verification: E2E test (`tests/e2e/specs/16-marreta.spec.ts`) validates homepage tile routing.
|
||||
## Operational Lessons
|
||||
- **Memory**: `free` command lies about available memory (doesn't account for pinned/reserved weights).
|
||||
- **Skills Discovery**: `omp` scans `~/.omp/agent/skills/`.
|
||||
- **Root-Cause**: PCIe x4 dock provides ~29 GB/s, not 130 GB/s.
|
||||
- **Logs**: Use `podman logs` or `journalctl --user -u` for rootless Quadlets.
|
||||
|
||||
@@ -1,3 +1,75 @@
|
||||
- frank-large (halogen) healthcheck failure signature: the unit's healthcheck is halogen-healthcheck api with a 10 s timeout. During an engine PING-silence stall (log: "halogen: the engine has not answered PING for 45s"), /health HANGS rather than 503ing, so the healthcheck exits 125 (timeout) and podman shows the container (unhealthy) with a small FailingStreak. This is the documented transient stall — it self-recovers (streak resets to 0, /health 200) within ~1-2 min; do not restart over it. A completion request landing in the stall window takes ~60 s and then returns 200. A leftover failed transient *-132a6f9850c6fbd8.service (systemd-run podman healthcheck) may linger in the user manager — reset-failed it. _(context: 2026-09-17 11:28-11:30 PDT: frank-large 4x FailingStreak exit 125, self-recovered by 11:31; 65 s completion probe straddled the tail.)_
|
||||
- Fleet transport: blankpad has NO passwordless SSH to FrankBook-Pro — dank@192.168.0.112 rejects publickey (the 2026-09-12 key install only covered dankclaw/dankserver). Do not burn calls probing Mac SSH. Verify the *arr stack from the LAN instead: port probes at 192.168.0.112:{9696 prowlarr, 8989 sonarr, 7878 radarr, 8686 lidarr, 8787 readarr, 8080 qbittorrent} — 302 (login redirect) and 200 are both "up". Container-level inspection on the Mac needs a key installed on the Mac or a Mac-side session. _(context: 2026-09-17 fleet-wide container audit; ssh dank@192.168.0.112 → Permission denied (publickey), six-port LAN probe all answered.)_
|
||||
- Long-context speculative-decode tuning on llama.cpp: at 250k-token contexts, INCREASING draft depth (n-max 4) and busy-wait (poll 100) made decode WORSE (230–257 t/s) than depth 3 / poll 50 (257–302 t/s) — each longer verification round re-reads the full-context attention of the full-attention layers every round, which outweighs the extra accepted token. The productive prefill lever on a 24 GiB 3090 was ubatch 512-1024 (+5% pp at 254k, better tg too); ubatch 2048 is impossible because a spec draft context allocates its OWN pp compute buffer (~3.3 GiB at ub 2048 for a 12B model) against only ~2.2 GiB headroom AFTER the main context fits — the OOM signature is common_speculative_init_result: failed to create MTP context / graph_reserve: failed to allocate compute pp buffers AFTER the main model loaded fine. Also: idle VRAM is identical across logical batch sizes (4096 vs 8192) at the same ubatch — compute buffers reserve at max graph size, so a bigger logical -b is free. Verify with salted API probes (cache_prompt:false + uuid prefix) using the timings keys of modern builds: prompt_per_second / predicted_per_second / draft_n / draft_n_accepted. (frank-small, dankclaw 3090, 2026-09-17, deployed as 8192/1024, n-max 3, poll 50) _(context: Optimizing frank-small (Gemma4-12B QAT + MTP sidecar, 4x262144 slots, q8 KV, RTX 3090) for max prefill+decode at max context.)_
|
||||
- In the serverAppConfig checkout, concurrent peer sessions can stash YOUR uncommitted WIP and then DROP that stash (a peer did exactly this 2026-09-17 while deploying frank-small, dropping a stash that held the frank-medium env+seam changes). Dropped stash commits are dangling but reachable: git fsck --no-reflogs lists them, and the files recover cleanly with git checkout dangling-sha -- paths (restores content, no ref touched). Check git stash list AND git fsck for your files before re-doing work after a peer session has been active. Related standing rule for this repo: the deploy gate rejects a dirty tree, so a peer's uncommitted WIP blocks your deploy — stash their files (theirs, not yours), deploy, pop. _(context: serverAppConfig 2026-09-17: peer's frank-small deploy stashed then dropped my frank-medium WIP; recovered via dangling stash commit cd0a554.)_
|
||||
- llama.cpp draft-mtp spec-decode cost law (calibrated 2026-09-17 on frank-medium, 2x R9700 layer split): the draft context shares the main model's weights (shares_model — no sidecar file, no second model copy), so the cost is one nextn layer's KV at 4096 B/position × the TARGET's ctx (the draft context is sized from the target, same law as any drafter) PLUS draft buffers that a formula-based fit model underestimates badly: at 393216 ctx the MTP total charge on the last-layer card measured 4.20 GiB vs the fit's ~1.7 GiB — 2.46 GiB hot, which moved the card's headroom from a predicted 3.54 to a real 1.08 GiB and forced the floor down to 1.0 GiB. Calibrate the fit constant to the measured rocm-smi load, not the formula. The MTP/DFlash2 trade on this host: MTP frees ~1.1 GiB on the drafter card (no 1.92 GiB sidecar + buffers) at the cost of ~11 tok/s (32.4 vs 43.5 measured) — that freed space is what made 2x196k contexts fit. _(context: frank-medium 2026-09-17: switched DFlash2 - MTP drafter to fit a 393216 pool (2x196608); ROCm1 loaded 30.78 GiB vs fit 28.32; seam constant recalibrated to 2,901,231,913 B.)_
|
||||
- llama.cpp --fit with an assistant-style MTP sidecar (Gemma4: mtp-gemma-4-12B-it.gguf, arch gemma4-assistant) prints at boot llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set + failed to measure the memory of the extra model, fitting without it — this is the fit probe, self-described normal in the log, and the draft loads against the real context seconds later. Do NOT restart over it. Also: Gemma4 SWA + q8_0 KV made --fit take the full 262144 window in ~12 GiB of a 24 GiB RTX 3090 (idle 11,996 MiB with 7.4 GiB of Q4_K_M payloads), and Gemma4's chat template accepts every reasoning_effort level (all seven probed 2026-09-17: 200), unlike Qwen3.8's xhigh/medium/low gate. q8 KV on CUDA required rebuilding build-cuda with -DGGML_CUDA_FA_ALL_QUANTS=ON. Deployed as frank-small on dankclaw :7771 with --spec-type draft-mtp,ngram-mod (acceptance 0.38-0.46, mean len ~2.2). _(context: frank-small Gemma4-12B QAT cutover on dankclaw 2026-09-17 (commits 4b7de59, 87caca4; runbook serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md))_
|
||||
- User preference (Frank, 2026-09-17): never do multi-step arithmetic by hand in prose — "stop doing math without python, you're an LLM. your handling numbers is like expecting determinism out of a pachinko board." All VRAM fits, per-token costs, gap/slack computations and the like must go through the eval tool (Python) and quote the computed numbers. Hand arithmetic is acceptable only for trivial single-step checks. _(context: frank-medium VRAM fit for 36:29 @ 327680 — I had computed card balances in-head before the user corrected)_
|
||||
- llama.cpp mainline (fb27a525, the frank-medium build) does NOT autoload mmproj*.gguf from the model directory: a server started without an explicit --mmproj flag comes up healthy and text-only, answers image requests with 500 "image input is not supported - hint: if this is unexpected, you may need to provide the mmproj", and the load log contains no mmproj line at all — so /health and the journal both look fine. Always pass --mmproj file explicitly (the frank-small pattern), and verify vision behaviorally (send an actual image) after any deploy that touches the projector, because health checks cannot see it. Cost of the trap: one full extra deploy cycle on 2026-09-17 (frank-medium 36:29/327680 cutover). _(context: frank-medium 2026-09-17: NO_MMPROJ=0 + mmproj-BF16.gguf beside the model, first deploy healthy, vision 500'd; fixed with explicit --mmproj in serve-frank-medium.sh)_
|
||||
- Measuring batch geometry and VRAM balance on a ROCm llama.cpp host — three durable rules (Dankserver 2026-09-16, frank-medium, 2x R9700, scripts/bench-batch-ubatch.sh): 1. SETTLE GATE, or your infeasibility verdict is an artifact. Loading a config, reading rocm-smi, killing it and immediately loading the next one fails with error loading model: unable to allocate ROCm0 buffer — amdgpu does not return ~26 GiB the moment the process dies, and a 180 s settle is NOT enough (600 s was still not enough in one run; the VRAM came back after the transient unit's cgroup was torn down). That failure is indistinguishable from "this configuration does not fit". Gate each load on rocm-smi showing both cards near-empty (awk over --showmeminfo vram --csv, wait until used 1 GiB per card), and run each config in its own transient unit so systemd's cgroup teardown does the killing. 2. A "plateau" claim must be tested from BOTH sides. Sampling only 512→2048 showed a shallow basin (1135-1170 tok/s) that read as "the knob does not matter"; extending downward found the actual PEAK at ubatch 512 (1193) with real losses on both sides — 256 → 1046 and 384 → 1068 (SMs not saturated), 2048 → 1077 (10 % off). The deployed 1536 was thus both 2.4 % slower than the optimum AND 3.71 GiB more expensive. 3. THE SPLIT IS A FUNCTION OF THE BATCH, because the compute buffers are not symmetric. Shrinking ubatch 1536 → 512 freed 1.30 GiB on the plain card but 2.41 GiB on the card holding the output layers and the drafter's graphs — so the balance point moved a layer (36:29 → 35:30 on a 65-layer hybrid) and the reachable minimum gap became one DeltaNet layer (0.46 GiB), not zero. Change the batch and you must re-measure the split. Method notes that made the result trustworthy: llama-bench records n_ubatch per test in its -o json output, so "is the knob even applied" is answerable from the artifact; and the VRAM columns scale monotonically with ubatch, an independent confirmation. The falsification test for _(context: Dankserver 2026-09-16: swept llama.cpp --batch-size x --ubatch-size on frank-medium (2x R9700 layer split, Qwen3.8-27B UD-Q8_K_XL + DFlash2) and applied the result. Operator had said "that sounds like something's wrong" about a flat prefill curve.)_
|
||||
- In serverAppConfig, other sessions commit to the same checkout while you work, so line-anchored edits rot within minutes. Twice on 2026-09-16 an edit landed in the wrong section for the same reason: (1) AGENTS.md was edited against a read taken many turns earlier, and the peer session's commits had shifted the file — the frank-large text ended up split between the Model Config section and the service section, needing a repair; (2) an append at N used an N taken from wc -l, but the file had no trailing newline so its last content line was N+1 — the new section was inserted *before* the paragraph's final line, orphaning it. Discipline that prevents this: re-read the target region (read tool) immediately before editing, take the tag and line numbers from that read, prefer text-anchor insertions (PUT N: where N is a line you just saw) over remembered numbers, and immediately after an edit whose result looks even slightly off, re-read the region and check the surrounding structure (headings, list continuity, closing braces) rather than trusting the edit's summary. git diff --stat plus a targeted sed -n on the changed range is the cheap verification; for multi-hunk edits in a large markdown file, verify by reading the file, not by reasoning about the diff. Also: wc -l undercounts files that lack a final newline — never use it as the append anchor. _(context: Dankclaw/serverAppConfig 2026-09-16: while documenting the x4 link and Q4_K_S results, a peer session was committing frank-medium work to the same checkout; a stale-anchor AGENTS.md edit split a section, and a wc -l-based append split a paragraph in the x4 note. Both were caught by re-reading the files and repaired before commit.)_
|
||||
- When a serving engine keeps some weights in host RAM, the decisive question is whether it *streams* them across PCIe or *computes* them there — the link decides the first and is irrelevant to the second. Dankclaw's 3090 is on Gen4 **x4** (measured H2D 6.72 GB/s, D2H 6.60 GB/s = 85 % of the 7.877 GB/s spec; idle-downshifts to 2.5 GT/s x4 and trains up under load). For an A3B MoE whose active experts are ~1.70 GiB/token (Q4_K_M, 75 GiB expert mass), streaming them costs 272 ms/token ⇒ **3.7 tok/s** (4.6 with a fifth resident), while computing the same bytes in host RAM at 256 GB/s costs 7.1 ms — the link is **31× slower than the memory behind it**. Measured alternatives on the same host: 20.46 tok/s for llama.cpp with experts computed on the CPU (-ot exps=CPU; the link then carries activations only), 25 for halogen, 26.5 for mainline on the iGPU. So "is the link fast enough?" is answerable in one line of arithmetic: per-token bytes the engine must transfer ÷ measured rate = seconds/token, and compare that against the compute-in-place path. Two measurement subtleties from the probe: cudaMemcpyAsync on one stream was *no* faster than sync, and pageable transfers ran within 1 % of pinned — i.e. the copies were link-limited rather than staging-limited, so don't assume the usual "pageable costs 2-3x" rule. Also, hardware ceilings must be read from the *platform port* (LnkCap of the upstream bridge) and not just the card: the 3090 is x16-capable, the Strix Halo GPP port under it is x8-capable, so the x4 lives in the adapter or cable and the realistic upgrade is x8 (2×, ≈7-9 tok/s) — still behind every alternative. And FreeToken specifically (expert LRU in host RAM = exactly the streaming shape) loads only modelopt block-FP8 HF safetensors for qwen3_5_moe, so a Q4_K_M GGUF is unloadable there before bandwidth is even the question. _(context: Dankclaw 2026-09-16: user asked whether the x4 link could support running the 111 GiB Q4_K_M Flash-Next model with FreeToken on the 3090 alone; built pcie-link-test.cu, stopped both services, measured, and derived the ~3.7 tok/s streaming ceiling vs 20.46 tok/s for llama.cpp's compute-in-RAM path.)_
|
||||
- FreeToken v0.1.2 is an "edge-native MoE serving engine that keeps EXPERT WEIGHTS IN HOST RAM with an LRU GPU cache" — so unlike llama.cpp's -ot exps=CPU (which COMPUTES experts in host RAM and moves only activations over the bus), FreeToken STREAMS expert weights across PCIe every token. That makes it the one design where link width is the critical path, and its own Dankserver measurements prove the mechanism: Qwen3.6-35B-A3B-FP8 on a 3090 gave 56.7 tok/s at ~89% expert residency but 25.2 tok/s at 5% residency, and the runbook itself notes the bottleneck "flips to PCIe expert streaming" at low residency — 25.2 tok/s × ~2.4 GB/token of misses ≈ 60 GB/s, i.e. an x16 Gen4 link saturated. Two consequences for Dankclaw, where the 3090 sits behind Strix Halo GPP bridge 00:03.1 (LnkCap 16GT/s **x8**, LnkSta x4; the platform's other ports are x1/x4/x16-for-iGPU, so there is no x16 slot to move it to): (1) x4 = 7.877 GB/s and x8 = 15.75 GB/s cap that design at roughly 4.6–10 tok/s for a 111 GiB A3B model, because ~1.7 GiB of active expert bytes per token cannot cross in time; (2) it cannot even load this file — FreeToken's loader matrix is modelopt block-FP8 HF safetensors for qwen3_5_moe only (compressed-tensors FP8 fails; GGUF is not supported at all). The measured ceiling for "Q4_K_M experts-in-RAM on just the 3090" is therefore the CPU-compute path at 20.46 tok/s decode / 60–81 t/s prefill, and the host's fastest single-model arrangement remains the iGPU (25.8 tok/s Q4_K_M, 26.5 Q4_K_S at 64k depth). Also durable: FreeToken builds for CUDA only (its _pinned_tensor kernel links cudart), and torch.cuda.is_available() can report False on a box whose kernels are fine. Separately, the model-payload rules that bit me on 2026-09-16: mradermacher ships large quants as .partNofM fragments (a .gguf filter hides Q4_K_S entirely); curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body (three 153-byte "shards" with identical sha256 and a cheerful DOWNLOAD_CODE=done in 20 _(context: Dankclaw 2026-09-16/17: user asked whether the 3090's PCIe Gen4 x4 link is fast enough to run the 111 GiB Flash-Next Q4_K_M under FreeToken on just the 3090. Answered from the FreeToken runbook's own mechanism and measurements, the observed Dankserver residency-vs-throughput table, lspci -vv topology of the Strix Halo GPP bridge (x8 cap, x4 trained) and the earlier measured x4 bandwidth; the same )_
|
||||
- Dankserver's frank-medium (2x Radeon AI PRO R9700, gfx1201/Navi 48) runs **ROCm/HIP, never Vulkan**: scripts/build-llama-rdna4.sh builds with -DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF -DGGML_HIP_RCCL=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1201 against a private bundled ROCm 7.14 tree (lib/rocblas, lib/hipblaslt, libamdhip64.so.7). Two log-reading traps that follow: (1) a HIP build still logs ggml_cuda_init: found 3 ROCm devices because upstream implements the HIP backend INSIDE ggml-cuda (so the CUDA name persists) — the giveaways are device names coming back as ROCm0/ROCm1 and ldd showing libamdhip64/librocblas/libhsa-runtime64; (2) ldd | grep -ci vulkan = 0 is the one-line proof of no Vulkan. Vulkan exists only on Dankclaw's separate build-dual tree (CUDA+Vulkan) used for the x4 3090+iGPU experiment; it has nothing to do with Dankserver. RCCL is compiled out by design because card #2 cannot dispatch an RCCL kernel (PCIe atomics), which is why the two cards are joined by llama.cpp -sm layer, not tensor/pipeline parallelism. _(context: 2026-09-16, Dankserver: user asked "are we using rocm or vulkan?" while a llama-bench batch x ubatch sweep was running on frank-medium. Verified from scripts/build-llama-rdna4.sh:85-88, llama-bench --version (0.4.1-dev, fb27a525d), and ldd on the binary.)_
|
||||
- In the serverAppConfig homelab repo, more than one agent session can be committing at once (I shared it with a peer working on frank-medium tonight). That breaks line-anchored edits: I edited AGENTS.md using a tag from a read several turns earlier, the peer's commits had shifted every line below my anchor, and my two hunks landed in the wrong sections — the first split the "Model Config (harness)" section from the "frank-large" heading, the second inserted a paragraph mid-sentence so the halogen description read "halogen-flash-server 0.5.8 as a rootless Quadlet again since 2026-09-16 [paragraph] (dankclaw/frank-large/quadlet/...), serving ...". The tell was in the edit response itself: neighbouring lines that had no business being adjacent. Discipline that prevents it: re-read the exact target region immediately before editing (or grep the target text for a fresh line/anchor) rather than reusing a tag from earlier in the session, and always glance at the returned context lines to confirm they are the ones you expect; repair a mis-landed edit with a range rewrite (CUT the stray lines, PUT the whole section back in order) rather than incremental patching. Second, durable purge practice learned the same night: before deleting a "model cache" on a serving host, (a) inventory what is actually there, (b) find which dirs *live* units mount (grep Volume=/MODEL_DIR in the units) and protect them by name, (c) verify each service payload is re-fetchable upstream before deleting — query the Hub API anonymously, because a gated repo returns 403 even with the host's token, (d) record byte-size fingerprints before deletion so the re-fetch can be compared afterwards (that is how I proved halogen's overlays had been stale in place: 2,477,677,120 → 2,572,466,560 and 2,383,306,048 → 2,478,095,488 while the checkpoint returned byte-identical), and (e) expect the bind-mount trap when clearing a container's model dir: the directory must exist but be empty, or podman fails with Error: sta _(context: Dankclaw/serverAppConfig 2026-09-16 night: operator ordered the model cache cleared with one-time rm permission; I purged ~600 GB (service payloads included) and re-fetched, which revealed the halogen sidecars had been stale in place. The same session's AGENTS.md doc update mis-landed because a peer session's commits had shifted the file between my read and my write.)_
|
||||
- You can characterize a live LLM endpoint's real performance WITHOUT stopping or benchmarking it, by mining the serving runtime's own per-request telemetry from journald: llama.cpp (mainline, verbosity 3) logs slot print_timing per task with prompt eval time, eval time, total time, graphs reused, draft acceptance = A (n accepted / n generated), mean len = L, plus newer n_gen = N, tg = X t/s, tg_3s = Y t/s; halogen-flash-server logs one line per request: serve_api: mtp N tok in Xs = Y t/s | R rounds, commit Z/round | prompt P (C cached), prefill Ss | detok Uus/tok. Root unit - sudo journalctl -u unit; rootless quadlet - journalctl --user -u unit (never sudo). This is the policy-compliant substitute on hosts whose rules forbid running a bench beside a serving GPU, and it measures the REAL workload rather than a synthetic one. To attribute PCIe cost, run nvidia-smi dmon -s t -c N -o T (columns rxpci/txpci in MB/s) in the background while issuing ONE ordinary request; measured on dankclaw frank-small mid-generation: peak ~230 MB/s tx against a 63 Gb/s (Gen4 x4) link = ~3% utilization, zero at idle - i.e. a 4-lane link costs a fully-VRAM-resident model nothing, because decode traffic is only the sampled logits stream. THE SSH TRAP: if you background the curl and your remote command exits, ssh closes the session and kills the client, which CANCELS the in-flight generation - the journal shows W srv stop: cancel task, id_task = N and NO print_timing line at all, so you get 200-looking HTTP with no telemetry. Always end the remote command with wait so the session outlives the request. Also: per-request numbers are content-dependent (measured MTP draft acceptance 0.46-0.76 on the same build, mean accepted length 2.39-3.27), so a live sample is not comparable to a deploy-time baseline without a masked llama-bench run. _(context: 2026-09-16, dankclaw: user asked for "the performance of the model" after a PCIe x4 link discussion. Measured frank-small (llama.cpp + MTP on the RTX 3090) and frank-large (halogen on the iGPU) from production journal telemetry because AGENTS.md forbids benchmarking beside a live service, and proved PCIe x4 is not a factor with nvidia-smi dmon during one real 200-token generation.)_
|
||||
- Running llama-bench on the Dank fleet (verified 2026-09-16 on dankserver, frank-medium's UD-Q8_K_XL + 2x R9700 layer split) — four facts that decide whether a sweep measures what it claims: 1. llama-bench has NO -c/--ctx-size. Source line: cparams.n_ctx = n_prompt + n_gen + n_depth. The context is a CONSEQUENCE of the test shape, so llama-bench can never test whether a config allocates beside the deployment's real KV (frank-medium's 16 GiB at 262144 over 2 slots). That check must be done by starting the real unit. Same reason there is no --parallel and no --spec-*: llama-bench measures single-sequence, drafterless numbers — its tg is the autoregressive baseline (~15.9 tok/s at 64k depth here) against the deployed 43.8 tok/s with DFlash2+ngram. -d N is a depth prefix that puts N tokens of KV in place before the measured run and is CACHED across reps via a state save/restore (llama_state_seq_*), so the fill is not re-paid per rep and is not part of the measurement. 2. Separator trap: in llama-bench, / separates devices WITHIN one configuration and , separates configurations to sweep; -ts copies this. -ts 36,29 silently becomes one value (36) and puts everything on one card, benching a different machine. llama-server takes the comma form for the same pair. Also useful: list-valued args accept ranges — -ub 512-2048+256. 3. A test that cannot run calls exit(1) IMMEDIATELY (failed depth/prompt/gen run → llama_free + exit(1)), so one infeasible ubatch in a big cross-product invocation takes every later cell with it and loses the results. Drive one invocation per cell, continue on failure, and resume by skipping cells that already have a non-empty output file. 4. memlock: the deploy units set LimitMEMLOCK=infinity because the ROCm backend pins staging buffers; an SSH shell on these hosts gets soft=hard=8 MiB (ulimit -l), so a bare llama-bench run can fail the load for a reason that has nothing to do with the model. Run it inside a transient unit: sudo systemd-run --unit=n - _(context: Dankserver 2026-09-16: operator asked for a batch x ubatch sweep at 64k context on frank-medium (full 4x7 cross-product, -p 65536 -n 128 -d 65536, -r 3 --no-warmup, no spec decoding). Harness written, dry-run tested with a stub llama-bench, deployed via git, run as a transient systemd unit.)_
|
||||
- Three traps hit in one model-payload fetch on the Dank fleet (2026-09-16), all worth knowing before the next swap. (1) mradermacher's GGUF repos ship the LARGE quants as .partNofM fragments, not .gguf: mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF holds Q4_K_M as part1of3/2of3/3of3 (37.00+37.00+36.97 = 110.97 GiB — which is exactly how the deployed single-file payload was assembled) and Q4_K_S the same way (35.00+35.00+34.09 = 104.09 GiB), while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, IQ4_XS) are single files. Listing the Hub tree and filtering for paths ending in .gguf therefore makes Q4_K_S look like it does not exist — I asserted that to the user twice before they pushed back. Filter on the quant name, not the extension. (2) A fetch script must use curl -f/--fail-with-body, compare byte counts to the Hub's declared size, and verify the LFS sha256; otherwise it reports success on an access error. curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body, so a gated repo produced three 153-byte "shards" with identical sha256 values and a cheerful DOWNLOAD_DONE in 20 seconds for 104 GiB — caught only because the time and the duplicate hashes were absurd. (3) A token passed as curl -H "Authorization: Bearer …" is visible in /proc/pid/cmdline, so pgrep -a curl while a download runs prints the secret into your transcript (it happened); use curl -K mode-600 file or an HF_TOKEN environment variable, and never pgrep -a a command carrying an auth header. Related: pgrep -f in a wait loop matches the loop's own command line when the pattern appears in it, so a "wait for the download to finish" loop can spin its full iteration count without ever observing completion. Gated mirrors: orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF carries the same Q4_K_S ladder but returns 403 even with the host's HF token until that account accepts the gate — mradermacher's copy of the same quant is the ungated source. _(context: Dankclaw 2026-09-16: user asked to download Q4_K_S of mradermacher/Qwen3.8-Flash-Next-Uncensored-i1 for a smaller-quant test; the fetch went through a wrong "the repo has no Q4_K_S" claim, a 403-on-gated-repo false success, and a token leak via pgrep -a, before landing a verified payload (111,769,915,680 bytes, parts hash-matched against the Hub's LFS oids).)_
|
||||
- Two durable traps from rebalancing frank-medium's tensor split (2026-09-16, Dankserver; revert DSpark-DFlash2): 1. A RECORDED ARTIFACT SIZE IS A CLAIM, NOT EVIDENCE — verify against the Hub's tree API. The fetch script declared the DSpark drafter as 1,359,284,737 bytes while the Hub's authoritative size was 1,455,376,576 (96 MB short, the LFS object's own size), so --verify-only failed on a payload that was in fact byte-correct (its sha256 DID match the LFS oid, because my recorded hash was taken from the host file, not the Hub — a self-consistent loop that proves nothing). The check that settles it in one call: curl -sL "https://huggingface.co/api/models/repo/tree/main?recursive=1" then read each entry's size and lfs.oid (a 128-char sha256, pointerSize:135 for the pointer file). Compare BOTH against stat -c%s and sha256sum on the host. Note the bare /api/models/repo endpoint does NOT list files; the ?blobs=true variant returned null siblings on these repos, and piping the tree into jq failed with "cannot use null as iterable" when the response was partial — parse it in python from a saved file instead. 2. WHEN A REPO CARRIES A FITTED VRAM MODEL (a test seam), FITTING AGAINST IT REQUIRES MATCHING ITS ARITHMETIC UNIT CONVENTION EXACTLY, or the constants you derive are meaningless. frank_medium_unit.bats expresses compute buffers as 1442 * 1024 * ubatch — i.e. the coefficient is ~1.4 MB per ubatch UNIT, not 1442 bytes — while my first fit (in python) used 1442 * ubatch. The result was a fitted card-0 constant of -54,541,757 B (a negative "constant", the tell) and a seam that failed the floor assertion by ~2 GiB: 1442*1024*1536 vs 1442*1536 is a 2.1 GiB difference. Read the assertion expression literally before fitting to it, and treat a negative constant as evidence of a unit error rather than something to leave in. Also worth keeping: bash -n file.bats is NOT a syntax check — bats' @test "..." { } form is not valid bash, so it fails on a valid file (line 32 in that f _(context: Dankserver 2026-09-16: reverting frank-medium's spec drafter from magnitudedev DSpark to z-lab DFlash2 (DSpark's window-less draft KV spanned the full 262144 ctx = 5.0 GiB) and re-measuring the tensor split. The revert freed ~4.3 GiB on ROCm1 and moved the balance from 40:25 to 36:29. Repo: /home/blank/Projects/serverAppConfig, files scripts/frank-medium.env, tests/frank_medium_unit.bats, scripts/)_
|
||||
- llama.cpp multi-GPU layer-split sizing, learned on Dankserver (2x R9700, 262144 ctx targeted) 2026-09-16: 1. A SPECULATIVE DRAFTER'S KV IS SIZED FROM THE TARGET, NOT FROM ITS OWN FILE. common/speculative.cpp builds the draft context with cparams.n_ctx = llama_n_ctx(ctx_tgt) and there is NO flag to cap it independently (the --spec-draft-* surface has no ctx). So a drafter that declares no sliding window caches the WHOLE target context on whatever device --spec-draft-device names: DSpark (5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB/position) costs 5.0 GiB at 262144, where DFlash2's dflash.attention.sliding_window = 2048 had held it to ~40 MiB. Read sliding_window (and block_size, n_ctx_train) out of the drafter's GGUF header BEFORE sizing. Signature of getting it wrong: the model LOADS, /health answers, and the FIRST draft decode dies with ROCm error: out of memory inside common_speculative_impl_draft_dflash::process -- an allocation failure after a successful load is a SIZING failure, not a broken drafter, and a crash-looping unit makes it look like the latter. 2. A BIGGER --ubatch-size IS ASYMMETRIC ACROSS A LAYER SPLIT. Measured 2x R9700 at 262144: ubatch 512 - 1536 cost the output/drafter card ~3.3 GiB against ~1.7 GiB on the other (that card holds the output layers AND the drafter's graphs). So a split balanced at one batch is UNBALANCED at another -- 38:27 was balanced at 2048/512 and left the draft-carrying card at 0.08 GiB free after the batch tripled. One -ts step is worth ~1.5 GiB at that size; the balance point must be found by measurement (38:27 - 3.78/0.08; 41:24 - 1.47/2.38; 40:25 - 1.88/1.88). Corollary: the balanced split is also what MAXIMISES ctx for a given per-card headroom floor, because the headroom has to come out of CTX_SIZE. Also measured: batch 8192/ubatch 2048 does NOT fit at 262144 (its 2020 MiB pp buffer fails to allocate on the busier card), while 4096/1024 fits but leaves only 1.11 GiB. 3. A GPU PAGE FAULT IS NOT AN OOM. One split _(context: Dankclaw/Dankserver 2026-09-16: swapping frank-medium's drafter from z-lab DFlash2 to magnitudedev DSpark, then re-specifying batch/parallel/ctx per operator instruction. Repo: serverAppConfig, commits 7e11d24..718a283.)_
|
||||
- Dank-fleet VRAM accounting, learned while answering "what's the VRAM breakdown?" (2026-09-16). (1) THE TWO DEVICES ON DANKCLAW ARE NOT THE SAME KIND OF MEMORY. The RTX 3090 is discrete: nvidia-smi --query-gpu=memory.total,memory.used --format=csv,noheader (24,576 MiB total; 22,582–22,592 MiB used by the single llama-server PID serving frank-small, confirmable with nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader). The gfx1151 iGPU is UMA: /sys/class/drm/card1/device/mem_info_gtt_total = 122,402,963,456 (114 GiB) and mem_info_gtt_used = ~36.2 GB (33.7 GiB) is the real "VRAM", while mem_info_vram_total = 2,147,483,648 (a 2 GiB carve-out window) with mem_info_vram_used ~208 MB. So iGPU VRAM IS host RAM, and a phrase like "halogen holds 95.4 GiB" is misleading: halogen's ledger shows 68 GiB of weights PINNED AS HOST PAGES (mlock/page cache, not GTT) plus ~33.7 GiB inside the device budget (11.0 GiB KV pool over 400,384 positions + working memory/scratch). Its own measurement lines are the best source: podman logs frank-large | grep -aiE "weights pinned|kv pool|left for everything|overlay". (2) THE REAL PER-BUFFER BREAKDOWN IS PRINTED BY llama.cpp AT INFO — llama-model.cpp:1827 ("%12s model buffer size = %8.2f MiB", per backend), llama-context.cpp:692 (compute buffer size), plus the KV self-size line — BUT NO ARTIFACT IN THIS FLEET CONTAINS IT: llama-bench suppresses the loader's INFO (its logs show only the device banner and the results table), and sudo journalctl -u frank-small | grep -c "model buffer size" = 0 across the unit's whole history. To actually get it, launch the model once through llama-cli or llama-server (not llama-bench) and keep the first ~200 lines; add -v/--verbose if the level filters it. Otherwise compose the breakdown by arithmetic: read the GGUF tensor table for group sizes (on Flash-Next: experts 75.0 GiB / 48 layers = 1.5625 GiB per layer, per_layer_token_embd 32.8 GiB, everything else 3.2 GiB / 48 = 0.067 GiB p _(context: Dankclaw 2026-09-16: after the 3090+iGPU split experiment, reconstructing what the VRAM looked like per device (measured totals from nvidia-smi + sysfs, halogen's ledger lines, GGUF tensor arithmetic) because no log carried llama.cpp's per-buffer accounting; also pinned down llama-bench's derived context lengths (n_ctx = n_prompt + n_gen + n_depth) for the runs whose parameters were being reviewed)_
|
||||
- Working convention on the Dank fleet for answering "what was running when it broke": experiment artifacts must survive on disk because /tmp does NOT (/tmp is cleared by a reboot — after the 2026-09-16 Dankclaw incident the x4*.sh experiment scripts were gone while their logs in /home/dank/models/logs/ survived). So launch experiment scripts from a persistent path and log to /home/dank/models/logs/tag.log. Reconstruction then works from three independent evidence sources: (1) log file mtimes (ls -l --time-style=+%m-%d_%H:%M:%S) minus each run's own real duration from time = that run's start, which orders/nests the runs; (2) set -x lines inside those logs are the authoritative record of the exact command line and parameters — but a log can be truncated mid-write if it is cleaned while a script still holds it, so a log's mtime is not proof the run *finished*; (3) the user journal gives the first-failure instant for any service (journalctl --user --since … | grep -m2 'invalid internal status'), and on this fleet container healthchecks shell out to the podman CLI, which makes them a sensitive early indicator that podman itself has broken. The answer to "what was running" is whichever invocation's [start, end] window contains that first-failure instant; with several overlapping resource-heavy invocations, check whether they overlapped at all (mine did not prove it: the sampler output was in /tmp and gone). Related trap when reading such logs through an agent tool: long command lines get display-truncated, so wrap them (sed 's/\(.\{130\}\)/\1\n /g') rather than concluding the parameters were short, and beware off-by-one log names (x4b.log vs x4b-experiment.log) silently skipping files in a loop. _(context: Dankclaw 2026-09-16, answering "what were the test parameters before the crash?" about the rootless-podman pause-process incident: reconstructed from log mtimes + per-run real durations + the first invalid internal status at 18:58:30, which pinned the in-flight run (launched 18:54:31) — after /tmp/x4*.sh had been wiped by the repair reboot.)_
|
||||
- Correction to my earlier note on the rootless-podman pause-process death (Dankclaw, 2026-09-16): the cause is NOT established. I had written that memory pressure from a GPU experiment that committed 116 of the host's 121 GiB killed it, but journalctl -k --since for that window records NO OOM kill and the kernel says nothing else about the pause process — so the mechanism was a guess presented as a finding. The operational content that IS verified stands: every podman command fails with invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process; /run/user/uid/libpod/tmp/alive holds a stale UUID token no process owns; container starts then fail because the rootless netns holding podman's bridges is gone (failed to bind udp listener on 10.89.x.1:53: Cannot assign requested address); podman system migrate panics in 5.7.0 (nil deref in teardownNetworkBackend); moving the stale token aside restores the CLI but not the bridges; a reboot repairs everything and all boot-persistent quadlets return on their own. The generalizable discipline: before publishing a cause, grep the evidence that would prove it (for a killed process: journalctl -k | grep -i oom), and if it is absent, publish "trigger not established" rather than a plausible story — an incident narrative that turns out to be invented is worse than an open question, because the next agent plans against it. _(context: Dankclaw 2026-09-16: writing up the pause-process incident in AGENTS.md, the x4-pair-experiment note and log.md, then checking journalctl -k and finding no OOM kill for the window — which falsified the cause I had already written down in three documents.)_
|
||||
- On a rootless-podman fleet host, the rootless "pause" process is load-bearing for ALL container management, and its death is not recoverable in place. Symptoms seen on Dankclaw 2026-09-16: every podman command (even podman ps, podman system df) fails with invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process; the alive file in /run/user/uid/libpod/tmp/ holds a stale UUID token (not a pid) that no process owns; every container start then fails with netavark: error while applying dns entries: aardvark-dns failed to start / failed to bind udp listener on 10.89.x.1:53: Cannot assign requested address because the rootless netns that held podman's bridge interfaces died with the pause process; and every Restart=always quadlet loops forever (28 of them here). Podman's own named remedy, podman system migrate, PANICS in 5.7.0 (panic: runtime error: invalid memory address or nil pointer dereference in libpod.(*Runtime).teardownNetworkBackend while tearing down a stopped container). Moving the stale alive/alive.lck aside makes podman respond again but the missing bridge interfaces remain, so containers still fail. The reliable repair is a REBOOT: /run/user clears, the pause process and rootless netns are recreated, and every boot-persistent quadlet returns by itself (verify loginctl show-user u -p Linger → Linger=yes and [Install] WantedBy=default.target present in each unit BEFORE rebooting). Plausible trigger in that incident: memory pressure from a GPU experiment that committed 116 of the host's 121 GiB — the pause process is ordinary OOM-killable host state. Also from the same recovery: a container cannot reach a Network=host service at the host's LAN IP, because netavark DNATs only PUBLISHED ports — from inside another container 192.168.0.143:published answers but 192.168.0.143:host-bind and the gateway 10.89.1.1 are refused; use host.containers.internal (169.254.1.2), which fixed a 502 o _(context: Dankclaw 2026-09-16 evening: after the x4-pair GPU experiment (which committed 116 of 121 GiB) the rootless podman pause process was gone; all 28 rootless quadlets were down/looping and the Caddy front door with them. Diagnosed the pause token + missing rootless netns, found podman system migrate panicking, repaired by reboot, then fixed a hermes vhost 502 caused by the netavark published-port-onl)_
|
||||
- llama.cpp's llama-bench separates a device/tensor-split LIST with / and SWEEP values with , (-dev dev0/dev1/..., -ts ts0/ts1/..). So -dev CUDA0,Vulkan1 is TWO single-device configurations, never a two-device run — and -ts 0.85,0.15 is two sweep values, each becoming [0.85,0,0,...] / [0.15,0,0,...], i.e. device 0 takes every layer. Write -dev A/B -ts x/y. On 2026-09-16 this artifact produced a completely wrong fleet conclusion on Dankclaw ("CUDA + Vulkan cannot serve one model"): the "abort" was a single-device run with -ot exps=Vulkan1 pointing at a device absent from the scheduler's list, and an "OOM at 25% split" was really all 111 GiB on the 24 GiB card. Tell-tale: llama-bench's own dev column prints the devices actually used per row — read it. Corollary that generalizes: before concluding anything from an abort or a resource failure, prove the harness parsed your intent (echo the resolved config, or check the tool's own reported parameters), and for placement-dependent numbers get an independent witness (sample the other device's memory with nvidia-smi while the run is loaded) rather than trusting the label. The real blocker in that episode was different and simpler: per-tensor cross-device placement is killed by --override-tensor itself — has_tensor_overrides() is a required term of the scheduler's pipeline_parallel flag (llama-context.cpp:433, the only read site), which sets n_copies in ggml_backend_sched_new; with n_copies == 1 a pre-allocated weight on a non-primary device has no backend and ggml-backend.cpp:941 aborts with the leaf's own op name (NONE). Two-device LAYER splits (no -ot) work fine, including cross-vendor CUDA+Vulkan. _(context: Dankclaw 2026-09-16: x4-pair experiment (3090 + Strix Halo iGPU under mainline llama.cpp, dual CUDA+Vulkan build 37b53fd); pure-Vulkan and cross-vendor layer splits then measured at 27.55 / 27.23 tok/s @64k depth vs 25.77 iGPU-only.)_
|
||||
- Authoritative per-flag documentation for a llama.cpp build = its own common/arg.cpp: every flag is defined as add_opt(common_arg({"-x", "--flag"}, "TYPE", "multi-line help", parser)) with the help text inline (modern master, e.g. fb27a525 on Dankserver, keeps ALL arg definitions in this one file; older trees split them across common/common.cpp + tools). Two traps: (1) llama-server --help on this master does NOT print help and exit — with no --model it silently starts a router-mode server on 127.0.0.1:8080 and stays there, so grep the source instead of probing the binary; (2) --help-style patterns won't match — grep for the literal quoted flag string, e.g. grep -n -A9 -e '"--split-mode"' common/arg.cpp. Recipe: ssh host 'P=$(find buildroot -maxdepth 5 -name arg.cpp -path "*common*" | head -1); cat "$P"' /tmp/arg.cpp (the checkout path can move under you mid-session — resolve it in the same call), then extract per flag locally. _(context: frank-medium exhaustive settings review, 2026-09-16: extracted all 44 live flags' help text from common/arg.cpp in the fb27a525 tree on Dankserver after --help started a server instead of printing.)_
|
||||
- On the Dankclaw/Dankware fleet (and llama.cpp generally), a single model CANNOT be served by two different *device* backends — e.g. trunk+KV on the 3090 via CUDA and MoE experts on the Strix Halo iGPU via Vulkan — and this is a hard structural limit, not a tuning problem. Two failure modes, both measured 2026-09-16 on mainline llama.cpp 37b53fd: (1) per-tensor -ot pattern=device across device backends aborts in the scheduler with ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) — a weight in one device buffer cannot be consumed by an op the scheduler placed on another device; (2) per-layer cross-backend splits (-sm layer -ts, or layer-scoped -ot) fail at model load with only the generic "failed to load model". The CPU is the ONLY side that works as half of a heterogeneous split, because the CPU backend is the universal fallback and host buffers can be copied — which is exactly why -ot exps=CPU works. So on a one-NVIDIA-plus-one-AMD host, the 3090 can be the trunk OR the expert host but never the complement of an iGPU-resident peer; every working multi-device llama.cpp deployment in this fleet is either single-vendor (frank-medium's two R9700s via -sm layer) or device+CPU. The surprising measurement that came out of the same run, and which supersedes the old "mainline llama.cpp is slow on gfx1151" assumption: a CUDA+Vulkan dual-backend build with EVERYTHING on the iGPU (-dev Vulkan1 -ngl 999 -ot per_layer_token_embd=CPU) did 32.64 tok/s at ~0 depth and 25.77 tok/s at 64k depth, 456.7 tok/s fresh prefill, with the iGPU sampled at 99-100% busy and the 3090 idle at 0%. That MATCHES halogen's deployed ~25 tok/s (the ik-era "mainline = 1.01 tok/s" figure was the CPU+3090 hybrid, never the iGPU). It is nonetheless not deployable on Dankclaw: it commits 116 of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB PLE page cache) against halogen's 95.4 GiB held with 14.7 GiB spare besi _(context: Dankclaw 2026-09-16: dual-backend llama.cpp experiment testing whether the RTX 3090 (CUDA) plus the Strix Halo iGPU (Vulkan, RADV) could serve one 177B MoE (Flash-Next) together over the 3090's Gen4 x4 link. Result: not expressible; the iGPU-only configuration matches halogen but eats the host.)_
|
||||
- Deciding whether to split one model across two heterogeneous accelerators on a single host — the arithmetic that settles it, from the 2026-09-16 Dankclaw design pass (RTX 3090 + Strix Halo gfx1151 iGPU, RTX 3090 on a PCIe Gen4 **x4** M.2 dock): 1. LINK FACTS (measured): the 3090's LnkCap is 16GT/s x16 but LnkSta is x4, and it **idle-downshifts to Gen1 (2.5GT/s) x4 ≈ 1 GB/s**, reporting Speed 16GT/s, Width x4 only under load (7.88 GB/s). So the link is a *latency* resource, not a bandwidth one: a hidden state is ~10 KiB (Qwen3.8-27B embedding_length 5120 x 2 B), i.e. **1.3 µs per hop**, and 2 hops/layer × 64 layers ≈ 1 MiB/token ≈ 128 µs — ~0.3 % of a 40 ms token. Byte count is never the problem; **hop COUNT is** (ik_llama.cpp measured graph splits = 100/token with the GPU at 5 % SM). Design rule: partition by **where the weights live**, never by where activations flow; weights stay resident and only activations cross. 2. WHAT ACTUALLY PAYS: at batch-1 decode the KV cache is an **O(context) read every token** (Flash-Next: 28 KiB/position measured, so 5.5 GiB/token at 200k ≈ 21 ms of a 40 ms token at the LPDDR's shared ~256 GB/s vs 5.9 ms at the 3090's 936 GB/s). Expert weights are read once per token and are a rounding error beside it. So the highest-leverage move is **KV + its attention compute on the fast-VRAM device**, and KV must be co-located with its attention (separating them would push 5.5 GiB/token over a 7.88 GB/s link = ~700 ms/token, fatal). 3. WHAT DOESN'T: prefill is compute/amortized-weight bound, so moving weights to faster memory buys nothing and a device stage costs throughput. Calibration measured on this box, same model: moving 12.5 % of the expert mass to the GPU (-ncmoe 42) gave **decode +12 % (4.03 → 4.54 tok/s) but prefill 192 → 32 tok/s**. My bandwidth model predicts +10 % for that case, so it is usable for decode projections — and prefill is the casualty to budget for. 4. HARDWARE ASYMMETRY: on a UMA APU the iGPU is **not an independent memo _(context: Dankclaw 2026-09-16: after re-splitting frank-large (halogen on the iGPU) and frank-small (llama.cpp on the 3090), Frank asked what a *useful* 3090 + Strix Halo combination would be over the 3090's x4 link, then to elaborate the one-process split. Verified with live probes: lspci LnkSta/LnkCap idle vs loaded, sysfs VRAM/GTT windows, GGUF tokenizer identity between Qwen3.8-27B and Flash-Next, frank)_
|
||||
- Speculative decoding is only worth deploying when you check the ARITHMETIC against the incumbent, not when it "sounds free". Durable method (verified on Dankclaw 2026-09-16): round cost = K·d_s + ~1.1·t_s (d_s = drafter per-token step, t_s = target per-token step); tokens/round A = (1−p^(K+1))/(1−p); rate = A/round; compare against the *incumbent's* rate (a target with its own MTP/EAGLE head is already speculating). Two structural facts that decide most cases: (1) a PLAIN autoregressive drafter spends K sequential forwards per round, so it must be 10–50× faster than the target to pay — a 27B drafting for a 177B MoE (which is sparse and therefore fast) fails: measured 12 ms/drafter-token vs 67 ms/target-step gives break-even p ≈ 0.70 at K=3; (2) trained multi-token heads (MTP/EAGLE/DFlash) draft K candidates in ~one pass, which is why they are the drafter of record. Acceptance reality: the fleet's TRAINED z-lab DFlash2 drafter accepted only ~41% at depth 7 on the matched 27B target — a bar external/cross-finetune pairs won't clear. Also durable: cross-device speculation is cheap (token ids once per round, not activations) and the drafter's VRAM on a discrete card is an ADDITIVE pool, unlike an APU iGPU whose GTT window is a mapping of host LPDDR (no extra capacity or bandwidth). Cross-model pairs require a byte-identical tokenizer — cheap to verify by hashing the GGUF tokenizer.ggml.tokens list (Dankclaw's Qwen3.8-27B and Qwen3.8-Flash-Next GGUFs matched exactly: vocab 248320, merges 247587, pre=qwen35, same sha256 over the first 2000 tokens), but tokenizer identity alone does not make a drafter cheap or accepted. _(context: Dankclaw 2026-09-16: user asked whether a 27B on the 3090 could draft for Flash-Next on the iGPU, then whether a DFlash drafter could be hooked to halogen. Verified halogen's drafter surface is closed (serve_api.py DRAFTERS/DRAFTER_CAP, live 400 for dflash2), llama.cpp exposes --spec-draft-device/-devd, and computed break-even tables from measured step times (frank-small 11.96 ms/token; halogen ~6)_
|
||||
- llama.cpp speculative decoding, durable rules learned 2026-09-16 on Dankserver (frank-medium, Qwen3.8-27B UD-Q8_K_XL + zLayer split): (1) CHAINING IS GATED BY ctx_dft — common/speculative.cpp adds the draftless ngram family unconditionally but gates draft-mtp/draft-dflash/draft-dspark/draft-eagle3 on a draft CONTEXT existing, so a chain may hold one context-backed drafter plus any number of ngram types (draft-dflash,ngram-mod works), never two context-backed ones (draft-mtp,draft-dflash dies at load: "context type MTP requested but model doesn't contain MTP layers" — MTP builds against whatever model the draft context holds). Fixed internal order: ngram types first, then drafters, so CLI order is irrelevant. (2) MTP's head lives in the TARGET GGUF (blk.n.nextn.*) and only loads when MTP IS the draft model; with draft-dflash those tensors show as "unused tensor blk.64.nextn..." which is correct behaviour, NOT a truncated download. (3) ngram-mod prints NO startup banner in this build — a working chain looks unengaged in the journal; identify it behaviourally (draft_n0 only where text repeats) or via the per-task "draft acceptance = ... mean len = ..." line. It costs no VRAM (64 MiB host RAM). (4) Its window decides which traffic it can see: the fleet's 48/64 (n_min/n_max) wins because shorter windows propose on ordinary prose fragments, and rejected proposals still cost verification (43.9 - 36.0 tok/s); measured chain result: 2.6x the drafter alone on prompt-echo traffic, byte-identical to it on prose. (5) Measuring this is shape-driven and easy to get wrong: a "repeat verbatim" prompt with thinking ON puts the copy inside reasoning_content, so content is empty, cross-config identity checks compare '' to '', and you "verify" an echo you never looked at; disable thinking per request (chat_template_kwargs.enable_thinking=false) for echo/structured shapes. (6) Any harness that stops a production unit must restore it in a finally — mine crashed mid-run (a botched edit del _(context: serverAppConfig 2026-09-16: frank-medium moved from vllm-radiance replicas to mainline llama.cpp (layer split across 2x R9700, DFlash2 drafter) and then to a chained draft-dflash,ngram-mod. Findings live in serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md and AGENTS.md; harness is scripts/bench-spec-ab.py.)_
|
||||
- Speculative-decoding interfaces on the Dank fleet (verified 2026-09-16): HALOGEN (frank-large, iGPU) has a CLOSED drafter set — DRAFTERS = {"serial": 0, "mtp": 1, "dflash2": 2} in /halogen/tools/serve_api.py, wire values in the front-end→engine GEN protocol; DRAFTER_CAP gates mtp/dflash2; an unknown name is a deliberate 400 ("drafter 'x' unknown; use serial | mtp | dflash2"), and the entrypoint states there is no separate draft checkpoint for this model, so no external drafter can attach. Selection: per-request drafter field (omitted → engine applies the HALOGEN_DRAFTER env default), plus the image's own bench drafters max_tokens effort reps / sweep -d serial,mtp modes. Our deployment already runs the mtp drafter: engine verbose lines read mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round. LLaMA.CPP (frank-small, 3090) is the opposite: --spec-type takes a COMMA-SEPARATED LIST (none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,...), so drafters COMPOSE rather than replace, and there is a full external-draft surface (-md/--model-draft, -ngld/--gpu-layers-draft, -otd/--override-tensor-draft, --spec-draft-n-max, -ctkd/-ctvd, draft-cpu-moe/ncmoe). The RULE that generalizes: speculative decoding only exists inside the process that owns the target's KV, because verification needs a forkable/roll-back-able cache; hence "use device A as a drafter for device B" is only expressible as one llama.cpp process with both backends (needs a CUDA+Vulkan build — ours has GGML_VULKAN=OFF, GGML_RPC=OFF), and never as an HTTP-level arrangement. Also: the fast device belongs on the TARGET (the expensive stage), not the drafter. _(context: Answering "is it possible to use the 3090 as a drafter?" for Dankclaw's two endpoints (2026-09-16). Read from the halogen image's serve_api.py + entrypoint.sh and this llama.cpp build's --help.)_
|
||||
- Dankclaw's Strix Halo (Ryzen AI Max+ 395, Radeon 8060S/gfx1151) does NOT have a second memory pool for the iGPU — it has two *windows* onto the one 121.2 GiB LPDDR5X pool (8000 MT/s x 256-bit = 256 GB/s, verified via dmidecode; the 3090's 24 GB GDDR6X is 936 GB/s and genuinely additive). Probe: cat /sys/class/drm/card1/device/mem_info_vram_total = 2.0 GiB (BIOS UMA carve-out, genuinely private), .../mem_info_gtt_total = 114.0 GiB (GTT aperture mapping SYSTEM pages on demand); rocm-smi --showmeminfo all shows the same plus GTT Used, which tracks the engine's device-side footprint. Decisive proof that GPU memory here IS host RAM: halogen's own fit check is written 121.2 - 67.7 resident weights - reserve = device budget, and its ledger's "68.0 GiB locked in RAM + 11.0 KV pool + 16.5 working" shows up as GTT Used while the CPU counts the same pages. Consequences for anyone proposing a split: (1) capacity is not additive — moving layers "onto the iGPU" maps the same LPDDR pages through GTT, so a 117 GB model (84 GB experts + 33 GB PLE) still needs 117 GB; only the 2 GiB carve-out is private; (2) bandwidth is not additive — both engines share the 256 GB/s; (3) fencing more via amdgpu.gttsize (parm verified: "Size of the GTT userspace domain in megabytes") or the BIOS frame-buffer size subtracts from CPU-visible RAM, so it partitions rather than adds. This is why the split that paid on this box was device-vs-host (3090 GDDR6X for attention/dense/KV, LPDDR for experts via -ot exps=CPU), why iGPU-as-a-pipeline-stage buys nothing, and why "second memory pool" should be stated as "second compute engine over the same bus" — the iGPU's real value is as a whole-model runtime (halogen's ~4x prefill over llama.cpp on this SKU), not as a peer in a multi-device split. _(context: Answering "why is the iGPU not a second memory pool?" and "why was pipeline parallelism not attempted?" for Dankclaw (2026-09-16). Verified live on the host plus dmidecode/modinfo; the substantive correction was that the iGPU has addressable windows (2 GiB carve-out + 114 GiB GTT of a 121.2 GiB pool) but no independent capacity or bandwidth.)_
|
||||
- frank-medium on dankserver (2x Radeon AI PRO R9700, gfx1201) is now one mainline-llama.cpp process, layer-split across both cards, and the reason is worth remembering: **vLLM cannot use both cards for one model here in ANY of its parallelisms** — TP needs an allreduce and PP's stage handoff is GroupCoordinator.send_tensor_dict, which sends metadata over the gloo CPU group but the hidden-state TENSORS over the RCCL device group (vllm/distributed/parallel_state.py, the comment "run on the device backend where is_completed() is reliable"). That is the same p2p primitive this host cannot dispatch, because ROCr refuses a kernel that declares a hidden hostcall buffer on a device without PCIe atomics (gate at rocclr/device/rocm/rocvirtual.cpp, case KernelParameterDescriptor::HiddenHostcallBuffer), and card #2 sits behind the B650 chipset (AtomicOpsCtl: ReqEn-, bridges Routing-). So "use PP to dodge the atomics problem" is exactly backwards. **llama.cpp's --split-mode layer is the path that works**: layers are distributed and the boundary activation is a per-backend copy, once per forward pass — no collective is ever launched. Measured here: 1991 t/s prefill / 18.8 tok/s decode layer-split vs 272 / 26.3 tensor (butterfly fallback). The ~$30 M.2→x4 riser into the free CPU-attached M2_2 is still the only way to get TENSOR parallelism (≈+40% decode), not the way to "use both cards" — that battle is already won in software. Three operational traps from the same cutover, all measured: 1. **rocBLAS/hipBLASLt bundle depth.** Bundling a ROCm runtime beside a llama.cpp binary with cp -r SRC DST/ where DST does not exist creates DST as a copy of SRC and flattens one level; the engine then loads 29 GiB of weights, answers /health, and dies on the FIRST request with rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1201. Device code must live at $ORIGIN/../lib/rocblas/library and .../hipblaslt/library/arch. 2. **An env fi _(context: 2026-09-16 frank-medium cutover on dankserver: vLLM (radiance MXFP4 replicas, then an FP8 + PP attempt) → mainline llama.cpp fb27a525 layer split + DFlash2, 262144 ctx/1 slot. Produced scripts/build-llama-rdna4.sh, scripts/download-frank-medium-gguf.sh, scripts/frank-medium.{env,service}, scripts/serve-frank-medium.sh, scripts/bench-spec-ab.py, tests/frank_medium_unit.bats, serverAppConfigs-docs/L)_
|
||||
- Dankclaw (GMKtec EVO-X2, Strix Halo) has a hard 4-lane budget for BOTH M.2 2280 PCIe 4.0 x4 slots, and the RTX 3090 lives in one of them on an M.2-PCIe riser: kernel boot logs "pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)". The GPU's root port 00:03.1 advertises LnkCap Width x8 (APU GPP bridge IP max) but only 4 lanes are wired at the socket, so the link trains x4; port 00:02.5 (NVMe) is x4 too. Therefore swapping the SSD and the GPU between the two M.2 slots cannot widen the GPU link - both are x4, and the GPU is already behind the wider-capable bridge. THE TRAP: lspci shows the idle NVIDIA GPU as "LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded)" because the NVIDIA driver downshifts the link to Gen1 for power when the GPU is idle - this is NOT a fault. Prove the real link by sampling NVML under load: nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv reads 4,4 while a generation request is in flight (and 1,4 at idle). Corroborating evidence: LnkCtl2 Target Link Speed: 16GT/s on both ends, and nvidia-smi hostmax/gpumax = 4. Also: this box's SMBIOS type-9 slot table is junk (it labels the RTL8125 Ethernet port and the SD reader as "M.2 Socket 3" slots and lists a phantom free slot at bus address 0200:00:03.2, a PCI domain that does not exist) - trust live lspci, not dmidecode -t slot. There is no OCuLink and no add-in slot on the EVO-X2; two M.2 2280 slots is the whole expansion story. _(context: 2026-09-16, user asked whether swapping the dankclaw SSD and the RTX 3090 between M.2 slots would let the GPU run at full speed. Verified live over ssh with lspci -vv, journalctl -k boot log, dmidecode -t slot, and NVML sampling during a real frank-small generation.)_
|
||||
- Dankclaw (GMKtec EVO-X2, NucBox_EVO-X2, Strix Halo APU) has exactly TWO M.2 2280 slots, both PCIe 4.0 x4 on the SoC root complex (no chipset), SATA M.2 unsupported, 8TB max per slot. Both are OCCUPIED: slot 1 = 2TB AirDisk NVMe (MAXIO MAP1608 DRAM-less) at full Gen4 x4 (16GT/s), holding the root FS and all model weights; slot 2 = RTX 3090 (frank-small) in a dock/riser, and that dock negotiates only **Gen1 x4** (2.5GT/s, downgraded from the card's x16 Gen4 LnkCap) — the dock/riser is the likely cap, ~$30 Gen4 riser class of fix. Consequence: unlike dankserver (free CPU-attached M.2_2 = R9700 riser target), dankclaw has NO free M.2 slot, so there is no multi-GPU-via-M.2 path on this box — the only other GPU is the integrated gfx1151. PCI addresses: NVMe c3:00.0, 3090 c4:00.0. _(context: 2026-09-16, live ssh dankclaw: lsblk/lspci/dmidecode; user asked "tell me about the m.2 slots on dankclaw".)_
|
||||
- On a host where the second GPU cannot dispatch RCCL (no PCIe atomic routing on its bridge chain), vLLM cannot use both cards for one model in ANY of its parallelisms: tensor parallelism needs a per-layer allreduce, and **pipeline parallelism needs the same class of primitive** — vLLM's PP stage handoff sends metadata over the gloo CPU group but the hidden-state tensors over the *device* backend (vllm/distributed/parallel_state.py, isend_tensor_dict/recv_tensor_dict; the comment at the tensor-send path says these "run on the device backend where is_completed() is reliable"). So "use PP to dodge the atomics problem" is exactly backwards. The working software path for two cards on one model is **llama.cpp --split-mode layer**, which launches no collective at all: layers are distributed in order and only the boundary activation crosses PCIe, once per forward pass. Also note the kyuz0/amd-r9700-vllm-toolboxes images are stale for this purpose (Docker Hub latest = 2026-06-13, dev = 2026-08-12 vLLM 0.27.1 with dflash/ but no dflash2/), and that switching to such an image changes nothing about the atomics gate because the gate lives in ROCclr (rocclr/device/rocm/rocvirtual.cpp, case KernelParameterDescriptor::HiddenHostcallBuffer → !pcie_atomics_ → hipErrorIllegalState) and the shipped RCCL build is the same one that already failed. Measured on dankserver 2026-09-16: layer split 1991 t/s prefill / 18.8 tok/s decode vs tensor 272 / 26.3 (butterfly fallback). _(context: serverAppConfig 2026-09-16: frank-medium cutover from a vllm-radiance MXFP4 replica pair to mainline llama.cpp layer-split across two R9700s (the user asked first for vLLM + PP, then approved the llama.cpp pivot).)_
|
||||
- frank-medium (dankserver, 2x R9700 gfx1201) is now ONE mainline-llama.cpp process layer-split across both cards — and the decisive insight is that **vLLM's pipeline parallelism is not an escape from the PCIe-atomics wall**: PP sends its hidden-state handoff on the *device* backend (RCCL send/recv, vllm/distributed/parallel_state.py ~1117-1119), the same primitive card #2 cannot dispatch (AtomicOpsCtl: ReqEn- behind Routing- chipset bridges). llama.cpp's --split-mode layer needs no collective at all (layers distributed, boundary activation is a per-backend copy), so it is the one software path that puts both cards on one model: 1991 t/s prefill layer-split vs 272 tensor (butterfly fallback), and it delivered 1.0-1.4k tok/s prefill at 4k-140k tokens with 43.8 tok/s decode. Switching to the kyuz0 R9700 toolbox image changes nothing about this — its librccl is byte-identical in version to the radiance image's (RCCL 2.30.4 compiled with ROCm 7.14.0.0-9999-2b22ab01), and the toolbox's published tags are stale for DFlash2 (latest=2026-06-13, dev=2026-08-12 = vLLM 0.27.1, has spec_decode/dflash/ but no dflash2/); DFlash2 landed in llama.cpp via PR #27342 and is in main at fb27a525. Three durable traps from this deployment: (1) **Bundling a ROCm runtime next to a llama.cpp binary**: cp -r SRC DST/ with a missing DST creates DST as a copy of SRC and drops one directory level — rocBLAS/hipBLASLt look for device code at a fixed depth (../lib/rocblas/library, ../lib/hipblaslt/library/arch), so the flattened copy loads the model, answers /health, and dies on the FIRST request with rocBLAS error: Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201. Declare the destination then copy contents. (2) **DFlash2 in llama.cpp**: --spec-type draft-dflash + --spec-draft-model DFlash2 gguf; the init line block_size=8, n_extract=5, sample_from_anchor=true is the proof it took the DFlash2 path, n_max must be = block_size-1 = 7, and with draft-dflash the target _(context: 2026-09-16, dankserver frank-medium cutover: user asked for kyuz0 R9700 toolbox vLLM + d0xin FP8 + z-lab DFlash2 + pipeline parallelism; investigation showed PP needs the same RCCL p2p that the atomics wall blocks and that a 29.8 GB FP8 checkpoint cannot fit one 31.86 GiB card, so the user pivoted to llama.cpp + ajgazin UD-Q8_K_XL + the DFlash2 GGUF drafter + layer split as the sole LLM runtime. R)_
|
||||
- Dankclaw/Dank fleet serving conventions established 2026-09-16 (serverAppConfig), all verified live: 1. PORT RELEASE RACE (deployer convention). Starting an LLM replacement while the predecessor still holds the published port fails: pasta (rootless podman's userspace net stack) binds the published port at container CREATE, and the ik/LLM server logged "Deactivated successfully" ~1s before pasta still reported Address already in use (unit exit 126); only Restart=on-failure recovered it. Repo fix: wait_port_free port in scripts/deploy-remote.sh polls ss -Htan "sport = :$port" (listener + accepted + TIME_WAIT) and fails loudly with the socket table. Ordering matters: stop this service's OWN quadlet BEFORE waiting, or a redeploy deadlocks against itself. 2. THINKING-LEVEL TRAP. A model's own chat template can reject the harness's reasoning level: Qwen3.8's template accepts only xhigh|medium|low for reasoning_effort and raises — HTTP 500 naming Jinja, not the model. Subagents run at high, so every subagent on that endpoint died until compat.reasoningEffortMap folded minimal/low→low, medium→medium, high/max→xhigh in models.yml. Halogen accepts the wider set (verified minimal/low/medium/high/xhigh all 200), so per-provider entries legitimately differ. Corollary: probe each reasoning level against a new endpoint before routing roles at it. 3. TOOLS DEPEND ON THE RUNTIME, NOT THE MODEL. The ik_llama.cpp runtime rejected every request carrying tools (500 tools param requires --jinja flag), silently breaking every tool-using subagent on that endpoint; mainline llama.cpp defaults --jinja on, and halogen accepts tools and returns tool_calls. Always test a tool-call round trip before pointing task/smol roles at an endpoint. 4. TWO ENDPOINTS, ONE HOST = two providers in .omp/agent/models.yml (one provider per HOST:PORT; the file cannot express same-host-different-port), plus a separate modelRoles entry per role. The harness config lives with the harness on the dev box (/home/blank _(context: Dankclaw re-split 2026-09-16: halogen restored as frank-large (pool 400384 = 2x200192, 2 slots, MAX_TOK 24576 for the co-tenant) and frank-small created (mainline llama.cpp on the RTX 3090, Qwen3.8-27B GSQ-RCO IQ3_S -mtp + BF16 mmproj, port 7771). Failures: pasta bind race on cutover, subagent tools 500 from ik, reasoning_effort high rejected by the Qwen3.8 template.)_
|
||||
- Dankclaw now serves TWO LLM endpoints and the split produced three durable traps. (1) SHAPE: frank-large = halogen-flash-server 0.5.8 rootless Quadlet on :7770 (gfx1151 iGPU), retuned 2026-09-16 to HALOGEN_CTX=200192 + KV_POOL_POSITIONS=400384 (pool == 2 × ctx, so two 200k conversations are exactly resident) + KV_SLOTS=2 + MAX_TOK=24576 + HOST_RESERVE_GIB=20, VISION_TOWER unset; frank-small = mainline llama.cpp 37b53fd root systemd unit on :7771 (RTX 3090), 131072 ctx / 1 slot / f16 KV, alias frank-small. Measured: halogen budget 121−67.7−20=33.5 vs need 27.2 (no clamp), 95.4 GiB held, 14.7 GiB left, co-tenant stall NOTE prints again with 4,350→2,199 compaction stalls; frank-small 22,362 MiB of 24,576. (2) PORT RACE: rootless podman publishes through pasta, which binds the published port at container CREATE — a just-stopped predecessor still holds it (exit 126, "Address already in use"), so the deployer must stop the service's own unit, then poll ss -Htan 'sport = :PORT' empty (listener + accepted + TIME_WAIT) before installing/starting, and must stop its own quadlet first or a redeploy deadlocks against the gate. (3) TEMPLATE/TOOL TRAPS that break subagents invisibly: ik_llama.cpp serving frank-large returned 500 tools param requires --jinja flag for every request carrying tools (every tool-using agent on that endpoint was broken and it read like a model fault; mainline llama.cpp 0.4.1 defaults --jinja on, halogen accepts tools); and Qwen3.8's own GGUF template accepts ONLY xhigh|medium|low for reasoning_effort and raises a 500 otherwise — harness subagents run at high, so models.yml needs compat.reasoningEffortMap {minimal:low, low:low, medium:medium, high:xhigh, max:xhigh} for any endpoint whose template is that strict (halogen accepts all five, so the two provider entries legitimately differ). One provider per HOST:PORT in models.yml, and one role per endpoint: smol/tiny/task/vision → frank-small, slow/plan/designer/advisor → frank-large. _(context: 2026-09-16: re-split Dankclaw's inference (halogen restored as frank-large; new llama.cpp frank-small on the 3090), live cutover + docs; three defects found by driving the fleet rather than reading config.)_
|
||||
- dankserver's two R9700s cannot cooperate through RCCL, and the escape is a ~$30 piece of hardware. Causes, all verified: (1) ROCclr gates the hostcall buffer RCCL collective kernels need on a *queried* hardware property — Device::checkAtomicSupport() in rocdevice.cpp:181 reads LinkAttribute::kLinkAtomicSupport from the CPU fine-grain pool and only then sets info_.pcie_atomics_ = true; there is no else branch and no env/driver/sysfs override anywhere near the gate (grepped ROCclr for getenv/HSA_*/ROCR_*), and the value is consumed at rocdevice create time so setpci writes are useless. (2) The board's second x16 slot is chipset-attached (AtomicOpsCap: Routing- on both bridges, endpoint ReqEn-). ASUS TUF GAMING B650-E WIFI lane wiring from ASUS's own techspec: PCIEX16_1 = CPU x16; PCIEX16_2 = B650 chipset, x16 slot in **x1 mode**; M.2_1 = CPU PCIe 5.0 x4 (boot NVMe, occupied); **M.2_2 = CPU PCIe 4.0 x4 and free**; M.2_3 = chipset x4. No x8/x8 bifurcation on this board. So an M.2→PCIe x4 riser into M.2_2 is the cheapest route to two CPU-attached cards and therefore to real TP=2; a CPU x8/x8 board (X670E/X870E class) is the better-bandwidth alternative. Bandwidth is not the blocker: a 27B TP=2 allreduce at batch 4 moves ~10 MB per token-step, ~1.4 ms at Gen4 x4. (3) Empirically, RCCL launches NOTHING involving the chipset card: rank0 (CPU card) completes broadcast/all_reduce/barrier while rank1 fails, and dist.send/recv — the primitive pipeline parallelism would use — fails too, so PP=2 is dead as well, not just TP=2. (4) vLLM's one collective-free path (IPC custom allreduce, custom_ar=True compiled in the radiance image) is gated off for ROCm RDNA4 purely by a hardcoded CUDA-sm-keyed table CUSTOM_ALL_REDUCE_MAX_SIZES = {9.0, 10.0, 10.3, 10.7} while RocmPlatform.get_device_capability() returns DeviceCapability(major=12, minor=0) → '12.0' not in table → False; patching that table is NOT sufficient, because broadcast/barrier/all_gather on the TP init path have no non-RCCL _(context: 2026-09-16 dual-R9700 TP=2 research on dankserver (ASUS TUF GAMING B650-E WIFI, Ryzen AM5, 2x Radeon AI PRO R9700). Produced the ranked options and the source-level proofs.)_
|
||||
- NEVER run RCCL/NCCL collective experiments on a live GPU host that is serving production. On 2026-09-16 I ran a 2-rank RCCL probe (torch.distributed, backend=nccl, both R9700s) on dankserver — the production LLM host — *while frank-medium was serving*, reasoning that the probe needed only ~200 MB of free VRAM and that a failure would just error out. It hit the known PCIe-atomics wall ('rank1 broadcast/all_reduce/barrier/send-recv FAIL', 'rank0 OK'), the process hung until my 200-300 s timeout killed it, and within ~15-20 minutes the whole host was hard down: no ARP reply (ip neigh FAILED from a peer), 100% ICMP loss from two subnets, every port closed, and it did not come back over 25+ minutes — i.e. powered off or kernel-wedged, needing a physical power cycle (consumer board, no BMC). A hung AQL dispatch / failed hostcall on the atomics-less card is a plausible cause of a GPU-driver wedge that takes the box down, and this host has a documented precedent (the 2026-08 pinned-host-buffer incident which 'starved co-tenants into unreachability until a power cycle'). The correct procedure for any multi-GPU collective experiment is the benchmarking policy: sudo systemctl stop frank-medium && sudo systemctl mask frank-medium first (mask, never disable), then experiment, then unmask and verify /health — and ideally experiment on a host that is not serving users at all. Two secondary lessons from the same hour: (1) when a host goes unreachable, ip neigh show ip from a peer distinguishes 'powered off / NIC dead' (FAILED) from 'kernel alive but wedged' — ARP usually still answers on a merely wedged box; (2) validate the *naming* of your own probes before reporting an outage: I briefly believed Gitea was down on dankclaw because systemctl --user is-active gitea returned 'inactive', but the unit is gitea-server and the container was 'Up 9 hours (healthy)' — a check against a non-existent unit name produced a false outage report. _(context: dankserver hard-down for 25+ min on 2026-09-16 during the dual-R9700 TP=2 research pass, ~20 min after running a bare 2-rank RCCL collective/P2P probe on its GPUs while frank-medium was live.)_
|
||||
- In serverAppConfig, the highest-yield overnight work (4 of 4 passes found something real) came from one audit family: **surfaces whose signals were already wrong before I arrived**. Concrete recurring defects and their fixes: (1) Homepage status cards can be red *by construction* — the checker runs on Dankclaw, so it reaches only that host's loopback, its container names, and *published* LAN addresses; mariadb/valkey/Postgres probed container-internal DB ports and could never be up. Rule now in Infrastructure/Homepage.md: a card must be probeable from the monitor's host or be wip (Gluetun was the pre-existing example). (2) Prometheus jobs comfyui, services, gitea scrape HTML or a 404 /metrics — a scrape cannot express "is this HTTP service answering?"; remedy is the blackbox exporter (/probe?target=&module=), which also gives guarded vhosts an uptime signal since Prometheus cannot solve an Anubis challenge. (3) **Retiring a file makes every assertion that greps it pass vacuously** — with dankclaw/compose.yaml gone, four bats tests (observability_quadlet, comfyui_rocm, caddyfile×2, authentik dead var) passed by absence (a missing path makes grep exit 2 and ! succeed). Fix pattern: assert the invariant directly ("no dankclaw compose file competes with the quadlet units") *and* keep a conditional live check for a re-added entry — then prove teeth by recreating the file and watching the tests go red. (4) A linter must be validated against the resolver it models before acting on its output: my vault-link checker produced 45 false positives (Obsidian wikilinks are case-insensitive, [[folder/Note]] resolves by path suffix not stem, %20 in markdown links is URL-encoding) before reaching the 2 real breaks. It now lives at scripts/check-vault-links.py with a docs_completeness seam. (5) Your own change invalidates prose and unit comments: check the repo's rule files first — AGENTS.md claimed "ComfyUI runs CPU-only" (false since the GPU unit was restored) and "observability RUN _(context: serverAppConfig overnight session 2026-09-16: found a permanently-red homepage board, three unprobeable Prometheus jobs, four hollow tests after retiring dankclaw/compose.yaml, three dangling vault links, and a flatly wrong AGENTS.md lesson about ComfyUI. Suite 315 green, e2e 32.)_
|
||||
- In serverAppConfig, grep-style bats seams and doc claims silently rot whenever a file they reference is retired — the audit that keeps finding real defects is "which signal here cannot be correct?" Three concrete traps, all hit on 2026-09-16: (1) VACUOUS ASSERTIONS — after dankclaw/compose.yaml was retired, four tests that ended in ! grep -qE '^\s+svc:' "$REPO_ROOT/dankclaw/compose.yaml" passed because grep exits non-zero on a MISSING FILE, so the negation succeeded. Fix pattern: assert the real invariant ([ ! -f "$cf" ]) in ONE place plus keep a live conditional check ([ ! -f "$cf" ] || ! grep ...) for a re-added entry, and TEETH-TEST by recreating the file and watching the tests go red. (2) PERMANENTLY-RED PROBES: homepage cards and Prometheus jobs probing container-internal or HTML endpoints can never report up (mariadb 192.168.0.235:3306, valkey/postgres 127.0.0.1:6379/:5432 unpublished; comfyui/services scrape / = HTML; gitea /metrics 404). Rule: a monitor signal must be probeable from the monitor's host or be marked wip/none — an always-red signal trains the operator to ignore the surface. (3) STALE CLAIMS AFTER YOUR OWN CHANGE: retiring a file or adding a feature invalidates prose — AGENTS.md was telling agents "ComfyUI runs CPU-only, no ROCm wheels for gfx1151" after ComfyUI had been restored with GPU access (an agent trusting it would re-break it), and it still said observability was "RUNNING BUT UNCONFIGURED" plus named halogen as the active frank-large runbook. Grep the repo for every reference to anything you retire/change, not just code. When auditing a docs vault, validate the LINTER against the resolver it models before acting: a naive Obsidian link checker reported 45 breaks of which only 3 were real (case-insensitivity, path-suffix resolution for [[folder/Note]], and %20 URL-decoding in markdown links were the three false-positive classes). scripts/check-vault-links.py (docstring records all three) plus the no unresolved links in the docs vault seam _(context: Overnight 2026-09-16 passes on serverAppConfig: homepage board audit (3 impossible cards), vault link integrity + checker, four hollow bats assertions re-armed, AGENTS.md/doc stale-claim sweep after retiring dankclaw/compose.yaml and adding alert rules.)_
|
||||
- Fleet convention: a monitoring/status signal must be probeable from the vantage point of whatever produces it, or it must be marked intentional (wip) — an always-red signal teaches the operator to ignore the surface it lives on. Instances found on 2026-09-16: (1) the homepage status checker runs on Dankclaw, so it can reach that host's loopback, its container names, and published LAN addresses — but NOT container-internal ports of another host; three cards (mariadb probing 192.168.0.235:3306, valkey 127.0.0.1:6379, Postgres 127.0.0.1:5432) were red by construction while the databases were healthy, and publishing a DB port on the LAN to make a badge green is not a trade this fleet makes, so they became wip with the reason in their description (Gluetun/WireGuard-UDP was the pre-existing example of the same call); (2) four Prometheus jobs read down and three always will — comfyui and the two services probes scrape HTML pages (Prometheus refuses to parse HTML, error 'unsupported Content-Type text/html'), and gitea's /metrics is a 404 — none is an outage, and the tempting fix (deleting the job) is refused because an absent scrape and a healthy one look identical; the documented remedy is a blackbox exporter in probe mode, and Gitea's metrics enabled or the job repointed; (3) a bats test failed on a clean tree for many sessions ('comfyui Containerfile ...') because its selector did find | head -1 and read the wrong file — a permanently red test trains everyone to ignore red, so it was fixed at the root rather than excused. Diagnostic pattern that works: for every signal that is not green, ask what produces it and from where; if the producer cannot reach the target, it is a config defect, not an outage — classify before touching. When fixing, prefer honest state (wip / documented) over widening exposure or deleting the signal. _(context: Dankclaw/Dankserver 2026-09-16 night: audited the homepage board (31 cards, 3 impossible-red DB cards), the Prometheus target list (7/11 up, 3 impossible), and the bats suite (one chronic red test); fixed each and documented the rule in the Homepage and Observability pages.)_
|
||||
- Migrating a compose-only container to a boot-persistent Quadlet on this fleet has three mechanical traps that bit on 2026-09-16, all invisible until something later breaks — a skill for the general procedure already exists (compose-to-quadlet-cutover), so these are the additions to remember. (1) THE CONTAINER, NOT THE SERVICE, IS WHAT FAILS: restart: unless-stopped (or Restart=always) restarts the *existing* container with its *old* config; when the compose service later gains GPU/devices/volumes, the running container keeps the stale config forever. ComfyUI died 9,146 times this way — created before devices:/group_add: were added, so it started with no /dev/kfd, no /dev/dri, no keep-groups and failed on "No HIP GPUs are available" while the YAML on disk looked perfect. Diagnosis: podman inspect c --format '{{.Config.CreateCommand}}' (the argv actually used) and {{.Created}} — NOT .HostConfig.Devices, which reported [] for a container that demonstrably had the devices (verified: torch saw the GPU, --device /dev/kfd was in the create command). Fix: remove the stale container so the unit's own can take the name. (2) A HAND-WRITTEN UNIT COLLIDES BY NAME with the one the generator produces (~/.config/systemd/user/comfyui.service shelling out to podman compose up -d vs the generated comfyui.service); retire it — stop, disable, move to ~/.AGENTTRASH — and never let both exist. (3) ONE OWNER PER CONTAINER NAME: delete the compose service, and rewire the deploy path in the same commit — planner allowlist (DANKCLAW_SERVICES/DANKSERVER_SERVICES) + a path rule for target/svc/**, the dispatcher's service_is_allowed entry, a handler that installs/restarts/health-waits, and install_quadlet pointed at "$TARGET/observability/quadlet"-style per-target trees rather than a hardcoded host. When a target ends up with zero compose services, retire the file rather than leaving a services: key with nothing under it (the compose-validity seam asserts ≥1 service). (4) SEAMS THAT READ COMPOSE _(context: Dankclaw/Dankserver 2026-09-16: migrated Dankserver node-exporter and Dankclaw ComfyUI from compose to Quadlet (ComfyUI had 9,146 restarts from a stale device-less container), retired Dankclaw's now-empty compose.yaml, and found two services whose test seams read the stale compose while the unit diverged.)_
|
||||
- Before tightening a container's bind address, enumerate EVERY consumer of it — a bind is a contract with something, not just an exposure choice. On 2026-09-16 I set HERMES_WEBUI_HOST=127.0.0.1 on dankclaw's hermes-webui (reasoning: the compose service it replaced published loopback, and the repo policy favours internal-only), and broke hermes.franksdank.cloud with a 502: that unit is HOST-NETWORKED (it shares the loopback with the chrome-devtools CDP daemon), so Caddy cannot reach it by container name and fronts the guarded vhost with reverse_proxy 192.168.0.143:8787 — the host's LAN address, recorded in dankclaw/Caddyfile with the same explanation. With network_mode host there is no Podman port forwarder, so the app's bind IS the only path in: the wildcard serves both consumers (Caddy on the LAN address, the SSH tunnel on the host's localhost), and loopback serves only the latter. The doc's own gotcha already said this was "an exposure change to decide deliberately, not a silent fix" — read the service's doc before changing its bind. Two process lessons from the same near-miss: (1) the e2e suite stayed 100% green through it because playwright.config.ts matches specs BY NUMBER PREFIX, not glob, so a new spec file that is not registered in a project silently never runs — register the prefix and let tests/e2e_scaffold.bats fail if any spec matches no project; (2) for every guarded vhost, at least one gate should assert that the vhost reaches its app through the guard chain (status 500 is the load-bearing check: a dead upstream is a 502 from Caddy), especially for services whose upstream is a host address rather than a container name. _(context: Dankclaw 2026-09-16: tightened hermes-webui's bind to loopback, broke the hermes.franksdank.cloud vhost (502) with the whole e2e suite green; reverted, added specs/17-hermes-vhost.spec.ts, registered it, and added a guard that every spec matches a playwright project.)_
|
||||
- On this fleet, "which container will not come back after a reboot?" has a one-line answer: every running container should carry a PODMAN_SYSTEMD_UNIT label; anything without one is compose-only or bare and will die silently at the next boot/shutdown. Audit with for c in $(podman ps --format '{{.Names}}'); do printf '%s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"; done and compare against the host's ~/.config/containers/systemd/*.container and the repo's host/svc/quadlet/. THREE separate instances of this failure mode were found on 2026-09-16 alone: (1) Dankserver's node-exporter — compose-only, Exited (143) for 37 h, no unit, Prometheus's node-dankserver job scraping nothing, and the compose form bound the wildcard address; fixed by adding dankserver/observability/quadlet/node-exporter.container (loopback + LAN 9100, the deliberate LAN listener posture). (2) ComfyUI on Dankclaw — its container was created BEFORE the GPU declarations were added to the compose service and restart: unless-stopped then kept restarting that stale, device-less container for 9,146 restarts (RuntimeError: No HIP GPUs are available); the service was fine, the container was old. Fixed by a Quadlet unit + deleting the compose service. (3) hermes/hermes-dashboard/hermes-webui were ALREADY quadlet-owned (labels proved it) while dankclaw/compose.yaml still defined them — a second owner for the same container names, which is how a stray podman compose up creates a duplicate that fights the unit. Generalisations worth keeping: **editing a compose service does not replace a running container** (restart policy will keep the old config alive forever — recreate explicitly); **podman inspect --format '{{.HostConfig.Devices}}' is not a reliable device-passthrough check** (it read devices=0 on a container that demonstrably had --device /dev/kfd — use {{.Config.CreateCommand}} for the real argv, and confirm from inside with e.g. podman exec c python3 -c "impo _(context: 2026-09-16 night session on the serverAppConfig fleet (Dankclaw + Dankserver): restored Dankserver's node-exporter (37 h dark) and ComfyUI (9,146 restarts, 7 h down) as boot-persistent Quadlets, audited every running container's systemd owner, and found that all three hermes services were already units while compose.yaml still duplicated them. Verified fixes live: node_* targets up, ComfyUI health)_
|
||||
- The recurring outage class on this fleet is a container whose RUNNING config has drifted from its DEFINITION, kept alive by a restart policy: compose restart: unless-stopped (or a hand-written systemd unit calling podman compose up -d) restarts the stale container forever instead of recreating it from the edited definition, so a service can sit broken for days looking merely "unhealthy". Found twice on 2026-09-16: Dankserver's node-exporter had been Exited (143) for 37h with no unit at all (compose-only, dead since a reboot), and ComfyUI had restarted 9,146 times because its container was created 2026-09-15 14:12 BEFORE the /dev/kfd //dev/dri/keep-groups declarations were added to the compose service — so torch reported "No HIP GPUs are available" on every start while the *definition* was correct. Diagnostic trap that nearly produced a wrong root cause: podman inspect --format '{{.HostConfig.Devices}}' reported EMPTY for a container that demonstrably had --device flags; the trustworthy view is --format '{{.Config.CreateCommand}}' (the actual argv). Also: find who started a container via the PODMAN_SYSTEMD_UNIT label — a podman-compose@… label means compose owns it, regardless of what unit files exist. The durable fix in this repo is the convention already written down: one owner per container name (delete the compose service), a repo-tracked Quadlet under host/svc/quadlet/ installed by the deployer, and retirement of any legacy compose-era unit file into ~/.AGENTTRASH (a leftover ~/.config/systemd/user/svc.service COLLIDES by name with the unit the Quadlet generator produces). Remediation ordering that worked: build image - install unit - stop/disable+move the legacy unit - podman rm -f the stale container - daemon-reload - restart unit - health-wait - then verify the real capability INSIDE the container (e.g. torch.cuda.is_available()), not just that it is Up. _(context: serverAppConfigs, 2026-09-16 night: restored Dankserver node-exporter (37h dark, compose-only) and ComfyUI (9,146 restarts, stale device-less container) by migrating both to repo-tracked Quadlet units; also documented that three Prometheus jobs (comfyui + two services probes scraping HTML, gitea /metrics 404) can never be green.)_
|
||||
- Two durable facts from the 2026-09-16 night (Dankclaw/Dankserver fleet). (1) TRAP: podman inspect c --format "{{len .HostConfig.Devices}}" reports 0 and .HostConfig.GroupAdd reports [] even for a container that WAS started with --device /dev/kfd --device /dev/dri --group-add keep-groups. I nearly misdiagnosed ComfyUI as "no GPU devices configured" from that field. The ground truth is the actual argv: podman inspect c --format "{{.Config.CreateCommand}}" (or podman exec c python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" for the behavioural truth — which printed True | Radeon 8060S Graphics once fixed). (2) ROOT-CAUSE PATTERN for "a service crash-loops after a reboot while its config looks right": a container created by an EARLIER revision of the compose service (before devices:/group_add: were added) is kept alive by restart: unless-stopped; podman compose up -d does not recreate it, so it restarts forever with the OLD config (ComfyUI: RuntimeError: No HIP GPUs are available, 9,146 restarts, ~7h down). Decisive check: compare the container's creation timestamp against when the config gained the missing lines, and read .Config.CreateCommand for the flags it actually has. Fix on this fleet is not "recreate the compose container" but migrating the service to a repo-tracked Quadlet (the repo's own rule: every container is a boot-persistent, self-healing unit with repo provenance). Result: ComfyUI now runs from dankclaw/comfyui/quadlet/comfyui.container — image comfyui-rocm:gfx1151, /dev/kfd + /dev/dri, GroupAdd=keep-groups, four comfyui-data volumes, loopback-only publish 127.0.0.1:8188 (unguarded service = internal-only; homepage card probes loopback), Restart=always, WantedBy=default.target, healthcheck via a mounted file (the podman 5.x v1 generator splits HealthCmd on spaces, so inline python3 -c "..." is a quoting trap). Unit is generated, healthy, restarts=0, and curl 127.0.0.1:8188 → 200. Also fixed the same night: Dankser _(context: Dankclaw ComfyUI outage found while auditing Prometheus targets during the 2026-09-16 frank-medium night shift: 9,146 restarts, stale device-less compose container, migrated to a Quadlet; verified hip available True. Dankserver node-exporter restored the same way.)_
|
||||
- Fleet state after the 2026-09-16 night session on serverAppConfigs (frank-medium + observability work): (1) DANKSERVER NODE-EXPORTER RESTORED — it had run only from dankserver/compose.yaml, died Exited (143) at a reboot and left Prometheus' node-dankserver job scraping nothing for ~37 h; it is now dankserver/observability/quadlet/node-exporter.container (generated, Restart=always, binds exactly 127.0.0.1:9100 + 192.168.0.235:9100, no wildcard — verified with ss -ltn), the compose service is deleted (one owner per container name), and deploy_observability now installs from "$TARGET/observability/quadlet" instead of a hardcoded Dankclaw path. Dankclaw keeps five units there, Dankserver one. (2) PENDING FOR FRANK — ComfyUI on Dankclaw is crash-looping: 9,146 restarts, exit 0, log says RuntimeError: No HIP GPUs are available, unit shows active (exited) while the container respawns; deliberately not touched because the fix depends on which posture Frank wants (its documented posture is CPU-only --cpu, which contradicts that error). (3) THREE PROMETHEUS TARGETS ARE CONFIGURATION DEFECTS, NOT OUTAGES: gitea returns 404 on /metrics (service itself fine), and the two services targets (SearXNG :8081, OpenWebUI :3005) answer 200 with Content-Type text/html, which Prometheus 3 refuses to parse — "blackbox-style probes" without a blackbox exporter are inert; the observability doc's own blackbox-exporter note is the fix. So "7 of 11 targets up" is the expected reading, and an alert on up==0 across all jobs would be permanent noise (the LLM rules are scoped to frank-* and are unaffected). (4) Also this session: tests are GREEN (313, ALL TESTS PASSED) — the long-standing "comfyui test 73 fails on a clean tree, ignore it" belief is now false and must not be repeated; the homepage card for Dankserver :7770 was mislabelled frank-small (retired) and is now frank-medium; Grafana has a provisioned Prometheus datasource + an LLM fleet dashboard; Prometheus evaluates 5 LLM alert rules but _(context: serverAppConfigs, 2026-09-16 pre-dawn session: frank-medium (2x R9700, vllm-radiance, two TP=1 replicas behind an adaptive-routing proxy) plus an observability audit that found the node-exporter gap, the mislabelled homepage card, a red test that had been excused for sessions, and a crash-looping ComfyUI.)_
|
||||
- Durable fact for this repo (serverAppConfigs): the measurement harnesses for the CURRENT LLM serving stack now live in git — do not rewrite them in /tmp. Four files, added 2026-09-16 with their traps documented in each header: scripts/bench-vllm-corpus.py (builds a code-shaped corpus from the repo's own sources; synthetic --dataset-name random prompts misreport speculative draft acceptance badly — measured 2.9-3.6 tokens/update on random against ~5.8 on code, so tuning SPEC depth on random data picks the wrong value), scripts/bench-vllm-serve.sh (concurrency sweep via vllm bench serve from inside the serving image; it exists mainly to encode two traps — the client needs --tokenizer /models/... with /models mounted, because with only --model frank-medium it tries to resolve that alias on HuggingFace and dies before sending a request, producing a run that looks like a benchmark and reports nothing; and --ignore-eos output is not comparable with runs that lack it), scripts/bench-turn-latency.py (per-turn TTFT for an agent-loop shape — the instrument that decided the deployed prefix-affinity routing policy, and the fastest way to demonstrate cold-vs-warm prefill: measured 12.5 s cold vs 1.4 s warm on the same ~50k-token prompt), and scripts/bench-llm-soak.py (sustained multi-family load; this is what found the two-card capacity wall). Prompts in every harness carry a salt as their leading bytes, because that changes both the router's identity window and vLLM's prefix-cache key — without it a run silently inherits a warm cache from the previous run and reports fantasy numbers. Also durable: scripts/*.py are now compiled by the scripts_syntax seam (previously only scripts/*.sh were checked), and the whole suite is green (311 tests) with no known-failure exceptions to explain away. Older llama.cpp-era harnesses (bench-decode*, bench-spec-decode.py, bench-threads-batch*, generate-prompts.py) are still in scripts/ and target a different runtime's endpoint — check which stack _(context: Night session 2026-09-16 on Dankserver frank-medium (vllm-radiance MXFP4, two R9700 replicas behind an adaptive-routing proxy): after building and using throwaway measurement scripts for hours, the useful four were committed to scripts/ and smoke-tested against the live fleet, and the script-syntax seam gained its missing python half.)_
|
||||
- CORRECTION to the long-standing belief recorded in this project's memory that "./tests/run.sh exits 1 on a clean tree because of a pre-existing comfyui test-73 failure, and that is expected, so don't treat a red suite as blocking." That is now FALSE as of 2026-09-16: the test was fixed properly and the suite is GREEN (310 tests, ALL TESTS PASSED). Root cause of the old failure was a selector bug in tests/comfyui_rocm.bats — find dankclaw -name 'Containerfile*' | head -1 returned whichever Containerfile the directory listing happened to put first (dankclaw/alloy/Containerfile), so the assertion read a file that never claimed to be ComfyUI's and failed; the same selector would have passed on a wrong file. The fix resolves the Containerfile the way the build does — build.context joined with build.dockerfile, read from dankclaw/compose.yaml — so the test checks the file the deploy actually builds. The generalisable lesson: in this repo a test that guesses a path with find ... | head -1 is a bug waiting to be believed as "a known failure"; read the contract (compose, unit file, config) instead of the filesystem. Also: a permanently red test trains every operator and every future agent to ignore red — treat "known failure" as a defect with an owner, not as a baseline. A dead duplicate was retired alongside it (dankclaw/comfyui/Containerfile was referenced by nothing — compose builds Containerfile.comfyui), staged in .AGENTTRASH/ awaiting Frank's purge decision. _(context: serverAppConfig repo, 2026-09-16 night session: fixed tests/comfyui_rocm.bats test "comfyui Containerfile is based on the gfx1151 torch image" after it had failed across many sessions; suite went from 285 ok + 1 not ok to 310 ok, ALL TESTS PASSED.)_
|
||||
- Dankserver frank-medium as of 2026-09-16 is NOT a single tensor-parallel process: it is two single-GPU TP=1 replicas of the vllm-radiance MXFP4 stack (frank-medium-a on R9700 #1 :7771, frank-medium-b on #2 :7772) behind a prefix-affinity proxy (frank-medium-proxy :7770, routing.py policy + affinity-proxy.py server), each at 114688 context / 4 sequences / CHUNK=8192 / KV_MEM=0, with the DFlash2 drafter at SPEC=7. TP=2 is impossible on this host for a hardware reason: RCCL's collective kernels carry a hidden hostcall-buffer parameter and ROCr refuses to dispatch them on a device without PCIe atomics, and the second R9700 sits behind chipset bridges 05:00.0/06:09.0 advertising AtomicOpsCap: Routing- (read-only) — so no software enables atomics there while the first card, on CPU root port 00:01.1, has them. The practical capacity ceiling per card is 122,636 KV tokens (the engine's own ledger), i.e. about two 30k-token conversations; three concurrent 50-80k contexts thrash the prefix cache (turns alternate ~1.4 s warm / 22-45 s cold) and neither load-balanced placement nor per-card admission control fixes it, because a running sequence and its cached prefix draw on the same pool. CHUNK is not a lever: available KV is 5.44 GiB at 8192 versus 4.22 at 16384 and 4.16 at 4096 (which refuses to boot), so 8192 is a sweet spot. Long-prefix caching itself is healthy — the same ~50k-token prompt sent twice to an idle card is 22.3 s then 1.4 s — and the engine's lifetime "Prefix cache hit rate" line is an average over all traffic ever scraped, not a verdict on any request. The fleet's Prometheus scrapes both LLM endpoints (frank-medium and frank-large) and five rules watch them; nothing is notified because there is no Alertmanager, so alerts only appear on Prometheus' /alerts and the dashboard's ALERTS panel. _(context: Night-long session on Dankserver frank-medium: TP=2 ruled out by PCIe atomic routing, deployment reshaped as two TP=1 replicas + adaptive-routing proxy, capacity measured, four optimization hypotheses tested (routing and failover shipped; placement, admission, CHUNK variants rejected with numbers), then Prometheus scraping + Grafana provisioning + five alert rules added.)_
|
||||
- On the Dankclaw fleet, Grafana is configured as CODE, not by clicking: shared/grafana/provisioning/{datasources,dashboards} is installed to /opt/grafana/provisioning by copy_grafana_provisioning() in scripts/deploy-remote.sh (the grafana quadlet already mounts that path read-only), and scripts/deploy-plan.py maps any shared/grafana/** path to the grafana service, so ./scripts/deploy.sh dankclaw --service grafana is the whole apply step. VERIFICATION TRAP (cost me a false negative): a *provisioned* dashboard does NOT appear in the legacy dashboard table of grafana.db on the current Grafana build — dashboards live in unified storage behind the dashboard.grafana.app API group — so querying that table returns empty and looks like the dashboard failed to load. The datasource DOES appear in the data_source table, which makes the asymmetry easy to misread. Correct verification, in order: podman logs --since 10m grafana | grep -i "inserting datasource\|provision dashboards" (the provisioning paths log), then the API — curl -u admin:admin http://127.0.0.1:3002/api/search?type=dash-db for dashboards and /api/datasources/proxy/uid/uid/api/v1/query?query=... to prove the datasource actually reaches Prometheus. Grafana on this host publishes only on loopback :3002 (the off-host path is the guarded vhost), and its admin login is still the default admin:admin — worth changing, and worth remembering because every metric/dashboard assertion needs it. One more habit that paid off twice in this repo: before writing any dashboard panel or config comment, grep the metric names out of the live /metrics endpoint (curl -s host:port/metrics | awk '{print $1}' | sort -u | grep word); a comment naming vllm:prefix_cache_hit_rate shipped once and that series does not exist (the real pair is prefix_cache_hits_total / prefix_cache_queries_total). _(context: Dankclaw 2026-09-16: provisioned the Prometheus datasource plus an "LLM fleet — capacity and reuse" dashboard from git (shared/grafana/provisioning/*), after discovering Grafana had zero datasources so the newly scraped vLLM series were only reachable through the Prometheus API. Verified via the Grafana API after the legacy dashboard table gave a false empty.)_
|
||||
- frank-medium on Dankserver (192.168.0.235:7770) is no longer one tensor-parallel process: since 2026-09-16 it is TWO single-GPU vLLM replicas (frank-medium-a on :7771, frank-medium-b on :7772) behind a prefix-affinity proxy on :7770, because TP=2 cannot launch a collective on this host (RCCL kernels need PCIe atomics; the second R9700 sits behind chipset bridges advertising AtomicOpsCap: Routing-). Each replica is TP=1 at 114688 context / 4 sequences / CHUNK=8192 / SPEC=7 with vLLM-profiled KV. Concrete consequences a future session must not re-derive: the harness entry in /home/blank/.omp/agent/models.yml is contextWindow 114688 / maxTokens 32768 and omp --model frank-medium -p "..." works end to end; per-card KV is 122,636 tokens, so two concurrent large contexts are comfortable and three thrash; the deploy handler installs a/b/proxy and moves any stale TP=2 unit into ~/.AGENTTRASH; Prometheus now scrapes both LLM endpoints (frank-medium vLLM, frank-large llama.cpp) on their LAN addresses; and the trio is proven boot-persistent by an actual unattended reboot (active 59 s after boot, healthy ~3.5 min, no manual step). Full evidence lives in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md — read it before assuming TP=2 or the retired ROCmFPX deployment. _(context: frank-medium deployment and fleet wiring, 2026-09-15/16: TP=2 → two TP=1 replicas + adaptive proxy, harness/models.yml update, Prometheus scrape jobs, reboot verification.)_
|
||||
- Benchmarking an LLM serving deployment honestly (used repeatedly on frank-medium 2026-09-16, applies to any future swap incl. frank-large): (1) vllm bench serve needs an explicit --tokenizer local path with /models mounted — passing only --model alias makes the client resolve the alias on HuggingFace and die before sending a request, a green-looking run that produces no numbers; (2) synthetic --dataset-name random under-reports speculative decoding badly (2.9-3.6 tok/update measured vs 5.8 on code) so tuning SPEC depth on it picks the wrong default for a code workload — build the corpus from the repo's own sources instead (JSONL of {"prompt": instruction + file body}); (3) vLLM's Prefix cache hit rate: log line is a LIFETIME average over every prompt that replica ever saw, not a per-request verdict — to learn whether long-prefix reuse works, send the same prompt twice to an idle card and compare TTFTs (measured: 22.3 s cold → 1.4 s warm on a 50k-token prefix); (4) A/B arms must use different leading bytes (a per-arm salt at the front of every prompt), otherwise the second arm silently reuses the first arm's warm cache; (5) repeat each arm at least twice — I saw 5-10% run-to-run variance, and a single measurement was enough to make me nearly ship the wrong SPEC depth; (6) when a proxy sits in front of several replicas, measure the workload shape the fleet actually runs (sequential conversation turns) separately from batch benchmarks: a cold batch of distinct prompts favors spreading across cards, sequential turns favor pinning to one, and the two conclusions pull in opposite directions. _(context: frank-medium (2x R9700, two TP=1 vLLM replicas behind a proxy) optimization pass, 2026-09-16: corpus design from repo sources, salted per-arm corpora, repeated arms, sequential-turn client vs batch benchmarks, engine counter semantics.)_
|
||||
- frank-medium on Dankserver (2x R9700 gfx1201, since 2026-09-16) serves as TWO single-GPU TP=1 replicas behind a proxy on :7770 (adaptive routing + failover), because TP=2 is impossible on this AM5 box: RCCL collective kernels need PCIe atomics and card #2's chipset path advertises AtomicOpsCap: Routing- (read-only). Per-replica shape: MAXLEN=114688, MAXSEQS=4, CHUNK=8192, KV_MEM=0 (passed through literally — blanking it makes vLLM size KV from GPU_UTIL alone and OOM), SPEC=7. Engine ledger facts: **GPU KV cache size: 122,636 tokens per card** (5.44 GiB), 1.07x concurrency at full context. Measured and REJECTED tonight, do not retry without new evidence: (1) load-balanced family placement — no improvement (heavy 25.4→46.7 s median); (2) large-request admission control, one 16k-token request per card — no clear win (heavy 25.4→48.2 s, mediums improved on small samples); (3) CHUNK=4096 or 16384 to free KV — available KV is NON-monotonic in chunk size (4.16 GiB at 4096 where the engine refuses to boot, 5.44 at 8192, 4.22 at 16384), so 8192 is a sweet spot; (4) SPEC=5 for batch throughput — 7 stays (46.2 vs 41.5 tok/s single-stream). The wall is capacity arithmetic: live sequences and cached prefixes share one pool, so a card holding two 50k-token families must carry 2x50k cached + 50k live = 150k against 122,636 — co-tenants evict each other regardless of placement or serialization. Caching itself is healthy (a 50k-token prefix: 22.3 s cold then 1.4 s warm, with a stable or varying tail). Trap: the engine's "Prefix cache hit rate: N%" log line is a LIFETIME average over every prompt that replica has seen, not a per-request verdict. Verified elsewhere tonight: the trio survives an unattended reboot (units generated, Linger=yes, active 59 s after boot, replicas healthy ~3.5 min in), and the harness path works (omp --model frank-medium -p --no-tools → correct reply in ~8-10 s) with models.yml contextWindow 114688 / maxTokens 32768. _(context: Dankserver frank-medium, 2026-09-16 overnight session: two-replica MXFP4 deployment, routing/failover shipped, four optimization hypotheses measured (two shipped, two rejected), capacity ceiling quantified, reboot-tested.)_
|
||||
- On a single-GPU vLLM replica, a request's LIVE sequence and its cached prefix draw on the same KV pool — so placement/routing policy cannot rescue an overloaded working set. Measured on 2x R9700 radiance MXFP4 (122,636 KV tokens per card, 1.07x concurrency at 114k context): one 80k-token family plus two 50k-token families, each turn re-sending its document, thrash the cache no matter how families are placed — counter placement: heavy median 25.4 s (23% cached), mediums 34.7/45.2 s (14/15% cached); load-balanced placement (new families to the card carrying less context, continuations still pinned): heavy 46.7 s (20% cached), mediums 23.4/34.3 s (0% cached). No improvement, change reverted. Why: a card running two 50k-token generations has already spent its 122,636 tokens on live sequences before any cache entry is considered, so placement only decides WHICH card is over capacity. Lever for more concurrent large contexts = fewer in flight (context compaction) or another card, never a routing tweak. Two measurement disciplines that made this decision possible, both worth repeating: (1) design the A/B so the two policies actually differ — with tiny "light" families nothing is under pressure and both arms look identical, so size the families to straddle the card's KV ceiling (e.g. 80k + 50k + 50k against 122k); (2) before trusting an arm, unit-check that the mechanism separates as intended (here: heavy-card0, both mediums-card1, confirmed by calling the routing function directly) — otherwise "no improvement" may just mean the feature never fired. Same discipline caught two other hypotheses in the same session: CHUNK=4096 also reduced available KV (4.16 GiB) rather than freeing it, so 8192 is a sweet spot, not a compromise. _(context: 2026-09-16, Dankserver frank-medium: A/B of load-balanced vs counter placement for two radiance MXFP4 replicas behind an adaptive-routing proxy; hypothesis rejected on measurement and reverted, documented in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md under "Placement does not rescue an overloaded working set".)_
|
||||
- frank-medium (Dankserver, 2x R9700) runs as TWO single-GPU vLLM-radiance replicas behind an adaptive proxy — TP=2 is impossible on this AM5 box (RCCL needs PCIe atomics; card 2's chipset path has AtomicOpsCap Routing-, read-only). Durable numbers measured 2026-09-16: each replica holds 5.44 GiB KV = 122,636 tokens at KV_MEM=0/MAXLEN=114688/CHUNK=8192/MAXSEQS=4/GPU_UTIL=0.98 (1.07x concurrency at full context). The arena/KV budget is NOT monotonic in CHUNK: available KV is 5.44 GiB at 8192 but 4.22 GiB at 16384 and 4.16 GiB at 4096 (where the engine refuses to boot: "5.0 GiB KV cache is needed"), so 8192 is a sweet spot and CHUNK is NOT a lever for freeing KV — more concurrent large contexts need another card. A 25-min soak of three concurrent large-context agent loops (each turn re-sending a ~30k-token document) gave 155/155 requests answered, zero errors/failovers, but only 25% cache-hit turns (median 1190 ms) against 75% re-prefills (median 13.9 s): the ceiling shows up as latency, not failure. Boot persistence is PROVEN by rebooting dankserver unattended: units active 59 s after boot, both replicas healthy ~3.5 min after boot, harness (omp --model frank-medium -p) working, no manual step — though SSH took ~9 min to return while systemd reports a 2m14s boot. The stack's /v1/completions returns raw continuations (no chat template), so api: openai-completions entries work only because the harness builds its own prompt; verify harness paths with omp --model alias -p --no-tools rather than assuming. OPERATIONAL TRAP hit twice: shipping files straight into the host checkout (/home/dank/serverAppConfigs) makes the next ./scripts/deploy.sh fail with "Your local changes ... would be overwritten by merge" — restore that path (git checkout -- path, or move the hand-copied file aside) before deploying, or the deploy silently stops at the pull. _(context: Dankserver frank-medium 2026-09-16 (overnight session): two TP=1 radiance MXFP4 replicas behind an adaptive-routing proxy on :7770; bench/soak/reboot verification; repo commits up to 99ed1ac.)_
|
||||
- Two counterintuitive, measured facts about vLLM KV capacity on the frank-medium radiance stack (Dankserver, 2x R9700, TP=1 replicas, 27B MXFP4, 114688 ctx) — both cost hours if re-derived from reasoning: 1. Available KV is NOT monotonic in the prefill chunk size. Measured per card: CHUNK=4096 - 4.16 GiB available KV and the engine REFUSES TO BOOT ("To serve at least one request with the model's max seq len (114688), 5.0 GiB KV cache is needed, which is larger than the available KV cache memory (4.16 GiB)"); CHUNK=8192 - 5.44 GiB (the sweet spot, and the deployed value); CHUNK=16384 - 4.22 GiB. So shrinking the chunk to "free memory for KV" is wrong in both directions. Get the real capacity from the engine's own boot ledger, never from arithmetic: podman logs container | grep -E "GPU KV cache size|Maximum concurrency|Available KV cache memory" - here "GPU KV cache size: 122,636 tokens" and "Maximum concurrency for 114,688 tokens per request: 1.07x". 2. 122,636 KV tokens per card is about TWO 30k-token conversations with their cached prefixes. A 25-minute soak with three concurrent agent loops (each turn re-sending a ~30k-token document, growing history) came back perfectly stable — 155 requests, 155 answers, 0 failures, 0 proxy failovers — but only 25% of turns hit the prefix cache; the other 75% paid a 13.9 s median re-prefill because a third family on the same card evicts. Capacity exhaustion on this fleet shows up as latency, not errors. Related: the boot warning "Add 3 padding layers, may waste at most 60.00% KV cache memory" is real — the mamba/attention hybrid wastes KV, so the usable budget is well under the nominal pool. Also: the harness path is verifiable directly with omp --model frank-medium -p --no-tools --no-session "prompt" (9.8 s round trip here) — that exercises models.yml provider/baseUrl/alias end to end, which reading YAML cannot. Note the provider uses api: openai-completions, i.e. /v1/completions, where this stack returns a RAW continuation (no _(context: Dankserver frank-medium 2026-09-16, after the two-replica + adaptive-routing deployment: soak of three concurrent agent loops, CHUNK sweep (4096/8192/16384), engine boot-ledger capacity read, and a real omp harness call.)_
|
||||
- A reverse proxy in front of N LLM replicas that trusts a periodically-refreshed health table will hand clients 5xx errors during every restart window: with a 10 s health TTL, requests routed to a card that just went down fail before the table notices — and a deploy restarts all replicas. Fix: on connection failure (nothing written to the client yet), demote that replica immediately and retry the request on another live replica; log the failover, because a silent one hides a downed card. Safe for stateless OpenAI-compatible endpoints (chat/completions carries no session state) but NEVER retry after streaming has begun. Verify behaviourally: stop one replica and fire a burst of requests in the same second — all must return 200 with exactly one failover line in the proxy log. Measured on frank-medium (2x R9700, adaptive routing): 6/6 requests 200, one failover log line, before the change they returned 502. Also observed: a stopped radiance replica takes ~150 s to come back with a warm compile cache, so the deploy health wait must budget for it. _(context: Dankserver frank-medium 2026-09-16: closed the 502-during-restart window in the two-replica proxy (routing.py + affinity-proxy.py), verified by stopping card 1 and firing a burst; deployed green with the e2e gate 30/30.)_
|
||||
- Benchmarking a speculative-decoding LLM endpoint on this fleet taught a discipline that changed three conclusions in one night: (1) Each configuration under test needs its OWN salted corpus — prefix caching persists inside a running engine, so reusing a document across configs silently measures a warm cache; put a unique token sequence at the very start of every prompt so both the router hash and vLLM's cache key change. (2) Synthetic random prompts under-report speculative decoding badly: the same model/server measured 2.9-3.6 tok/update on --dataset-name random and 3.86 tok/update on a corpus built from the repo's own source files, so a code/RAG/agent workload must be measured with a code-shaped corpus (vllm bench --dataset-name custom with JSONL {"prompt": ...}). (3) Run each setting at least twice: the same config measured 118.7 and 127.0 tok/s (conc 4) and 136.1 / 129.4 (the other config) — ~5-7% run-to-run variance, so a single-run difference under ~10% is not a decision, and the two settings' ranges overlapped while the ordering held. (4) Conversation workloads need a sequential-turn client measuring per-turn TTFT, not a batch benchmark: a batch of distinct prompts parallelises prefill across replicas and even makes round-robin look better, while the agent-loop shape (same prefix, growing history) is what exposes cache behaviour. Separately, a sizing fact for the radiance/vLLM stack on a 32 GiB card: the prefill arena scales with --max-num-batched-tokens and shares one device budget with the KV cache, so CHUNK=16384 leaves 4.22 GiB of KV where 114688 context needs 5.21 GiB and the engine refuses to start — CHUNK and MAXLEN are a matched pair, not independent knobs. _(context: Dankserver frank-medium 2026-09-16/17: tuning two single-GPU radiance MXFP4 replicas (SPEC depth 7 vs 5, CHUNK 8192 vs 16384) and their adaptive-routing proxy; every number in the runbook came from these corpora and repeats.)_
|
||||
- On the vllm-radiance MXFP4 stack (frank-medium on Dankserver's R9700s), two KV-sizing traps cost deploy cycles and are counterintuitive: (1) KV_MEM=0 must be passed THROUGH to the engine, not blanked. The entrypoint expands ${KV_MEM:+--kv-cache-memory "$KV_MEM"}; the image's patched KV sizing reads an explicit 0 as "profile this shape". If the variable is blanked instead (so the flag is omitted), vLLM sizes the KV cache from --gpu-memory-utilization alone, overshoots, and the allocation dies with torch.OutOfMemoryError right after the weights load — measured: "Tried to allocate 2.16 GiB. GPU has 116 MiB free, 29.34 GiB allocated by PyTorch". Symptom shape: the quadlet crash-loops every ~2 min ("Engine core initialization failed", status=1) while the same config runs fine when started by hand with the older entrypoint — compare entrypoint versions when a hand-run works and the unit does not. Do not "fix" the entrypoint by making the expansion empty. (2) CHUNK (--max-num-batched-tokens) is coupled to the context: the prefill ARENA grows with the chunk size and shares one device budget with the KV cache. On a single 32 GiB card at MAXLEN=114688, CHUNK=16384 leaves 4.22 GiB of KV against the 5.21 GiB needed and the engine refuses to start ("To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)"). Raising CHUNK therefore means cutting MAXLEN. Per-card single-card prefill at CHUNK=8192 measured ~2.5k tok/s (~15k-token prompt, 5992 ms TTFT), so the trade was not worth taking. Also: 262144 context needs 9.61 GiB of KV and 131072 needs 5.58 GiB, which is why one card tops out at 114688 in this shape. _(context: Dankserver frank-medium, 2026-09-16: two TP=1 radiance MXFP4 replicas, 114688 context each. These two traps are documented in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md; capturing them in memory because both look like engine bugs rather than sizing rules.)_
|
||||
- When fronting multiple single-GPU vLLM replicas with a proxy, plain round-robin destroys prefix-cache hits: an agent loop re-sends the same system prompt + file context every turn, so alternating replicas makes every other turn pay a cold prefill. Measured on 2x R9700 (90k-token contexts): round-robin warm-turn TTFT 2676 ms vs pinned 1096 ms. But pinning everything is also wrong — a cold batch then loses a card's prefill parallelism (16 requests: 18.6 s spread vs 30.1 s pinned). The right rule is workload-shaped: pin *continuations* (requests carrying 1 message = an agent loop resending context) to the replica that served the prefix, and spread everything else (one-shot questions, batch members, parallel subagents) by counter. Measured result of that hybrid on the same host: 1095 ms warm turns AND 19.97 s burst. Two traps found by measuring, not reasoning: (1) hashing a fresh prefix to choose a card sends every member of a same-prefix burst to the same card (29.8 s) — spread must come from a counter, only continuations may be pinned; (2) the conversation identity is the first N KiB of the prompt (4096 by default) — if the prefix is shorter than that window, turns hash differently and never pin. Also keep the routing policy in its own stdlib-only module: a proxy that imports aiohttp at module scope cannot be unit-tested in an environment without it, and the first version of the routing code shipped a bug (stale index - HTTP 500 on every continuation) that only a policy-level test catches cheaply. _(context: Dankserver frank-medium 2026-09-16: two TP=1 radiance MXFP4 replicas behind a proxy on :7770 (TP=2 impossible — PCIe atomics). Implemented routing.py + affinity-proxy.py, tests/frank_medium_proxy.bats, deployed via deploy.sh with the e2e gate.)_
|
||||
- Multi-GPU RCCL collectives on AMD require PCIe atomics on EVERY participating GPU, and on AMD desktop platforms the second x16 card usually sits behind chipset bridges that advertise AtomicOpsCap: Routing- (read-only). RCCL's collective kernels carry a hidden hostcall-buffer parameter; ROCr refuses to give them one when the device lacks atomics (rocclr/device/rocm/rocvirtual.cpp:3963, gated on pcie_atomics_ set once at driver probe from the CPU-agent link info), and the launch dies as hipErrorIllegalState ("the operation cannot be performed in the present state") with NO mention of atomics. Diagnostics: AMD_LOG_LEVEL=3 prints the truth (Pcie atomics not enabled, hostcall not supported → AQL dispatch failed! → hipLaunchKernel: Returned hipErrorIllegalState); lspci -vv -s gpu on the endpoint AND every upstream bridge must show AtomicOpsCap: Routing+; the endpoint's Device Control 2 (setpci -s gpu CAP_EXP+28.w) must have bit 6 set. Setting that bit by hand does NOT help — the driver consumes it at probe time. Traps: a single-rank (world_size=1) collective succeeds on either card because NCCL short-circuits, so one-card tests prove nothing; the failing kernel version was ruled out across 7.0.0-28/-30/-31; a from-source RCCL rebuild reproduces the identical failure, so the library is never the bug. Workaround when only one card is atomic-capable: two single-GPU TP=1 replicas behind a round-robin proxy (vLLM's OpenAI API is stateless per request). Related KV trap found the same night: passing --kv-cache-memory 0 explicitly means "profile this shape" (radiance-patched vLLM); BLANKING the variable so the flag is omitted makes vLLM size KV from --gpu-memory-utilization alone and the allocation OOMs right after weights load. _(context: Dankserver frank-medium (2x R9700, AM5) 2026-09-16: TP=2 radiance MXFP4 could not launch any collective; root cause traced to PCIe atomic routing on the second card's chipset path; deployment changed to two TP=1 replicas + round-robin proxy, deployed green with benches (conc1 42.8, conc4 147.6, conc8 230.8 tok/s; TPOT 23.6-28.8 ms; 32k prefill ~2.8k tok/s per card).)_
|
||||
- serverAppConfig deploy.sh pre-flight gotcha (2026-09-16): it requires a WHOLE-TREE clean git status --porcelain (untracked included) AND the main checkout on branch main — a linked git worktree fails the "must run from the main branch" check, so worktrees cannot bypass a dirty tree. When another agent's uncommitted files block a needed deploy, do NOT commit their WIP; use a targeted stash cycle: git stash push -m "deploy-window: park peer WIP" -- peer files && ./scripts/deploy.sh target ... && git stash pop. The stash survives a conflicting pop (kept on failure), the peer's edits are restored byte-identical, and the window is seconds. Check git status --short afterward to confirm the peer's files are back. _(context: 2026-09-15 frank-large deploy: deploy.sh failed twice (dirty tree from peer files; worktree rejected by main-branch check), then succeeded via targeted stash cycle; peer WIP verified restored (git status identical, stash dropped cleanly).)_
|
||||
- CORRECTION (2026-09-16): the "e2e 11-admin.spec fails on 192.168.0.112:8989 from blankpad — environmental, expect 29/30" lesson is STALE. On 2026-09-16 tests/e2e/run.sh passed 30/30 from blankpad, including all three 11-admin specs, with no known intervention. Do not pre-declare the admin-SSO specs environmentally doomed or treat 29/30 as the expected baseline — run the suite and read the actual result. The old diagnosis (blankpad TCP-path to the arr ports blocked while peers succeeded) no longer reproduces; if an admin spec fails again, re-diagnose fresh rather than citing the old blocker. _(context: 2026-09-16 frank-large ik cutover wrap-up: tests/e2e/run.sh from blankpad returned 30/30 passed including specs/11-admin.spec.ts against 192.168.0.112 raw ports; repo commits 795b0b3/ad7a9c4.)_
|
||||
- frank-large ik_llama.cpp performance campaign (2026-09-16): decode ~3.8-4.3 t/s is gated by a SERIAL per-token critical path — ~100 CPU↔GPU backend splits/token with the 3090 at 5% SM utilization (launch/sync-bound across the PCIe x4 dock); thread count is irrelevant (T1≈T16); prefill ~192 t/s sits at the CPU AVX512-VNNI compute ceiling for CPU-resident experts. The 30 t/s halogen bar is unreachable on this model+hardware: ik's CUDA-graph capture aborts on qwen4exp's per-layer ggml_repeat_4d hc-broadcast ops (capture-illegal op), mainline llama.cpp 37b53fd runs 1.0 t/s (CPU MoE kernels 4x slower than ik's iqk), the GGUF has no MTP head (mradermacher stripped it), and 84 GiB of experts cannot fit 24 GiB VRAM. Re-evaluate when mainline's hc-ops + graph handling mature (ggml-org PR #28901). DANGERS learned: (1) systemd Restart=always respawns a pkill'd llama-server ~4s later — manual experiments MUST sudo systemctl stop the unit first or measurements get phantom instances; (2) ik --defer-experts does NOT keep experts off the GPU under -ngl 999 (only PLE defers) → cudaMalloc OOM crash at 79.7 GiB; (3) ik -ot/-ncmoe overrides disable mmap for the whole load (create_tensors returns false); (4) -ncmoe N without GGML_CUDA_NO_PINNED_WEIGHTS=1 page-locks ~108 GiB (the co-tenant killer); (5) never run -j16 builds concurrently with decode measurements. _(context: 2026-09-15/16 overnight frank-large decode campaign on dankclaw: goal was halogen's ~30 t/s decode; measured evidence chain (perf pause-spin analysis, T1=T16 sweep, nvidia-smi 5% SM during decode, graph splits=100, mainline comparison 1.0 t/s, GGUF MTP-head check) recorded in serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md, commit 795b0b3.)_
|
||||
- serverAppConfig repo convention (established 2026-09-16, commit f227786): AGENTS.md is evergreen rules + key lessons only. Fleet-swap history, retired-runtime detail, and per-flag sizing rationale do NOT go back into AGENTS.md — they live in serverAppConfigs-docs/LLM/ runbooks, and AGENTS.md points at them (Runbook Index table). The "Key Lessons Learned" section holds only durable, cross-runtime operational facts (ROCm layer-split, HIP env collisions, mmap/PLE, swap ordering, quadlet/podman gotchas). When a runtime swap or retirement happens: update the ACTIVE topology table and the Runbook Index row, add any new durable lesson, and write the detail into the runbook — do not grow AGENTS.md history sections. Known pre-existing test failure: tests/run.sh not ok 73 (comfyui Containerfile selector bug, test-side) is unrelated to doc changes and does not block commits. _(context: AGENTS.md rewrite session 2026-09-16; Frank approved "lessons + pointer table" for retired runtimes and "condense to operational-critical" for halogen detail via grilling. Commit f227786 (734→412 lines).)_
|
||||
- blankpad skill directories (2026-09-16): the pi-era user skills at ~/.pi/agent/skills and ~/.agents/skills were wiped; omp discovers user skills at ~/.omp/agent/skills (global) and .omp/skills (project), both of which now exist again. Restored 9 skills from source repos: 8 from github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff — code-review and handoff were mislabeled "Local" in the old omp-docs/pi/skills.md; they are from mattpocock) + find-skills from github.com/vercel-labs/skills. Copied as plain dirs (skill/name/SKILL.md, spec-compliant frontmatter) — the npx skills CLI was bypassed because it auto-detects the wrong agent (claude) and doesn't know omp. UNRECOVERABLE until dankware (192.168.0.226, offline, no SSH alias) is back online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base. New skills are discovered on the NEXT session, not the current one. _(context: blankpad skill restoration session 2026-09-16: user noticed /skill:grilling unresolved. Traced pi-era skills (~/.pi/agent/skills + ~/.agents/skills, installed via vercel-labs skills CLI) as wiped; recovered 9 of 13 from source repos on GitHub into ~/.omp/agent/skills.)_
|
||||
- The AGENTS.md fleet-topology snapshot embedded in session context can lag the repo: on 2026-09-15 the serverAppConfig tree at HEAD (commits d53962f/4183e31) described a NEWER layout than the context file — frank-medium (Dankserver 2x Radeon AI PRO R9700, gfx1201) replacing frank-small, and frank-large moved to ik_llama.cpp serving orcarouter/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M on Dankclaw's RTX 3090. Before reasoning about the LLM fleet from the AGENTS.md context (service names, models, runtimes, ports), verify against git log --oneline + the tracked serve/env files at HEAD; the context file is a snapshot of a past date. Related trap from the same session: the working tree carried UNCOMMITTED radiance-era edits (scripts/deploy-plan.py, scripts/deploy-remote.sh, tests/deployment_scoping.bats) plus a DELETED test (tests/frank_large_quadlet.bats — deletion must go through .AGENTTRASH per policy, never rm); reconcile git status before any deploy step. _(context: 2026-09-15 frank-medium (dankserver 2x R9700, vllm-radiance MXFP4) cutover session: interrupted while reconciling working-tree state before host-side trash staging. The system-prompt AGENTS.md describes the 2026-09-11/13 fleet; commits d53962f/6085a99 ("serve the new R9700s, retire frank-small") plus scripts/frank-large.env (2026-09-15 header, ik_llama.cpp, dankclaw RTX 3090) show the fleet was re)_
|
||||
- FLEET STATE UPDATE (2026-09-15): frank-small (FreeToken/CUDA, RTX 3090) on dankserver is RETIRED. The 3090 was removed and replaced with two AMD Radeon AI PRO R9700 (Navi 48, gfx1201, 32,624 MiB each; no NVIDIA device remains; host iGPU gfx1036 hidden from ROCm via HIP_VISIBLE_DEVICES=0,1). The :7770 dankserver endpoint is now frank-medium: charlie12345/ROCmFPX llama.cpp (c49ebdb, built gfx1200;gfx1201 with the fork's self-contained ROCm 7.14 TheRock toolchain, RPATH-bundled — host /opt/rocm-6.3.3 was purged and never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (26.28 GiB Q8_0 + MTP Q4_0 draft pinned ROCm1) with layer split across both cards. Deploy path: ./scripts/deploy.sh dankserver --service frank-medium (unit is a root systemd unit symlinked to the repo file; first deploy retires the old frank-small unit). Clients already cut over: blankpad ~/.omp/agent/{models.yml,config.yml} and Hermes live config on dankclaw (provider key stays 'dankserver-small' as label-only). Any memory/doc still describing dankserver as FreeToken/CUDA frank-small is stale. Trash staged awaiting purge approval: repo scripts/.AGENTTRASH/{frank-small.env,frank-small.service,serve-frank-small.sh}, dankserver ~/.AGENTTRASH/{20260915-frank-small-retire,20260915-rocm633-cleanup}, dankclaw hermes models_cache.json-20260915. _(context: 2026-09-15 GPU swap on dankserver (RTX 3090 out, 2x Radeon AI PRO R9700 gfx1201 in); frank-medium deployed via repo git flow, commits 6085a99/d53962f/d4024a9/db29c4d; runbook serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md; supersedes the fleet-topology lines in older memory (frank-small FreeToken/CUDA on dankserver).)_
|
||||
- Dankserver frank-medium (2x Radeon AI PRO R9700, gfx1201) ops facts, measured 2026-09-15: (1) llama-bench multi-GPU gotcha — '-dev ROCm0,ROCm1' runs two SEPARATE single-card benches; a real 2-GPU bench needs '-dev ROCm0/ROCm1'. (2) Parallelism on this RCCL-less ROCmFPX build: layer split (1991 t/s prefill, ~17.6-18.8 t/s tg) beats tensor split (~272 t/s prefill, 7x worse) because tensor mode falls back to the meta-backend butterfly allreduce; with MTP serving, layer also wins decode (35.7-46.2 vs 28.7 tok/s). Draft head must be pinned to ROCm1 — unpinned draft under tensor mode aborts KV build (GGML_ASSERT(buffer)). (3) First-request-after-restart on the engine can go silent ~45-60s (health engine_unresponsive, 503s) then self-recovers — do not restart over it. (4) Request-level speculative:{n_max:0} does NOT disable drafting. (5) Hermes provider KEY 'dankserver-small' is label-only and deliberately NOT renamed (DB references) even though its model id is now frank-medium; repo example dankclaw/hermes/config.yaml.example synced in db29c4d. (6) Trash still staged awaiting purge approval: repo scripts/.AGENTTRASH/{frank-small.env,frank-small.service,serve-frank-small.sh}, dankserver ~/.AGENTTRASH/{20260915-frank-small-retire,20260915-rocm633-cleanup}, dankclaw hermes models_cache.json-20260915. _(context: 2026-09-15 frank-medium bring-up on Dankserver (2x Radeon AI PRO R9700, charlie12345/ROCmFPX llama.cpp, kingjones Q8_0_ROCMFPX-AGENT); runbook at serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md, commits 6085a99/d53962f/d4024a9/db29c4d.)_
|
||||
- frank-large (halogen-flash-server, Dankclaw) pool history and post-restart behavior, measured 2026-09-13: pool trimmed 524288 → 401408 (deployed, repo commits 91406af + 27e6335) to restore co-tenant headroom. Measured boot ledger at 401408/MAX_TOK=32768/reserve 15: 68.0 weights + 11.0 KV + 21.0 working = 100.0 GiB held, host left 9.1 GiB / 1396 contiguous 2 MiB blocks (vs 7.1 / 310 at 524288), engine printed need ~31.5 GiB vs modelled ~33.4, and the co-tenant stall NOTE no longer prints (threshold is somewhere below 9.1 GiB / 1396 blocks). TWO new facts: (1) On the FIRST request after any (re)start the engine can go silent ~45-60 s — /health flips to engine_unresponsive "no PONG within 30s" and the front-end 503s, then self-recovers. Do not restart the unit over it; the quadlet HealthStartPeriod=300s covers it. First completion attempt timed out during this window; a retry 20 s later returned instantly. (2) Host-left gain (+2.0 GiB) came in below the modelled +3.4 because the engine holds freed KV space as working slack (working 21.0 at 401k vs 20.6-ish at 524k with arena). The omp models.yml on blankpad (/home/blank/.omp/agent/models.yml) had frank-large maxTokens: 262144 — above halogen's 65536 cap and a 503-queue hazard after the pool cut; fixed to 65536, and the stale flm (dankclaw-npu) provider + its modelRoles (vision/tiny/task) were removed per the AGENTS.md route map (vision→frank-large, tiny→frank-small, task→frank-large:medium). _(context: frank-large KV pool trim to 401408 on Dankclaw, 2026-09-13, followed by OMP models.yml/config.yml cleanup on blankpad.)_
|
||||
- Dankclaw host swap (as of 2026-09-12): 64 GiB file at /swap.img, ext4 on /dev/nvme0n1p2, fallocated hole-free, driven by /etc/fstab line 12 (/swap.img none swap sw 0 0) which generates the swap.img.swap systemd unit — fstab is the only control point, there is no hand-written unit and no repo file, so deploy.sh cannot see swap state. vm.swappiness=60 (distro default, UNTESTED against this workload), zswap disabled. Raised from 8 GiB; 748 GiB still free on /. The old 8.1 GiB file is staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img awaiting Frank's approval to purge. Important framing: swap on this host protects the ~28 co-tenant containers, NOT frank-large — its ~68 GiB of pinned weights are not relieved by swap, and paging the KV pool (which IS anonymous) would be worse than the OOM kill it prevents, so if the box thrashes the knob is lower swappiness or a cgroup cap, not more swap. _(context: Raised swap on Dankclaw 2026-09-12 after the frank-large reserve-15 config left only ~7.1 GiB for co-tenants. Documented in serverAppConfigs-docs/Dankclaw/Swap Configuration.md and AGENTS.md memory-model section.)_
|
||||
- In the serverAppConfig repo, ./tests/run.sh exits 1 on a **clean tree** because of one pre-existing failure: tests/comfyui_rocm.bats test 73 "comfyui Containerfile is based on the gfx1151 torch image". It is a test-side selector bug, not a product regression — its find ... | head -1 picks dankclaw/alloy/Containerfile instead of the ComfyUI one, so the assertion reads the wrong file. Verify before blaming your own change: a good run reports 285 ok plus this single not ok 73. Confirmed present at HEAD across several sessions (2026-09-12 frank-large work), independent of frank-large/Caddy/quadlet edits. Do NOT treat a green deploy as blocked by it, and do NOT "fix" it by weakening the assertion — the correct fix is to target the actual ComfyUI Containerfile path. Note deploy.sh gates on its own per-service health plus the Playwright e2e suite, not on tests/run.sh, so this failure does not block deploys. _(context: Every ./tests/run.sh run during the 2026-09-12 frank-large KV-pool/reserve work returned exit=1 with exactly not ok 73 comfyui Containerfile is based on the gfx1151 torch image while 285 tests passed, including on commits where nothing ComfyUI-related changed.)_
|
||||
|
||||
@@ -1 +1 @@
|
||||
dankclaw hosts frank-large (rootless Quadlet, 103.4 GiB RAM, 3 KV slots, greedy default) and marreta (podman compose, Caddy/Anubis/Authentik guard chain). Key ops: probe marreta internally to avoid netns HTTP 000; set temperature: 0 on frank-small for determinism; preserve_thinking is client-side per-request. Watch for file-cache uninterruptible sleep and missing pyyaml on system python3. Authentik flows must link Application(slug=marreta) to prevent Rust outpost crashes.
|
||||
dankclaw/dankserver systems featuring ROCmFPX and llama.cpp. frank-large (HALOGEN) uses a 524288 KV pool and is managed via rootless Quadlets (mandatory for persistence). Key hardware fix: M.2 riser for R9700 chipset PCIe issues. Key software behavior: Halogen stalls are documented (don't restart), first-request silence is normal, and HIP env vars (TEMP/HOST) must be handled carefully. Marreta is secured via an Authentik/Anubis auth chain.
|
||||
|
||||
@@ -1,5 +1,169 @@
|
||||
# Raw Memories
|
||||
|
||||
## 01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3
|
||||
updated_at: 1789599423
|
||||
## Rootless Podman restart policy pitfall
|
||||
- `restart: unless-stopped` only survives *crashes*, not *explicit stops* (e.g., kernel power-off sends a polite stop signal).
|
||||
- Result: bare containers without a Quadlet unit are gone after reboot with no alert.
|
||||
|
||||
## Repo `serverAppConfig` deployment pattern
|
||||
- Every container must have a Quadlet `.container` file (git-tracked) to be boot-persistent and self-healing.
|
||||
- The unit file is the container's birth certificate + pension plan + ghost in one.
|
||||
|
||||
## Historical incidents (from story)
|
||||
- Aug 26: Caddy never returned after reboot (no unit).
|
||||
- Sep 11: Five observability services found unitless after reboot.
|
||||
|
||||
## Persona note
|
||||
- Model: `frank-medium` via llamacpp (dankserver). Persona: Soviet-style computer systems engineer with Munroe-esque humor. Uses "Comrade" address.
|
||||
|
||||
## 01a0a7ad-42e2-739d-bc1c-dc405ca02cbd
|
||||
updated_at: 1789574671
|
||||
## Dankserver Dual R9700 RCCL Chipset Issue
|
||||
|
||||
**Hardware**: 2x AMD R9700 on ASUS board. Card #1 on CPU-attached PCIe x16. Card #2 on chipset-attached PCIe x16 slot (PCIEX16_2, which is x1 mode per ASUS spec).
|
||||
|
||||
**Problem**: RCCL (ROCm Collective Communications Library) fails on the chipset-attached card. Bare 2-rank torch.distributed probe results:
|
||||
- broadcast: OK (CPU card) / FAIL (chipset card)
|
||||
- all_reduce: OK / FAIL
|
||||
- barrier: OK / FAIL
|
||||
- send/recv (P2P): hang / FAIL
|
||||
|
||||
**Implication**: send/recv failure kills pipeline parallelism too (vLLM's PP handoff IS that primitive). No software-only escape.
|
||||
|
||||
**Root cause**: Chipset-attached PCIe lanes don't route atomic operations the way RCCL's kernels require. CPU-attached lanes do.
|
||||
|
||||
**Fix (recommended)**: M.2→PCIe x4 riser (~$30) to move card #2 to M.2_2 slot. ASUS spec confirms M.2_1 and M.2_2 are CPU-attached; M.2_2 is free/empty. Restores atomic-op routing → RCCL works → TP=2 functional.
|
||||
|
||||
**Alternative**: Board with CPU x8/x8 bifurcation (~$250-400).
|
||||
|
||||
**Fallback**: llama.cpp layer split (no collectives needed, but needs GGUF and loses vLLM batching).
|
||||
|
||||
**Bandwidth analysis**: 27B TP=2 allreduce at batch 4 ≈ 10 MB per token-step ≈ 1.4 ms at Gen4 x4. Bandwidth is NOT the bottleneck; atomics routing is.
|
||||
|
||||
## Software Stack
|
||||
- vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14
|
||||
- Shipped RCCL exposes no ncclDevComm* symbols → torch's symmetric-memory path can't run
|
||||
- Open hypothesis (unverified): recompiling RCCL with -mprintf-kind=buffered might work, but no live printf/assert/__assertfail found in device source
|
||||
|
||||
## Operational Notes
|
||||
- dankserver went down during probing; needs physical power cycle
|
||||
- Boot persistence should auto-restore: frank-medium trio, media stack, node-exporter
|
||||
- To distinguish 'off' from 'wedged': ip neigh from peer (FAILED = off/NIC dead; wedged kernel still answers ARP)
|
||||
- AGENTS.md benchmarking policy updated to cover any GPU experiment, not just benchmarks
|
||||
|
||||
## Model Target
|
||||
- kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
|
||||
- Fork: charlie12345/rocmfpx (llama.cpp ROCm fork)
|
||||
|
||||
## Skill Note
|
||||
- 'grilling' skill does NOT exist in available skills list (returned 'Unknown skill: grilling')
|
||||
|
||||
## 01a0a87f-6b17-7492-995c-8c3e7b37e241
|
||||
updated_at: 1789552363
|
||||
## Hardware Topology (dankclaw)
|
||||
- RTX 3090 (24 GiB VRAM) in PCIe 4.0 x4 M.2 dock (downgraded from x16, measured ~29 GB/s sequential)
|
||||
- AMD Strix Halo gfx1151 (64 GiB SVM-backed unified memory, RADV Vulkan driver; ROCm wheels do NOT support gfx1151)
|
||||
- Combined GPU budget: ~88 GiB
|
||||
- OS: Ubuntu 24.04
|
||||
- Kernel cmdline: amd_iommu=off (for prefill gains; retired frank-npu)
|
||||
|
||||
## ik_llama.cpp Build & Deployment
|
||||
- Built with CUDA (for 3090) + Vulkan (for Strix Halo) backends
|
||||
- Serves mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF at i1-Q4_K_M
|
||||
- Deployed as root systemd unit (Restart=always) on port 7770, NOT a quadlet
|
||||
- Context window: 262144 tokens
|
||||
- Device split: --split-mode layer with -dev CUDA0,Vulkan0
|
||||
- Shipped config: T16 (16 threads), no -ncmoe, no -thp
|
||||
- Commits: 795b0b3, ad7a9c4
|
||||
- Runbook: serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md
|
||||
|
||||
## Performance Envelope (verified single-instance)
|
||||
| Config | Decode (t/s) | Prefill (t/s) | Notes |
|
||||
|---|---|---|---|
|
||||
| Shipped T16 | 3.8-4.0 | ~192 (1.8k-32k ctx) | Production config |
|
||||
| T8/T4/T2/T1 | 4.18/4.20/4.26/3.97 | small-prompt only | Thread count barely matters |
|
||||
| -ncmoe 42 | 4.54 | — | 22.0/24.6 GiB VRAM, fragile at 262k ctx |
|
||||
| -ncmoe 48 -thp | 4.60 | — | 110 GiB anon host buffer, co-tenant killer |
|
||||
| --prefetch-experts | 4.03 | — | No effect (cache already resident) |
|
||||
| Mainline llama.cpp 37b53fd | 1.01 | — | Ruled out entirely |
|
||||
|
||||
## Root-Cause: Why 30 t/s Decode Is Unreachable
|
||||
1. Decode issues ~100 graph splits per token
|
||||
2. Each split = GPU launch + sync across PCIe x4 (3090 in dock)
|
||||
3. GPU SM utilization during decode: ~5%
|
||||
4. ~90% of cycles = pause (thread barrier spin); T1 ≈ T16
|
||||
5. CUDA graphs would collapse the overhead BUT capture ABORTS: qwen4exp hc ops use per-layer ggml_repeat_4d which is capture-illegal
|
||||
6. Speculative decoding unavailable: GGUF has no MTP head (mradermacher stripped NextN from the checkpoint)
|
||||
7. All experts on GPU: 84 GiB > 24 GiB 3090 VRAM — impossible
|
||||
8. Path forward: mainline llama.cpp PR #28901 (hc-ops + capture-safe graphs) — re-evaluate when it lands
|
||||
|
||||
## Prefill Ceiling
|
||||
- ~192 t/s is the ceiling: compute-bound on CPU AVX512-VNNI
|
||||
- Expert reads amortize across the batch; math saturates cores
|
||||
- Halogen hit 30 t/s decode by holding entire 115.55 GiB w4b checkpoint resident in LPDDR inside one proprietary engine; 24 GB GPU + CPU split cannot replicate that residency with any public runtime
|
||||
|
||||
## Operational Traps
|
||||
- Unit has Restart=always: pkill'd servers respawn mid-experiment → phantom instances invalidate benchmarks. Always verify single-instance before measuring.
|
||||
- Concurrent -j16 builds can starve probe processes
|
||||
- PCIe x4 dock: do NOT assume x16 bandwidth; measured ~29 GB/s (not 130 GB/s)
|
||||
- -ncmoe 42: +15-20% decode but only 2.5 GiB VRAM margin at 262k ctx — OOM risk
|
||||
- -ncmoe 48 -thp: 110 GiB anon host buffer — will kill co-tenants
|
||||
|
||||
## Trash (staged for user approval)
|
||||
- dankclaw ~/.AGENTTRASH/20260916-ik-partfiles/ — 111 GiB (3 download parts, superseded)
|
||||
- dankclaw ~/.AGENTTRASH/20260915-ik-swap/ — retired halogen quadlet
|
||||
- repo tests/.AGENTTRASH/frank_large_quadlet.bats, scripts/.AGENTTRASH/frank-small*
|
||||
|
||||
## 01a0a902-7f58-77f4-9392-764f6cfbd59e
|
||||
updated_at: 1789544414
|
||||
## AGENTS.md Rewrite Pattern (serverAppConfigs)
|
||||
- Collapse chronological swap history into: (1) active-topology table (current state only), (2) runbook pointer table mapping each retired runtime → its docs/LLM/ runbook path.
|
||||
- Add a 'Key Lessons Learned' section containing ONLY durable technical constraints/pitfalls (not narrative).
|
||||
- In config sections (e.g. halogen), keep only invariants that survive any config change (pool/reserve matched pair, client contract, memory-model rule); push sizing history to runbook.
|
||||
- Fix structural bugs (e.g. broken merged table rows) as part of the cleanup.
|
||||
|
||||
## Durable LLM/ROCm/Quadlet Lessons (from AGENTS.md distillation)
|
||||
- Layer-split-not-tensor: for multi-GPU ROCm, split by layer not tensor-parallel.
|
||||
- mmap/PLE constraints apply to model loading on ROCm.
|
||||
- HIP env collision: `TEMP` and `HOST` env vars can conflict with HIP runtime.
|
||||
- `GGML_CUDA_FA_ALL_QUANTS` env var needed for flash-attention with quantized KV.
|
||||
- MTP (Multi-Token Prediction) draft head has loader incompatibility on ROCm.
|
||||
- For crash diagnosis: use gdb over logs (logs are often incomplete/misleading for segfaults).
|
||||
- Swap-resize safe ordering: resize before use, not after.
|
||||
- `free` command lies about available memory (doesn't account for reserved/pinned).
|
||||
- `systemctl disable` can delete unit files (not just disable them).
|
||||
- `pkill -x` for exact-name process kill (avoids substring matches).
|
||||
- Rootless quadlet log retrieval has specific gotchas (see skill: rootless-quadlet-log-retrieval).
|
||||
|
||||
## omp Skill Discovery
|
||||
- omp ONLY discovers user skills at `~/.omp/agent/skills/`.
|
||||
- pi-era locations `~/.pi/agent/skills/` and `~/.agents/skills/` (vercel-labs `skills` CLI layout) are NOT scanned by omp.
|
||||
- `npx skills` CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
|
||||
- Newly installed skills surface on the NEXT session, not the current one.
|
||||
- Skill sources: github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff), github.com/vercel-labs/skills (find-skills).
|
||||
- 4 skills unrecoverable without dankware (192.168.0.226) online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base.
|
||||
|
||||
## 01a09e67-630f-74ed-a910-e997e01fbe54
|
||||
updated_at: 1789368883
|
||||
Source: serverAppConfigs-docs/LLM/halogen-flash-server.md (halogen-flash-server runbook), sections Memory model, Two failure modes, Co-tenancy, and troubleshooting table.
|
||||
|
||||
Core stall advice: With HALOGEN_FLASH_PIN_TRUNK=1 (default), the halogen container holds most host memory. If a co-tenant takes the remainder and host memory runs out, both the engine and the co-tenant can stall for minutes at full CPU with no disk activity and no output. This is documented behavior, not a crash, and it needs no restart. Do not treat full-CPU/no-output multi-minute stalls as OOM/crash/restart cases.
|
||||
|
||||
Last resort: HALOGEN_FLASH_PIN_TRUNK=0 gives the pinned memory back to the host but costs several times the decode speed; use only as last resort, not routine tuning.
|
||||
|
||||
Diagnostics/monitoring traps: free, MemAvailable, and Grafana can read wrong by about 68 GiB because locked weights are counted as reclaimable. A request that never returns then returns 503 engine_busy is a different admission issue: the pool could not reserve prompt + max_tokens.
|
||||
|
||||
Restart first-touch behavior: after restart, the first request can make the engine silent for ~45-60 s (has not answered PING, /health engine_unresponsive) while it first-touches the fresh KV pool; it self-recovers. HealthStartPeriod=300s covers this; do not restart the unit; retrying after the window succeeds.
|
||||
|
||||
KV pool verification: use podman logs frank-large | grep 'kv pool:' and match reserved count to requested count; /health does not prove pool size. If under budget, nothing may print; absence of WARNING is not evidence of headroom.
|
||||
|
||||
Memory freeing/trimming: cut the KV pool or arena (examples: 262144 hands back ~3.6 GiB from 401408; MAX_TOK 16384 = ~8.4 GiB). Measured host-left gain can be less than modelled because the engine keeps freed space as working slack, but contiguous block count may improve substantially; stall warning keys on that. Example verified trim: kv_pool_positions=401408 exact/no clamp, held 100.0 GiB (68.0 weights + 11.0 KV + 21.0 working), host left 9.1 GiB / 1396 contiguous blocks vs previous 7.1 GiB / 310, stall NOTE no longer printed, health ok, real completion succeeded.
|
||||
|
||||
Client config constraint (blankpad OMP ~/.omp/agent/): frank-large maxTokens must be <=65536 because halogen returns 400 above cap; prompt + max_tokens must fit the KV pool or the request queues ~3600 s then 503s. Retired dead flm/dankclaw-npu provider on :7771; roles: vision -> frank-large, tiny -> frank-small, task -> frank-large:medium.
|
||||
|
||||
Deployment/environment notes: generated Quadlet units cannot be systemctl --user enable on this host; boot persistence uses [Install] WantedBy=default.target plus linger. Before replacing the halogen container/unit, remove the old Quadlet unit from ~/.config/containers/systemd/ to avoid duplicate port 7770 claims. Known environmental e2e breakage: blankpad -> 192.168.0.112 Sonarr path issue; deploy used --skip-e2e. Tests: 285+ green except known pre-existing test-73 ComfyUI selector bug.
|
||||
|
||||
## 01a09835-91cf-72e7-8ece-e5316d4de48b
|
||||
updated_at: 1789286336
|
||||
Expanded frank-large KV pool from 262144 to 524288 positions (~14.4 GiB). Reduced HALOGEN_KV_SLOTS from 4 to 3 (~111 MiB freed per slot). Held memory stable at 103.4 GiB (68 weights + 14.4 KV pool + 21 working) within 121 GiB host budget. Admission is pool-bound, not slot-bound; caps at 3 simultaneous full-length requests.
|
||||
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a09e67-630f-74ed-a910-e997e01fbe54
|
||||
updated_at: 1789368883
|
||||
|
||||
Halogen-flash-server docs say multi-minute full-CPU stalls with no disk activity/output under co-tenancy/pinned trunk are documented behavior, not crashes; do not restart. Last resort: HALOGEN_FLASH_PIN_TRUNK=0 returns memory but costs several times decode speed. Related durable constraints: first request after restart may be silent ~45-60s and self-recovers; frank-large maxTokens must be <=65536 and prompt+max_tokens must fit the KV pool or requests queue then 503.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0a7ad-42e2-739d-bc1c-dc405ca02cbd
|
||||
updated_at: 1789574671
|
||||
|
||||
Configured dual AMD R9700 GPUs on dankserver for ROCmFPX llama.cpp. Discovered RCCL fails on chipset-attached PCIe slot (broadcast/allreduce/send/recv all fail), killing both TP and PP. Solution: $30 M.2→PCIe x4 riser to move card #2 to CPU-attached M.2_2 slot. Documented in serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md (commit cc29875). Host went down during probing; needs power cycle.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0a87f-6b17-7492-995c-8c3e7b37e241
|
||||
updated_at: 1789552363
|
||||
|
||||
Deployed mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF (i1-Q4_K_M) on dankclaw via ik_llama.cpp with CUDA (3090) + Vulkan (Strix Halo gfx1151). Shipped at ~3.8-4.0 t/s decode, ~192 t/s prefill. 30 t/s decode proven unreachable: 100 graph splits/token cause PCIe x4 launch/sync bottleneck; CUDA graph capture aborts on qwen4exp hc ops (ggml_repeat_4d is capture-illegal). Mainline PR #28901 is the only path to 30 t/s class. Root systemd unit on :7770, contextWindow 262144.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0a902-7f58-77f4-9392-764f6cfbd59e
|
||||
updated_at: 1789544414
|
||||
|
||||
Rewrote serverAppConfigs AGENTS.md (734→412 lines, commit f227786): collapsed swap-history into active-topology table + runbook pointer table, added 'Key Lessons Learned' section with durable ROCm/llama.cpp/quadlet pitfalls, condensed halogen section to invariants only. Also diagnosed and fixed missing omp skills: pi-era skills in ~/.pi/agent/skills/ and ~/.agents/skills/ are invisible to omp (which only scans ~/.omp/agent/skills/); restored 9 of 13 from mattpocock/skills and vercel-labs/skills repos.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3
|
||||
updated_at: 1789599423
|
||||
|
||||
Testing thread. Story response surfaced durable technical context: rootless Podman `restart: unless-stopped` does NOT survive explicit shutdown (only crashes), so bare containers vanish on reboot. Repo `serverAppConfig` mandates Quadlet `.container` files for boot persistence. Historical incidents: Aug 26 Caddy lost after reboot; Sep 11 five observability services found unitless.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
Verifying the Marreta auth chain:
|
||||
1. Probe `http://localhost:8017/` from host (expected 000, listener is in Caddy netns).
|
||||
2. Check `authentik-proxy-provider-app-link` for null flows.
|
||||
3. Ensure non-browser clients can bypass JS challenges or use backend datasource URLs.
|
||||
@@ -1,17 +0,0 @@
|
||||
# frank-large Operations
|
||||
Deployed as rootless Quadlet on dankclaw. ~68 GiB weights + 14.4 GiB KV pool (524k positions, 3 slots). Total ~103.4 GiB.
|
||||
## Ports
|
||||
- Engine: :8730 (loopback)
|
||||
- Front-end: :8731
|
||||
- Published: :7770
|
||||
## Logging & Env
|
||||
- `podman logs frank-large` / `journalctl --user -u frank-large`
|
||||
- Avoid `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE` (HIP bare TEMP)
|
||||
## Sampling & API
|
||||
- Large: greedy | Small: sampling (temp=1.0, top_k=20, top_p=0.95)
|
||||
- Small determinism: `temperature: 0` or `--sampling-defaults none`
|
||||
- `preserve_thinking`: client-side per-request (`chat_template_kwargs`)
|
||||
- `max_tokens: 200` w/ empty content = thinking consumed budget
|
||||
- File-cache shortage → uninterruptible sleep. `host_reserve_gib=24` protects n-gram table.
|
||||
## Tests
|
||||
- 285/286 pass (test 73 comfyui bug). `pyyaml` missing on system py3; `uv` available.
|
||||
-5
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"model": "frank-small",
|
||||
"temperature": 0,
|
||||
"chat_template_kwargs": {"preserve_thinking": true}
|
||||
}
|
||||
-6
@@ -1,6 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Verify KV pool slots and memory stability
|
||||
docker inspect frank-large --format '{{.Config.Env}}'
|
||||
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
echo "Slots: HALOGEN_KV_SLOTS"
|
||||
echo "Memory: ~103.4 GiB (68W + 14.4KV + 21W)"
|
||||
-6
@@ -1,6 +0,0 @@
|
||||
[Service]
|
||||
ExecStart=/usr/bin/podman run ...
|
||||
Environment=TEMP=...
|
||||
Environment=HALOGEN_KV_SLOTS=3
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
Verify if the request fits the allocated KV pool:
|
||||
1. Run `podman logs frank-large | grep 'kv pool:'`.
|
||||
2. Compare 'reserved' count to 'requested' count.
|
||||
3. Note: /health does not prove pool size; absence of WARNING is not evidence of headroom.
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
# Marreta & Authentik Deployment
|
||||
Podman compose on dankclaw. Shares gluetun egress with searxng.
|
||||
## Guard Chain
|
||||
Caddy TLS → Anubis (:8923) → Authentik outpost → internal :8017 → marreta
|
||||
## Auth Fix & Pitfalls
|
||||
- Login-wall caused by null `ProxyProvider.application` & flows crashing Rust outpost.
|
||||
- Fix: Link `Application(slug=marreta)`, populate auth/authz/invalidation flows.
|
||||
- Probe: `localhost:8017` returns HTTP 000 (Caddy netns). Use internal probe or grep config.
|
||||
- Non-browser clients fail JS challenges. Audit service-to-service URLs for public guarded hostnames.
|
||||
- Exempt: IdP, Gitea (git smart-HTTP), self-auth services.
|
||||
## Verification
|
||||
- E2E: `tests/e2e/specs/16-marreta.spec.ts` asserts homepage tile passes guard chain.
|
||||
-2
@@ -1,2 +0,0 @@
|
||||
Run: npx playwright test tests/e2e/specs/16-marreta.spec.ts
|
||||
Assert: Homepage tile passes Anubis/ProxyProvider guard chain without login-wall parking.
|
||||
-5
@@ -1,5 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Health check through netns
|
||||
podman exec marreta_caddy wget -qO- http://localhost:8017/ || echo "HTTP 000 expected"
|
||||
# Grep config for route flip
|
||||
cat /etc/caddy/config.json | grep "reverse_proxy"
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
application:
|
||||
name: Marreta
|
||||
slug: marreta
|
||||
provider: marreta-proxy
|
||||
flows:
|
||||
authentication: enabled
|
||||
authorization: enabled
|
||||
invalidation: enabled
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
To retrieve logs for a rootless Quadlet service (e.g., frank-large):
|
||||
1. Use `podman logs frank-large` for container-specific output.
|
||||
2. Use `journalctl --user -u frank-large` for systemd-integrated output.
|
||||
3. Use `podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'` to filter for memory/allocations.
|
||||
+100
-15
@@ -2,19 +2,52 @@
|
||||
# - apiKey "EMPTY" -> auth: none (keyless local endpoint)
|
||||
# - thinkingLevelMap {max: xhigh} -> compat.reasoningEffortMap
|
||||
providers:
|
||||
# The provider NAMES are historical (they were llama.cpp endpoints when the
|
||||
# fleet ran llama.cpp): frank-medium is vllm-radiance now, and frank-large is
|
||||
# halogen-flash-server again. Renaming is deliberately avoided because every
|
||||
# modelRoles entry in config.yml embeds the provider name, and a partial
|
||||
# rename fails at call time — the label is cosmetic, the references are not.
|
||||
# One provider per HOST:PORT, so Dankclaw's second endpoint (:7771) is a
|
||||
# separate block rather than a second model inside the :7770 one.
|
||||
"llamacpp (dankserver)":
|
||||
baseUrl: http://192.168.0.235:7770/v1
|
||||
api: openai-completions
|
||||
auth: none
|
||||
models:
|
||||
- id: frank-small
|
||||
name: "frank-small (dankserver)"
|
||||
contextWindow: 262144
|
||||
maxTokens: 262144
|
||||
- id: frank-medium
|
||||
name: "frank-medium (dankserver)"
|
||||
# Dankserver: both Radeon AI PRO R9700 (gfx1201) serving one model —
|
||||
# mainline llama.cpp, `--split-mode layer`, on
|
||||
# ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with
|
||||
# the z-lab DFlash2 Q8_0 drafter. Tensor parallelism is impossible on
|
||||
# this host (the second card's chipset path has no PCIe atomic routing,
|
||||
# so it cannot dispatch an RCCL collective — and vLLM's pipeline
|
||||
# parallelism needs the same p2p primitive), which is why one process
|
||||
# layer-splits across both cards instead. Since 2026-09-16 this is the
|
||||
# box's only LLM runtime; the vllm-radiance replica pair is retired.
|
||||
contextWindow: 196608
|
||||
# The KV pool is 393216 positions over TWO slots, so one conversation
|
||||
# carries 196608 -- the house rule (see frank-large) is to declare the
|
||||
# per-conversation window, not the pool. The checkpoint's trained
|
||||
# window is 262144 and the GGUF has no rope scaling, so the top 131k
|
||||
# of a conversation extrapolates on the base rope. 32768 keeps prompt
|
||||
# + budget comfortably inside it; thinking counts against the budget.
|
||||
maxTokens: 32768
|
||||
reasoning: true
|
||||
input: [text]
|
||||
input: [text, image]
|
||||
compat:
|
||||
# Qwen3.8's chat template accepts ONLY xhigh, medium and low for
|
||||
# reasoning_effort and RAISES on anything else — verified against this
|
||||
# endpoint on 2026-09-16: low/medium/xhigh answer 200, `high` returns
|
||||
# a 500 from the template. The harness's levels must therefore be
|
||||
# folded onto that set, exactly as the sibling frank-small entry does,
|
||||
# or any role whose level is unmapped fails on its first request
|
||||
# (subagents run at `high`).
|
||||
reasoningEffortMap:
|
||||
minimal: low
|
||||
low: low
|
||||
medium: medium
|
||||
high: xhigh
|
||||
max: xhigh
|
||||
|
||||
"llamacpp (dankclaw)":
|
||||
@@ -24,20 +57,25 @@ providers:
|
||||
models:
|
||||
- id: frank-large
|
||||
name: "frank-large (dankclaw-gpu)"
|
||||
contextWindow: 200000
|
||||
# Halogen caps max_tokens at HALOGEN_MAX_TOKENS_CAP=65536 (a request
|
||||
# above it is a 400). Admission reserves prompt + max_tokens against
|
||||
# the KV pool (401,408 positions since 2026-09-13): 200,000 + 65,536
|
||||
# = 265,536, comfortably inside. Overshooting the pool hangs for the
|
||||
# 3600 s queue timeout before a 503.
|
||||
# halogen-flash-server. The model's native window is 262,144, but this
|
||||
# deployment caps a conversation at 200,192 — exactly half of its
|
||||
# 400,384-position KV pool, so two 200k conversations are resident at
|
||||
# once. Declaring the half rather than the native window is what keeps
|
||||
# prompt + budget inside one conversation's region of the pool.
|
||||
contextWindow: 200192
|
||||
# HALOGEN_MAX_TOKENS_CAP: above this a request is a 400 rather than a
|
||||
# silent truncation, and thinking counts against the budget.
|
||||
maxTokens: 65536
|
||||
reasoning: true
|
||||
input: [text, image]
|
||||
# Text only as of 2026-09-16: the vision tower moved to frank-small and
|
||||
# this server refuses image content parts.
|
||||
input: [text]
|
||||
compat:
|
||||
reasoningEffortMap:
|
||||
max: xhigh
|
||||
# Qwen3.8-Flash-Next recommended sampling; halogen has no server-side
|
||||
# sampler defaults, so without this the endpoint decodes greedy.
|
||||
# Qwen3.8-Flash-Next recommended sampling. Halogen has no server-side
|
||||
# sampler defaults (a request that sends no temperature decodes
|
||||
# greedily), so this override is what makes ordinary calls sample.
|
||||
# extraBody is merged last (after compat policy) and wins.
|
||||
extraBody:
|
||||
temperature: 1.0
|
||||
@@ -47,5 +85,52 @@ providers:
|
||||
presence_penalty: 0.0
|
||||
repetition_penalty: 1.0
|
||||
# Keep prior thinking blocks in the rendered prompt across turns;
|
||||
# halogen honours it top-level (verified: 275-token prompt delta vs false).
|
||||
# halogen honours this key top-level (verified: 275-token prompt
|
||||
# delta with it false).
|
||||
preserve_thinking: true
|
||||
|
||||
"llamacpp (dankclaw-3090)":
|
||||
baseUrl: http://192.168.0.143:7771/v1
|
||||
api: openai-completions
|
||||
auth: none
|
||||
models:
|
||||
- id: frank-small
|
||||
name: "frank-small (dankclaw-3090)"
|
||||
# Dankclaw's RTX 3090: mainline llama.cpp serving
|
||||
# HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M.
|
||||
# The MTP head ships as a separate assistant sidecar
|
||||
# (mtp-gemma-4-12B-it.gguf) loaded via --model-draft and chained
|
||||
# behind the draftless ngram-mod matcher; q8_0 KV with the model's
|
||||
# sliding-window pattern keeps each full 262144-token window cheap,
|
||||
# and the explicit CTX_SIZE=1048576 / PARALLEL=4 gives FOUR full
|
||||
# windows (measured 22.3 GiB of the 24 GiB card at load; a 5th slot
|
||||
# does not fit). The BF16 projector (with audio encoder) is loaded
|
||||
# too, so this stays multimodal (frank-medium gained vision on
|
||||
# 2026-09-17 as well; frank-large has no vision tower).
|
||||
contextWindow: 262144
|
||||
# Each slot IS the full trained window: prompt + budget must fit
|
||||
# inside it, and thinking counts against the budget.
|
||||
maxTokens: 32768
|
||||
reasoning: true
|
||||
input: [text, image]
|
||||
compat:
|
||||
# Gemma4's chat template accepts every reasoning_effort level as of
|
||||
# 2026-09-17 (all seven probed: 200), so this map is inert. Kept
|
||||
# anyway: it cost nothing, and it is the safety net if the template
|
||||
# ever gates again the way Qwen3.8's did (subagents run at `high`).
|
||||
reasoningEffortMap:
|
||||
minimal: low
|
||||
low: low
|
||||
medium: medium
|
||||
high: xhigh
|
||||
max: xhigh
|
||||
# Gemma's reference local-recipe sampling (temp 1.0 / top_p 0.95);
|
||||
# the server's own default is greedy for clients that stay silent.
|
||||
# Thinking is already retained across turns by the server's
|
||||
# --reasoning-preserve default, and the key below is accepted
|
||||
# top-level (verified 200, not a 400), so it is passed to pin the
|
||||
# behaviour at the request rather than rely on a server default.
|
||||
extraBody:
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
preserve_thinking: true
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: code-review
|
||||
description: "Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes: Standards (does the code follow this repo's documented coding standards?) and Spec (does the code match what the originating issue/spec asked for?). Runs both reviews in parallel sub-agents and reports them side by side. Use when the user wants to review a branch, a PR, work-in-progress changes, or asks to \"review since X\"."
|
||||
---
|
||||
|
||||
Two-axis review of the diff between `HEAD` and a fixed point the user supplies:
|
||||
|
||||
- **Standards**: does the code conform to this repo's documented coding standards?
|
||||
- **Spec**: does the code faithfully implement the originating issue / spec?
|
||||
|
||||
Both axes run as **parallel sub-agents** so they don't pollute each other's context, then this skill aggregates their findings.
|
||||
|
||||
The issue tracker should have been provided to you. If `docs/agents/issue-tracker.md` is missing, tell the user to run `/setup-matt-pocock-skills`.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Pin the fixed point
|
||||
|
||||
Whatever the user said is the fixed point (a commit SHA, branch name, tag, `main`, `HEAD~5`, etc.). If they didn't specify one, ask for it.
|
||||
|
||||
Capture the diff command once: `git diff <fixed-point>...HEAD` (three-dot, so the comparison is against the merge-base). Also note the list of commits via `git log <fixed-point>..HEAD --oneline`.
|
||||
|
||||
Before going further, confirm the fixed point resolves (`git rev-parse <fixed-point>`) and the diff is non-empty. A bad ref or empty diff should fail here, not inside two parallel sub-agents.
|
||||
|
||||
### 2. Identify the spec source
|
||||
|
||||
Look for the originating spec, in this order:
|
||||
|
||||
1. Issue references in the commit messages (`#123`, `Closes #45`, GitLab `!67`, etc.), fetched via the workflow in `docs/agents/issue-tracker.md`.
|
||||
2. A path the user passed as an argument.
|
||||
3. A spec file under `docs/`, `specs/`, or `.scratch/` matching the branch name or feature.
|
||||
4. If nothing is found, ask the user where the spec is. If they say there isn't one, the **Spec** sub-agent will skip and report "no spec available".
|
||||
|
||||
### 3. Identify the standards sources
|
||||
|
||||
Anything in the repo that documents how code should be written, such as `CODING_STANDARDS.md` or `CONTRIBUTING.md`.
|
||||
|
||||
On top of whatever the repo documents, the Standards axis always carries the **smell baseline** below: a fixed set of Fowler code smells (_Refactoring_, ch.3) that applies even when a repo documents nothing. Two rules bind it:
|
||||
|
||||
- **The repo overrides.** A documented repo standard always wins; where it endorses something the baseline would flag, suppress the smell.
|
||||
- **Always a judgement call.** Each smell is a labelled heuristic ("possible Feature Envy"), never a hard violation. Like any standard here, skip anything tooling already enforces.
|
||||
|
||||
Each smell reads *what it is* → *how to fix*; match it against the diff:
|
||||
|
||||
- **Mysterious Name**: a function, variable, or type whose name doesn't reveal what it does or holds. → rename it; if no honest name comes, the design's murky.
|
||||
- **Duplicated Code**: the same logic shape appears in more than one hunk or file in the change. → extract the shared shape, call it from both.
|
||||
- **Feature Envy**: a method that reaches into another object's data more than its own. → move the method onto the data it envies.
|
||||
- **Data Clumps**: the same few fields or params keep travelling together (a type wanting to be born). → bundle them into one type, pass that.
|
||||
- **Primitive Obsession**: a primitive or string standing in for a domain concept that deserves its own type. → give the concept its own small type.
|
||||
- **Repeated Switches**: the same `switch`/`if`-cascade on the same type recurs across the change. → replace with polymorphism, or one map both sites share.
|
||||
- **Shotgun Surgery**: one logical change forces scattered edits across many files in the diff. → gather what changes together into one module.
|
||||
- **Divergent Change**: one file or module is edited for several unrelated reasons. → split so each module changes for one reason.
|
||||
- **Speculative Generality**: abstraction, parameters, or hooks added for needs the spec doesn't have. → delete it; inline back until a real need shows.
|
||||
- **Message Chains**: long `a.b().c().d()` navigation the caller shouldn't depend on. → hide the walk behind one method on the first object.
|
||||
- **Middle Man**: a class or function that mostly just delegates onward. → cut it, call the real target direct.
|
||||
- **Refused Bequest**: a subclass or implementer that ignores or overrides most of what it inherits. → drop the inheritance, use composition.
|
||||
|
||||
### 4. Spawn both sub-agents in parallel
|
||||
|
||||
**Standards sub-agent prompt** should include:
|
||||
|
||||
- The full diff command and commit list.
|
||||
- The list of standards-source files you found in step 3, **plus the smell baseline from step 3** pasted in full (the sub-agent has no other access to it).
|
||||
- The brief: "Report, per file/hunk where relevant, (a) every place the diff violates a documented standard: cite the standard (file + the rule); and (b) any baseline smell you spot: name it and quote the hunk. Distinguish hard violations from judgement calls: documented-standard breaches can be hard, but baseline smells are always judgement calls, and a documented repo standard overrides the baseline. Skip anything tooling enforces. Under 400 words."
|
||||
|
||||
**Spec sub-agent prompt** should include:
|
||||
|
||||
- The diff command and commit list.
|
||||
- The path or fetched contents of the spec.
|
||||
- The brief: "Report: (a) requirements the spec asked for that are missing or partial; (b) behaviour in the diff that wasn't asked for (scope creep); (c) requirements that look implemented but where the implementation looks wrong. Quote the spec line for each finding. Under 400 words."
|
||||
|
||||
If the spec is missing, skip the Spec sub-agent and note this in the final report.
|
||||
|
||||
### 5. Aggregate
|
||||
|
||||
Present the two reports under `## Standards` and `## Spec` headings, verbatim or lightly cleaned. Do **not** merge or rerank findings, because the two axes are deliberately separate (see _Why two axes_).
|
||||
|
||||
End with a one-line summary: total findings per axis, and the worst issue _within each axis_ (if any). Don't pick a single winner across axes: that's the reranking the separation exists to prevent.
|
||||
|
||||
## Why two axes
|
||||
|
||||
A change can pass one axis and fail the other:
|
||||
|
||||
- Code that follows every standard but implements the wrong thing → **Standards pass, Spec fail.**
|
||||
- Code that does exactly what the issue asked but breaks the project's conventions → **Spec pass, Standards fail.**
|
||||
|
||||
Reporting them separately stops one axis from masking the other.
|
||||
@@ -0,0 +1,3 @@
|
||||
interface:
|
||||
display_name: "Code Review"
|
||||
short_description: "Review a diff on standards and spec"
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
name: diagnosing-bugs
|
||||
description: Diagnosis loop for hard bugs and performance regressions. Use when the user says "diagnose"/"debug this", or reports something broken/throwing/failing/slow.
|
||||
---
|
||||
|
||||
# Diagnosing Bugs
|
||||
|
||||
A discipline for hard bugs. Skip phases only when explicitly justified.
|
||||
|
||||
When exploring the codebase, read `CONTEXT.md` (if it exists) to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.
|
||||
|
||||
## Redact
|
||||
|
||||
This skill has you show commands, outputs and captured artifacts. **Redact every secret first**: write `<REDACTED>` in its place. Build loops against env vars, so the credential stays in the environment rather than in what you show. Captured artifacts carry auth headers: quote only the lines that carry the signal.
|
||||
|
||||
If the redacted output is not enough to diagnose the bug, say so and ask the user.
|
||||
|
||||
## Phase 1: Build a feedback loop
|
||||
|
||||
**This is the skill.** Everything else is mechanical. If you have a **tight** pass/fail signal for the bug (one that goes red on _this_ bug), you will find the cause; bisection, hypothesis-testing, and instrumentation all just consume it. If you don't have one, no amount of staring at code will save you.
|
||||
|
||||
Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**
|
||||
|
||||
### Ways to construct one, in roughly this order
|
||||
|
||||
1. **Failing test** at whatever seam reaches the bug: unit, integration, e2e.
|
||||
2. **Curl / HTTP script** against a running dev server.
|
||||
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
|
||||
4. **Headless browser script** (Playwright / Puppeteer) that drives the UI and asserts on DOM/console/network.
|
||||
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
|
||||
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
|
||||
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
|
||||
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
|
||||
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
|
||||
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
|
||||
|
||||
Build the right feedback loop, and the bug is 90% fixed.
|
||||
|
||||
### Tighten the loop
|
||||
|
||||
Treat the loop as a product. Once you have _a_ loop, **tighten** it:
|
||||
|
||||
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
|
||||
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
|
||||
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
|
||||
|
||||
A 30-second flaky loop is barely better than no loop; a 2-second deterministic one is tight, a debugging superpower.
|
||||
|
||||
### Non-deterministic bugs
|
||||
|
||||
The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not, so keep raising the rate until it's debuggable.
|
||||
|
||||
### When you genuinely cannot build a loop
|
||||
|
||||
Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a redacted captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.
|
||||
|
||||
### Completion criterion: a tight loop that goes red
|
||||
|
||||
Phase 1 is done when the loop is **tight** and **red-capable**: you can name **one command** (a script path, a test invocation, a curl) that you have **already run at least once** (show the invocation and its output, redacted), and that is:
|
||||
|
||||
- [ ] **Red-capable**: it drives the actual bug code path and asserts the **user's exact symptom**, so it can go red on this bug and green once fixed. Not "runs without erroring"; it must be able to _catch this specific bug_.
|
||||
- [ ] **Deterministic**: same verdict every run (flaky bugs: a pinned, high reproduction rate, per above).
|
||||
- [ ] **Fast**: seconds, not minutes.
|
||||
- [ ] **Agent-runnable**: you can run it unattended; a human in the loop only via `scripts/hitl-loop.template.sh`.
|
||||
|
||||
If you catch yourself reading code to build a theory before this command exists, **stop: jumping straight to a hypothesis is the exact failure this skill prevents.** No red-capable command, no Phase 2.
|
||||
|
||||
## Phase 2: Reproduce + minimise
|
||||
|
||||
Run the loop. Watch it go red as the bug appears.
|
||||
|
||||
Confirm:
|
||||
|
||||
- [ ] The loop produces the failure mode the **user** described, not a different failure that happens to be nearby. Wrong bug = wrong fix.
|
||||
- [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
|
||||
- [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.
|
||||
|
||||
### Minimise
|
||||
|
||||
Once it's red, shrink the repro to the **smallest scenario that still goes red**. Cut inputs, callers, config, data, and steps **one at a time**, re-running the loop after each cut, and keep only what's load-bearing for the failure.
|
||||
|
||||
Why bother: a minimal repro shrinks the hypothesis space in Phase 3 (fewer moving parts left to suspect) and becomes the clean regression test in Phase 5.
|
||||
|
||||
Done when **every remaining element is load-bearing**: removing any one of them makes the loop go green.
|
||||
|
||||
Do not proceed until you have reproduced **and** minimised.
|
||||
|
||||
## Phase 3: Hypothesise
|
||||
|
||||
Generate **3–5 ranked hypotheses** before testing any of them. Single-hypothesis generation anchors on the first plausible idea.
|
||||
|
||||
Each hypothesis must be **falsifiable**: state the prediction it makes.
|
||||
|
||||
> Format: "If <X> is the cause, then <changing Y> will make the bug disappear / <changing Z> will make it worse."
|
||||
|
||||
If you cannot state the prediction, the hypothesis is a vibe: discard or sharpen it.
|
||||
|
||||
**Show the ranked list to the user before testing.** They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it; proceed with your ranking if the user is AFK.
|
||||
|
||||
## Phase 4: Instrument
|
||||
|
||||
Each probe must map to a specific prediction from Phase 3. **Change one variable at a time.**
|
||||
|
||||
Tool preference:
|
||||
|
||||
1. **Debugger / REPL inspection** if the env supports it. One breakpoint beats ten logs.
|
||||
2. **Targeted logs** at the boundaries that distinguish hypotheses.
|
||||
3. Never "log everything and grep".
|
||||
|
||||
**Tag every debug log** with a unique prefix, e.g. `[DEBUG-a4f2]`. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.
|
||||
|
||||
**Perf branch.** For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, `performance.now()`, profiler, query plan), then bisect. Measure first, fix second.
|
||||
|
||||
## Phase 5: Fix + regression test
|
||||
|
||||
Write the regression test **before the fix**, but only if there is a **correct seam** for it.
|
||||
|
||||
A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.
|
||||
|
||||
**If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.
|
||||
|
||||
If a correct seam exists:
|
||||
|
||||
1. Turn the minimised repro into a failing test at that seam.
|
||||
2. Watch it fail.
|
||||
3. Apply the fix.
|
||||
4. Watch it pass.
|
||||
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.
|
||||
|
||||
## Phase 6: Cleanup
|
||||
|
||||
Required before declaring done:
|
||||
|
||||
- [ ] Original repro no longer reproduces (re-run the Phase 1 loop)
|
||||
- [ ] Regression test passes (or absence of seam is documented)
|
||||
- [ ] All `[DEBUG-...]` instrumentation removed (`grep` the prefix)
|
||||
- [ ] Throwaway prototypes deleted (or moved to a clearly-marked debug location)
|
||||
- [ ] The hypothesis that turned out correct is stated in the commit / PR message, so the next debugger learns
|
||||
@@ -0,0 +1,3 @@
|
||||
interface:
|
||||
display_name: "Diagnosing Bugs"
|
||||
short_description: "Diagnose hard bugs and regressions"
|
||||
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# Human-in-the-loop reproduction loop.
|
||||
# Copy this file, edit the steps below, and run it.
|
||||
# The agent runs the script; the user follows prompts in their terminal.
|
||||
#
|
||||
# Usage:
|
||||
# bash hitl-loop.template.sh
|
||||
#
|
||||
# Two helpers:
|
||||
# step "<instruction>" → show instruction, wait for Enter
|
||||
# capture VAR "<question>" → show question, read response into VAR
|
||||
#
|
||||
# At the end, captured values are printed as KEY=VALUE for the agent to parse.
|
||||
#
|
||||
# `capture` prints its value back to the terminal, where the agent reads it,
|
||||
# so capture observations, and leave signing in to the user as a `step`.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
step() {
|
||||
printf '\n>>> %s\n' "$1"
|
||||
read -r -p " [Enter when done] " _
|
||||
}
|
||||
|
||||
capture() {
|
||||
local var="$1" question="$2" answer
|
||||
printf '\n>>> %s\n' "$question"
|
||||
read -r -p " > " answer
|
||||
printf -v "$var" '%s' "$answer"
|
||||
}
|
||||
|
||||
# --- edit below ---------------------------------------------------------
|
||||
|
||||
step "Open the app at http://localhost:3000 and sign in."
|
||||
|
||||
capture ERRORED "Click the 'Export' button. Did it throw an error? (y/n)"
|
||||
|
||||
capture ERROR_MSG "Paste the error message (or 'none'):"
|
||||
|
||||
# --- edit above ---------------------------------------------------------
|
||||
|
||||
printf '\n--- Captured ---\n'
|
||||
printf 'ERRORED=%s\n' "$ERRORED"
|
||||
printf 'ERROR_MSG=%s\n' "$ERROR_MSG"
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
name: find-skills
|
||||
description: Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.
|
||||
---
|
||||
|
||||
# Find Skills
|
||||
|
||||
This skill helps you discover and install skills from the open agent skills ecosystem.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when the user:
|
||||
|
||||
- Asks "how do I do X" where X might be a common task with an existing skill
|
||||
- Says "find a skill for X" or "is there a skill for X"
|
||||
- Asks "can you do X" where X is a specialized capability
|
||||
- Expresses interest in extending agent capabilities
|
||||
- Wants to search for tools, templates, or workflows
|
||||
- Mentions they wish they had help with a specific domain (design, testing, deployment, etc.)
|
||||
|
||||
## What is the Skills CLI?
|
||||
|
||||
The Skills CLI (`npx skills`) is the package manager for the open agent skills ecosystem. Skills are modular packages that extend agent capabilities with specialized knowledge, workflows, and tools.
|
||||
|
||||
**Key commands:**
|
||||
|
||||
- `npx skills find [query] [--owner <owner>]` - Search for skills interactively or by keyword, optionally scoped to a GitHub owner
|
||||
- `npx skills add <package>` - Install a skill from GitHub or other sources
|
||||
- `npx skills update` - Update all installed skills
|
||||
|
||||
**Browse skills at:** https://skills.sh/
|
||||
|
||||
## How to Help Users Find Skills
|
||||
|
||||
### Step 1: Understand What They Need
|
||||
|
||||
When a user asks for help with something, identify:
|
||||
|
||||
1. The domain (e.g., React, testing, design, deployment)
|
||||
2. The specific task (e.g., writing tests, creating animations, reviewing PRs)
|
||||
3. Whether this is a common enough task that a skill likely exists
|
||||
|
||||
### Step 2: Check the Leaderboard First
|
||||
|
||||
Before running a CLI search, check the [skills.sh leaderboard](https://skills.sh/) to see if a well-known skill already exists for the domain. The leaderboard ranks skills by total installs, surfacing the most popular and battle-tested options.
|
||||
|
||||
For example, top skills for web development include:
|
||||
- `vercel-labs/agent-skills` — React, Next.js, web design (100K+ installs each)
|
||||
- `anthropics/skills` — Frontend design, document processing (100K+ installs)
|
||||
|
||||
### Step 3: Search for Skills
|
||||
|
||||
If the leaderboard doesn't cover the user's need, run the find command:
|
||||
|
||||
```bash
|
||||
npx skills find [query] [--owner <owner>]
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
- User asks "how do I make my React app faster?" → `npx skills find react performance`
|
||||
- User asks "can you help me with PR reviews?" → `npx skills find pr review`
|
||||
- User asks "I need to create a changelog" → `npx skills find changelog`
|
||||
|
||||
### Step 4: Verify Quality Before Recommending
|
||||
|
||||
**Do not recommend a skill based solely on search results.** Always verify:
|
||||
|
||||
1. **Install count** — Prefer skills with 1K+ installs. Be cautious with anything under 100.
|
||||
2. **Source reputation** — Official sources (`vercel-labs`, `anthropics`, `microsoft`) are more trustworthy than unknown authors.
|
||||
3. **GitHub stars** — Check the source repository. A skill from a repo with <100 stars should be treated with skepticism.
|
||||
|
||||
### Step 5: Present Options to the User
|
||||
|
||||
When you find relevant skills, present them to the user with:
|
||||
|
||||
1. The skill name and what it does
|
||||
2. The install count and source
|
||||
3. The install command they can run
|
||||
4. A link to learn more at skills.sh
|
||||
|
||||
Example response:
|
||||
|
||||
```
|
||||
I found a skill that might help! The "react-best-practices" skill provides
|
||||
React and Next.js performance optimization guidelines from Vercel Engineering.
|
||||
(185K installs)
|
||||
|
||||
To install it:
|
||||
npx skills add vercel-labs/agent-skills@react-best-practices
|
||||
|
||||
Learn more: https://skills.sh/vercel-labs/agent-skills/react-best-practices
|
||||
```
|
||||
|
||||
### Step 6: Offer to Install
|
||||
|
||||
If the user wants to proceed, you can install the skill for them:
|
||||
|
||||
```bash
|
||||
npx skills add <owner/repo@skill> -g -y
|
||||
```
|
||||
|
||||
The `-g` flag installs globally (user-level) and `-y` skips confirmation prompts.
|
||||
|
||||
## Common Skill Categories
|
||||
|
||||
When searching, consider these common categories:
|
||||
|
||||
| Category | Example Queries |
|
||||
| --------------- | ---------------------------------------- |
|
||||
| Web Development | react, nextjs, typescript, css, tailwind |
|
||||
| Testing | testing, jest, playwright, e2e |
|
||||
| DevOps | deploy, docker, kubernetes, ci-cd |
|
||||
| Documentation | docs, readme, changelog, api-docs |
|
||||
| Code Quality | review, lint, refactor, best-practices |
|
||||
| Design | ui, ux, design-system, accessibility |
|
||||
| Productivity | workflow, automation, git |
|
||||
|
||||
## Tips for Effective Searches
|
||||
|
||||
1. **Use specific keywords**: "react testing" is better than just "testing"
|
||||
2. **Try alternative terms**: If "deploy" doesn't work, try "deployment" or "ci-cd"
|
||||
3. **Check popular sources**: Many skills come from `vercel-labs/agent-skills` or `ComposioHQ/awesome-claude-skills`
|
||||
|
||||
## When No Skills Are Found
|
||||
|
||||
If no relevant skills exist:
|
||||
|
||||
1. Acknowledge that no existing skill was found
|
||||
2. Offer to help with the task directly using your general capabilities
|
||||
3. Suggest the user could create their own skill with `npx skills init`
|
||||
|
||||
Example:
|
||||
|
||||
```
|
||||
I searched for skills related to "xyz" but didn't find any matches.
|
||||
I can still help you with this task directly! Would you like me to proceed?
|
||||
|
||||
If this is something you do often, you could create your own skill:
|
||||
npx skills init my-xyz-skill
|
||||
```
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: grilling
|
||||
description: Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrases.
|
||||
---
|
||||
|
||||
Interview the user relentlessly until you reach a shared understanding. Map this as a **design tree**: every decision branches into the decisions that hang off it.
|
||||
|
||||
Work the tree in **rounds**. The **frontier** is every decision whose prerequisites are already settled: the questions you can ask _now_ without guessing at answers you haven't heard yet. Ask the whole frontier in one round: number each question and give your recommended answer. Then wait for the user's answers before the next round.
|
||||
|
||||
Format a round like so:
|
||||
|
||||
```
|
||||
❓ **Q1** - **<question title>**: <question body, might be multiple paragraphs, including multiple choices>
|
||||
|
||||
➡️ <your recommended answer>
|
||||
|
||||
---
|
||||
|
||||
❓ **Q2** - **<question title>**: <question body, might be multiple paragraphs, including multiple choices>
|
||||
|
||||
➡️ <your recommended answer>
|
||||
```
|
||||
|
||||
Each round the user answers reshapes the tree: settled decisions push the frontier outward and unblock questions that depended on them. Recompute the frontier and ask the next round. A question whose answer depends on another question still open in this round belongs to a _later_ round, not this one.
|
||||
|
||||
Finding _facts_ is your job, never the user's. When a frontier question needs a fact from the environment (filesystem, tools, etc.), dispatch a sub-agent to find it; don't ask the user for anything you could look up yourself. Don't block on it: a running exploration is an unsettled prerequisite, so only the questions downstream of it wait for the sub-agent to report; ask the rest of the frontier now. The _decisions_ are the user's: put each to them and wait.
|
||||
|
||||
The session is done when the frontier is empty: every branch of the design tree visited, nothing left silently assumed. Do not act on it until the user confirms you have reached a shared understanding.
|
||||
@@ -0,0 +1,3 @@
|
||||
interface:
|
||||
display_name: "Grilling"
|
||||
short_description: "Stress-test thinking a round of questions at a time"
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
name: handoff
|
||||
description: Compact the current conversation into a handoff document for another agent to pick up.
|
||||
argument-hint: "What will the next session be used for?"
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
Write a handoff document summarising the current conversation so a fresh agent can continue the work. Save to the temporary directory of the user's OS - not the current workspace.
|
||||
|
||||
Include a "suggested skills" section in the document, naming which skills the next agent should call the Skill tool for.
|
||||
|
||||
Do not duplicate content already captured in other artifacts (specs, plans, ADRs, issues, commits, diffs). Reference them by path or URL instead.
|
||||
|
||||
Redact any sensitive information, such as API keys, passwords, or personally identifiable information.
|
||||
|
||||
If the user passed arguments, treat them as a description of what the next session will focus on and tailor the doc accordingly.
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Handoff"
|
||||
short_description: "Compact a conversation into a handoff"
|
||||
policy:
|
||||
allow_implicit_invocation: false
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: implement
|
||||
description: "Implement a piece of work based on a spec or set of tickets."
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
Implement the work described by the user in the spec or tickets.
|
||||
|
||||
Use /tdd where possible, at pre-agreed seams.
|
||||
|
||||
Run typechecking regularly, single test files regularly, and the full test suite once at the end.
|
||||
|
||||
Once done, use /code-review to review the work.
|
||||
|
||||
Commit your work to the current branch.
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Implement"
|
||||
short_description: "Build work from a spec or tickets"
|
||||
policy:
|
||||
allow_implicit_invocation: false
|
||||
@@ -0,0 +1,67 @@
|
||||
# Logic Prototype
|
||||
|
||||
A single, self-contained HTML file (a **shareable demo**) that lets anyone drive a state model by clicking buttons. Use this when the question is about **business logic, state transitions, or data shape**: the kind of thing that looks reasonable on paper but only feels wrong once you push it through real cases.
|
||||
|
||||
Because it's one file with nothing to install, you can hand it to a non-developer (a designer, a PM, a domain expert) and let them feel the model for themselves. So it speaks their language, not the code's.
|
||||
|
||||
## When this is the right shape
|
||||
|
||||
- "I'm not sure if this state machine handles the edge case where X then Y."
|
||||
- "Does this data model actually let me represent the case where..."
|
||||
- "I want to feel out what the API should look like before writing it."
|
||||
- Anything where someone wants to **press buttons and watch state change**.
|
||||
|
||||
If the question is "what should this look like," this is the wrong branch. Use [UI.md](UI.md).
|
||||
|
||||
## Process
|
||||
|
||||
### 1. State the question
|
||||
|
||||
Before writing code, write down what state model and what question you're prototyping. One paragraph, at the top of the demo (in a visible intro, not just a comment). A logic prototype that answers the wrong question is pure waste, so make the question explicit so it can be checked later, whether the user is watching now or returning to it AFK.
|
||||
|
||||
### 2. Isolate the logic in a portable module
|
||||
|
||||
Put the actual logic (the bit that's answering the question) in a single `<script>` block written as a small, pure module that could be lifted out and dropped into the real codebase later. The page around it is throwaway; this module isn't.
|
||||
|
||||
The right shape depends on the question:
|
||||
|
||||
- **A pure reducer**: `(state, action) => state`. Good when actions are discrete events and state is a single value.
|
||||
- **A state machine**: explicit states and transitions. Good when "which actions are even legal right now" is part of the question.
|
||||
- **A small set of pure functions** over a plain data type. Good when there's no implicit current state, just transformations.
|
||||
- **A class or module with a clear method surface** when the logic genuinely owns ongoing internal state.
|
||||
|
||||
Pick whichever shape best fits the question being asked, *not* whichever is easiest to wire to a page. Keep it pure: no DOM, no `document`, no button handlers reaching inside it. The page calls into it; nothing flows the other direction. This is what makes the prototype useful past its own lifetime: once the question's answered, the validated reducer / machine / function set lifts into the real module on its own.
|
||||
|
||||
### 3. Build the shareable HTML file
|
||||
|
||||
One file, plain HTML/CSS/JS: no framework, no bundler, no server, everything inline so it opens by double-click and survives being emailed around. Anyone should be able to run it by opening it.
|
||||
|
||||
Write it for a non-developer. Every label is in **domain language**, not code: buttons and state read like the business, not the reducer. Explain in plain words what's happening.
|
||||
|
||||
Lay it out with a clean hierarchy, top to bottom:
|
||||
|
||||
1. **Title and one-line explanation** of what this demo lets you explore (the question from step 1).
|
||||
2. **Current state**: the full relevant state, rendered as a readable panel (labelled fields, not a raw JSON dump), re-rendered after every click so the change is visible. Where it helps a non-developer follow, call out what just changed.
|
||||
3. **Free-play buttons**: one button per action, always available, so anyone can poke at the model in any order. Each click dispatches its action and re-renders the state.
|
||||
4. **Guided walkthroughs**: a set of **scenarios**, one per tab. Each tab holds a short plain-language description of the scenario (the situation it sets up and what to watch for) and underneath it, the ordered **buttons to press** for that scenario. Each step is a real button: clicking it performs that action and moves to the next step. Starting a walkthrough resets to a known initial state so the scenario runs the same way every time.
|
||||
|
||||
Choose scenarios that demonstrate the awkward cases, the ones hard to reason about on paper: the happy path, a tricky edge case, an attempt at something that should be illegal.
|
||||
|
||||
Keep it beautiful but restrained: clean typography, generous spacing, one accent colour. No animations, no gimmicks: nothing that competes with the state and the buttons.
|
||||
|
||||
### 4. Hand it over
|
||||
|
||||
Send them the file, or open it for them. They'll click through the walkthroughs and free-play whenever they get to it; the interesting moments are when they say "wait, that shouldn't be possible" or "huh, I assumed X would be different"; those are the bugs in the _idea_, which is the whole point. If they want new actions or a new scenario, add them. Prototypes evolve.
|
||||
|
||||
### 5. Capture the answer and the prototype
|
||||
|
||||
Once the prototype has answered its question, capture the answer, then capture the prototype the way the [SKILL](SKILL.md) describes. The logic-specific mapping: the validated reducer / machine / function set lifts into the real module (the decision, absorbed); the HTML shell rides along to the throwaway branch that keeps the prototype as a primary source, and being one self-contained file, it stays trivially re-runnable there.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't add tests.** A prototype that needs tests is no longer a prototype.
|
||||
- **Don't wire it to the real database.** Use in-memory state unless the question is specifically about persistence.
|
||||
- **Don't generalise.** No "what if we wanted to support X later." The prototype answers one question.
|
||||
- **Don't blur the logic and the page together.** If the pure module references the DOM, `document`, or button handlers, it's no longer liftable. Keep the page as a thin shell over a pure module.
|
||||
- **Don't reach for a framework, bundler, or server.** One file the recipient double-clicks; a React app or a dev server defeats "shareable".
|
||||
- **Don't ship the HTML shell into production.** The page is optimised for being clicked through by hand. The logic module behind it is the bit worth keeping.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
name: prototype
|
||||
description: Build a throwaway prototype to answer a design question. Use when the user wants to sanity-check whether a state model or logic feels right, or explore what a UI should look like.
|
||||
---
|
||||
|
||||
# Prototype
|
||||
|
||||
A prototype is **throwaway code that answers a question**. The question decides the shape.
|
||||
|
||||
## Pick a branch
|
||||
|
||||
Identify which question is being answered, using the user's prompt, the surrounding code, or by asking if the user is around:
|
||||
|
||||
- **"Does this logic / state model feel right?"** → [LOGIC.md](LOGIC.md). Build a single shareable HTML file (free-play buttons plus tabbed guided walkthroughs) that pushes the state machine through cases that are hard to reason about on paper, and that a non-developer can drive.
|
||||
- **"What should this look like?"** → [UI.md](UI.md). Generate several radically different UI variations on a single route, switchable via a URL search param and a floating bottom bar.
|
||||
|
||||
The two branches produce very different artifacts, so getting this wrong wastes the whole prototype. If the question is genuinely ambiguous and the user isn't reachable, default to whichever branch better matches the surrounding code (a backend module → logic; a page or component → UI) and state the assumption at the top of the prototype.
|
||||
|
||||
## Rules that apply to both
|
||||
|
||||
1. **Throwaway from day one, and clearly marked as such.** Locate the prototype code close to where it will actually be used (next to the module or page it's prototyping for) so context is obvious, but name it so a casual reader can see it's a prototype, not production. For throwaway UI routes, obey whatever routing convention the project already uses; don't invent a new top-level structure.
|
||||
2. **Trivial to run.** A UI prototype starts from one command in the project's task runner: `pnpm <name>`, `python <path>`, `bun <path>`, etc. A logic demo is a single HTML file the user double-clicks. Either way, no thinking required to start it.
|
||||
3. **No persistence by default.** State lives in memory. Persistence is the thing the prototype is _checking_, not something it should depend on. If the question explicitly involves a database, hit a scratch DB or a local file with a clear "PROTOTYPE, wipe me" name.
|
||||
4. **Skip the polish.** No tests, no error handling beyond what makes the prototype _runnable_, no abstractions. The point is to learn something fast.
|
||||
5. **Surface the state.** After every action (logic) or on every variant switch (UI), print or render the full relevant state so the user can see what changed.
|
||||
6. **Capture it when done.** Fold any validated decision into the real code, then capture the prototype itself as a **primary source**: commit it to a throwaway branch, out of main, and leave a context pointer to that branch on the implementation issue. Capture the answer too (the verdict and the question it settled) in the issue or a commit. The main branch keeps only the validated decision.
|
||||
@@ -0,0 +1,112 @@
|
||||
# UI Prototype
|
||||
|
||||
Generate **several radically different UI variations** on a single route, switchable from a floating bottom bar. The user flips between variants in the browser, picks one (or steals bits from each), then throws the rest away.
|
||||
|
||||
If the question is about logic/state rather than what something looks like, this is the wrong branch. Use [LOGIC.md](LOGIC.md).
|
||||
|
||||
## When this is the right shape
|
||||
|
||||
- "What should this page look like?"
|
||||
- "I want to see a few options for this dashboard before committing."
|
||||
- "Try a different layout for the settings screen."
|
||||
- Any time the user would otherwise spend a day picking between three vague mockups in their head.
|
||||
|
||||
## Two sub-shapes: strongly prefer sub-shape A
|
||||
|
||||
A UI prototype is much easier to judge when it's **butting up against the rest of the app**: real header, real sidebar, real data, real density. A throwaway route on its own is a vacuum: every variant looks fine in isolation. Default to sub-shape A whenever there's a plausible existing page to host the variants. Only reach for sub-shape B if the prototype genuinely has no nearby home.
|
||||
|
||||
### Sub-shape A: adjustment to an existing page (preferred)
|
||||
|
||||
The route already exists. Variants are rendered **on the same route**, gated by a `?variant=` URL search param. The existing data fetching, params, and auth all stay. Only the rendering swaps. This is the default; pick it unless there's a specific reason not to.
|
||||
|
||||
If the prototype is for something that doesn't yet have a page but *would naturally live inside one* (a new section of the dashboard, a new card on the settings screen, a new step in an existing flow), it's still sub-shape A. Mount the variants inside the host page.
|
||||
|
||||
### Sub-shape B: a new page (last resort)
|
||||
|
||||
Only use this when the thing being prototyped genuinely has no existing page to live inside (e.g. an entirely new top-level surface, or a flow that can't be embedded anywhere sensible).
|
||||
|
||||
Create a **throwaway route** following whatever routing convention the project already uses. Don't invent a new top-level structure. Name it so it's obviously a prototype (e.g. include the word `prototype` in the path or filename). Same `?variant=` pattern.
|
||||
|
||||
Before committing to sub-shape B, sanity-check: is there really no existing page this could be embedded in? An empty route hides design problems that a populated one would expose.
|
||||
|
||||
In both sub-shapes the floating bottom bar is identical.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. State the question and pick N
|
||||
|
||||
Default to **3 variants**. More than 5 stops being radically different and starts being noise, so cap there.
|
||||
|
||||
Write down the plan in one line, in the prototype's location or a top-of-file comment:
|
||||
|
||||
> "Three variants of the settings page, switchable via `?variant=`, on the existing `/settings` route."
|
||||
|
||||
This works whether the user is here to push back or not.
|
||||
|
||||
### 2. Generate radically different variants
|
||||
|
||||
Draft each variant. Hold each one to:
|
||||
|
||||
- The page's purpose and the data it has access to.
|
||||
- The project's component library / styling system (TailwindCSS, shadcn, MUI, plain CSS, whatever).
|
||||
- A clear exported component name, e.g. `VariantA`, `VariantB`, `VariantC`.
|
||||
|
||||
Variants must be **structurally different**: different layout, different information hierarchy, different primary affordance, not just different colours. Three slightly-tweaked card grids isn't a UI prototype, it's wallpaper. If two drafts come out too similar, redo one with explicit "do not use a card grid" guidance.
|
||||
|
||||
### 3. Wire them together
|
||||
|
||||
Create a single switcher component on the route:
|
||||
|
||||
```tsx
|
||||
// pseudo-code, adapt to the project's framework
|
||||
const variant = searchParams.get('variant') ?? 'A';
|
||||
return (
|
||||
<>
|
||||
{variant === 'A' && <VariantA {...data} />}
|
||||
{variant === 'B' && <VariantB {...data} />}
|
||||
{variant === 'C' && <VariantC {...data} />}
|
||||
<PrototypeSwitcher variants={['A','B','C']} current={variant} />
|
||||
</>
|
||||
);
|
||||
```
|
||||
|
||||
For sub-shape A (existing page): keep all the existing data fetching above the switcher; only the rendered subtree changes per variant.
|
||||
|
||||
For sub-shape B (new page): the throwaway route under `/prototype/<name>` mounts the same switcher.
|
||||
|
||||
### 4. Build the floating switcher
|
||||
|
||||
A small fixed-position bar at the bottom-centre of the screen with three pieces:
|
||||
|
||||
- **Left arrow**: cycles to the previous variant (wraps around).
|
||||
- **Variant label**: shows the current variant key and, if the variant exports a name, that name too. e.g. `B (Sidebar layout)`.
|
||||
- **Right arrow**: cycles forward (wraps around).
|
||||
|
||||
Behaviour:
|
||||
|
||||
- Clicking an arrow updates the URL search param (use the framework's router, e.g. `router.replace` on Next, `navigate` on React Router, etc) so the variant is shareable and reload-stable.
|
||||
- Keyboard: `←` and `→` arrow keys also cycle. Don't intercept arrow keys when an `<input>`, `<textarea>`, or `[contenteditable]` is focused.
|
||||
- Visually distinct from the page (e.g. high-contrast pill, subtle shadow) so it's obviously not part of the design being evaluated.
|
||||
- Hidden in production builds: gate on `process.env.NODE_ENV !== 'production'` or an equivalent check, so a stray prototype merge can't ship the bar to users.
|
||||
|
||||
Put the switcher in a single shared component so both sub-shapes can reuse it. Locate it wherever shared UI lives in the project.
|
||||
|
||||
### 5. Hand it over
|
||||
|
||||
Surface the URL (and the `?variant=` keys). The user will flip through whenever they get to it. The interesting feedback is usually **"I want the header from B with the sidebar from C"**, which is the actual design they want.
|
||||
|
||||
### 6. Capture the answer and clean up
|
||||
|
||||
Once a variant has won, capture the answer (which variant and why), then capture the prototype the way the [SKILL](SKILL.md) describes. Fold the winner into the real code and move the rest onto the throwaway branch, not into main:
|
||||
|
||||
- **Sub-shape A**: fold the winner into the existing page; drop the losing variants and the switcher from main.
|
||||
- **Sub-shape B**: promote the winning variant to a real route; drop the throwaway route and the switcher from main.
|
||||
|
||||
The full set of variants is the primary source, so it lands on the throwaway branch, not the bin, since variant components and the switcher left in the main branch rot fast and confuse the next reader.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Variants that differ only in colour or copy.** That's a tweak, not a prototype. Real variants disagree about structure.
|
||||
- **Sharing too much code between variants.** A shared `<Header>` is fine; a shared `<Layout>` defeats the point. Each variant should be free to throw out the layout.
|
||||
- **Wiring variants to real mutations.** Read-only prototypes are fine. If a variant needs to mutate, point it at a stub: the question is "what should this look like", not "does the backend work".
|
||||
- **Promoting the prototype directly to production.** The variant code was written under prototype constraints (no tests, minimal error handling). Rewrite it properly when you fold it in.
|
||||
@@ -0,0 +1,3 @@
|
||||
interface:
|
||||
display_name: "Prototype"
|
||||
short_description: "Prototype to answer a design question"
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: tdd
|
||||
description: Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
|
||||
---
|
||||
|
||||
# Test-Driven Development
|
||||
|
||||
TDD is the red → green loop. This skill is the reference that makes that loop produce tests worth keeping: what a good test is, where tests go, the anti-patterns, and the rules of the loop. Every section applies on every cycle: consult them before and during the loop, not after.
|
||||
|
||||
When exploring the codebase, read `CONTEXT.md` (if it exists) so test names and interface vocabulary match the project's domain language, and respect ADRs in the area you're touching.
|
||||
|
||||
## What a good test is
|
||||
|
||||
Tests verify behavior through public interfaces, not implementation details. Code can change entirely; tests shouldn't. A good test reads like a specification: "user can checkout with valid cart" tells you exactly what capability exists, and it survives refactors because it doesn't care about internal structure.
|
||||
|
||||
See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines.
|
||||
|
||||
## Seams: where tests go
|
||||
|
||||
A **seam** is the public boundary you test at: the interface where you observe behavior without reaching inside. Tests live at seams, never against internals.
|
||||
|
||||
**Test only at pre-agreed seams.** Before writing any test, write down the seams under test and confirm them with the user. No test is written at an unconfirmed seam. You can't test everything, so agreeing the seams up front is how testing effort lands on the critical paths and complex logic instead of every edge case.
|
||||
|
||||
Ask: "What's the public interface, and which seams should we test?"
|
||||
|
||||
When the shape of that interface is itself in question (how deep the module is, where the seam belongs, what the interface should expose), call the Skill tool with "codebase-design" for the vocabulary. It is the shared source of the module, interface, depth, seam, adapter, leverage and locality terms, and it is a reference to consult, not a session to run.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Implementation-coupled**: mocks internal collaborators, tests private methods, or verifies through a side channel (querying the database instead of using the interface). The tell: the test breaks when you refactor but behavior hasn't changed.
|
||||
- **Tautological**: the assertion recomputes the expected value the way the code does (`expect(add(a, b)).toBe(a + b)`, a snapshot derived by hand the same way, a constant asserted equal to itself), so it passes by construction and can never disagree with the code. Expected values must come from an independent source of truth: a known-good literal, a worked example, the spec.
|
||||
- **Horizontal slicing**: writing all tests first, then all implementation. Bulk tests verify _imagined_ behavior: you test the _shape_ of things rather than user-facing behavior, the tests go insensitive to real changes, and you commit to test structure before understanding the implementation. Work in **vertical slices** instead: one test → one implementation → repeat, each test a **tracer bullet** that responds to what the last cycle taught you.
|
||||
|
||||
## Rules of the loop
|
||||
|
||||
- **Red before green.** Write the failing test first, then only enough code to pass it. Don't anticipate future tests or add speculative features.
|
||||
- **One slice at a time.** One seam, one test, one minimal implementation per cycle.
|
||||
- **Refactoring is not part of the loop.** It belongs to the review stage (see the `code-review` skill), not the red → green implementation cycle.
|
||||
@@ -0,0 +1,3 @@
|
||||
interface:
|
||||
display_name: "TDD"
|
||||
short_description: "Test-driven red-green-refactor"
|
||||
@@ -0,0 +1,59 @@
|
||||
# When to Mock
|
||||
|
||||
Mock at **system boundaries** only:
|
||||
|
||||
- External APIs (payment, email, etc.)
|
||||
- Databases (sometimes - prefer test DB)
|
||||
- Time/randomness
|
||||
- File system (sometimes)
|
||||
|
||||
Don't mock:
|
||||
|
||||
- Your own classes/modules
|
||||
- Internal collaborators
|
||||
- Anything you control
|
||||
|
||||
## Designing for Mockability
|
||||
|
||||
At system boundaries, design interfaces that are easy to mock:
|
||||
|
||||
**1. Use dependency injection**
|
||||
|
||||
Pass external dependencies in rather than creating them internally:
|
||||
|
||||
```typescript
|
||||
// Easy to mock
|
||||
function processPayment(order, paymentClient) {
|
||||
return paymentClient.charge(order.total);
|
||||
}
|
||||
|
||||
// Hard to mock
|
||||
function processPayment(order) {
|
||||
const client = new StripeClient(process.env.STRIPE_KEY);
|
||||
return client.charge(order.total);
|
||||
}
|
||||
```
|
||||
|
||||
**2. Prefer SDK-style interfaces over generic fetchers**
|
||||
|
||||
Create specific functions for each external operation instead of one generic function with conditional logic:
|
||||
|
||||
```typescript
|
||||
// GOOD: Each function is independently mockable
|
||||
const api = {
|
||||
getUser: (id) => fetch(`/users/${id}`),
|
||||
getOrders: (userId) => fetch(`/users/${userId}/orders`),
|
||||
createOrder: (data) => fetch('/orders', { method: 'POST', body: data }),
|
||||
};
|
||||
|
||||
// BAD: Mocking requires conditional logic inside the mock
|
||||
const api = {
|
||||
fetch: (endpoint, options) => fetch(endpoint, options),
|
||||
};
|
||||
```
|
||||
|
||||
The SDK approach means:
|
||||
- Each mock returns one specific shape
|
||||
- No conditional logic in test setup
|
||||
- Easier to see which endpoints a test exercises
|
||||
- Type safety per endpoint
|
||||
@@ -0,0 +1,77 @@
|
||||
# Good and Bad Tests
|
||||
|
||||
## Good Tests
|
||||
|
||||
**Integration-style**: Test through real interfaces, not mocks of internal parts.
|
||||
|
||||
```typescript
|
||||
// GOOD: Tests observable behavior
|
||||
test("user can checkout with valid cart", async () => {
|
||||
const cart = createCart();
|
||||
cart.add(product);
|
||||
const result = await checkout(cart, paymentMethod);
|
||||
expect(result.status).toBe("confirmed");
|
||||
});
|
||||
```
|
||||
|
||||
Characteristics:
|
||||
|
||||
- Tests behavior users/callers care about
|
||||
- Uses public API only
|
||||
- Survives internal refactors
|
||||
- Describes WHAT, not HOW
|
||||
- One logical assertion per test
|
||||
|
||||
## Bad Tests
|
||||
|
||||
**Implementation-detail tests**: Coupled to internal structure.
|
||||
|
||||
```typescript
|
||||
// BAD: Tests implementation details
|
||||
test("checkout calls paymentService.process", async () => {
|
||||
const mockPayment = jest.mock(paymentService);
|
||||
await checkout(cart, payment);
|
||||
expect(mockPayment.process).toHaveBeenCalledWith(cart.total);
|
||||
});
|
||||
```
|
||||
|
||||
Red flags:
|
||||
|
||||
- Mocking internal collaborators
|
||||
- Testing private methods
|
||||
- Asserting on call counts/order
|
||||
- Test breaks when refactoring without behavior change
|
||||
- Test name describes HOW not WHAT
|
||||
- Verifying through external means instead of interface
|
||||
|
||||
```typescript
|
||||
// BAD: Bypasses interface to verify
|
||||
test("createUser saves to database", async () => {
|
||||
await createUser({ name: "Alice" });
|
||||
const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]);
|
||||
expect(row).toBeDefined();
|
||||
});
|
||||
|
||||
// GOOD: Verifies through interface
|
||||
test("createUser makes user retrievable", async () => {
|
||||
const user = await createUser({ name: "Alice" });
|
||||
const retrieved = await getUser(user.id);
|
||||
expect(retrieved.name).toBe("Alice");
|
||||
});
|
||||
```
|
||||
|
||||
**Tautological tests**: Expected value restates the implementation, so the test passes by construction.
|
||||
|
||||
```typescript
|
||||
// BAD: Expected value is recomputed the way the code computes it
|
||||
test("calculateTotal sums line items", () => {
|
||||
const items = [{ price: 10 }, { price: 5 }];
|
||||
const expected = items.reduce((sum, i) => sum + i.price, 0);
|
||||
expect(calculateTotal(items)).toBe(expected);
|
||||
});
|
||||
|
||||
// GOOD: Expected value is an independent, known literal
|
||||
test("calculateTotal sums line items", () => {
|
||||
expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15);
|
||||
});
|
||||
```
|
||||
@@ -0,0 +1,35 @@
|
||||
# GLOSSARY.md Format
|
||||
|
||||
`GLOSSARY.md` is the canonical language for this teaching workspace. All explainers, exercises, and learning records should adhere to its terminology. Building it is itself part of learning: compressing a concept into a tight definition is evidence the user understands it.
|
||||
|
||||
## Structure
|
||||
|
||||
```md
|
||||
# {Topic} Glossary
|
||||
|
||||
{One or two sentence description of the topic this glossary covers.}
|
||||
|
||||
## Terms
|
||||
|
||||
**Hypertrophy**:
|
||||
Muscle growth driven by mechanical tension and metabolic stress over repeated training sessions.
|
||||
_Avoid_: Bulking, getting big
|
||||
|
||||
**Progressive overload**:
|
||||
Systematically increasing the demand on a muscle over time, via load, volume, or intensity.
|
||||
_Avoid_: Pushing harder, levelling up
|
||||
|
||||
**RPE (Rate of Perceived Exertion)**:
|
||||
A 1–10 self-rating of how hard a set felt, where 10 is failure and 8 means two reps left in the tank.
|
||||
_Avoid_: Effort score, intensity rating
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **Add a term only when the user understands it.** The glossary is a record of compressed knowledge, not a dictionary the user reads to learn. If the user has just been introduced to a concept, wait until they can use it correctly before promoting it here.
|
||||
- **Be opinionated.** When several words exist for the same concept, pick the best one and list the rest as aliases to avoid. This is how language compresses.
|
||||
- **Keep definitions tight.** One or two sentences. Define what the term IS, not what it does or how to do it.
|
||||
- **Use the glossary's own terms inside definitions.** Once a term is in the glossary, prefer it everywhere, including inside other definitions. This is what makes complex terms easier to grasp later.
|
||||
- **Group under subheadings** when natural clusters emerge (e.g. `## Anatomy`, `## Programming`). A flat list is fine when terms cohere.
|
||||
- **Flag ambiguities explicitly.** If a term is used loosely in the wider field, note the resolution: "In this workspace, 'set' always means a working set; warm-ups are tracked separately."
|
||||
- **Revise as understanding deepens.** A definition the user wrote in week one may be wrong by week six. Update in place; do not leave stale entries.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Learning Record Format
|
||||
|
||||
Learning records live in `./learning-records/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. Create the directory lazily: only when the first record is written.
|
||||
|
||||
They are the teaching equivalent of ADRs: they capture non-obvious lessons, key insights, and stated prior knowledge that will steer future sessions. They are used to calculate the zone of proximal development.
|
||||
|
||||
## Template
|
||||
|
||||
```md
|
||||
# {Short title of what was learned or established}
|
||||
|
||||
{1-3 sentences: what was learned (or what prior knowledge was established), and why it matters for future sessions.}
|
||||
```
|
||||
|
||||
That is the whole format. A learning record can be a single paragraph. The value is recording _that_ this is now known and _why_ it changes what to teach next, not in filling out sections.
|
||||
|
||||
## Optional sections
|
||||
|
||||
Only include these when they add genuine value. Most records won't need them.
|
||||
|
||||
- **Status** frontmatter (`active | superseded by LR-NNNN`): useful when an earlier understanding turns out to be wrong and is replaced.
|
||||
- **Evidence**: how the user demonstrated the understanding (a question answered, an exercise completed, prior experience cited). Useful when the claim might be revisited.
|
||||
- **Implications**: what this unlocks or rules out for future sessions. Worth recording when non-obvious.
|
||||
|
||||
## Numbering
|
||||
|
||||
Scan `./learning-records/` for the highest existing number and increment by one.
|
||||
|
||||
## When to write a learning record
|
||||
|
||||
Write one when any of these is true:
|
||||
|
||||
1. **The user demonstrated genuine understanding of something non-trivial**: not just exposure, but evidence they can use the concept correctly. This sets a new floor for what to teach next.
|
||||
2. **The user disclosed prior knowledge**: "I already know X." Record it so future sessions don't re-teach it. Also record the _depth_ claimed.
|
||||
3. **A misconception was corrected**: the user previously believed something wrong and now sees why. These are high-value: they predict future stumbling blocks for related topics.
|
||||
4. **The mission shifted in response to learning**: the user discovered they cared about something different than they thought. Cross-link to [[MISSION.md]] and update it.
|
||||
|
||||
### What does _not_ qualify
|
||||
|
||||
- Material that was merely covered. Coverage is not learning. Wait for evidence.
|
||||
- Anything already captured tersely in [[GLOSSARY.md]] as a term definition. Don't duplicate.
|
||||
- Session-by-session activity logs. Learning records are not a journal: they are decision-grade insights.
|
||||
|
||||
## Supersession
|
||||
|
||||
When a later record contradicts an earlier one (the user's understanding deepened or corrected), mark the old record `Status: superseded by LR-NNNN` rather than deleting it. The history of how understanding evolved is itself useful signal.
|
||||
@@ -0,0 +1,31 @@
|
||||
# MISSION.md Format
|
||||
|
||||
`MISSION.md` lives at the workspace root. It captures the _reason_ the user is learning this topic. Every teaching decision (what to teach next, which resources to surface, which exercises to design) should trace back to this document.
|
||||
|
||||
## Template
|
||||
|
||||
```md
|
||||
# Mission: {Topic}
|
||||
|
||||
## Why
|
||||
{1-3 sentences. The concrete real-world goal the user is chasing. What changes in their life or work when they have this skill? Avoid abstract framings like "to understand X"; push for the underlying outcome.}
|
||||
|
||||
## Success looks like
|
||||
- {A specific, observable thing the user will be able to do}
|
||||
- {Another specific thing}
|
||||
- {…}
|
||||
|
||||
## Constraints
|
||||
- {Time, budget, prior commitments, learning preferences, anything that bounds the approach}
|
||||
|
||||
## Out of scope
|
||||
- {Adjacent topics the user explicitly does not want to chase right now, protecting the zone of proximal development}
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **One mission per workspace.** If the user wants to learn two unrelated things, that is two workspaces.
|
||||
- **Concrete over abstract.** "Run a half marathon by October" beats "get fitter." "Ship a Rust CLI to my team" beats "learn Rust."
|
||||
- **Push back on vagueness.** If the user cannot articulate why, interview them before writing anything. A bad mission is worse than no mission.
|
||||
- **Revise when reality shifts.** Missions change. When the user's goal moves, update this file: don't leave a stale mission steering future sessions.
|
||||
- **Keep it short.** If `MISSION.md` runs past a screen, it has stopped being a compass and started being a plan.
|
||||
@@ -0,0 +1,32 @@
|
||||
# RESOURCES.md Format
|
||||
|
||||
`RESOURCES.md` is the curated set of trusted sources for this topic. Knowledge for explainers should be drawn from here, not from parametric guesses. Wisdom comes from the communities listed here.
|
||||
|
||||
## Structure
|
||||
|
||||
```md
|
||||
# {Topic} Resources
|
||||
|
||||
## Knowledge
|
||||
|
||||
- [Book: _The Science and Practice of Strength Training_ by Zatsiorsky & Kraemer](https://example.com)
|
||||
Foundational text on programming and adaptation. Use for: anything to do with periodisation, recovery, intensity zones.
|
||||
- [Article: "How Much Should I Train?" by Greg Nuckols (Stronger By Science)](https://example.com)
|
||||
Evidence-based review of volume landmarks. Use for: weekly set targets per muscle group.
|
||||
|
||||
## Wisdom (Communities)
|
||||
|
||||
- [r/weightroom](https://reddit.com/r/weightroom)
|
||||
High-signal subreddit, moderated against bro-science. Use for: programme critique, plateau troubleshooting.
|
||||
- Local: Tuesday strength class at {gym name}
|
||||
Use for: real-time coaching feedback on lifts.
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **High-trust only.** Prefer primary sources, recognised experts, peer-reviewed work, and communities with strong moderation. If a resource is marketing dressed as education, leave it out.
|
||||
- **Annotate every entry.** A bare link is useless in three months. Add one line: what it covers and when to reach for it.
|
||||
- **Group by Knowledge / Wisdom.** Mirrors the philosophy in [SKILL.md](./SKILL.md). It is fine for a resource to appear in only one group.
|
||||
- **Surface gaps explicitly.** If no good resource exists for an area the mission needs, write a `## Gaps` section listing what is missing. This drives future search.
|
||||
- **Prune ruthlessly.** A resource that turned out to be wrong, shallow, or off-mission should be removed, not buried. Better five sharp sources than thirty mediocre ones.
|
||||
- **Record community preferences.** If the user has opted out of joining communities, note it here so future sessions don't keep proposing them.
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
name: teach
|
||||
description: Teach the user a new skill or concept, within this workspace.
|
||||
disable-model-invocation: true
|
||||
argument-hint: "What would you like to learn about?"
|
||||
---
|
||||
|
||||
The user has asked you to teach them something. This is a stateful request - they intend to learn the topic over multiple sessions.
|
||||
|
||||
## Teaching Workspace
|
||||
|
||||
Treat the current directory as a teaching workspace. The state of their learning is captured in this directory in several files:
|
||||
|
||||
- `MISSION.md`: A document capturing the _reason_ the user is interested in the topic. This should be used to ground all teaching. Use the format in [MISSION-FORMAT.md](./MISSION-FORMAT.md).
|
||||
- `./reference/*.html`: A directory of reference materials. These are the compressed learnings from the lessons - cheat sheets, reference algorithms, syntax, yoga poses, glossaries. They are the raw units of learning. They should be beautiful documents which print out well, and are designed for quick reference.
|
||||
- `RESOURCES.md`: A list of resources which can be explored to ground your teaching in contextual knowledge, or to acquire knowledge and wisdom. Use the format in [RESOURCES-FORMAT.md](./RESOURCES-FORMAT.md).
|
||||
- `./learning-records/*.md`: A directory of learning records, which capture what the user has learned. These are loosely equivalent to architectural decision records in software development - they capture non-obvious lessons and key insights that may need to be revised later, or drive future sessions. These should be used to calculate the zone of proximal development. They are titled `0001-<dash-case-name>.md`, where the number increments each time. Use the format in [LEARNING-RECORD-FORMAT.md](./LEARNING-RECORD-FORMAT.md).
|
||||
- `./lessons/*.html`: A directory of lessons. A **lesson** is a single, self-contained HTML output that teaches one tightly-scoped thing tied to the mission. This is the primary unit of teaching in this workspace.
|
||||
- `./assets/*`: Reusable **components** shared across lessons. See [Assets](#assets).
|
||||
- `NOTES.md`: A scratchpad for you to jot down user preferences, or working notes.
|
||||
|
||||
## Philosophy
|
||||
|
||||
To learn at a deep level, the user needs three things:
|
||||
|
||||
- **Knowledge**, captured from high-quality, high-trust resources
|
||||
- **Skills**, acquired through highly-relevant interactive lessons devised by you, based on the knowledge
|
||||
- **Wisdom**, which comes from interacting with other learners and practitioners
|
||||
|
||||
Before the `RESOURCES.md` is well-populated, your focus should be to find high-quality resources which will help the user acquire knowledge. Never trust your parametric knowledge.
|
||||
|
||||
Some topics may require more skills than knowledge. Learning more about theoretical physics might be more knowledge-based. For yoga, more skills-based.
|
||||
|
||||
### Fluency vs Storage Strength
|
||||
|
||||
You should be careful to split between two types of learning:
|
||||
|
||||
- **Fluency strength**: in-the-moment retrieval of knowledge
|
||||
- **Storage strength**: long-term retention of knowledge
|
||||
|
||||
Fluency can give the user an illusory sense of mastery, but storage strength is the real goal. Try to design lessons which build long-term retention by desirable difficulty:
|
||||
|
||||
- Using retrieval practice (recall from memory)
|
||||
- Spacing (distributing practice over time)
|
||||
- Interleaving (mixing up different but related topics in practice - for skills practice only)
|
||||
|
||||
## Lessons
|
||||
|
||||
A lesson is the main thing you produce: the unit in which knowledge and skills reach the user. Each lesson is one self-contained HTML file, saved to `./lessons/` and titled `0001-<dash-case-name>.html` where the number increments each time.
|
||||
|
||||
A lesson should be **beautiful**, with clean, readable typography and layout, since the user will return to these later to review. Think Tufte.
|
||||
|
||||
The lesson should be short, and completable very quickly. Learners' working memory is very small, and we need to stay within it. But each lesson should give the user a single tangible win that they can build on. It should be directly tied to the mission, and should be in the user's zone of proximal development.
|
||||
|
||||
If possible, open the lesson file for the user by running a CLI command.
|
||||
|
||||
Each lesson should link via HTML anchors to other lessons and reference documents.
|
||||
|
||||
Each lesson should recommend a primary source for the user to read or watch. This should be the most high-quality, high-trust resource you found on the topic.
|
||||
|
||||
Each lesson should contain a reminder to ask followup questions to the agent. The agent is their teacher, and can assist with anything that's unclear.
|
||||
|
||||
## Assets
|
||||
|
||||
Lessons are built from reusable **components**, stored in `./assets/`: stylesheets, quiz widgets, simulators, diagram helpers, and anything else a second lesson could reuse.
|
||||
|
||||
Reuse is the default, not the exception. Before authoring a lesson, read `./assets/` and build from the components already there. When a lesson needs something new and reusable, write it as a component in `./assets/` and link to it; never inline code a future lesson would duplicate.
|
||||
|
||||
A shared stylesheet is the first component every workspace earns: every lesson links it, so the lessons look like one consistent course rather than a pile of one-offs. As the workspace grows, so should the component library.
|
||||
|
||||
## The Mission
|
||||
|
||||
Every lesson should be tied into the mission - the reason that the user is interested in learning about the topic.
|
||||
|
||||
If the user is unclear about the mission, or the `MISSION.md` is not populated, your first job should be to question the user on why they want to learn this.
|
||||
|
||||
Failing to understand the mission will mean knowledge acquisition is not grounded in real-world goals. Lessons will feel too abstract. You will have no way of judging what the user should do next.
|
||||
|
||||
Missions may change as the user develops more skills and knowledge. This is normal - make sure to update the `MISSION.md` and add a learning record to capture the change. Confirm with the user before changing the mission.
|
||||
|
||||
## Zone Of Proximal Development
|
||||
|
||||
Each lesson, the user should always feel as if they are being challenged 'just enough'.
|
||||
|
||||
The user may specify an exact thing they want to learn. If they don't, figure out their zone of proximal development by:
|
||||
|
||||
- Reading their `learning-records`
|
||||
- Figuring out the right thing to teach them based on their mission
|
||||
- Teach the most relevant thing that fits in their zone of proximal development
|
||||
|
||||
## Knowledge
|
||||
|
||||
Lessons should be designed around a skill the user is going to learn. The knowledge in the lesson should be only what's required to acquire that skill. You teach the knowledge first, then get the user to practice the skills via an interactive feedback loop.
|
||||
|
||||
Knowledge should first be gathered from trusted resources. Use `RESOURCES.md` to keep track of them. Lessons should be littered with citations - links to external resources to back up any claim made. This increases the trustworthiness of the lesson.
|
||||
|
||||
For acquiring knowledge, difficulty is the enemy. It eats working memory you need for understanding.
|
||||
|
||||
## Skills
|
||||
|
||||
If knowledge is all about acquisition, skills are about durability and flexibility. Make the knowledge stick.
|
||||
|
||||
For skill acquisition, difficulty is the tool. Effortful retrieval is what builds storage strength. Skills should be taught through interactive lessons. There are several tools at your disposal:
|
||||
|
||||
- Interactive lessons, using quizzes and light in-browser tasks
|
||||
- Lessons which guide the user through a list of real-world steps to take (for instance, yoga poses)
|
||||
|
||||
Each of these should be based on a **feedback loop**, where the user receives feedback on their performance. This feedback loop should be as tight as possible, giving feedback immediately - and ideally automatically.
|
||||
|
||||
For quizzes, each answer should be exactly the same number of words (and characters, if possible). Don't give the user any clues about the answer through formatting.
|
||||
|
||||
## Acquiring Wisdom
|
||||
|
||||
Wisdom comes from true real-world interaction - testing your skills outside the learning environment.
|
||||
|
||||
When the user asks a question that appears to require wisdom, your default posture should be to attempt to answer - but to ultimately delegate to a **community**.
|
||||
|
||||
A community is a place (online or offline) where the user can test their skills in the real world. This might be a forum, a subreddit, a real-world class (budget permitting) or a local interest group.
|
||||
|
||||
You should attempt to find high-reputation communities the user can join. If the user expresses a preference that they don't want to join a community, respect it.
|
||||
|
||||
## Reference Documents
|
||||
|
||||
While creating lessons, you should also create reference documents. Lessons can reference these documents - they are useful for tracking raw units of knowledge useful across lessons.
|
||||
|
||||
Lessons will rarely be revisited later - reference documents will be. They should be the compressed essence of the lesson, in a format designed for quick reference.
|
||||
|
||||
Some learning topics lend themselves to reference:
|
||||
|
||||
- Syntax and code snippets for programming
|
||||
- Algorithms and flowcharts for processes
|
||||
- Yoga poses and sequences for yoga
|
||||
- Exercises and routines for fitness
|
||||
- Glossaries for any topic with its own nomenclature
|
||||
|
||||
Glossaries, in particular, are an essential reference. Once one is created, it should be adhered to in every lesson.
|
||||
|
||||
## `NOTES.md`
|
||||
|
||||
The user will sometimes express preferences of how they want to be taught, or things you should keep in mind. This is the place to record those preferences, so you can refer back to them when designing lessons or working with the user.
|
||||
@@ -0,0 +1,5 @@
|
||||
interface:
|
||||
display_name: "Teach"
|
||||
short_description: "Learn a concept in a guided workspace"
|
||||
policy:
|
||||
allow_implicit_invocation: false
|
||||
Reference in New Issue
Block a user