Compare commits
88
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6af3714bd0 | ||
|
|
495e2229f5 | ||
|
|
37ef4897ae | ||
|
|
06e549cb27 | ||
|
|
c96741794a | ||
|
|
e14d0906ea | ||
|
|
d5be16127b | ||
|
|
b7aef40266 | ||
|
|
fafdaa2617 | ||
|
|
7213068d91 | ||
|
|
05dbe8c5c1 | ||
|
|
806352f8f6 | ||
|
|
34a7af2847 | ||
|
|
635c64d773 | ||
|
|
64d9c54657 | ||
|
|
aec7f0f068 | ||
|
|
4023b21365 | ||
|
|
e14fc884a3 | ||
|
|
b3a84c5bb8 | ||
|
|
9e6254108a | ||
|
|
ce87958b77 | ||
|
|
802014fea9 | ||
|
|
e22acf4913 | ||
|
|
c14c64d6c0 | ||
|
|
6e3e5594eb | ||
|
|
36ff3742b4 | ||
|
|
fd6a07bb58 | ||
|
|
1921d76319 | ||
|
|
d302151a8d | ||
|
|
5b6f4381b0 | ||
|
|
cae49fcd09 | ||
|
|
ee292ffd10 | ||
|
|
dd4713fa81 | ||
|
|
a8727ddf0b | ||
|
|
2faeab665a | ||
|
|
62b3b92e69 | ||
|
|
ec6a707fe4 | ||
|
|
96553979fb | ||
|
|
ac5b377750 | ||
|
|
8f5a229cc6 | ||
|
|
ea514c8fd2 | ||
|
|
bc60fa33be | ||
|
|
e4ecc93f29 | ||
|
|
d055eb1e7f | ||
|
|
70346f3d57 | ||
|
|
1251cfbd40 | ||
|
|
e67bbd0de0 | ||
|
|
986081bf1d | ||
|
|
a9b3c495da | ||
|
|
c99eb8e11b | ||
|
|
cd20d6ddfd | ||
|
|
82dd0c7d09 | ||
|
|
99b9a6a17c | ||
|
|
39385636be | ||
|
|
d99c1c3c4f | ||
|
|
651924a4bc | ||
|
|
e86e60af39 | ||
|
|
01d9d3fbd5 | ||
|
|
570b5509d0 | ||
|
|
3579c7ebc5 | ||
|
|
0075ca3bbf | ||
|
|
bcd52bb0f4 | ||
|
|
edd7e0cf68 | ||
|
|
f033639b54 | ||
|
|
6fffd83957 | ||
|
|
dd8f3974c1 | ||
|
|
acf67182e9 | ||
|
|
38bf9c4897 | ||
|
|
54f15e2a58 | ||
|
|
1e7e212e5c | ||
|
|
475211d2e7 | ||
|
|
0258ef70f8 | ||
|
|
0c7fe8b4bf | ||
|
|
765b686859 | ||
|
|
332067a516 | ||
|
|
66e3f220e7 | ||
|
|
6134e1168e | ||
|
|
c379d16858 | ||
|
|
ca8b46241e | ||
|
|
3fa0242f56 | ||
|
|
fe039a0b73 | ||
|
|
d4092f2f65 | ||
|
|
cedef97c99 | ||
|
|
0ee08bc368 | ||
|
|
6664f9090a | ||
|
|
7de07ee35c | ||
|
|
fcc7226c3b | ||
|
|
a8011c3d4d |
@@ -100,15 +100,18 @@ secret in every agent's context window on every machine that pulls this repo.
|
||||
|
||||
| box | role | GPU | notes |
|
||||
|-----|------|-----|-------|
|
||||
| Dev workstation | client box, never runs inference | laptop iGPU + dGPU (non-unified) | mainline llama.cpp at `~/GitPackages/llama.cpp/build/bin/`; this repo lives here at `~/.omp` |
|
||||
| Dev workstation (`blankpad`) | client box, never runs inference | Lenovo laptop, AMD Ryzen 5 PRO 5650U / Cezanne APU (**unified memory**, s2idle sleep only, no dGPU) | this repo lives here at `~/.omp`; the earlier "iGPU + dGPU / `~/GitPackages/llama.cpp`" description belonged to the sold `dankware` box |
|
||||
| Dankclaw | bench target, secondary models + service stack | Strix Halo APU (**unified memory**) + RTX 3090 (moved over from Dankserver 2026-09) | prod services `frank-dragoon` (halogen-flash-server, :7770, iGPU) and `frank-zealot` (llama.cpp PrismML fork, :7771, 3090); runbooks in the serverAppConfigs vault (`~/Projects/serverAppConfig/serverAppConfigs-docs/LLM/`); ssh: `ssh dankclaw` |
|
||||
| Dankserver | main-model box | 2× Radeon AI PRO R9700 — non-unified (was RTX 3090 until 2026-09) | runs `frank-archon` (radiance, TP=2, :7770) — **this repo's operator session runs on it**; ssh: `ssh dankserver` |
|
||||
|
||||
## Environment gotchas
|
||||
|
||||
- Gitea API needs a token: `GITEA_TOKEN` env (gitignored `.env`). No `tea` CLI
|
||||
installed anywhere. Remote URLs embed the token locally in `.git/config` — that
|
||||
file is never pushed; it is not a substitute for the secret gate.
|
||||
installed anywhere. Remote URLs carry NO embedded token (2026-10-09
|
||||
credential-store cutover) — git and the API both read
|
||||
`~/.config/secrets/gitea` (`store --file`, URL-line format; see
|
||||
`omp-docs/tools/gitea/client-access.md`). That file is never pushed; it is
|
||||
not a substitute for the secret gate.
|
||||
- Passwordless sudo (`sudo -n`) works on Dankclaw and Dankserver for systemctl
|
||||
verbs only; user units need no sudo at all.
|
||||
- SSH into both servers is BatchMode-capable from here:
|
||||
|
||||
@@ -96,6 +96,17 @@ Never assume the user reports correct information. He is human, and humans make
|
||||
- Always practice Test Driven Development practices; do UNIT TESTS exist for the functionality about to be edited? Is there a LINTER? Is there a FORMATTER? Verify that they exist, and that they are configured as a pre-commit hook. If they do not exist, they MUST be written PRIOR to the code being written. Invoke `/skill:tdd` for preferred testing style. If tests do exist, ensure that they still match the planned changes. If you believe a PRE-EXISTING test must be changed prior to commencing work, ASK THE USER ABOUT IT.
|
||||
- Always ensure tests run on code changes after every commit, and all project tests run before every push using git hooks, EXCEPT in the event that the individual test in question takes an excessive amount of time. Always ask the user before disabling an individual due to running time or performance concerns.
|
||||
|
||||
|
||||
### 6. Keep the Todo List LIVE, Not Retrospective
|
||||
|
||||
The todo list is the user's cockpit view of your work. A list that only flips at the end is a history dump, not tracking - the user sees nothing while you work.
|
||||
|
||||
- Multi-step work: `todo init` BEFORE the first substantive step - the list must exist while the work is unseen, not after.
|
||||
- Batch the todo transition with the step it describes: `done` + `start` ride along in the same message as the first tool calls of the next step. NEVER defer flips until the whole bill of tasks is finished.
|
||||
- A task is `done` the moment its work is verified - not when the surrounding phase ends.
|
||||
- Blocked on external input (user answer, subagent, service)? `block` it with a reason so the stall is visible; `unblock` the instant it is actionable.
|
||||
- NEVER make a todo call the only call in a message - that is ceremony, not progress; pair every flip with real work.
|
||||
|
||||
## FORBIDDEN ACTS AND MISTAKES TO NOT REPEAT (DO NOT DO THESE):
|
||||
- Editing node_modules or similar import module directories. This is third party code that will be overwritten next time `npm update` or a similar command is run.
|
||||
- `git push` - unless directed, except if it's our local gitea instance on Dankclaw.
|
||||
|
||||
+18
-5
@@ -1,6 +1,10 @@
|
||||
providers:
|
||||
openrouterVariant: online
|
||||
fetch: auto
|
||||
maxInFlightRequests:
|
||||
llamacpp (dankserver): 3
|
||||
llamacpp (dankclaw): 2
|
||||
llamacpp (dankclaw-3090): 3
|
||||
modelRoles:
|
||||
slow: llamacpp (dankclaw)/frank-dragoon:medium
|
||||
vision: llamacpp (dankclaw-3090)/frank-zealot:xhigh
|
||||
@@ -13,7 +17,7 @@ modelRoles:
|
||||
default: llamacpp (dankserver)/frank-archon:medium
|
||||
smol: llamacpp (dankclaw-3090)/frank-zealot:medium
|
||||
web: web/searxng
|
||||
speech: local/kokoro
|
||||
speech: frank-tts (dankclaw)/frank-tts
|
||||
judge: openrouter/~typesafe/jev-latest:auto
|
||||
dictation: local/parakeet-tdt-0.6b-v3
|
||||
memory: llamacpp (dankclaw-3090)/frank-zealot:medium
|
||||
@@ -145,17 +149,25 @@ display:
|
||||
subagentLivePreview: true
|
||||
task:
|
||||
showResolvedModelBadge: true
|
||||
maxConcurrency: 3
|
||||
maxConcurrency: 2
|
||||
agentModelOverrides:
|
||||
designer: llamacpp (dankclaw-3090)/frank-zealot
|
||||
librarian: llamacpp (dankclaw-3090)/frank-zealot
|
||||
reviewer: llamacpp (dankserver)/frank-archon
|
||||
scout: llamacpp (dankserver)/frank-archon
|
||||
scout: llamacpp (dankclaw-3090)/frank-zealot
|
||||
security-reviewer: llamacpp (dankclaw)/frank-dragoon
|
||||
sonic: llamacpp (dankclaw-3090)/frank-zealot
|
||||
task: llamacpp (dankserver)/frank-archon
|
||||
seance: llamacpp (dankclaw-3090)/frank-zealot
|
||||
eager: default
|
||||
enableLsp: true
|
||||
maxRuntimeMs: 1800000
|
||||
softRequestBudget: 120
|
||||
maxRecursionDepth: 1
|
||||
disabledAgents: []
|
||||
isolation:
|
||||
enabled: true
|
||||
completionProbe: false
|
||||
snapcompact:
|
||||
systemPrompt: none
|
||||
toolResults: false
|
||||
@@ -203,8 +215,8 @@ vault:
|
||||
tts:
|
||||
localVoice: bm_george
|
||||
speech:
|
||||
enabled: true
|
||||
mode: yield
|
||||
enabled: false
|
||||
mode: all
|
||||
enhanced: true
|
||||
voice: am_fenrir
|
||||
secrets:
|
||||
@@ -232,6 +244,7 @@ followUpMode: one-at-a-time
|
||||
tools:
|
||||
approvalMode: yolo
|
||||
format: auto
|
||||
maxTimeout: 600
|
||||
marketplace:
|
||||
autoUpdate: notify
|
||||
bash:
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: amd-apu-vulkan-tenant-onboarding
|
||||
description: "Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, RADV heap gate, render-node resolution, build-side glslc/ICD deps, co-tenancy measurement."
|
||||
---
|
||||
|
||||
Use when a ggml-based engine (llama.cpp, qwentts.cpp, anything built on ggml) must move from CPU to the **iGPU** of an AMD APU (gfx1151 / Strix Halo / Ryzen AI Max) that **already serves another engine on the same card**. Answers two questions only a booted host can: what gate the second tenant must clear, and which files the build needs to clear it.
|
||||
|
||||
## 1. The usual VRAM reasoning is wrong on an APU
|
||||
|
||||
`/sys/class/drm/cardN/device/mem_info_vram_total` reports a small **carve-out** (2 GiB class), and `rocm-smi`'s "dedicated" figure is not the working set either. The real working set lives in **GTT / system pages**: `/proc/meminfo` → `GTT` and `System pages`. A second GPU tenant therefore consumes host RAM from the same pool the first tenant's weights and the OS use — no supplementary groups, no separate aperture to reserve.
|
||||
|
||||
- Price the tenant from a **booted** footprint, never from `weights + KV + buffers`: on an APU the compute buffers and any projector ride on GTT, which is host RAM; check `free` *after* the load, not before.
|
||||
- Never subtract a "dedicated" figure from it to find headroom.
|
||||
- Unheld fabric clock (`pp_dpm_mech` = `auto`) is a hazard only when the NPU joins too; a pure iGPU pair is unaffected — check it before adding a third tenant.
|
||||
- Swap on a host with GPU tenants protects co-tenants; paging a KV pool is worse than the OOM it prevents.
|
||||
|
||||
## 2. The gate that decides feasibility: the RADV heap a container sees
|
||||
|
||||
`rocm-smi` and `free` cannot see this number; only a container off the **engine's own base image** can:
|
||||
|
||||
```bash
|
||||
# one-off container, no model, no load, no compute — safe beside production
|
||||
podman run --rm --device /dev/dri --group-add keep-groups <engine-base> bash -c \
|
||||
'apt-get update -qq >/dev/null 2>&1; apt-get install -y -qq --no-install-recommends mesa-vulkan-drivers vulkan-tools glslang-tools >/dev/null 2>&1; vulkaninfo --text'
|
||||
```
|
||||
|
||||
Read `VkPhysicalDeviceMemoryProperties` → the `DEVICE_LOCAL` heap size. On a hybrid Strix Halo box Mesa splits the aperture (~1/3 : 2/3), so `DEVICE_LOCAL` is the budget the ggml tool will actually allocate against — measured 77.33 GiB *with the first tenant loaded*, against a 2 GiB DRM carve-out. Two traps, both measured:
|
||||
|
||||
- `vulkaninfo --batch` does not exist in Ubuntu 24.04's `vulkan-tools` (prints help, exits 1). Every `vulkaninfo` parse needs `--text`.
|
||||
- The container may have **no NVIDIA ICD**, so Vulkan index `0` *is* the iGPU — but do not trust ordering by index: `--text` and read `deviceName` + `driverID` (`RADV_GFX1151` vs `NVIDIA`). A hand-picked `/dev/dri/renderDN` is worse than useless on a hybrid host: **`renderD128` there is the NVIDIA card and `renderD129` the amdgpu iGPU**; resolve the node with `readlink -f /sys/class/drm/renderD*/device/driver` and pass `/dev/dri` wholesale, letting the driver pick by name.
|
||||
- A rootless container that drops its supplementary groups enumerates **nothing**. `--group-add keep-groups` (quadlet: `GroupAdd=keep-groups`) plus the host udev rule that makes the render nodes group-writable is the whole requirement; `--device /dev/dri` alone is not enough.
|
||||
|
||||
## 3. Build-side deps: the configure fails before the device ever matters
|
||||
|
||||
ggml's `src/ggml-vulkan/CMakeLists.txt` requires, in order: `find_package(Vulkan COMPONENTS glslc REQUIRED)` then `find_package(SPIRV-Headers CONFIG REQUIRED)`, and it compiles its shaders with **glslc**, never `glslangValidator`. On Ubuntu 24.04 (measured against the digest-pinned `ubuntu:24.04` base, 2026-10-19):
|
||||
|
||||
- `glslang-tools` ships `glslangValidator` and **NOT `glslc`** → `command -v glslc` empty, `find_package(Vulkan … glslc REQUIRED)` fails, and the Vulkan backend silently never exists.
|
||||
- The package is the standalone **`glslc`** (universe, `2023.8-1build1`). Working build-stage set: `libvulkan-dev glslc spirv-headers` — `libvulkan-dev` provides `vulkan/vulkan.h` + the `VulkanHeaders`/`VulkanLoader` CMake configs `find_package(Vulkan)` resolves to, `spirv-headers` ships `/usr/share/cmake/SPIRV-Headers/SPIRV-HeadersConfig.cmake`. A configure probe in the pinned base prints `Found Vulkan: … found components: glslc` then `Including Vulkan backend`.
|
||||
- **The archive's glslc is old.** 2023.8 compiles `GL_KHR_cooperative_matrix` (the path ggml's vectored quantized matmul needs) but reports `cooperative_matrix2`, `integer_dot_product`, `bfloat16`, `float_e2m1/e4m3` as unsupported, so those shader families are compiled out. If a Vulkan prefill reads as inexplicably slow, suspect the **shader compiler before the device**; the fix is a pinned glslang build passed as `-DVulkan_GLSLC_EXECUTABLE=<path>`, not a different runtime.
|
||||
- Runtime stage needs the loader **and an ICD**: `libvulkan1` + `mesa-vulkan-drivers`. Loader-without-ICD enumerates zero devices, and ggml-based servers die in device lookup — often *after* reading the weights, so the journal reads like a model fault.
|
||||
- Ship **no NVIDIA ICD** on purpose: one Vulkan device in the enumeration is what makes the *name* `Vulkan0` safe to write into config. Prefer the name over an index in every flag.
|
||||
|
||||
## 4. Prove enumeration without allocating anything
|
||||
|
||||
Do not boot a scratch second engine beside production to ask which devices exist — a second model load is a real GPU tenant, and fleet law stops that. Instead run the binary with a device name that cannot exist and read its error; it names what it enumerated, with no allocation and no model resident:
|
||||
|
||||
```bash
|
||||
podman run --rm --device /dev/dri --group-add keep-groups --entrypoint /app/tts-server <image> \
|
||||
--model /dev/null --codec /dev/null --device NoSuchDevice 2>&1 | tail -20
|
||||
```
|
||||
|
||||
## 5. The rollback must be a flag, not a rebuild
|
||||
|
||||
Put the device in the env file as a value the entrypoint passes only when set, and make **set-but-empty** mean "pass no flag at all" — the binary's own default (CPU, always built) is the rollback. `: "${VAR:=default}"` is the WRONG form: `:=` fires on an empty value too, so the documented rollback line `VAR=` is silently rewritten back to the GPU and the only way out is a rebuild. Use:
|
||||
|
||||
```bash
|
||||
if [[ ! -v QWENTTS_DEVICE ]]; then QWENTTS_DEVICE=Vulkan0; fi
|
||||
```
|
||||
|
||||
(`podman`'s `--env-file` passes `VAR=` through as set-and-empty, and `source` does the same, so both readers agree.) Pin it in a test: empty device ⇒ the flag is absent from argv.
|
||||
|
||||
## 6. Co-tenancy measurement (only after the tenant is live)
|
||||
|
||||
The gate is the **first tenant's** performance with a real request from the second in flight — not the second tenant's own speed. And price the tenant from a booted footprint, never from weights+KV+buffers: on an APU the compute buffers and any projector ride on GTT, which is host RAM; check `free` after the load, not before.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: audit-never-run-container-deploy
|
||||
description: "Audit a locally built podman/quadlet deploy path that has never actually run: prove base-image assumptions, env-file reader divergence, and image-hash coverage before the first deploy."
|
||||
---
|
||||
|
||||
# Audit a never-run container deploy path
|
||||
|
||||
A deployable that has never been built or started has never been *executed*: every
|
||||
fault in it is invisible to tests that stub the runtime. Audit it as static code
|
||||
against the tools it calls, then prove each suspicion cheaply against the real
|
||||
tool. The faults worth hunting are the ones that **report success while doing
|
||||
nothing** — those are the ones that survive a green `/health` and a green e2e run.
|
||||
|
||||
## The three fault classes (all measured 2026-10-18, frank-tts-engine)
|
||||
|
||||
1. **A command the base image does not ship.** e.g. `adduser` in a `RUN` line
|
||||
against a trimmed `ubuntu:24.04` — absent, and not pulled in by
|
||||
`ca-certificates curl libgomp1`. Dies at build time (loud), but only the first
|
||||
time anyone builds it.
|
||||
Prove it in the base, never from memory:
|
||||
`podman run --rm <base> sh -c "command -v adduser || echo absent; useradd --system --uid 10000 --home-dir /app --shell /usr/sbin/nologin tts && id tts"`
|
||||
Note the accepted cosmetic warning (`uid 10000 is greater than SYS_UID_MAX 999`)
|
||||
in a comment, or a future agent "fixes" it by dropping `--system` and the uid
|
||||
stops matching the sibling container.
|
||||
2. **Two readers of one env file disagree.** systemd units that pass
|
||||
`EnvironmentFile=` hand it to podman `--env-file`, which does **no shell
|
||||
parsing** — it takes the literal remainder of the line — while bash `source`
|
||||
strips one quoting layer. So `WARM_TEXT="Warm up, one two three."` reaches the
|
||||
container *with its quotes*, and a script that interpolates it into a JSON body
|
||||
produces invalid JSON: the container boots, `active`, every voice registered,
|
||||
every voice cold. Fix in the consumer: strip exactly one matched pair **only**
|
||||
when both ends are the same quote AND the inner content carries another quote or
|
||||
backslash (never mangle a value that legitimately contains quotes), then escape.
|
||||
3. **A build-skip hash that under-covers the image.** `build_compose_image`-style
|
||||
helpers skip `podman build` when a label hash matches; hashing the `Containerfile`
|
||||
alone means editing a `COPY`'d app/script rebuilds **nothing** — deploy reports
|
||||
success, `/health` answers, container serves old code. Each case arm must name
|
||||
the paths its `Containerfile` copies (`extra=(…)`) and hash Containerfile+extras.
|
||||
Arms that have no extras keep their old hash, so they do not needlessly rebuild.
|
||||
|
||||
## Test it as behaviour, never as a grep of the handler
|
||||
|
||||
- Drive the **real** function: `awk '/^build_compose_image\(\) \{/,/^\}/' deploy-remote.sh > bci.sh`,
|
||||
`source` it, put a stub `podman` first on `PATH`, assert the observable question —
|
||||
*did the deploy ask podman to build, or did it skip?*
|
||||
- Stub traps that cost real time: podman is invoked `podman image inspect <image> --format …`
|
||||
so the tag is **`$3`**, not `$2`; tags contain `/` so a state file named after one
|
||||
needs `${tag//\//_}` or the write silently lands in a nonexistent directory and
|
||||
every image looks permanently unbuilt; `--label cfhash=…` arrives **before** `-t <tag>`,
|
||||
so collect the label and write it after the arg loop.
|
||||
- For an env-file bug, write one test that reads the file **both** ways — `source`
|
||||
and the `--env-file` way (parse `KEY=literal remainder`) — and assert the program
|
||||
receives the same value in both shapes, including a value that carries its own
|
||||
quotes and a backslash. A suite that only sources the file is how the bug shipped.
|
||||
- Prove red: `git show HEAD:scripts/deploy-remote.sh > scripts/deploy-remote.sh`,
|
||||
run the new test (must fail at the exact intended step), restore. Keep each fix in
|
||||
its own commit and run the seam at each one so the branch bisects.
|
||||
|
||||
## Base-image and host facts to check while auditing
|
||||
|
||||
- `useradd` lives in `login(1)` (base carries it); `adduser` does not.
|
||||
- Rootless podman **cannot raise rlimits** — `--ulimit nofile=…` above the host hard
|
||||
limit dies at container create, not at runtime.
|
||||
- A referenced `Network=` `.network` file the generator cannot find means the
|
||||
container unit is **not produced at all**.
|
||||
- Two containers talk by DNS name only if they attach to a **common** network;
|
||||
check both unit files rather than trusting a comment that says they share one.
|
||||
- Container DNS name + the network that owns it, never `host.containers.internal`
|
||||
or a published loopback port, for service-to-service traffic.
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not deploy the audited thing because the audit is finished: first build of a
|
||||
heavy image (recursive source clone ~10 min) plus a new unit on a host serving
|
||||
people is the operator's call.
|
||||
- Do not add a Prometheus target for a container with no `/metrics` — say the probe
|
||||
is a decision (blackbox HTTP), do not fake a scrape.
|
||||
@@ -1,82 +1,32 @@
|
||||
---
|
||||
name: bounding-subagent-runtime
|
||||
description: "Cap and diagnose runaway or hung omp subagents: task.maxRuntimeMs, softRequestBudget, agentIdleTtlMs, per-agent model overrides, the cfg save-approval quirk, and the finished-work-but-never-exits nested-fanout deadlock."
|
||||
description: "Cap and diagnose runaway or hung omp subagents: task.maxRuntimeMs, softRequestBudget, maxConcurrency, maxRecursionDepth (the 0-trap), per-provider in-flight ceilings."
|
||||
---
|
||||
|
||||
# Bounding subagent runtime (omp harness)
|
||||
## When
|
||||
Subagent runs long, fan out too wide, or you must (re)declare the fleet's subagent limit policy. Also when asked "what limits can subagents have".
|
||||
|
||||
Use when a `task`/`eval` subagent runs absurdly long (tens of minutes on a small job),
|
||||
when it **finishes its deliverable and still never exits**, when you want a hard cap
|
||||
before spawning, or when you must tell a busy agent to stop.
|
||||
## Read first
|
||||
- Repo KB page `omp-docs/pi/subagent-limits.md` — the live policy of record (values + rationale + deliberately-not-set list). Do not re-derive; read it.
|
||||
- `cfg://task` and `cfg://providers/maxInFlightRequests` — effective values, not the stock defaults.
|
||||
|
||||
## The knobs (`cfg://task/...`)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
## The knobs (cfg:// writes; `/save` suffix persists)
|
||||
| Knob | Effect | Gotcha |
|
||||
|---|---|---|
|
||||
| `maxRuntimeMs` | `0` (unlimited) | **Hard wall-clock limit per subagent (ms).** The only true time cap. |
|
||||
| `softRequestBudget` | `200` | Soft cap on assistant **requests** per run; force-stop at **1.5×**. |
|
||||
| `softRequestBudgetNotice` | `true` | Injects one "wrap up" steering message at the soft mark. |
|
||||
| `agentIdleTtlMs` | `420000` | Parks an **idle** agent after 7 min. Useless against a busy-but-slow agent. |
|
||||
| `completionProbe` | `true` | Asks a working subagent to self-report completeness at 2/5/10 requests. |
|
||||
| `maxConcurrency` | `32` (often lowered) | Concurrent subagents; sizes GPU/endpoint load. |
|
||||
| `agentModelOverrides` | `{}` | Per-agent model routing, e.g. `{"scout":"provider/model-id"}`. |
|
||||
| `tools.maxTimeout` | `0` | Caps the timeout the agent may set on a **tool**, not on the agent. |
|
||||
| `task.maxConcurrency` | session semaphore; excess queues | stock default 32 — "2" is not a default |
|
||||
| `task.maxRecursionDepth` | nesting cap | predicate is `depth < n`; **1 = children cannot spawn (correct for "no nesting"); 0 disables spawning for the MAIN session too — never reach for 0**; negative = unlimited |
|
||||
| `task.maxRuntimeMs` | hard wall clock, overrun = `aborted` | default 0 = off |
|
||||
| `task.softRequestBudget` | wrap-up notice at N, force-yield at 1.5N | scout/sonic may impose lower built-ins |
|
||||
| `task.agentIdleTtlMs` | park idle finished agents | `<= 0` disables parking |
|
||||
| `providers.maxInFlightRequests` | concurrent LLM requests per provider id | keys are the quoted block labels from `agent/models.yml` (e.g. `"llamacpp (dankclaw-3090)"`), flat map lookup; absent/≤0 = unlimited; scope = this machine's config root only — other workstations and non-omp clients are NOT counted; excess queues |
|
||||
| `tools.maxTimeout` | ceiling on any agent-requested tool timeout | kills `timeout: 0` runaways |
|
||||
| `task.isolation.enabled` | makes per-spawn `isolated` flag available | opt-in per spawn, not a blanket sandbox; needs git repo; off in plan mode |
|
||||
| `task.completionProbe` | periodic completeness side-requests | token sink on shared endpoints |
|
||||
|
||||
Reasonable fleet defaults: `maxRuntimeMs = 900000` (15 min), leave the request budget
|
||||
alone unless the failure mode is a looping search agent (then lower it — request count,
|
||||
not the clock, is what bounds a 200-call research loop).
|
||||
## Verify semantics from the bundle, not the label
|
||||
Setting descriptions lie by omission. `~/.bun/install/global/node_modules/@oh-my-pi/pi-coding-agent/dist/cli.js`:
|
||||
- depth gate: `grep -o '.\{130\}xC\b.\{130\}'` finds `U_(n,s)`; `grep -o 'function U_(e,t){.\{0,60\}'` → `e<0||t<e` (n=cap, s=taskDepth).
|
||||
- in-flight lookup: `grep -o '.\{150\}axInFlightRequests.\{200\}'` → flat `[providerId]` lookup, non-number/≤0 = unlimited, clamp ≥1.
|
||||
|
||||
**Verify the cap instead of assuming it.** `read cfg://task/maxRuntimeMs` at the start of
|
||||
any session that plans to rely on it. A session-only write is gone next session, and an
|
||||
unanswered `/save` approval means the value never persisted — so the fleet can run hours
|
||||
with the cap at `0` while everyone believes it is 15 minutes. Nothing else bounds a child.
|
||||
|
||||
## Three failure modes, three different fixes
|
||||
|
||||
- **Runaway loop** (hundreds of sequential tool calls): lower `softRequestBudget`; the
|
||||
clock cap only truncates it late.
|
||||
- **Slow model, legitimate work** (dozens of calls on a low-t/s endpoint = an hour of wall
|
||||
clock): fix routing, not the clock — check `cfg://task/agentModelOverrides` and move the
|
||||
agent class onto a faster/shared endpoint. Watch the slot math: subagent endpoints share
|
||||
KV slots with the interactive session, so `maxConcurrency + 1` sequences on an N-slot
|
||||
endpoint degrades everyone's decode.
|
||||
- **Finished work, no exit (nested fan-out deadlock).** A subagent that spawned **its own**
|
||||
children blocks on their barriers. If those children die out-of-band (killed, starved, or
|
||||
waiting on an answer whose subject changed mid-flight), the parent waits forever with its
|
||||
deliverable already written. No instruction text prevents this — not the repo AGENTS.md,
|
||||
not the task prompt; only the clock cap or a kill ends it.
|
||||
|
||||
Signature: parent and children report **the same elapsed wall clock**, and the parent's
|
||||
on-disk output is complete. Census (agent registry + `ps`): group by elapsed time, look
|
||||
for identical `etimes` across a parent/child set, then judge the **artifacts** instead of
|
||||
the process — balanced code fences, no truncated tables, trailing newline, front matter or
|
||||
the page's H1+status convention intact. Complete artifacts + still "running" = deadlock,
|
||||
not work.
|
||||
|
||||
Repair: kill the **subtree** and confirm the children leave (a killed parent does not
|
||||
cancel children that came from a separate `tasks[]` batch). Then **flatten**: the
|
||||
top-level session does the fan-out and children never spawn children. Re-run only the
|
||||
slices whose artifacts are missing — integrate from what is already on disk.
|
||||
|
||||
## cfg write mechanics (bit me once)
|
||||
|
||||
- `write cfg://task/maxRuntimeMs 900000` → applies **session-only**, replies "NOT saved".
|
||||
- Persist = `write cfg://task/maxRuntimeMs/save` with the same content, and that write
|
||||
needs its **own interactive approval**. If the approval prompt is not answered in time
|
||||
the value stays session-only and the tool says *NEVER retry unless the user asks* —
|
||||
report it in the reply instead.
|
||||
- Revert a session value by writing the old value back (e.g. `0`).
|
||||
|
||||
## Order of operations when one is already running late
|
||||
|
||||
1. `read proc://` → registry of running agents; `read proc://<id>` for status/output.
|
||||
2. `read agent://<id>` → live progress ("Latest assistant text") — often shows it is
|
||||
mid-batch, not hung. If the latest text is already a wrap-up/summary, treat it as
|
||||
**done** and collect its output instead of waiting for an exit event.
|
||||
3. `write agent://<id>` an explicit wrap-up order: stop searching, answer what is
|
||||
sourced, mark the rest `NOT VERIFIED — needs <what>`, cap the report length.
|
||||
4. Only then `write proc://<id>/kill` — the cap does **not** retroactively bind agents
|
||||
spawned before the change, so a long runner must be ordered or killed, never just
|
||||
re-configured. After any kill, re-read the registry: unreaped children and orphaned
|
||||
host-side processes (a scratch server holding GPU/RAM) are the expensive residue, and
|
||||
`pkill -f` patterns that also match production services are forbidden — kill by pid.
|
||||
## Landing a policy change
|
||||
cfg `/save` writes persist immediately; `agent/config.yml` is machine-live tier (heartbeat commits it). Human-facing policy changes land as an `omp-docs` page update via branch → PR (docs tier), `git worktree add` from `origin/main` if the current checkout is branch-locked by workspace policy.
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: corpus-voice-skill-rewrite
|
||||
description: "Rewrite a skill to emulate an author corpus's voice: full read, style-DNA distillation from writing-for-agents levers, positive-phrasing pass, sharpened completion criteria"
|
||||
---
|
||||
|
||||
# Corpus-voice skill rewrite
|
||||
|
||||
Rewrite a skill so it reads as if a chosen author wrote it. Done for `grilling` against the `mattpocock/skills` corpus (2026-10-18, command-center PR #20); behaviour preserved, voice swapped.
|
||||
|
||||
## Where the corpus lives
|
||||
|
||||
Installed third-party skills: `~/.agents/skills/<name>/SKILL.md` (the skills-CLI global store; `~/.agents/.skill-lock.json` names sources). Whole mattpocock corpus ≈ 2,800 lines — small enough to read inline in ~4 `cat` batches via bash; do NOT delegate the reading, style lives in the exact prose.
|
||||
|
||||
Read the author's own meta-skill FIRST if one exists (`writing-for-agents` for mattpocock — it states his levers explicitly), then the corpus practicing them.
|
||||
|
||||
## Style DNA checklist (mattpocock; generalise per author)
|
||||
|
||||
- **Leading words**: compact pretrained concepts repeated as tokens, never sentences (_red_, _seam_, _fog of war_, _brief_, _frontier_). Define each once, reuse everywhere. Prefer words already in the author's other skills so the corpus shares one language.
|
||||
- **Vocabulary-first**: bolded glossary terms, then used exactly; explicit "_Avoid_: component, service".
|
||||
- **Positive phrasing**: prohibitions rewritten as target behaviours; keep a prohibition only when no positive form exists (hard guardrails).
|
||||
- **Sharpened completion criteria**: fuzzy bounds ("understanding reached") → checkable + exhaustive ("frontier empty and nothing sits blocked", "one artefact per question, counted before sending").
|
||||
- **Zero history**: no provenance, attribution, or absorbed-from lineage in the body.
|
||||
- **No-op hunting**: delete harness defaults and anything the environment already confesses (formats, rendering, caps). Gotchas the environment CANNOT confess (exclusive dialogs, Esc-abort semantics) stay inline — they're the cache worth keeping.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Read entire corpus; tabulate levers with per-skill evidence before writing.
|
||||
2. Rewrite preserving settled behaviour 1:1 — voice only; diff the two versions for semantic drift.
|
||||
3. Audit pass: grep the new text for history words (extracted/absorbed/from/upstream), machine-specific constants, repo-specific paths.
|
||||
4. Run repo checks, commit as `style(<scope>)`, note the lever→instance mapping in the PR comment so the reviewer can grade the emulation.
|
||||
@@ -1,49 +1,62 @@
|
||||
---
|
||||
name: dankclaw-pihole-lan-dns-record
|
||||
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list path), plus the public Porkbun A record step and client cache gotchas."
|
||||
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list + repo addn-hosts, plus the public Porkbun A record side) — and triage \"LAN names stopped resolving\" including the FTL boot-time record-load failure."
|
||||
---
|
||||
|
||||
Use when a new `*.franksdank.cloud` service resolves publicly/LAN-needs a local address, on Dankclaw (Pi-hole v6 container, quadlet `pihole.container`).
|
||||
# Dankclaw Pi-hole LAN DNS record
|
||||
|
||||
## Ground truth layout (verified 2026-10-11)
|
||||
## What this fixes
|
||||
|
||||
- Quadlet: `~/.config/containers/systemd/pihole.container`; volumes `/home/dank/Containers/pihole/{etc-pihole,etc-dockers,addn-hosts}`.
|
||||
- **LAN records live in `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list`** — plain hosts format: `192.168.0.143 gitea.franksdank.cloud`. This is the boring, working path.
|
||||
- `pihole.toml` `[[dns.records]]` array is EMPTY on this host — do not expect v6 UI-style records there.
|
||||
- `gravity.db` exists but has NO `domain` table (v6 layout) — do not insert local records into SQLite.
|
||||
A new `*.franksdank.cloud` service resolves publicly (via Porkbun → `franksdank.cloud` A record → Caddy on Dankclaw `192.168.0.143`) but **does not resolve on the LAN**. Pi-hole is the LAN DNS server (`dankclaw_default` bridge, published port 53) and serves local records only from files it imports.
|
||||
|
||||
## Procedure (30 s, proven)
|
||||
## Live truth
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'echo "192.168.0.143 <name>.franksdank.cloud" | sudo tee -a /home/dank/Containers/pihole/etc-pihole/hosts/custom.list >/dev/null \
|
||||
&& podman exec pihole pihole restartdns'
|
||||
ssh dankclaw 'getent hosts <name>.franksdank.cloud' # must resolve
|
||||
dig +short <name>.franksdank.cloud @192.168.0.143 # from any LAN box
|
||||
- **The repo owns the records.** `dankclaw/pihole/addn-hosts` (hosts-style lines: `192.168.0.143 <name>`) is copied by `deploy-remote.sh` to `/home/dank/Containers/pihole/addn-hosts` and bind-mounted **read-only** into the container at `/etc/pihole/hosts/addn-hosts` (`Volume=/home/dank/Containers/pihole/addn-hosts:/etc/pihole/hosts/addn-hosts:ro` in `dankclaw/pihole/quadlet/pihole.container`). Add the name there, in the repo, then deploy — that is the durable path. `tests/pihole.bats` validates every line (IPv4, LAN range, name suffix, Caddyfile cross-check).
|
||||
- `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list` also exists on the host (older mechanism, mode `640` owned by the container uid, root reads it with `sudo`). It is **not** the repo's owner file — do not hand-edit it as the fix.
|
||||
- `pihole.toml` carries a `dns.hosts = [ "192.168.0.143 …", … ]` array too (FTL's own in-config records). Both sources exist; if LAN names fail, neither is being applied.
|
||||
- Pi-hole runs as a rootless quadlet named `pihole` — query it with `podman exec pihole …`, never `sudo podman` (per-user storage law).
|
||||
- `pihole.franksdank.cloud` resolves through Pi-hole itself (loop/edge), so a vhost record pointing at `192.168.0.143` is correct for every Caddy-fronted service.
|
||||
|
||||
## Procedure (repo-first)
|
||||
|
||||
1. Append the line to `dankclaw/pihole/addn-hosts`:
|
||||
```
|
||||
192.168.0.143 <name>.franksdank.cloud
|
||||
```
|
||||
One record per vhost; keep it in sync with the Caddyfile (the bats seam cross-checks the two).
|
||||
2. `./tests/run.sh` (the `pihole` seam validates the file), commit, push.
|
||||
3. Deploy so the copy step runs: `./scripts/deploy.sh dankclaw --service pihole`.
|
||||
4. Verify from the box: `getent hosts <name>.franksdank.cloud` → `192.168.0.143`.
|
||||
5. Public edge record (optional; skip when every subscriber is LAN-only): Porkbun `POST /api/v3/json` with `apikey`/`secretapikey`/`command=add`/`type=A`/`hostname=<name>`/`data=76.49.118.136` (see `dankclaw-porkbun-edge`).
|
||||
|
||||
Client-side proof (avoid the resolver trap):
|
||||
|
||||
```
|
||||
sudo resolvectl flush-caches # systemd-resolved otherwise keeps the negative cache
|
||||
curl -sv --max-time 5 https://<name>.franksdank.cloud 2>&1 | grep 'Connected to'
|
||||
```
|
||||
|
||||
Hosts files are read-only-readable only via sudo (dir owned by uid 100999). `sudo -n` works on dankclaw.
|
||||
Expected: `Connected to <name>.franksdank.cloud (192.168.0.143)` — LAN route, not the public IP.
|
||||
|
||||
## API maze — DON'T re-explore
|
||||
## Triage: every LAN name broke at once (2026-10-19 law)
|
||||
|
||||
- FTL API base is `/api` at the web port (8083) **with `Host: pi.hole` header**; `/admin/api/*` → bad_request hint; bare 127.0.0.1 without the vhost → 404 `not_found`.
|
||||
- Login: `POST /api/login {"password": <WEBPASSWORD from /home/dank/Containers/pihole/env>}` → session in `X-Cors` header/cookie.
|
||||
- `POST /api/apps/domains/custom` → **404 on this build** (FTL version predates it); `/api/customdns/custom` also absent. There is no container sqlite3. Use custom.list.
|
||||
Signature: **all** `*.franksdank.cloud` return NXDOMAIN / Playwright `ERR_NAME_NOT_RESOLVED` while **external names still resolve**. The external-OK/local-fail split is the tell: FTL is answering external names from cache and holds **no local records at all** after a container/host restart — the mounts and files are intact and readable inside the container (`ls -l /etc/pihole/hosts/`, `grep -c . …/addn-hosts`), `dns.hosts` in `pihole.toml` is complete. The mechanism is unestablished; do not go hunting for a lost file.
|
||||
|
||||
## Public record (off-LAN reachability)
|
||||
|
||||
Porkbun API, keys in `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`):
|
||||
|
||||
```bash
|
||||
curl -sL -X POST https://api.porkbun.com/api/json3/domain/api/create/franksdank.cloud \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"apikey\":\"$K\",\"secretapi\":\"$S\",\"name\":\"<name>\",\"type\":\"A\",\"content\":\"71.227.227.172\",\"ttl\":\"600\"}"
|
||||
```
|
||||
ssh dankclaw 'getent hosts grafana.franksdank.cloud' # empty => FTL, not the client
|
||||
ssh dankclaw 'podman exec pihole pihole restartdns reload' # fixes it in seconds
|
||||
```
|
||||
|
||||
- `json3` path 302-redirects → **`-L` required**; no-`-s` to see status.
|
||||
- Porkbun edge (openresty) intermittently 503s — transient; retry later, don't burn turns. Public record is OPTIONAL when LAN-only subscribers exist.
|
||||
- Wildcard `*.franksdank.cloud` TLS is already covered: Caddy + Porkbun DNS-01 (caddy-porkbun image), so a new subdomain needs NO new cert work — just the Caddyfile site block + reload.
|
||||
Distinguish before acting: a **single** client failing while others work is FTL rate-limiting that client — `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`; a `10.0.2.x` client address is a pasta container's DNS forwarder (SNAT), not a machine. Full causal page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md` (**Records** pitfall + Failure modes table).
|
||||
|
||||
## Client-side cache gotchas
|
||||
## Gotchas
|
||||
|
||||
- Workstation (systemd-resolved): negative cache survives Pi-hole changes; `sudo resolvectl flush-caches` needs a TTY (faillock-sensitive — no `-S` guessing). Non-interactive proof: `curl --resolve <name>.franksdank.cloud:443:192.168.0.143 https://<name>.franksdank.cloud/...`; cache self-expires (SOA negative TTL).
|
||||
- Dankclaw/Dankserver resolve via Pi-hole directly — verify there first.
|
||||
- Do NOT insert local records into SQLite. v6's `gravity.db` `domain` table has NO `type=2` custom-DNS rows in this build.
|
||||
- `pihole restartdns reload` re-imports the hosts files **and** `dns.hosts`; that is the remedy for the boot-load failure above. Plain `pihole restartdns` also restarts the resolver — brief LAN-wide DNS blip.
|
||||
- `dig @<ip> <name>` from the workstation bypasses systemd-resolved — use it to check FTL's answer directly (`+short @192.168.0.143`).
|
||||
|
||||
## Cross-reference
|
||||
|
||||
- Fleet DNS flow: `dankclaw/Caddyfile` (Caddy binds `:443` on the host, serves `*.franksdank.cloud` vhosts)
|
||||
- Pi-hole service page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md`, flow page `Infrastructure/LAN-DNS.md`
|
||||
- Public edge: managed skill `dankclaw-porkbun-edge`
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: dankvoice-bats-serialization
|
||||
description: "Run dankvoice bats suites without hangs or false failures: serialize dist rebuilds, never edit src mid-run, and rebuild before committing the tracked bundle."
|
||||
---
|
||||
|
||||
Use when running the dankvoice `bats tests/` suite on the shared box, when a suite run hangs or times out, or when the "committed dist is byte-identical to a fresh build" pin test fails after a commit.
|
||||
|
||||
## The two serialization hazards
|
||||
|
||||
### 1. `setup()` rebuilds `dist/` from `src/`
|
||||
Every test runs `cd "$REPO" && bun run build` in `setup()`. Consequences:
|
||||
|
||||
- **Never edit `src/` while a suite is running.** The next test's `setup()` compiles whatever is on disk at that instant, so an in-flight edit gets attributed to the run and produces a failure with no owner. Finish the gate, then write.
|
||||
- **Two suites on the same checkout race on `dist/`.** Serialize: run one, wait for its report, then start the next. Multiple worktrees of the repo each build their own `dist/` and do not collide.
|
||||
- Suite runtime is ~85 s for 55 tests; a backgrounded job with a 600 s deadline is the right shape (`timeout: 600`, `async: true`). Never poll it — `setup()` output lands in the conversation when the job ends.
|
||||
|
||||
### 2. `dist/` is tracked, and a pin test enforces it
|
||||
One test rebuilds and compares against the committed bundle. Therefore the commit order is:
|
||||
|
||||
```bash
|
||||
bun run build # BEFORE staging
|
||||
git add -A && git commit -m '…'
|
||||
bats tests/ # gate AFTER the commit
|
||||
```
|
||||
|
||||
Committing `src/` changes without rebuilding first fails the pin test (`not ok … byte-identical to a fresh build`) — the tracked bundle is the *previous* state. If that happens on an unpushed branch, fix it by amend rather than a follow-up commit:
|
||||
|
||||
```bash
|
||||
bun run build && git add dist && git commit --amend --no-edit
|
||||
```
|
||||
|
||||
Verify freshness any time the pin is in doubt:
|
||||
|
||||
```bash
|
||||
bun run build >/dev/null 2>&1 && git status --porcelain dist # empty = committed == fresh build
|
||||
```
|
||||
|
||||
## If a backgrounded gate job vanishes
|
||||
A crash can kill the job before its report; `read proc://bg_N` returning "not found" means it is gone, not that it failed. Reconstruct what landed before resuming — `git log --oneline -3`, `git status --porcelain`, then the freshness check above — and rerun the suite; the commit may well have succeeded while the gate never ran.
|
||||
|
||||
## Gotchas
|
||||
- `DV_HOLD` is seconds (`sleep "${DV_HOLD:-0.1}"`), so `DV_HOLD=3` really holds the stub for three seconds.
|
||||
- Isolation tests must not count shared state: any assertion that sweeps `$TMPDIR` or `~/.omp/cache` observes every other live session on the box (each narrating session creates `dankvoice-*` scratch dirs). Point `TMPDIR` at a private directory instead — see the `cutting off mid-utterance leaves no orphaned audio behind` test.
|
||||
- `bun run build` does not typecheck. Run `bunx tsc --noEmit` separately when moving code between modules; the bundler happily emits a bundle that references a deleted identifier.
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: dankvoice-bundle-swap-red-proof
|
||||
description: "Prove a dankvoice behavior test is red against the pre-change bundle by pointing DV_EXT at git show HEAD:dist/index.js, with the driver extracted from the bats heredoc into a pinned scratch directory. Use before claiming a new bats assertion is red-capable."
|
||||
---
|
||||
|
||||
Prove a new `tests/behavior.bats` assertion is **red-capable** — that it fails against the *previous* build, not just passes against the new one — without stashing, branching, or editing `src/`.
|
||||
|
||||
## Why
|
||||
|
||||
`tests/behavior.bats` is a black-box suite: the plugin bundle is loaded through `DV_EXT`, so a *different bundle* is the red side. Stashing the fix would break the `dist` byte-identity pin test and the shared worktree.
|
||||
|
||||
## Working recipe
|
||||
|
||||
```bash
|
||||
cd ~/Projects/dankvoice # or the feature worktree: ~/Projects/wt-dv-*
|
||||
bun run build # current src = the green side
|
||||
git show HEAD:dist/index.js > /tmp/old-dist.js # the red side (pre-change bundle)
|
||||
|
||||
W=/tmp/dv-redproof # PINNED name; see traps below
|
||||
rm -rf "$W" && mkdir -p "$W/bin"
|
||||
cat > "$W/player.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$DV_STUB_LOG"
|
||||
for arg in "$@"; do case "$arg" in *.wav) cp "$arg" "$DV_COPY" ;; esac; done
|
||||
sleep "${DV_HOLD:-0.1}"
|
||||
SH
|
||||
chmod +x "$W/player.sh"
|
||||
ln -s "$W/player.sh" "$W/bin/pw-play"
|
||||
printf '#!/usr/bin/env bash\nprintf "116\\talsa_output.usb-KT.0\\taudio/sink\\t*\\n"\n' > "$W/bin/wpctl"
|
||||
chmod +x "$W/bin/wpctl"
|
||||
# The driver lives inside the bats file as a heredoc: extract it verbatim.
|
||||
end=$(grep -n '^TS$' tests/behavior.bats | cut -d: -f1)
|
||||
sed -n "86,$((end - 1))p" tests/behavior.bats > "$W/driver.ts"
|
||||
|
||||
export DV_EXT="$PWD/dist/index.js" DV_STUB="$W/player.sh" DV_STUB_LOG="$W/player.log" \
|
||||
DV_COPY="$W/played.wav" DV_OMPCONFIG=/dev/null DANKVOICE_CONFIG="$W/settings.json" \
|
||||
DV_MODELS="$W/models.yml" DV_WPCTL=x DV_SELECT=""
|
||||
for ext in /tmp/old-dist.js "$PWD/dist/index.js"; do
|
||||
rm -f "$W/player.log" "$DANKVOICE_CONFIG"
|
||||
DV_EXT="$ext" bun "$W/driver.ts" <scenario> 2>&1 | grep -E '^PLAYS|^CALLS|^NOTIFY|^SETTINGS'
|
||||
done
|
||||
```
|
||||
|
||||
The scenario name is the driver's third argv (`speak`, `auto`, `repeat`, `humanizecycle`, `badurl`, …); the bats `-f` test body's env (`DV_PLAYER_BIN`, `DV_SETTINGS_JSON`, `DANKVOICE_VERBOSITY`, …) is passed the same way. `DV_SELECT` holds the menu answers, `|`-separated, consumed in order.
|
||||
|
||||
## Traps, each one cost a turn
|
||||
|
||||
- **The scratch directory must be pinned by name.** `ls -dt /tmp/tmp.*` picks up *other sessions'* temp dirs, and then `player.sh` and `driver.ts` live in different directories — the run silently does the wrong thing.
|
||||
- **The stub must append to `$DV_STUB_LOG`.** The driver counts player invocations from that file; a stub that only `sleep`s reports `PLAYS 0` for *both* bundles and the whole proof reads as "fix does nothing".
|
||||
- **`DV_EXT` is required.** Missing it yields `Cannot find package 'undefined'` from the driver, which looks like a broken driver, not a missing env var.
|
||||
- **`DV_SETTINGS_JSON` supports `__PORT__`** (the driver substitutes the fixture port), so a preseed can name a prefixed route — needed whenever the test is about *which* endpoint got resolved.
|
||||
- **Never run this while a `bats` run is in flight** in the same repo: `setup()` runs `bun run build`, so two runs clobber `dist/` and the pin test.
|
||||
|
||||
## Reading the result
|
||||
|
||||
Red side must show the *defect* (`PLAYS 0`, `644`, a chat `CALL` with the wrong prefix, `Invalid URL`), green side the fix — assert on printed markers only (`PLAYS`, `CALLS`, `CALL <url>`, `NOTIFY`, `SETTINGS`). `CALLS` counts `/audio/speech` calls only; the rewrite call appears as its own `CALL /…/chat/completions … sys=<full system prompt>` line, which is how prompt-level knobs (verbosity, heckle) are observed.
|
||||
|
||||
Prefer this over a synthetic leak: if the check already caught a real leftover/failure once, that event *is* the sensitivity proof — say so in the commit message instead of faking one.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: dankvoice-menu-knob-workflow
|
||||
description: "Add a persisted /say knob to dankvoice and test its menu row through the bats driver (DV_SELECT picks, INPUT stub, dist-pin rebuild gate)."
|
||||
---
|
||||
|
||||
Adding a setting to `dankvoice` (`~/Projects/dankvoice`, omp speech extension) means five touchpoints plus one behavioral test. Skip one and TypeScript or a silent drop catches it; skip the test and the row ships dead.
|
||||
|
||||
## The five touchpoints (order matters)
|
||||
|
||||
1. `src/settings.ts` — field on `Settings` (optional) **and** a type-guarded line in `readSettings`. A field without a guard line is written to disk and never read back; tsc stays clean, so nothing else catches it.
|
||||
2. `src/config.ts` — `Config` field (required, defaulted), resolution line, entry in the returned object, and the knob documented in the file header comment (that header is the README's pointer for "full knob list").
|
||||
- Convention: explicit env wins over settings (`env.X === "1" ? true : env.X === "0" ? false : (settings.x ?? false)`); booleans use `1`/`0`, enumerations use the literal value.
|
||||
3. `src/index.ts` — `persistSettings()` key list (a knob missing there never survives the session) **and** the getter in the `new Speaker({...})` deps object.
|
||||
4. `src/utterance.ts` — getter on `SpeakerSettings` if the Speaker reads it at speak time.
|
||||
5. `src/menu.ts` — the row. Rows carry their own `run`; never dispatch on `choice === options[N]`.
|
||||
|
||||
Gate at the call site, not the prompt, when a mode must not apply to operator-typed text: `Speaker.speak(ctx, text, thinking, auto)` — `auto=true` only from `message_end`/`agent_end`; `/say <text>` passes nothing. Prompt-only gating leaks.
|
||||
|
||||
## Test the row behaviorally, never as a config assertion
|
||||
|
||||
`tests/behavior.bats` drives the built `dist/index.js` against a mock harness. Selects and typed text both come from one pipe-separated queue: `DV_SELECT="row label|sub choice|typed text"`, consumed in call order; a consumed-exhausted `select` returns `options[0]`, so always name the row by its **rendered label** (`"Verbosity: normal (change)"`, `"Intro: none (change)"`).
|
||||
|
||||
- `ui.input(title, placeholder)` in the driver mock logs `INPUT <title> <placeholder>` and takes the next queue entry. If the mock lacks `input`, no typed value was ever testable — add the stub before writing the test.
|
||||
- Scenario pattern that proves a pick works (`case "knobs"`): `say.handler("menu")` → `say.handler("on")` → two `agentEnd(answer(...))` with `settle()` between. Then assert on observable output, not on the settings file:
|
||||
- prompt-shaping knob: `[[ "$output" == *"ONE spoken sentence"* ]]` (directive reached the rewrite request)
|
||||
- framing: `[[ "$output" == *"input=Update, chief!"* ]]`
|
||||
- throttle/framing counting: `[[ "$output" == *"CALLS 1"* ]]`
|
||||
- A guard test whose negative case would pass trivially must also assert its premise, e.g. truncation: assert `CALLS 1` so "no outro" cannot be a green caused by nothing ever being cut.
|
||||
|
||||
Red first, in a batch: write all the new `@test` blocks, run `bats tests/behavior.bats -f "<pattern>"`, confirm the new ones fail, then implement.
|
||||
|
||||
## Gates and gotchas
|
||||
|
||||
- **Two `bats` runs at once corrupt each other**: `setup()` runs `bun run build`, so a concurrent run rebuilds `dist/` mid-suite and the byte-identical pin test fails spuriously. Serialize.
|
||||
- **Edit `src/` → rebuild before commit**: the pin test compares committed `dist/` to a fresh build. Sequence is `bun run build && bunx tsc --noEmit && bats -f <pattern>`, then `git add -A` (dist included). A failure there is normally a stale bundle, not a bad assertion.
|
||||
- `VAR=value some_function args` in bats does reach the child process, so knob tests need no `export`.
|
||||
- `edit` tool: inserting after a line whose content you know but that the current tag never displayed gets refused — the refusal echoes the real line; if it matches intent, re-issue the identical edit instead of re-reading.
|
||||
- Ship: branch `feat/<issue#>-<slug>` off fresh `origin/main`, one commit per knob, PR targets `main`, merge only on Frank's explicit OK (`do=merge`, `merge_style=no-ff`), then `omp plugin upgrade dankvoice` and `cmp` installed `dist/index.js` against the merged one.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: dankvoice-silence-triage
|
||||
description: "Triage silent or late dankvoice auto-speak on the fleet: arming key, /say grammar, endpoint/sink, event granularity, humanizer latency, plus the black-box bats driver."
|
||||
---
|
||||
|
||||
# Triage "dankvoice is silent" on the fleet
|
||||
|
||||
Symptom that recurs: auto-speak only speaks when `/say <text>` is typed
|
||||
explicitly. Work the gates in this order; each has an objective check.
|
||||
|
||||
1. **Armed?** `cat ~/.omp/cache/dankvoice.json | jq .auto` — must be `true`
|
||||
(or `DANKVOICE_AUTO=1`). Arming is read at **startup only** and `/say on`
|
||||
never persists, so a session started before the key was added is silent even
|
||||
though fresh sessions work. `/say start` is *not* a subcommand: unknown
|
||||
arguments are spoken verbatim (it read out the word "start"). `/say on` is
|
||||
the verb; `/say status` must print `auto-speak on`.
|
||||
2. **Endpoint + sink?** `/say status` must show a real endpoint (not
|
||||
`NO ENDPOINT`) and the saved sink must exist:
|
||||
`pw-dump | grep -o 'alsa_output\.[^"]*' | sort -u`.
|
||||
3. **Did it synthesize?** `pgrep -a pw-play` during a turn: one process per
|
||||
segment, `--target <sink>`, wav under `/tmp/dankvoice-*/say-<epochms>.wav`.
|
||||
One temp **directory per utterance chain** — count dirs to count utterances,
|
||||
pids to count segments.
|
||||
4. **Event granularity (the structural cause).** `agent_end` fires once per
|
||||
*prompt* when the loop settles and is skipped when the harness maps a
|
||||
non-terminal settle (`willContinue`) — long runs with background jobs or
|
||||
subagents therefore narrate nothing. Mid-run narration needs `turn_end`
|
||||
(fires per assistant message + tool results). Verify delivery, don't infer
|
||||
it: skill `omp-extension-event-probe`.
|
||||
5. **Latency is not silence.** Narration passes through the humanizer
|
||||
(frank-zealot) before synthesis, so an utterance lands several seconds after
|
||||
its turn; under endpoint load that stretches to tens of seconds.
|
||||
`/say humanize off` = mechanical voice, near-instant. Judge cadence from
|
||||
probe-log timestamps, never from when audio starts.
|
||||
|
||||
## Design rules the plugin now encodes (dank/dankvoice#2→#3)
|
||||
|
||||
- Narrate `turn_end` per turn **plus** `agent_end` for the settled answer;
|
||||
dedupe identical text per run (reset on `agent_start`) so the answer speaks once.
|
||||
- A turn landing while the player is alive is **dropped** — no queue, no
|
||||
interrupting; a tool loop would cut every sentence off after a word.
|
||||
Busy test: `state.child !== undefined && state.child.exitCode === null`.
|
||||
- Tool-only turns (no prose, no narration-flagged thinking) stay silent.
|
||||
- `print`/`json` modes stay silent, so benches never chatter.
|
||||
- Settings writes must **round-trip the file** (`{...onDisk, ...owned keys}`).
|
||||
Rebuilding from owned keys deleted `"auto"` on any menu choice and silently
|
||||
un-armed every later session — the classic cause of "it worked yesterday".
|
||||
|
||||
## Black-box test harness (repo `dank/dankvoice`)
|
||||
|
||||
`tests/behavior.bats` drives the built artifact through an in-process driver:
|
||||
`const say = DV_EXT(ctx, cfg)[0]`, handlers via `const events = new Map();
|
||||
const pi = { on: (n,h)=>events.set(n,h) }`, then `events.get("turn_end")?.({
|
||||
turnIndex:0, message:{role:"assistant",content:[{type:"text",text:"…"}]},
|
||||
toolResults:[]}, ctx)`. Assert on `CALL`/`PLAYS`/`NOTIFY` lines and
|
||||
`SETTINGS <file contents>`; stubs are `DV_PLAYER_CMD` (logs voice+input, honours
|
||||
`DV_HOLD`) and `DV_REWRITE_URL` (404 → falls back to mechanical text). `setup()`
|
||||
runs `bun run build` per test → serialize suites on the shared `dist/`.
|
||||
`tests/packaging.bats` requires `dist/` **committed** (compares to HEAD), so
|
||||
build → commit → the drift pin goes green.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: docs-dedupe-verify-campaign
|
||||
description: "Run an exhaustive docs dedupe + fact-verification campaign in dank/command-center: scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR."
|
||||
---
|
||||
|
||||
# Exhaustive docs dedupe + verification campaign (dank/command-center)
|
||||
|
||||
## Frame
|
||||
- Read `omp-docs/references/okf-spec.md` + root `omp-docs/index.md` first; inventory with `find omp-docs -name '*.md' | wc -l`.
|
||||
- Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir).
|
||||
- Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/<name> -b docs/<issue>-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees.
|
||||
- Open the Gitea issue first (multi-moving-parts law); branch name `docs/<issue#>-<slug>`.
|
||||
|
||||
## Scout fleet (overlap detection)
|
||||
- One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task (skills trees, llama trees, pi vs packages, niri, sleep/power, core, gitea, gitea+cross-cutting sweep). Cap 2 concurrent; queue rest.
|
||||
- Ledger format per cluster → `local://dedupe/<cluster>.md`: inventory table (file | lines | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE), overlaps, facts-to-keep, inbound links, staleness.
|
||||
- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. NEVER apply a scout verdict without re-checking the file, the tests, and the changelog yourself.
|
||||
|
||||
## Fact verification
|
||||
- Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec <c> <tool> --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32).
|
||||
- External claims: web_search/web read yourself (no subagents per Frank's rule). Kill fabrications outright (invented distro `FreeToken`, nonexistent `ornith-llama-cpp-setup` tree, unattributable "44.5% peer-reviewed CUA") — delete/drop with an explicit note, never keep "plausible" numbers.
|
||||
- Policy drift (live ≠ tuned docs): flag in the doc + PR body as an operator decision; NEVER silently re-apply policy.
|
||||
|
||||
## Log compaction (log.md)
|
||||
- Entry law: dated entry = what changed + durable lesson + concept-page pointer; narratives recoverable via `git log --follow`. Record the law at the top of the log. Tag historical-era entries.
|
||||
|
||||
## Gates + landing
|
||||
- Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands.
|
||||
- Before opening the PR: `git diff --stat origin/main..HEAD | grep -i 'tmp\|scratch\|\.bak'` — scaffolding merged under a `.tmp-` name in the 2026-10-19 campaign and needed a follow-up PR to promote.
|
||||
- PR via API (skill `gitea-api-pr-operations` for token-extraction + merge payload); `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue.
|
||||
- After Frank's merge OK: propagate deployment `git -C ~/.omp fetch && git merge --ff-only origin/main`; reap campaign worktree (`git worktree remove --force`) + delete branch local/remote; close issue with landing comment.
|
||||
|
||||
## Line-edit op syntax trap (bit three times)
|
||||
- Valid ops: `PUT N.=M:` (body `+text` lines), `CUT N.=M` / `CUT N.=M @reg`, `PUT <N @reg`, `PUT >N:` / `PUT <N:`. There is NO `PUT N+=:` form — it lands in the file as a literal garbage line and shifts the insert to the wrong position. For gap-insert use a lone `PUT >N:` or `PUT <N:` per call; pre-compute line numbers against the file map (ops reference original lines within one call, but a failed op poisons positions).
|
||||
- Recovery recipe when a literal op line lands: read exact bounds, `CUT N.=N` (junk), `CUT A.=B @block` (misplaced section), then `PUT <M @block` at the true anchor, re-read tail, add spacing blank if a `---` now abuts a list line.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: docs-point-of-use-destruction
|
||||
description: "Documentation exists for point of use, not archival. Use before any dedupe/compact/verify-docs job, and whenever a census or \"only copy\" argument is being used to KEEP documentation: apply the purpose test and destroy obsolete measurements. Supersedes the preservation conclusions of vault-one-fact-one-owner-dedupe §3/§12."
|
||||
---
|
||||
|
||||
# Documentation exists for point of use
|
||||
|
||||
**Supersedes** `vault-one-fact-one-owner-dedupe` §3 ("port before cut") and §12
|
||||
("expect docs flat") wherever those conclude a page must be *kept*. Sole custody of a
|
||||
useless fact is still useless.
|
||||
|
||||
## The test, before you write and again before you keep
|
||||
|
||||
- Would the reader, doing a real task tomorrow, **act differently** because of this
|
||||
line? No → delete it.
|
||||
- Is it a measured rate (t/s, tok/s, ms, MiB-of-pool, acceptance %, a bench ladder) for
|
||||
a model / runtime / layout that is **not running now and not being compared now**?
|
||||
Destroy it. Not "file it in an Eras row", not "preserve it for history". AI moves
|
||||
faster than the vault: the number is wrong before anyone reads it, and a wrong number
|
||||
in a runbook gets believed.
|
||||
- Keep the **law** the measurement taught; drop the measurement. Qualitative causal
|
||||
stories survive; numbers do not.
|
||||
- What survives: live config and its exact numbers, procedures, failure signatures,
|
||||
security posture, interface contracts, rollback pointers.
|
||||
- **Losing information is an acceptable, usually correct outcome.** Frank said so
|
||||
explicitly on 2026-10-19 after rejecting a PR that had preserved 53 ledgers.
|
||||
|
||||
## Traps that lead agents to preserve junk
|
||||
|
||||
- **A census cannot decide what to keep.** `port-census.py`-style tools answer "where
|
||||
does this fact live", never "does anyone need it". "14–63 sole-record lines per page"
|
||||
was read as *irreplaceable* when the fact was *obsolete*. Ask usefulness, not custody.
|
||||
- **A census that includes the file under test proves the page is held by itself.**
|
||||
Exclude it, print the corpus size, assert absence; treat `0 unheld` on a long page as
|
||||
a tool bug until proven otherwise.
|
||||
- **"One fact, one owner" is a placement rule, not a preservation rule.** It says where
|
||||
a surviving fact lives; it never says the fact must survive.
|
||||
- **A change diary (`log.md`) is the largest archive in most vaults** (this one: 2,380
|
||||
lines, ~8 % of the vault). Git holds the chronology; a causal law belongs on the page
|
||||
that owns the thing it governs. Retire the diary.
|
||||
|
||||
## Execution that worked (2026-10-19, −13.7k lines)
|
||||
|
||||
1. Write the kill list as a reviewed artifact, one path per line, grouped by *why it is
|
||||
dead* — that list becomes the PR's argument and the trash manifest.
|
||||
2. Destroy whole pages first (`scripts/destroy-vault.py <list>`): move to
|
||||
`.AGENTTRASH/<date>-destruction/`, then delete every reference from survivors, whole
|
||||
line included when the dead link was its only content. Do not compare link targets
|
||||
with `pathlib.Path.stem` — it strips real suffixes (`sd.cpp`).
|
||||
3. Then strip numbers from what remains (`scripts/axe-measurements.py`): delete ledger
|
||||
sections (noun-LEDS the heading, nothing instructional after it), blank any table cell
|
||||
reporting a rate, fold Eras older than the current one into a single row pointing at
|
||||
git. Never touch a shape number or a Laws/Wire/Rollback section.
|
||||
4. Sweep indices (generator must exclude the trash dir, drop rows whose page cell became
|
||||
empty), re-lint, re-check links.
|
||||
5. Rewrite the governing policy in the same PR, or the next agent re-grows the vault:
|
||||
project `AGENTS.md` *Documentation Policy* + the global agent settings.
|
||||
|
||||
## Report honestly, per area
|
||||
|
||||
docs / scripts / operator-doc separately, measured against the merge-base. The tools are
|
||||
a cost that must be named, not hidden inside a headline "net" number — that is what got
|
||||
the previous pass rejected.
|
||||
|
||||
## Verify the repair tools themselves
|
||||
|
||||
One repair pass matched "line ends with a comma" and deleted the trailing comma from 660
|
||||
wrapped prose lines. Restore by matching lines that are byte-identical to `HEAD` except
|
||||
for trailing punctuation (`restore-punctuation.py`), keep the destructive rule out of the
|
||||
tool, and record the incident in its docstring.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: docs-refresh-scout-summaries
|
||||
description: "Refresh a stale documentation cluster without delegating the writing: scout subagents produce per-line verdict tables (holder verified by comma-insensitive grep), the parent makes every edit, and a port-before-cut census proves nothing was lost. Use for dedupe-our-docs, fix-the-stale-runbooks, pointers-instead-of-copied-numbers jobs."
|
||||
---
|
||||
|
||||
# Refresh stale docs with scout summaries, parent-side writes
|
||||
|
||||
Use when a cluster of documentation pages is stale and/or restates facts another page owns, and
|
||||
the fix touches many files. **Scouts read and classify; the parent writes.** Sibling skill
|
||||
`vault-one-fact-one-owner-dedupe` covers collapsing era pages into owner pages - its section 3
|
||||
(orphan census), section 7 (edit-tool hazard) and section 9 (verify/PR) apply unchanged.
|
||||
|
||||
## Why not writer subagents (measured cost)
|
||||
|
||||
Writer agents produce plausible output and cannot be trusted on the one claim that matters - the
|
||||
owner page already holds this fact. In one run they asserted every cut was grep-confirmed while
|
||||
FOUR cuts left pointers to pages that did not contain the fact (118->238 GiB, 16,095 tokens,
|
||||
0.739/0.554/0.427/0.340, PR #23897). Each became a broken promise the next reader trusts.
|
||||
Classification is the only part that parallelizes; writing is where taste (ownership, era
|
||||
naming, what counts as a law) lives. So fan out read-only scouts and keep the pen.
|
||||
|
||||
If writer agents were already spawned and are no longer wanted: `write agent://all` to stop
|
||||
them, then `git status` the target tree - `proc://` does not list agent jobs, and a finished
|
||||
worker may already have edited files.
|
||||
|
||||
## Pipeline
|
||||
|
||||
1. **Cluster** targets into groups of 3-4 pages sharing subject matter (one cluster per scout).
|
||||
Know the owner map first (endpoint/service/host pages, plus the repo documentation law).
|
||||
2. **Wave of 2 scouts** (harness cap is `task.maxConcurrency`, usually 2). Scout contract:
|
||||
|
||||
> READ-ONLY, no writes of any kind. Per file, one table row per candidate fact:
|
||||
> `| lines | verbatim figure + 6 words | holder elsewhere (file:line) or none | VERDICT |`
|
||||
> Verdicts: `CUT -> [[Owner]]` (holder verified), `KEEP(owned here)` (no holder, this page is
|
||||
> the record), `PORT(parent)` (rightful owner lacks it - name page and section), `STALE`
|
||||
> (contradicted by current state; quote the contradicting line), `INDEX` (locator page).
|
||||
> Verify every holder claim with a COMMA-INSENSITIVE grep: a holder writing 1,350,182
|
||||
> satisfies a page writing 1350182 and vice versa; 16,095 does NOT satisfy a check for 16095
|
||||
> unless commas are stripped. Never name a holder line you have not read.
|
||||
> End with a per-file tally: kept / cut / port / stale.
|
||||
|
||||
Put the law, the owner map and the verification rule in the batch-level `context`, never per
|
||||
task.
|
||||
3. **Write while the next wave runs**: edit cluster N from wave N, launch wave N+1 in the same
|
||||
turn. Never launch a wave you have nothing to write.
|
||||
4. **Verify each CUT yourself** before cutting - read the holder line. A scout row is a
|
||||
hypothesis, not evidence.
|
||||
5. **PORT items are your edits**: add the fact to the owner page section (Eras row / Laws / spec
|
||||
table), then cut it from the technique page in the same commit.
|
||||
6. **STALE items: check live state**, not another document - `ssh <host> systemctl is-active
|
||||
<svc>`, the endpoint `/health`, the env file. Then rewrite with a date, or drop transient
|
||||
live state an owner page already records.
|
||||
|
||||
## The ownership test that stops wasted ports
|
||||
|
||||
Two pages carrying one figure is the defect - a figure with exactly one holder is fine wherever
|
||||
it lives. A technique page that is the sole record of a campaign owns that campaign's arm
|
||||
tables; do not port such ledgers into an endpoint Eras table. Write the clause into the repo
|
||||
documentation law so the next pass stops porting single-owner facts. Corollary: a worked example
|
||||
may repeat a live number if it declares itself dated and names the owner (a snapshot, not fleet
|
||||
state - recompute from the live files).
|
||||
|
||||
## Census after each cluster (catches your own slips too)
|
||||
|
||||
Run the sibling skill's section 3 census over `git diff -U0` for the cluster (deleted lines x
|
||||
distinctive numeric keys vs the whole vault **plus AGENTS.md**), comma-insensitively; drive the
|
||||
count to zero or list each accepted loss in the commit message.
|
||||
|
||||
```python
|
||||
flat = corpus.replace(',', '')
|
||||
pres = lambda n: n in corpus or n.replace(',', '') in flat
|
||||
toks = lambda l: [n for n in re.findall(r'\d[\d,]*(?:\.\d+)?', l)
|
||||
if len(n.replace(',', '').split('.')[0]) >= 4
|
||||
or (len(n.split('.')[-1]) >= 2 and len(n.replace(',', '')) >= 5)]
|
||||
orphan = [l for l in deleted if toks(l) and not any(pres(n) for n in toks(l))]
|
||||
```
|
||||
|
||||
## Verify and land
|
||||
|
||||
- Vault link seam must print `0 unresolved links`; hard-wrapped `[[Two Word Page]]` links split
|
||||
across lines are the usual cause - unwrap them.
|
||||
- Full suite once at the end, not per cluster; a docs seam running while scouts are live may
|
||||
read half-written files, so re-run after the last edit.
|
||||
- Commit per cluster (docs-only means **no deploy**); push, open the PR through the API with a
|
||||
600-perm curl config, `xdg-open` the PR URL immediately.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: fixture-delay-window-race-test
|
||||
description: Pin a race/abort bug in a fixture-server test by delaying only the Nth request so the interruption lands inside the in-flight window; assert via a driver-installed unhandledRejection reporter.
|
||||
---
|
||||
|
||||
# Reproducing a timing-window bug against a fixture server
|
||||
|
||||
Use when a defect only appears in a window — a promise created before it is awaited, a kill/abort landing mid-flight, a prefetch interrupted — and a fixture HTTP server answers instantly, so the window never opens. Earned in `dankvoice` issue #7 (`[Unhandled Rejection] AbortError` from `/say stop` during a pipelined synthesis).
|
||||
|
||||
## Why naive attempts pass vacuously
|
||||
|
||||
1. **Uniform delay on the fixture route** (`DV_LATENCY_MS` for every request) makes *all* requests resolve at the same time, so by the time the interruption fires nothing is in flight. The test goes green against the broken code — worse than no test.
|
||||
2. Asserting on a log line count that also counts interruption noise: `dankvoice`'s driver prints `PLAYS <n>` as the number of **player log lines**, and the stub appends `KILLED` on SIGTERM — so a single playback with one interrupt reports `PLAYS 2`. Count the payload lines instead (`grep -c 'PLAYER .*\.wav$'`).
|
||||
|
||||
## The shape that works
|
||||
|
||||
- **Delay only requests after the first.** The first response gets the pipeline into playback; the second stays in flight, which is the window:
|
||||
```ts
|
||||
const spoken = received.filter((c) => c.url?.endsWith("/audio/speech")).length; // already pushed
|
||||
const delay = spoken > 1 ? Number(process.env.DV_SYNTH_MS ?? 0) : 0;
|
||||
if (delay > 0) setTimeout(answer, delay); else answer();
|
||||
```
|
||||
- **Make the interruption land inside it.** Player holds long (`DV_HOLD=2`), interruption at a fixed offset (`DV_STOP_AT=400`) with the delayed response far later (`DV_SYNTH_MS=1200`): margins wide enough that the abort is neither before the request nor after the response.
|
||||
- **Turn unhandled rejections into observable output** in the test driver instead of letting the runtime crash on them — that is also exactly what the real harness does to the user:
|
||||
```ts
|
||||
process.on("unhandledRejection", (reason) => {
|
||||
console.log(`UNHANDLED ${reason instanceof Error ? `${reason.name}: ${reason.message}` : String(reason)}`);
|
||||
});
|
||||
```
|
||||
Then assert `[[ "$output" != *"UNHANDLED"* ]]` plus the concrete behavior (segments actually played, player killed).
|
||||
- **Leak checks measure a delta, not an absolute:** count `dankvoice-*` temp dirs before and after the run and compare; an absolute `== 0` fails on another test's leftovers.
|
||||
|
||||
## Order
|
||||
|
||||
Write the scenario first and watch it fail **on the rejection assertion specifically** — if it fails on some other assertion (a count, a kill line), the window is wrong, not the bug. Then fix, then confirm red→green on that one assertion before running the full suite.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: fleet-service-runtime-census
|
||||
description: "Answer \"what is service running right now?\" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins."
|
||||
---
|
||||
|
||||
# Fleet service runtime census
|
||||
|
||||
Answer the question "what is `<service>` running currently?" with evidence, not from docs. Docs say what *should* run; the host says what *does*. Run all five probes, then report one table.
|
||||
|
||||
## 1. Probe the published port (works from any LAN box, no ssh)
|
||||
|
||||
```bash
|
||||
curl -s -m 10 http://<host-ip>:<port>/health; echo
|
||||
curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices # per-endpoint extras: /v1/models, /metrics
|
||||
```
|
||||
|
||||
The extra route matters more than `/health` for identity claims:
|
||||
- llama.cpp and radiance **echo the requested model name** — a 200 proves nothing; read `/v1/models`.
|
||||
- halogen 404s `/completion`, `/ping`, `/props`, `/tokenize`.
|
||||
- The frank-tts façade answers `/health` **without dialing the engine**, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census.
|
||||
|
||||
## 2. Units + containers on the host (ssh as the deploy user, no sudo)
|
||||
|
||||
```bash
|
||||
ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
|
||||
| grep -E "^(●| Active| Loaded)"; \
|
||||
podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
|
||||
podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'
|
||||
```
|
||||
|
||||
- `active (running) since …` + `Up N hours (healthy)` is the answer; `activating (auto-restart)` with a ~50 ms exit is a **preflight refusal**, not an engine fault.
|
||||
- **NEVER prefix these podman calls with `sudo`.** Containers live in the deploy user's rootless storage; `sudo podman inspect frank-tts-engine` returns `no such object` while the plain form succeeds. Treating that as "container missing" is a false finding.
|
||||
|
||||
## 3. Drop-ins — the override that no health gate sees
|
||||
|
||||
```bash
|
||||
ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
|
||||
systemctl --user show <engine>.service -p DropInPaths'
|
||||
```
|
||||
|
||||
`DropInPaths=` must be empty. A leftover bench drop-in (`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`) makes the next deploy boot a different model at a different shape while `/health` still answers 200 and the e2e gate passes.
|
||||
|
||||
## 4. Live env vs repo env — the only truth test
|
||||
|
||||
```bash
|
||||
# repo side (local checkout)
|
||||
grep -vE '^\s*#|^\s*$' scripts/<svc>.env
|
||||
# host side
|
||||
ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'
|
||||
```
|
||||
|
||||
Diff them. Per repo law the env file beside the serve script **is** the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check `git log --oneline -3 -- scripts/<svc>.env` to confirm the host isn't ahead of git.
|
||||
|
||||
## 5. Report
|
||||
|
||||
Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.
|
||||
|
||||
## Traps that produce false findings
|
||||
|
||||
- Digest-pinned images: the local image ID's *creation date* answers "is this stale" better than the tag; `latest` + `Pull=never` is a locally built image, not a registry pull.
|
||||
- `ssh` user context changes what podman sees (root vs rootless) — see step 2.
|
||||
- Two ports on one host are two providers in `models.yml`; a port mismatch is a rename or drift, not a config typo.
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: fleet-status-daily-report
|
||||
description: "Answer \"status update / what happened yesterday\" for the dank fleet: git-window record on origin/main plus a live per-host census (endpoints, units, RAM, DNS), with the reboot-attribution check that stops false incident causes."
|
||||
---
|
||||
|
||||
# Fleet status report (Dankclaw + Dankserver)
|
||||
|
||||
Use for "what's the state of the fleet", "what happened yesterday", or a post-incident sweep. Read-only.
|
||||
For one service's live state, prefer `fleet-service-runtime-census`; this is the fleet-wide sweep.
|
||||
|
||||
## 1. The day's record (repo is the ledger)
|
||||
|
||||
```bash
|
||||
git fetch origin --quiet
|
||||
git log --no-merges --since="YYYY-MM-DD 00:00" --until="YYYY-MM-DD+1 00:00" \
|
||||
--date=format:'%H:%M' --pretty='%ad %h %s' origin/main
|
||||
git log --merges --since=... --pretty='%ad %h %s' --date=format:'%H:%M' origin/main # PR numbers
|
||||
```
|
||||
Merge lines carry the PR number — that is the review trail; the non-merge log is what actually changed.
|
||||
Group commits into streams (endpoint names lead the subject) rather than listing 20 subjects.
|
||||
|
||||
## 2. Live census (parallel: one call per host + one endpoint sweep)
|
||||
|
||||
Endpoint sweep from any LAN box — `/health` is honest on all four (archon binds only after full load):
|
||||
|
||||
```bash
|
||||
for u in 192.168.0.143:7770 192.168.0.143:7771 192.168.0.235:7770 192.168.0.143:7772; do
|
||||
printf "%s %s\n" "$u" "$(curl -s -m 12 -w ' [%{http_code}]' http://$u/health | head -c 160)"; done
|
||||
curl -s http://192.168.0.143:7770/v1/models # dragoon lists EVERY engine it serves, incl. NPU-lane models
|
||||
```
|
||||
|
||||
Per host: `uptime -s; uptime -p`; `systemctl show <llm units> -p ActiveEnterTimestamp -p NRestarts -p DropInPaths`
|
||||
(`sudo` for the root LLM units, `--user` for the quadlets); `list-units --failed`; `free -h`;
|
||||
`journalctl -k --since "<evening>" -p err`; the host's own engine log tail.
|
||||
|
||||
## 3. Attribution rules (where reports go wrong)
|
||||
|
||||
- **Never call a host reboot from a container's start timestamp.** Check `uptime -s` first; if uptime spans the
|
||||
event, the trigger was a **container restart** — find it with
|
||||
`journalctl --user -u <name>.service --since … --until …` + `podman inspect <c> --format '{{.State.StartedAt}}'`.
|
||||
(This exact mistake mislabelled a 2026-10-19 Pi-hole incident as boot-caused; the law's remedy belongs after
|
||||
*any* restart, which is a different action item.)
|
||||
- `podman stats` is a 28-container wall — sort/limit it; RSS of the LLM tenants is what matters.
|
||||
- This podman's inspect template has **no `.Config.PodName`, no `.Config.GroupAdd`, and `.HostConfig.Devices`
|
||||
can read `[]` even when a quadlet `AddDevice=` worked**. Prove GPU use from the engine's own log
|
||||
(`ggml_vulkan: Found 1 Vulkan devices: … Radeon 8060S (RADV GFX1151)`), not from inspect.
|
||||
- Kernel lines: **count** occurrences (`journalctl -k --since '7 days' | grep -c "NETDEV WATCHDOG"`) before
|
||||
calling a one-off a trend.
|
||||
- Failed units: `systemctl show <u> -p ExecMainExitTimestamp -p ExecMainStatus --value` separates
|
||||
"broken since boot" from "broke yesterday" — and a nightly failure (e.g. logrotate) is a live risk, not noise.
|
||||
|
||||
## 4. Report shape
|
||||
|
||||
Stream table (what landed) → live-state table (what is true now) → **corrections to my own earlier record** →
|
||||
open items that need Frank's decision, each with the evidence line and the one action. Flag anything that would
|
||||
touch host config outside the repo as a decision, never a silent fix.
|
||||
@@ -1,125 +1,81 @@
|
||||
---
|
||||
name: frank-tts-add-cloned-voice
|
||||
description: "Add or repair a cloned voice on the dank fleet: /home/dank/voices/<name> pack, name rules, boot-warm registration, and the under-a-second proof"
|
||||
description: "Add or repair a cloned voice on the dank fleet: pack builder, the reference.wav contract, the boot warm hook (argv-cap and env-quote laws), and how to prove a voice really answers."
|
||||
---
|
||||
|
||||
Use for "give frank-tts a new voice", "voice X is missing/unknown", or "the first
|
||||
request of the day takes 5 s". Runbook of record: vault pages [[Frank TTS]] (voice
|
||||
packs, naming) and [[Frank TTS Engine]] (the warm registry). Everything below runs
|
||||
as the deploy user on **Dankclaw** (`ssh -o BatchMode=yes dankclaw '…'`); the
|
||||
containers are **rootless** (`systemctl --user …`, `podman …` without sudo).
|
||||
# Adding (or repairing) a cloned voice on the dank fleet
|
||||
|
||||
## Naming (the question that gets asked first)
|
||||
Repo law lives in `serverAppConfigs-docs/Dankclaw/Voice Packs.md` and
|
||||
`Frank TTS Engine.md` — read those for the pack format. This skill is the
|
||||
**operating sequence plus the traps that cost real time on 2026-10-18**.
|
||||
|
||||
A voice is addressed **by its directory name** — `voice=narrator1` on
|
||||
`POST /v1/audio/speech` (omp's TTS device may send `voice_id`, same value). Use:
|
||||
## Stack, fast
|
||||
|
||||
- `^[a-z0-9][a-z0-9._-]*$`, **lowercase**, ≤ 64 chars. No slash, no `..`, no
|
||||
space, no uppercase; must not start with `.`/`_`/`-`.
|
||||
- Why lowercase rather than just "alphanumeric": the shim accepts
|
||||
`[A-Za-z0-9][A-Za-z0-9._-]{0,63}` (`app.py`, `VOICE_NAME_RE`) but the **engine
|
||||
lowercases names**, so `Narrator1` resolves on one side and not the other. The
|
||||
boot warmer refuses a mixed-case pack loudly instead of half-working.
|
||||
- Names are user-visible: a 400 carries them in `available_voices`. Fleet
|
||||
convention is `narrator1`, `narrator2`, … for the open-notebook podcast
|
||||
speakers — open-notebook is configured with exactly `narrator1/2/3`, so renaming
|
||||
those breaks podcast rendering, not just TTS — and a lowercase personal name for
|
||||
a person's own voice.
|
||||
- `frank-tts-engine` — arm A `qwentts.cpp tts-server`, container-internal `:8880`,
|
||||
publishes **no port**. Reach it with `podman exec frank-tts-engine curl …`.
|
||||
- `frank-tts` — OpenAI façade on host `:7772`; resolves voice **by name**.
|
||||
- Packs: `/home/dank/voices/<name>/` with `reference.wav` (+ manifest, `.spk`,
|
||||
`.rvq`). Builder: `scripts/build-voice-pack.sh` (crop 19.84–25 s, first 10 s of
|
||||
any source is forbidden).
|
||||
- Weights: `/home/dank/models/Qwen3-TTS-GGUF/` —
|
||||
`scripts/download-frank-tts-engine-qwentts.sh [--verify-only]` (1.4 GiB, ~18 s
|
||||
on the LAN; verifies byte count + Hub LFS sha256).
|
||||
|
||||
## Pack layout (host state — never commit it)
|
||||
## Sequence
|
||||
|
||||
`/home/dank/voices/<name>/` is a voice **iff** it holds `reference.wav`:
|
||||
1. Build the pack (dry-run first — sizes are arithmetic and asserted).
|
||||
2. Deploy the engine only if its files changed:
|
||||
`./scripts/deploy.sh dankclaw --service frank-tts-engine`
|
||||
(first run builds the image: ~10 min recursive source clone).
|
||||
3. Register/warm: `podman exec frank-tts-engine /app/warm-voices.sh`
|
||||
— registers **every** pack each run (the engine replaces a name it already
|
||||
holds; `tools/tts-server.cpp:148`), then synthesizes twice per voice.
|
||||
4. Prove from the consumer side, not the config:
|
||||
`curl -s localhost:7772/v1/audio/voices` then a real
|
||||
`POST /v1/audio/speech` and listen to the bytes.
|
||||
|
||||
```
|
||||
reference.wav 24 kHz mono pcm_s16le, 18-25 s of clean single-speaker audio
|
||||
reference.spk engine x-vector: enc_dim 2048 f32 = 8192 B
|
||||
reference.rvq RVQ codes: 22 B per 12.5 Hz frame (5456-6864 B)
|
||||
manifest.json source path + sha256, crop start/end, engine commit, name
|
||||
```
|
||||
## Trap 1 — the kernel's argument cap (this broke the first deploy)
|
||||
|
||||
Build it; do not hand-assemble and never fabricate `.spk`/`.rvq` from a WAV:
|
||||
`/usr/bin/curl: Argument list too long` in the warm log ⇒ a request body was
|
||||
passed as **one argv string**. Linux caps a single argument at
|
||||
`MAX_ARG_STRLEN = 131 072 bytes`; a ~1 MB `reference.wav` base64's to ~1.27 MB.
|
||||
curl never runs, and hooks report it as `HTTP no response` while `/health` is
|
||||
green. Fix = write the body to a file and use `--data-binary @file`; feed the
|
||||
payload through a redirection, not `$(base64 …)`.
|
||||
Regression-test shape: **real** curl + real listener + a fixture >128 KB, so the
|
||||
kernel (not a stub) produces the failure. Any pack over ~96 KB trips it.
|
||||
|
||||
```bash
|
||||
scripts/build-voice-pack.sh --all /home/dank/voices # everything in VOICE_PACKS
|
||||
scripts/build-voice-pack.sh --source SRC --offset SEC --duration SEC --sha256 HEX NAME /home/dank/voices
|
||||
scripts/build-voice-pack.sh --dry-run ... # validate + crop, no extractor
|
||||
```
|
||||
## Trap 2 — two readers of the env file disagree about quotes
|
||||
|
||||
Knobs live in `scripts/voice-packs.env` (one `_SOURCE`/`_OFFSET`/`_DURATION`/
|
||||
`_SHA256` set per voice). Three things it enforces that an operator gets wrong:
|
||||
`bash set -a; source` strips `"…"`. podman's `--env-file` (how the unit really
|
||||
reads it) does **not**. A value interpolated into a JSON body therefore arrives
|
||||
quoted and invalidates it. Test both readers, not just `source`.
|
||||
|
||||
1. **The 10-second gate.** A crop starting inside the first 10 s of the source is
|
||||
refused before anything is written (project source-material law), and the
|
||||
manifest is re-checked against the source's sha256 afterwards.
|
||||
2. **The effective crop window is 19.84-25 s, not 18-25 s.** An 18.0 s crop is a
|
||||
legal `reference.wav` but yields only 22 × 225 = 4 950 B of `.rvq`, under the
|
||||
5 456 B floor the codec config implies — so the builder says so up front.
|
||||
3. **It never deletes.** An existing pack is refused unless `--force`, and
|
||||
`--force` moves the old directory to `$PACKS_DIR/.AGENTTRASH/<voice>-<UTC>/`.
|
||||
## Trap 3 — locally built images skip rebuilds on a stale hash
|
||||
|
||||
Files must be world-readable (the container runs as uid 10000): `644` files,
|
||||
`755` dirs. `mkdir -p /home/dank/voices` as the deploy user if it is absent —
|
||||
rootless podman auto-creates a missing bind source **root-owned**, and the
|
||||
endpoint then sees no voices at all.
|
||||
`build_compose_image` compares a `cfhash` label; it must cover every file the
|
||||
`Containerfile` `COPY`s, or an edited `app.py`/hook deploys "successfully" while
|
||||
the container runs old code.
|
||||
|
||||
## Register and warm (the part people forget)
|
||||
## Measured numbers (2026-10-18, deployed build — retire the trial's ladder)
|
||||
|
||||
The engine keeps its voice registry **in memory**, and the **first synthesis of a
|
||||
registered voice decodes the whole reference clip** to snapshot codec state:
|
||||
6.2-8.0 s, first audio byte at ~5 530 ms. Later requests restore in 0.5 ms.
|
||||
Re-registering does **not** warm — only synthesizing does. So:
|
||||
| thing | value |
|
||||
|---|---|
|
||||
| first synthesis of a freshly registered voice | 1 883 ms |
|
||||
| later short requests | 1 226–1 283 ms |
|
||||
| ~110-word request (repeat) | 9.4 s / 9.8 s — tracks generated audio, not state |
|
||||
| whole boot warm, 3 packs | 15.2 s |
|
||||
| engine resident | ~4.0 GiB |
|
||||
|
||||
```bash
|
||||
# registers every pack that has a reference.wav, then speaks the warm sentence once
|
||||
podman exec frank-tts-engine /app/warm-voices.sh
|
||||
systemctl --user --no-pager --full status frank-tts-engine.service | tail -20
|
||||
```
|
||||
The trial's `6.2–8.0 s cold → 0.5 ms warm` claim does **not** reproduce; warming
|
||||
is for registration (registry is memory-only, re-register ≠ warm) plus ~0.6 s.
|
||||
|
||||
The unit's
|
||||
`ExecStartPost=-/usr/bin/podman exec frank-tts-engine /app/warm-voices.sh` does
|
||||
the same at every boot, so a fresh boot is already warm; the explicit call above
|
||||
is for **adding one voice to a running engine** and needs no restart — a restart
|
||||
drops every registration and re-pays the cold cost for the whole library. The
|
||||
`podman exec` form is not cosmetic: the engine **publishes no port**, so there is
|
||||
no host address to curl.
|
||||
## Ops notes
|
||||
|
||||
## Proof (do not report "added" without it)
|
||||
|
||||
```bash
|
||||
curl -s localhost:7772/v1/audio/voices # name is listed
|
||||
time curl -s -o /tmp/a.wav localhost:7772/v1/audio/speech \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"model":"tts-1","voice":"NAME","input":"Checking the cloned voice.","response_format":"wav"}'
|
||||
time curl -s -o /tmp/b.wav localhost:7772/v1/audio/speech \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"model":"tts-1","voice":"NAME","input":"And the second request.","response_format":"wav"}'
|
||||
file /tmp/a.wav /tmp/b.wav # really audio
|
||||
```
|
||||
|
||||
The warmer's journal lines carry `voice NAME cold=…ms warm=…ms` — quote that pair.
|
||||
Note that streaming and buffered output differ at the same seed (92.2 % of samples,
|
||||
-41.6 dB RMS): never compare two paths or two "identical" renders, only that each
|
||||
is audio and that the second request is fast.
|
||||
|
||||
## Failure modes worth naming
|
||||
|
||||
- `400 unknown voice …; available voices: …` → the name is not a pack: wrong
|
||||
spelling/case, missing `reference.wav`, or `/home/dank/voices` root-owned/empty.
|
||||
Trust the 400's list over any manifest.
|
||||
- 400 with `(none: no voice pack with a reference.wav in /voices)` → mount problem,
|
||||
not a voice problem.
|
||||
- First request slow **after** a boot → the warm pass failed; read the unit's
|
||||
journal (the warmer exits 0 and logs when there is no voice directory, so
|
||||
silence there is not success either).
|
||||
- Endpoint answers but synthesis fails 502 → the façade must dial
|
||||
`http://frank-tts-engine:8880` over the shared bridge
|
||||
(`frank-tts-engine.network`); a `host.containers.internal` / `127.0.0.1:8880`
|
||||
upstream can never work between containers, because netavark DNATs published
|
||||
ports only.
|
||||
|
||||
## Never
|
||||
|
||||
Never write under `/voices` from the endpoint (mounted `:ro` by design), never
|
||||
`git add` a voice pack, never `rm` a pack (repo policy: move to `.AGENTTRASH/`),
|
||||
and never benchmark against this endpoint while it serves — stop and mask the
|
||||
units first.
|
||||
- `systemctl --user stop` **removes** the quadlet container — `podman start` then
|
||||
finds nothing. To run the engine without its `ExecStartPost` hook, use a
|
||||
separate container, not stop+start.
|
||||
- `podman cp` of a fixed script into a running container is a **trial only**: the
|
||||
layer dies with the container. Merge + redeploy to make it durable.
|
||||
- Nothing on the voice pair exposes `/metrics`; Prometheus can't scrape it (a
|
||||
blackbox HTTP probe on 7772 is the only honest option).
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
name: frank-tts-consumer-voice-triage
|
||||
description: "Triage a consumer (OpenWebUI, omp, open-notebook) rejecting or ignoring a frank-tts voice name: prove whether the server or the browser-side Kokoro.js engine produced the error."
|
||||
---
|
||||
|
||||
# frank-tts consumer voice triage
|
||||
|
||||
A consumer says a voice name is invalid, speaks in the wrong voice, or is
|
||||
silent. `frank-tts` (`192.168.0.143:7772`, shim
|
||||
`dankclaw/voiceservice/app.py`) resolves `voice=<name>` against
|
||||
`/home/dank/voices/<name>/reference.wav` and answers **400 + the list of
|
||||
voices that exist** for a name it cannot resolve — never a silent fallback.
|
||||
Any error naming a *different* voice list did not come from it.
|
||||
|
||||
**Signature that decides the case immediately:** an enum of ~29 names
|
||||
`af_heart, af_alloy, … am_*, bf_*, bm_* … bm_fable` is **Kokoro's fixed pack**,
|
||||
the voice list of **Kokoro.js running in the browser** (OpenWebUI ships it
|
||||
in its own front end as TTS engine value `browser-kokoro`, with a personal
|
||||
"Kokoro.js dtype" setting; model `onnx-community/Kokoro-82M-v1.0-ONNX`).
|
||||
`narrator1/2/3` are frank-tts cloned voices. Kokoro cannot clone.
|
||||
|
||||
## Instrument ladder (stop at the first that answers)
|
||||
|
||||
1. **What the endpoint actually accepts** (never infer from a consumer's text):
|
||||
`curl -s http://127.0.0.1:7772/v1/audio/voices` → JSON array of pack names.
|
||||
2. **Speak with the consumer's exact body, from inside the consumer's
|
||||
container** — same network, same shape. Write the JSON to a file and
|
||||
`--data-binary @file` (nested `sh -c` quoting silently sends an empty
|
||||
body and yields a bogus `400 malformed request body`):
|
||||
```
|
||||
printf '%s' '{"model":"frank-tts","input":"probe","voice":"narrator1","response_format":"mp3","speed":1.0}' > /tmp/p.json
|
||||
podman cp /tmp/p.json <consumer>:/tmp/p.json
|
||||
podman exec <consumer> curl -sS -o /tmp/o.bin -w 'http=%{http_code} type=%{content_type} bytes=%{size_download}\n' \
|
||||
-X POST http://host.containers.internal:7772/v1/audio/speech \
|
||||
-H 'Content-Type: application/json' --data-binary @/tmp/p.json
|
||||
```
|
||||
`200 audio/mpeg|audio/wav` ⇒ the server path is healthy; the failure is elsewhere.
|
||||
3. **Did the request arrive at all?** `podman logs --tail 400 <consumer> | grep -c audio/speech`.
|
||||
A server-path failure leaves a log line **and** a frank-tts error body; a
|
||||
client-side engine leaves **neither**. This contrast is the discriminator.
|
||||
4. **Where the rejection string lives** (proves the emitter without guessing):
|
||||
```
|
||||
podman exec <consumer> sh -c 'grep -ho ".\{200\}Should be one of.\{80\}" /app/build/_app/immutable/chunks/*.js | head'
|
||||
podman exec <consumer> sh -c 'grep -ho ".\{60\}af_heart.\{200\}" /app/build/_app/immutable/chunks/*.js | head -2'
|
||||
```
|
||||
Matched inside the front-end bundle ⇒ browser-side validator, not the API.
|
||||
5. **Config authority sweep** (OpenWebUI): admin config is the `config`
|
||||
table (one key per row, JSON `value`; expected
|
||||
`audio.tts.engine=openai`, `audio.tts.model`, `audio.tts.voice`,
|
||||
`audio.tts.openai.api_base_url`); per-user overrides are the `settings`
|
||||
JSON column of table `user` (singular, no `is_active`; some rows hold the
|
||||
string `null` → guard it). `audio.stt.*` and `audio.tts.*` are separate.
|
||||
No `sqlite3` CLI in the image or on the host: `podman exec -i <c> python -`
|
||||
with the stdlib module. **`podman exec` needs `-i` to accept a heredoc
|
||||
script** — without it the script is silently dropped and you get no output.
|
||||
|
||||
## Known traps
|
||||
|
||||
- OpenWebUI is a **rootless** quadlet: `sudo podman …` reports *no such
|
||||
container*. Read it rootless.
|
||||
- `podman ps` in this podman build has no `.ImageName` field — use `{{.Image}}`.
|
||||
- The in-browser engine choice is **per-browser storage**: empty
|
||||
`user.settings` + correct admin config + zero backend hits is the normal
|
||||
shape of this bug. Fix is UI: Settings → Audio → TTS Engine → **Default**.
|
||||
- `GET /api/config` publishes no `audio` block, and nonexistent `/api/...`
|
||||
paths return the SPA HTML with `200` — never conclude "config missing" from
|
||||
an HTML body.
|
||||
- TTS is memory-registry sensitive: the engine's voice registry is
|
||||
memory-only, so "every name returns 400" after a restart points at the warm
|
||||
pass, not at the caller (see [[frank-tts-add-cloned-voice]]).
|
||||
|
||||
## Local playback side (omp `/say`, manual probes)
|
||||
|
||||
If synthesis succeeds (`200`, sane WAV bytes) but nothing is audible, the
|
||||
fault is the PipeWire graph, not the endpoint:
|
||||
`wpctl status` (default sink marked `*`, and *Default Configured Devices* —
|
||||
a remembered-but-offline `bluez_output.*` there is the classic silent
|
||||
route), `wpctl get-volume @DEFAULT_AUDIO_SINK@`, `pw-play -v <file>` (expect
|
||||
`stream drained` + exit 0), and target a device explicitly with
|
||||
`pw-play --target <node-id>` to identify which hardware is live. A one-word
|
||||
clip (~0.7 s) is easy to miss — probe with a multi-second sentence.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
name: franksign-worktree-db-cred-trap
|
||||
description: "Diagnose franksign e2e seam 5 / health 503 db:down in a fresh worktree: setup-dev.sh minted a password that mismatches the shared franksign-db container."
|
||||
---
|
||||
|
||||
# franksign fresh-worktree DB credential trap
|
||||
|
||||
## Symptom
|
||||
`tests/run.sh` in a fresh franksign worktree fails seam 5 ("e2e server did not
|
||||
become healthy") while the server log says `franksign-api listening on :3100`.
|
||||
Manual probe shows the server IS up:
|
||||
|
||||
```bash
|
||||
PORT=3100 node --env-file=.env apps/api/dist/cli.js >/tmp/manual-e2e.log 2>&1 &
|
||||
sleep 2; curl -s http://127.0.0.1:3100/health
|
||||
# -> 503 {"status":"degraded","db":"down",...}
|
||||
```
|
||||
|
||||
Not a regression, not a :3100 port collision (check `ss -tlnp | grep 3100` to rule that out).
|
||||
|
||||
## Root cause
|
||||
`infra/setup-dev.sh` mints a fresh `POSTGRES_PASSWORD` per checkout
|
||||
(`if [ ! -f infra/postgres.env ]`), but `franksign-db` + `franksign-pgdata`
|
||||
are shared box state owned by the main checkout
|
||||
(`~/Projects/franksign`). The new worktree's `.env` DATABASE_URL then fails
|
||||
TCP auth against the initialized volume.
|
||||
|
||||
The setup script masks it: its readiness gate is
|
||||
`podman exec franksign-db pg_isready`, which uses the in-container local
|
||||
socket and passes regardless of password — so setup prints "postgres ready"
|
||||
while the checkout is dead on arrival.
|
||||
|
||||
## Fix (git-ignored files, no commit needed)
|
||||
Copy the shared credentials from the owning checkout:
|
||||
|
||||
```bash
|
||||
grep DATABASE_URL /home/blank/Projects/franksign/.env # source of truth
|
||||
# PUT that exact DATABASE_URL line into this worktree's .env
|
||||
# PUT the matching POSTGRES_PASSWORD into this worktree's infra/postgres.env
|
||||
```
|
||||
|
||||
Gotcha: the shared password may contain literal `$$`. Keep it byte-for-byte —
|
||||
systemd `EnvironmentFile` and `node --env-file` both read values literally,
|
||||
so `$$` in the file IS `$$` in the password. No escaping, no de-escaping.
|
||||
|
||||
## Verify
|
||||
```bash
|
||||
PORT=3100 node --env-file=.env apps/api/dist/cli.js & # then
|
||||
curl -fsS http://127.0.0.1:3100/health # -> {"status":"ok","db":"up"}
|
||||
tests/run.sh # 14/14
|
||||
```
|
||||
|
||||
## Permanent fix (proposed, not yet landed)
|
||||
`setup-dev.sh` should reuse `~/.config/containers/systemd/postgres.env`
|
||||
(symlink to the owning checkout) when present instead of minting new creds.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
name: gguf-lora-adapter-fit-census
|
||||
description: "Read-only census before applying a GGUF LoRA/adapter to a llama.cpp (or fork) endpoint: Hub metadata, declared base provenance, runtime flag support, and the behavioural probe that proves the effect."
|
||||
---
|
||||
|
||||
# GGUF LoRA / adapter fit census (read-only, before you promise a file to a payload)
|
||||
|
||||
A 3 MB adapter is a payload change: it edits the compute graph at load. Do the census
|
||||
before editing an endpoint's trio, and prove the EFFECT behaviourally — no metadata,
|
||||
card table, or health gate can see it.
|
||||
|
||||
## 1. Census the Hub repo first (no auth for public repos)
|
||||
|
||||
```bash
|
||||
curl -sS "https://huggingface.co/api/models/<org>/<repo>?blobs=true" | python3 -c "
|
||||
import json,sys; d=json.load(sys.stdin)
|
||||
print(d.get('pipeline_tag'), d.get('library_name'))
|
||||
print([t for t in d.get('tags',[])][:20]) # base_model / base_model:adapter rows matter
|
||||
for s in d['siblings']: print(s['rfilename'], s.get('size'), (s.get('lfs') or {}).get('oid'))"
|
||||
curl -sS "https://huggingface.co/<org>/<repo>/raw/main/README.md" > /tmp/card.md # grep for --lora flags
|
||||
```
|
||||
|
||||
- `base_model` / `base_model_relation: adapter` name the pack the adapter was built for.
|
||||
- **A file with `lfs.oid = None` is a plain git object**: pin the sha256 of the Hub's bytes,
|
||||
not an LFS oid. Fetcher law still applies: `curl -fsSL --fail-with-body`, byte count,
|
||||
sha256, `--verify-only`, no resume.
|
||||
|
||||
## 2. Read the adapter's own GGUF KV section (settles provenance in one read)
|
||||
|
||||
Small python read of the KV pairs — no torch, no download beyond the adapter itself:
|
||||
magic `GGUF`, `u32 ver`, `u64 n_tensors`, `u64 n_kv`; key = `u64 len` + bytes; type `u32`
|
||||
(0 u8,1 i8,2 u16,3 i16,4 u32,5 i32,6 f32,7 bool,8 str,9 array,10 u64,11 i64,12 f64);
|
||||
array = `u32 elem_type` + `u64 count`.
|
||||
|
||||
Keys worth printing: `general.type` (`adapter`), `adapter.type` (`lora`),
|
||||
`adapter.lora_alpha`, and whatever provenance the publisher wrote — e.g. PrismML/abliteration
|
||||
packs carry `prism.abliterate.base_gguf`, `base_sha256`, `direction_row`, `target_layers`,
|
||||
`target_embedding`, `sites`, `basis`.
|
||||
|
||||
**Provenance is metadata, not a gate.** The loader matches **tensor names, not hashes**, so an
|
||||
adapter fit against a sibling quantization of the same model still applies — which is a
|
||||
decision to surface to the operator, not a failure to hide. Compare `base_sha256` against the
|
||||
sha the endpoint's own fetcher pins; if they differ, say so in the PR body, the env header and
|
||||
the endpoint page, and prove the effect behaviourally instead.
|
||||
|
||||
Two publisher facts that decide whether a fork can carry the file at all: the build it was
|
||||
verified on, and what it deliberately does NOT touch (an embedding rank is the usual crash —
|
||||
`token_embd` left alone is a graph constraint, not an omission).
|
||||
|
||||
## 3. Ask the runtime binary, not your memory
|
||||
|
||||
```bash
|
||||
<llama-server> --help | grep -iE "lora|adapter"
|
||||
```
|
||||
Expect `--lora FNAME`, `--lora-scaled FNAME:SCALE,...`, `--lora-init-without-apply`
|
||||
(+ `POST /lora-adapters` for runtime scale). Prefer `--lora-scaled <file>:<scale>` as ONE
|
||||
argv token so a scale can never reach the engine without its file, and follow the endpoint's
|
||||
existing opt-out convention (the `MMPROJ_FILE=` shape: `:`-less default so an explicitly empty
|
||||
value means "off, on purpose"). Gate the deployer on the file's existence with the adapter's
|
||||
OWN fetcher named in the error text.
|
||||
|
||||
## 4. Prove the effect: refusal/behaviour probe, before AND after
|
||||
|
||||
The probe is not a bench, so the no-bench law does not block it — and the **before** half must
|
||||
be captured while the old shape is still live.
|
||||
|
||||
- Same prompts, `temperature` low, both before and after; classify with a refusal-marker regex
|
||||
and print only the opening characters, never the generated body.
|
||||
- **Budget ≥ 2 048 tokens on a thinking model.** A 600-token request can be swallowed whole by
|
||||
the reasoning channel: `content` comes back empty with `finish_reason: length` and
|
||||
`reasoning_content` full — which reads exactly like a refusal that never happened.
|
||||
- A leading `⚠️ Disclaimer`/hedge after the adapter is style, not the circuit: judge the marker
|
||||
sentence, not the tone.
|
||||
|
||||
## 5. Post-deploy reads (the ledger is the instrument)
|
||||
|
||||
- Boot log shows `llama_adapter_lora_init_impl: loading lora adapter from '<file>'` and then
|
||||
**dumps the adapter's KV block** — the provenance mismatch is printed at every start, so the
|
||||
runbook should point at the journal instead of restating it.
|
||||
- Confirm the flag on argv: `tr '\0' ' ' < /proc/$(systemctl show <svc> -p MainPID --value)/cmdline`;
|
||||
confirm `systemctl show <svc> -p DropInPaths --value` is empty (a leftover bench drop-in outranks
|
||||
everything you just deployed).
|
||||
- Shape must be untouched by an adapter: `n_slots` / `n_ctx_slot` / KV lines unchanged, and
|
||||
`/metrics` still answering (a llama.cpp endpoint is invisible to Prometheus without `--metrics`).
|
||||
- Root units log to the journal; rootless podman storage is read WITHOUT sudo (`sudo podman
|
||||
inspect` sees a different storage namespace and reports "no such object").
|
||||
|
||||
## Yields
|
||||
|
||||
One sentence per question: does the file exist and at what size/sha; which pack it declares as
|
||||
its base and whether that matches the served payload; which flag the runtime actually offers;
|
||||
what the before/after probe classified. Anything else is a decision for the operator, and the
|
||||
rollback should be one env line on a file the endpoint already loads.
|
||||
@@ -1,105 +1,73 @@
|
||||
---
|
||||
name: gitea-api-pr-operations
|
||||
description: "Open, list, merge, and verify pull requests (and diagnose issue-API scope failures) on the Dankclaw Gitea via its REST API from a workstation, keeping the token out of argv (curl -K config file, mode 600) — includes the credential-store extraction one-liner, the merge endpoint's Do-field law and Content-Type-header trap, the suggested-URL-is-not-a-PR gotcha, and the insufficient-token-scope trap."
|
||||
description: "\"Open, list, merge, and diagnose issues/PRs on the Dankclaw Gitea API (192.168.0.143:3000, repo dank/command-center) with the token stored in ~/.config/secrets/gitea — incl. the required merge payload, where review comments actually live on 1.27.3, and how to recover the session id for the mandatory body footer.\""
|
||||
---
|
||||
|
||||
# Gitea API PR operations (Dankclaw)
|
||||
# Gitea API PR operations (Dankclaw `dank/command-center`)
|
||||
|
||||
`git push` on this fleet prints `Create a new pull request for ...` — that is a
|
||||
**suggestion, not a PR**. Verify (`GET /pulls?state=open` → expect `[]`) before
|
||||
claiming a PR "should exist"; create it via the API when asked.
|
||||
## Where
|
||||
- Token: `~/.config/secrets/gitea` (`http://user:TOKEN@192.168.0.143` single line).
|
||||
Extract WITHOUT echoing:
|
||||
`tok=$(sed -nE 's#^http://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)`
|
||||
- Build a 600-perm temp curl config and use `curl -K`:
|
||||
`cfg=$(mktemp) && chmod 600 "$cfg" && printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"`
|
||||
Delete `$cfg` after use. NEVER pass `-H "Authorization: token $tok"` (the literal
|
||||
header lands in transcript + logs). NEVER echo the token.
|
||||
- Base URL: `http://192.168.0.143:3000/api/v1/repos/dank/command-center`.
|
||||
|
||||
## Endpoint + auth
|
||||
- API base: `http://192.168.0.143:3000/api/v1/repos/dank/<repo>`
|
||||
- Token lives in the git credential store configured by
|
||||
`git config --get credential.helper` → `store --file ~/.config/secrets/gitea`,
|
||||
line format `http://user:TOKEN@192.168.0.143:3000`.
|
||||
- The repo-local `.env` `GITEA_TOKEN` (main checkout, gitignored) carries
|
||||
**`read:issue` too** (measured 2026-10-18): filing issues needs THAT token,
|
||||
the credential-store one 403s on issues.
|
||||
## Recipes (all verified live 2026-10-18)
|
||||
- **Open PR** (token only needed for private repos; this one reads publicly but
|
||||
POST needs auth): `POST /pulls` JSON `{title, head, base:"main", body}`.
|
||||
- **PATCH PR title/body**: `PATCH /pulls/<n>` with JSON body `{title}` and/or
|
||||
`{body}`. Returns 201 with the full PR. Use `jq -n --arg` to build JSON; for
|
||||
multi-line bodies write the text to a temp file and `--arg b "$(cat b.txt)"`.
|
||||
(jq CLI: a bare `---` argument after the program is parsed as a flag → error;
|
||||
fold footers into the --arg string instead.)
|
||||
- **List open PRs**: `GET /pulls?state=open` → `jq -r '.[] | "#\(.number) \(.head.ref) \(.title)"'`.
|
||||
- **Read PR**: `GET /pulls/<n>` (no token; public repo) → `jq -c '{state,merged,mergeable,head:.head.ref}'`.
|
||||
`mergeable:false` is often Gitea's *not-yet-checked* state, not a conflict — prove a real
|
||||
conflict locally with `git merge-tree --write-tree HEAD origin/main` before believing it.
|
||||
- **MERGE**: `POST /pulls/<n>/merge` with `-H 'Content-Type: application/json'`
|
||||
and body **`{"Do":"merge"}`** — this build REQUIRES the `Do` field:
|
||||
`{}` or `{"DoNotMerge":false,...}` → 422; `{"Do":"merge"}` → 200 +
|
||||
`{"state":"closed","merged":true,"merge_commit_sha":"..."}`. Verify with a plain
|
||||
GET /pulls/<n> afterwards (merge endpoint's own stdout may not parse).
|
||||
Merge is the `--no-ff` merge commit on main; then `git -C ~/.omp fetch && git
|
||||
merge --ff-only origin/main` propagates the live deployment.
|
||||
- **Issues**: same shape at `/issues` (`POST {title, body}`), `/issues/<n>/comments`
|
||||
(`POST {body}`, `PATCH .../comments/<id>`) — both work with the temp-cfg auth.
|
||||
|
||||
Extract the token (the `http://` scheme contains a colon — scheme-stripping
|
||||
awk regexes silently grab the port instead; use this anchored sed):
|
||||
## Review comments: the API lies on 1.27.3 (learned 2026-10-18)
|
||||
- `GET /pulls/<n>/comments` (inline review comments) → **404 `not found` on every
|
||||
route** (token or not, `?pull_request_id=` or not) even when a review with
|
||||
comments exists.
|
||||
- `GET /pulls/<n>/reviews` works (shows `state`, `comments_count`) but bodies are empty.
|
||||
- **The comments are readable from the PR HTML**:
|
||||
`curl -sS http://192.168.0.143:3000/dank/command-center/pulls/<n> -o /tmp/pr.html`
|
||||
then extract `<div class="content comment-content">` blocks
|
||||
(python: `re.finditer(r'comment-content', src)` → widen ±800 chars, strip tags).
|
||||
Duplicates appear (suggested-change UI shows text twice); dedupe mentally.
|
||||
- Per-conversation resolve API also 404s — flag resolutions in an issue comment instead.
|
||||
|
||||
```bash
|
||||
tok=$(sed -nE 's#^http://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)
|
||||
[ -n "$tok" ] && echo "token-extracted(len=${#tok})" # length check, never echo the value
|
||||
```
|
||||
## Body-footer law (AGENTS.md)
|
||||
Every issue/PR body ends with `---` line then `cwd: <wd> · session: <id>`. Keep
|
||||
it inside the --arg string (not a `---` CLI arg).
|
||||
|
||||
## Keep the token out of argv (repo secret law)
|
||||
`curl -H "Authorization: token $tok"` is world-readable in `/proc/<pid>/cmdline`.
|
||||
Use a mode-600 config file:
|
||||
`cwd` is THIS session's own working directory (note a build worktree separately if they
|
||||
differ). When `/status` is unreachable (headless/subagent context), recover the session id
|
||||
from the session store instead of inventing one:
|
||||
`ls -t ~/.omp/agent/sessions/<cwd-slug>/ | head -2` → `<timestamp>_<uuid>` (the same uuid as
|
||||
`<uuid>.jsonl`); the slug is the cwd with `/` replaced by `-`. Subagent files live under the
|
||||
**parent** session's directory, so the newest entry there is the parent id.
|
||||
|
||||
```bash
|
||||
cfg=$(mktemp); chmod 600 "$cfg"
|
||||
printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
|
||||
```
|
||||
|
||||
- POST JSON needs a second header line: `header = "Content-Type: application/json"`.
|
||||
There IS NO `content-type` config option — curl rejects the file.
|
||||
- `rm -f "$cfg"` when done (a mode-600 token file left in /tmp is its own leak).
|
||||
|
||||
## Operations
|
||||
```bash
|
||||
# list open PRs
|
||||
curl -sS -K "$cfg" ".../pulls?state=open" | jq -r '.[] | [.number,.head.name,.base.name,.title] | @tsv'
|
||||
# create (build the payload as a FILE, see below)
|
||||
curl -sS -K "$cfg" -o /tmp/pr.json -w '%{http_code}\n' -X POST -d @/tmp/pr-payload.json ".../pulls"
|
||||
# verify: 201 is not enough — check mergeable
|
||||
curl -sS -K "$cfg" ".../pulls/1" | jq '{number,state,merged,mergeable,base:.base.ref,head:.head.ref,url:.html_url}'
|
||||
```
|
||||
|
||||
## Merging: the field is `Do`, and it is a STRING naming the strategy
|
||||
|
||||
Measured 2026-10-18 on Gitea at 192.168.0.143:3000 (`POST .../pulls/<n>/merge`).
|
||||
`MergePullRequestOption` is `{From, To, When, Do}` — **there is no `DoMerge` and
|
||||
no `merge_method`**. Every wrong shape is a 422 with a distinct message, so read it:
|
||||
|
||||
|body sent|http|message|
|
||||
|---|---|---|
|
||||
|`{"DoMerge":true,"merge_method":"merge"}`|422|`[Do]: Required`|
|
||||
|`{"Do":true,"merge_method":"merge"}`|422|`[]: json: cannot unmarshal JSON boolean into Go string within "/Do"`|
|
||||
|`{"Do":"true"}`|422|`[Do]: In…` (validation: not a strategy name)|
|
||||
|form body `Do=true&merge_method=merge`|422|`[Do]: Required` (the value must be a strategy name)|
|
||||
|`{"Do":"merge"}`|**200**|merged|
|
||||
|
||||
```bash
|
||||
curl -sS -K "$cfg" -H 'Content-Type: application/json' -X POST -d '{"Do":"merge"}' ".../pulls/4/merge"
|
||||
# then prove it: .merged true, .state closed, .merge_commit_sha set, and origin/main moved
|
||||
curl -sS ".../pulls/4" | jq -c '{number,state,merged,merge_commit_sha}'
|
||||
```
|
||||
|
||||
Strategy names: `merge`, `rebase`, `rebase-merge`, `squash`, `manually` — a
|
||||
strategy the repo does not allow is rejected too.
|
||||
|
||||
**Merge trap (measured 2026-10-18):** on `POST /merge` the `Content-Type` header
|
||||
must go on the **command line** (`-H 'Content-Type: application/json'`). Same
|
||||
curl that succeeds on `POST /pulls` with Content-Type supplied only via the `-K`
|
||||
config file returns **405 with an EMPTY body** on `/merge`. A 405 there is this
|
||||
trap (or a base-moved mergeability recheck — GET the PR, see `mergeable:true`,
|
||||
then retry with `-H`), NOT a permission problem.
|
||||
|
||||
## Building issue/PR bodies without quote carnage
|
||||
Hand-written `jq -n --arg t "..." '{title:$t, body:"..."}'` one-liners with long
|
||||
markdown bodies WILL break on a stray quote (and the broken quoting can swallow
|
||||
the following commands). Build bodies as files (heredoc or write tool), then:
|
||||
```bash
|
||||
b=$(jq -Rs . body.md)
|
||||
jq -n --arg t "$title" --argjson b "$b" '{title:$t, body:$b, head:"<branch>", base:"main"}' > payload.json
|
||||
curl ... -d @payload.json
|
||||
```
|
||||
|
||||
## Gotchas measured 2026-10-14 / 2026-10-17 / 2026-10-18
|
||||
- Unauthenticated/bad token → HTTP 401 body `{"message":"not found"}`; `jq '.[]|[.number,..]'` then fails with *cannot index "not found"* — that is an auth failure, not a missing repo.
|
||||
- **Insufficient token scope ≠ missing data.** The credential-store token
|
||||
(measured 2026-10-17) carries only `write:repository,write:user`; the issues
|
||||
API answers HTTP 403 `{"message":"token does not have at least one of required
|
||||
scope(s), required=[read:issue], token scope=write:repository,write:user"}`.
|
||||
jq surfaces that JSON as the *cannot index* error — do NOT conclude the issue
|
||||
list is empty, and note POSTs under this token create NOTHING (verify, don't
|
||||
assume retry safety). Filing issues needs a fresh token minted in the Gitea
|
||||
web UI with `read:issue`+`write:issue` (Frank-class credential action; ask) —
|
||||
OR use the main-checkout `.env` `GITEA_TOKEN`, which already has issue scope.
|
||||
PR endpoints work fine with either token (`pulls` POST/GET/merge all 200/201).
|
||||
- On the created-PR response `.head.name` can be null; verify with the `GET /pulls/<n>` and use `.head.ref`.
|
||||
- **Swagger JSON is disabled on this instance** — `/api/swagger.v1.json`, `/api/swagger.json` and `/api/v1/swagger.json` all 404. Do not go looking for the schema there; the 422 messages name the field and its Go type, which is the faster instrument.
|
||||
## Gotchas
|
||||
- Content-Type: POSTs with `-d @file.json` MUST carry
|
||||
`-H 'Content-Type: application/json'` on the command line. `-H` entries put in
|
||||
the curl config file are ignored for this (header ordering quirk).
|
||||
- `xdg-open <pr-url>` lands the PR in Frank's Firefox the moment it is opened —
|
||||
law, not courtesy.
|
||||
- `jq: cannot index "not found" with "user"` = endpoint 404'd into an error
|
||||
object; check `-w '%{http_code}'` before blaming jq.
|
||||
- The token in `.git/config`'s remote URL is the same secret; the credential
|
||||
file (`~/.config/secrets/gitea`) is the one git actually consults (global
|
||||
credential.helper) and the one to reuse for API calls.
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
name: gitea-merge-form-encoded
|
||||
description: "Use when a Gitea API merge on Dankclaw (192.168.0.143:3000) returns HTTP 422 \"[Do]: Required\" or 405 \"Please try again later\" — the merge endpoint needs form-encoded fields with the lowercase field name do, and a behind-main branch must be merged forward before it will merge."
|
||||
---
|
||||
|
||||
Gitea on Dankclaw (192.168.0.143:3000) — merge quirks, measured 2026-10-19 on `dank/serverAppConfigs` and again on `dank/dankvoice`, re-earned the same day on serverAppConfigs PR #27.
|
||||
|
||||
## 1. The payload: form-encoded, lowercase `do`
|
||||
|
||||
Creating a PR accepts a JSON body. **Merging does not**: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge` with a JSON body (`{"merge_style":"no-ff", ...}`) or with those values as request headers returns `422 {"message":"[Do]: Required"}` — four variants of that all fail identically, so do not "fix" it by changing the style name or moving the headers around.
|
||||
|
||||
**The field name is lowercase `do`.** `--data-urlencode 'Do=merge'` (capital D, which the error message spells) returns the same `422 [Do]: Required`. Do not retry with case variants — the web-UI form field is `do`.
|
||||
|
||||
Working call (form-encoded):
|
||||
|
||||
```bash
|
||||
tok=$(sed -n 's|^https\{0,1\}://[^:]*:\([^@]*\)@192\.168\.0\.143.*$|\1|p' ~/.config/secrets/gitea | head -1) # git credential.helper store file
|
||||
curl -sS -X POST -w 'http %{http_code}\n' -K <(printf 'header = "Authorization: token %s"\n' "$tok") \
|
||||
--data-urlencode 'do=merge' \
|
||||
--data-urlencode 'merge_style=no-ff' \
|
||||
http://192.168.0.143:3000/api/v1/repos/dank/<repo>/pulls/<N>/merge
|
||||
# 200 with an empty body == merged
|
||||
```
|
||||
|
||||
- `merge_style` values are the dashed names: `no-ff`, `merge`, `rebase`, `squash`.
|
||||
- Keep the token out of argv: `-K <(printf …)` feeds the `Authorization` header through stdin (a `-H "Authorization: token …"` lands in `/proc/<pid>/cmdline`, world-readable — the 2026-09-16 HF-token leak class).
|
||||
- Verify by fast-forwarding a checkout (`git fetch && git merge --ff-only origin/main`) rather than re-reading the PR.
|
||||
- Fleet law still applies: never merge without an explicit current OK from Frank, and open the PR URL in his browser (`xdg-open`) the moment it exists.
|
||||
|
||||
## 2. `405 {"message":"Please try again later"}` = the branch is behind main
|
||||
|
||||
A correctly formed merge call returns **405**, not 422, when Gitea cannot test-merge the head into the base. Read it as "merge main forward", not "retry later" — no sleep-retry ever clears it.
|
||||
|
||||
Confirm first: `GET /pulls/{n}` shows `"mergeable": false` (and `merged: false`, `state: open`) for exactly this state.
|
||||
|
||||
The fix is the repo's own trunk-based law — merge forward, never rebase:
|
||||
|
||||
```bash
|
||||
git worktree add ../wt-prN -B prN <head-branch>
|
||||
cd ../wt-prN && git merge --no-edit origin/main # resolve, see below
|
||||
./tests/run.sh # the seam is the arbiter, run it on the RESOLVED tree
|
||||
git push -q origin prN:refs/heads/<head-branch> # same branch name; Gitea reuses the open PR
|
||||
# then re-POST the merge — 200 this time
|
||||
```
|
||||
|
||||
Notes earned on PR #27 (2026-10-19):
|
||||
|
||||
- A `git worktree` per PR keeps the deployed checkout untouched; clean up with `git worktree remove --force ../wt-prN && git branch -D prN`.
|
||||
- `git status --short` during a merge lists every incoming file as staged — that is the merge commit's content, not a dirty tree.
|
||||
- Push a local branch to the PR's branch name with an explicit refspec (`prN:refs/heads/docs/…`); the push output prints `Visit the existing pull request`, which is how you confirm you hit the right head.
|
||||
- Gitea needs a few seconds after the push to recompute mergeability; a `sleep 3` before re-POSTing avoids a second 405.
|
||||
|
||||
## 3. Resolving vault conflicts against a compaction wave
|
||||
|
||||
When main has run a docs compaction (page deletions, description shortening), two conflict shapes recur:
|
||||
|
||||
- **`description:`** — `scripts/check-okf.py` enforces **DESC: ≤ 120 chars**. A branch's longer description must be shortened, not argued for; keep main's shape and fold the new topic into the wording.
|
||||
- **A `[[Page]]` link whose target page main destroyed** — `git ls-tree -r --name-only origin/main -- serverAppConfigs-docs | grep -i '<page>'` settles it. Drop the dead link; do not resurrect a page. Compaction left link-stripped stub bullets behind (`- the compile fix for …`, dash + two spaces): `git grep -nE '^- [A-Z]' origin/main -- 'serverAppConfigs-docs/**/*.md'` counts them, and the seam cannot see them because a stub is no longer a link to validate.
|
||||
|
||||
After any frontmatter change, regenerate: `./scripts/gen-vault-index.py serverAppConfigs-docs` and commit the index rows — the `vault indices are generated, not hand-carried` seam diffs regeneration against the checkout. The sub-index sync can emit one extra space after `|` for a quoted description; commit what the generator emits, since the seam compares against the generator.
|
||||
|
||||
## Credential-store token extraction (same file, adjacent trap)
|
||||
|
||||
`~/.config/secrets/gitea` holds one line shaped `http://user:token@192.168.0.143:3000`. Splitting it with `awk -F'/'` puts the host in `$3`, not `$2`, and `awk -F'/' '{print $NF}'` yields the host+path, not the token — an empty/garbage token makes **private repos answer `404 not found`** (Gitea hides them from anonymous), which reads like a wrong URL rather than an auth failure. Extract with:
|
||||
|
||||
```bash
|
||||
tok=$(sed -n 's|^https\{0,1\}://[^:]*:\([^@]*\)@192\.168\.0\.143.*$|\1|p' ~/.config/secrets/gitea | head -1)
|
||||
echo "token length: ${#tok}" # 40 — print the length, never the value
|
||||
```
|
||||
|
||||
Other API quirks measured the same day:
|
||||
|
||||
- `GET /pulls?state=open` can return an error object where `state=all` returns the list; `state=all` list entries may omit `head`/`base` — fetch `/pulls/{n}` for those.
|
||||
- POST `/pulls` for a head/base pair that already has a PR → `409` naming `issue_id: N`; the PR number is that `issue_id`, so PATCH it instead of POSTing again.
|
||||
- Issues and PRs share the issue index per repo, so `#36`-style numbers are repo-scoped: a footer fix aimed at `dank/command-center#36` 404s when pointed at `dank/dankvoice`.
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
name: gitea-pr-api-cycle
|
||||
description: "\"Open/update/merge PRs and issues on the Dankclaw Gitea API (192.168.0.143:3000) with the token-safe sequence, the form-encoded merge field, the empty-token-returns-404 trap, and post-merge live-checkout fast-forward plus the plugin-package-name upgrade gotcha.\""
|
||||
---
|
||||
|
||||
# Gitea PR/issue cycle — Dankclaw API
|
||||
|
||||
Friction-earned 2026-10-18 (PRs #15–#20, issue #18) and re-earned 2026-10-19 on `dank/dankvoice` PRs #4–#6: the merge payload claim below was wrong, and an empty token masqueraded as a missing repo.
|
||||
|
||||
## Where
|
||||
- API base: `http://192.168.0.143:3000/api/v1/repos/dank/<repo>/{pulls,issues}` — repo varies (`command-center`, `dankvoice`, `serverAppConfigs`); numbering is per-repo, so a footer citing `#36` means nothing without the repo.
|
||||
- Token: never inline, never in argv (`-H "Authorization: token …"` lands in `/proc/<pid>/cmdline`, world-readable). Extract into a 600-perm curl config and hand it to curl with `-K`:
|
||||
```bash
|
||||
tok=$(sed -nE 's#^https?://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)
|
||||
[ "${#tok}" -ge 40 ] || { echo "token extraction failed"; exit 1; }
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
|
||||
curl -sS -K "$cfg" … ; shred -u "$cfg"
|
||||
```
|
||||
- **Empty token answers `404 {"message":"not found"}`, NOT 401** — Gitea hides the private repo. A 404 on a path you have used before means auth, not a typo'd repo. The trap that produces the empty token: `awk -F'/' … '{print $NF}'` on the store line (`scheme://user:token@host/`) — the host is in `$3`, the match never fires, you get a silent empty variable. Always assert the length before POSTing.
|
||||
- Multi-line bodies: build JSON in python (`json.dumps({"title":…, "body":open(f).read(),…})`) or `jq -n --arg b "$(cat b.txt)"`. NEVER pass `---` as a bare jq argument (parsed as a flag). Escaped quotes inside `python3 -c '…'` break on `\"` — write the script to a file or use a heredoc.
|
||||
- Always capture status: `-o r.json -w '%{http_code}'`; a bare `-o` hides failures.
|
||||
|
||||
## Create
|
||||
`POST /pulls` with `{title, body, base:"main", head:<branch>}` — JSON is accepted here (unlike merge).
|
||||
- **409 `pull request already exists for these targets [issue_id: N, …]`** — a PR for that head/base exists already. Common cause: the push message printed `/pulls/new/<branch>` and someone (Frank, or an `xdg-open` you ran) clicked Create. `GET /pulls?state=all`, find it, then `PATCH /pulls/N` with your title+body instead of re-posting.
|
||||
- `GET /pulls?state=all` on this Gitea omits `head`/`base` per entry — read `/pulls/N` for those, or key off `number`/`user`.
|
||||
- Issues: same shape, `POST /issues` `{title, body}`; `PATCH /issues/N` fixes a body retroactively (works for PRs too, since a PR is an issue).
|
||||
|
||||
## Merge (the 422 trap) — corrected
|
||||
`POST /pulls/N/merge` needs **form-encoded** fields; the merge form field name is lowercase **`do`**:
|
||||
|
||||
```bash
|
||||
curl -sS -X POST -K "$cfg" \
|
||||
--data-urlencode 'do=merge' --data-urlencode 'merge_style=no-ff' \
|
||||
http://192.168.0.143:3000/api/v1/repos/dank/<repo>/pulls/N/merge
|
||||
```
|
||||
|
||||
JSON `{"Do":"merge"}` (what this skill used to claim) and header variants all return `422 {"message":"[Do]: Required"}`. `merge_style` ∈ `no-ff|merge|rebase|squash`. Success returns **200 with an empty body**; verify by landing a checkout rather than re-reading the PR. Merge only with Frank's explicit per-PR OK.
|
||||
|
||||
## Post-merge landing
|
||||
- Command-center: the checkout IS the deployment — `git -C ~/.omp fetch -q origin && git -C ~/.omp merge --ff-only origin/main`.
|
||||
- Package repos (`~/Projects/<pkg>`): same `--ff-only`, then reinstall (`omp plugin upgrade <name>`) and `cmp` the installed `dist/index.js` against the repo `dist/index.js` to prove the live artifact is the merged one. Check `ls -ld node_modules` after landing anything that touched it.
|
||||
- **`omp plugin upgrade` takes the npm package name, not the short name.** `omp plugin upgrade dankvoice` fails with `Failed to upgrade dankvoice: dankvoice is not installed` while the plugin is plainly installed; the package is `omp-dankvoice`. Find the truth with `ls ~/.omp/plugins/node_modules` or `find ~/.omp -path '*<name>/dist/index.js' -not -path '*/Projects/*'` — the installed tree is unscoped (`~/.omp/plugins/node_modules/omp-dankvoice/dist/index.js`), not `@dank/dankvoice`.
|
||||
- **`cmp` exiting 1 BEFORE the upgrade is the expected state** (installed bundle is the pre-merge build, diff reported at some char/line). Do not treat it as a failed deploy and do not "fix" the build; run the upgrade, then `cmp` again for byte identity, and grep the new bundle for a string only the merged code contains (e.g. `grep -c "Heckle instead of summarizing"`) as a second, semantic confirmation.
|
||||
- Prune: `git worktree remove --force <path>` (force: untracked deps link), `git branch -D`, `git push origin --delete <branch>` (allowed: local Gitea).
|
||||
|
||||
## Attribution footer (AGENTS.md)
|
||||
Every issue/PR body ends with `---`, then `cwd: <session cwd> · session: <session id>`. Read the real values, never invent an id.
|
||||
- **The session id is the UUID suffix of the LIVE transcript `.jsonl`, not the session directory name.** Transcript dirs are keyed by cwd (e.g. `-.omp-wt-wt-20261019-020309-6d1fc1f`) and ONE cwd can hold several sessions (resume / continuation forks), so `ls -t ~/.omp/agent/sessions/` cannot tell you which is yours and `ls -t <dir>` is equally blind. Pick the file being appended to right now: `ls -l --time-style=+%H:%M:%S <dir>/*.jsonl` beside `date +%H:%M:%S` — the mtime that equals the current second is the live session. The id is the part after `_` in `<timestamp>_<uuid>.jsonl`.
|
||||
- A `history.db` query for `session_id` filtered by `cwd` usually agrees, but it is not authoritative for WHICH of several sessions in that cwd is live — use the jsonl mtime test.
|
||||
- `local/` does NOT disambiguate: a forked transcript dir carries copies of the same `local://` artifacts, so matching artifact names proves nothing about which session is live.
|
||||
- `cwd` is that session's own working directory (the path the harness states), **not** the build worktree you committed in — when they differ, name the build worktree inside the body and keep `cwd` the session's.
|
||||
- Resolve `local://<name>.md` with `read local://<name>.md` (its header prints the absolute path) instead of guessing the directory.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: gitea-review-comment-html-scrape
|
||||
description: "Read inline review comments on the Dankclaw Gitea when pulls/N/comments 404s (Gitea 1.27.3): scrape comment-content blocks and diff anchors from the PR HTML page"
|
||||
---
|
||||
|
||||
# Gitea review-comment extraction when the API 404s
|
||||
|
||||
On the Dankclaw Gitea (192.168.0.143:3000, Gitea 1.27.3) the review-comment API is dead for inline comments:
|
||||
`/api/v1/repos/{owner}/{repo}/pulls/{N}/comments` returns 404 `not found` **even with a valid token**, and `pulls/comments?pull_id=` / `?pull_request_id=` variants 404 too. `/pulls/{N}/reviews` works but its bodies are empty for inline reviews (read `comments_count` for how many exist); `/issues/{N}/comments` shows only conversation comments, never inline ones.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Confirm the review exists and how many inline comments it holds:
|
||||
`curl -sS http://192.168.0.143:3000/api/v1/repos/dank/command-center/pulls/{N}/reviews | jq -c '.[] | {id, state, comments_count}'`
|
||||
2. Scrape the PR page instead (no token needed for a public repo):
|
||||
`curl -sS http://192.168.0.143:3000/dank/command-center/pulls/{N} -o /tmp/pr.html`
|
||||
3. Extract each comment's text — every inline comment renders a `comment-content` block; dedupe (each comment appears twice in the HTML):
|
||||
|
||||
```python
|
||||
import re, html
|
||||
src = open('/tmp/pr.html').read()
|
||||
seen = set()
|
||||
for m in re.finditer(r'comment-content', src):
|
||||
t = html.unescape(re.sub(r'<[^>]+>', ' ', src[m.start()-100:m.start()+800]))
|
||||
t = re.sub(r'\s+', ' ', t).strip()[:400]
|
||||
if t not in seen:
|
||||
seen.add(t); print('>>', t)
|
||||
```
|
||||
|
||||
The comment text appears twice back-to-back inside each block ("text text") — read it once. Blocks start after `Copy Link|Copy Source`; ignore those chrome words.
|
||||
|
||||
4. **Locate the file/line anchor** (comment-content blocks carry no `data-path`/`data-line` attrs on this build): search the raw HTML for the comment string, walk *backward* ~4–7 KB, strip tags; the nearby `agent/...` path, `@@ -a,b +c,d @@` hunk header, and surrounding diff context lines identify the anchored file and the quoted source line.
|
||||
|
||||
## Replying / housekeeping
|
||||
|
||||
- Per-conversation resolve API does not exist on this build (404s) — reply to the round as a normal issue comment (`POST /repos/{o}/{r}/issues/{N}/comments`, token via `curl -K` temp cfg, `header = "Authorization: token …"`, 600 perms, `jq -n --arg b "$(cat b.txt)" '{body:$b}'`) listing which note got which fix commit.
|
||||
- Edit a posted comment: `PATCH /repos/{o}/{r}/issues/{N}/comments/{id}` with `{"body": ...}` — use to fix your own typos rather than double-posting.
|
||||
- Token source on this box: `~/.config/secrets/gitea` (grep the remote-URL pattern); never echo it.
|
||||
|
||||
## Why this works
|
||||
|
||||
The PR page is server-rendered with full inline comment bodies; Gitea's "Copy Source" UI element guarantees the raw text is present pre-markdown-render.
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: gitea-token-from-http-credential-store
|
||||
description: "Extract the Dankclaw Gitea token from ~/.config/secrets/gitea for API calls to PRIVATE repos, and diagnose the empty-token 404 that looks like a missing repo."
|
||||
---
|
||||
|
||||
The Gitea credential store line on Frank's boxes is **http** (not https) with a percent-encoded port:
|
||||
|
||||
```
|
||||
http://dank:<40-char-token>@192.168.0.143%3a3000
|
||||
```
|
||||
|
||||
## Extraction that actually works
|
||||
|
||||
```bash
|
||||
tok=$(sed -nE 's#^[a-z]+://[^:]+:([^@]+)@.*#\1#p' ~/.config/secrets/gitea | head -1)
|
||||
curl -s -H "Authorization: token $tok" http://192.168.0.143:3000/api/v1/<path>
|
||||
```
|
||||
|
||||
Common wrong pattern (silent failure): an `awk`/`grep` anchored on `https://` or on `@192.168.0.143:` matches nothing, leaving `tok=""` — verify with `echo "tok len ${#tok}"` before believing any API result.
|
||||
|
||||
## The 404 trap
|
||||
|
||||
`dank/command-center` is **public**: an empty token still returns real issue data. `dank/dankvoice` is **private**: the same empty token returns `404` on `/api/v1/repos/<owner>/<repo>` and `{}`-shaped bodies (`title: undefined`) on issues. A 404 with a zero-length token is an auth problem, NOT a missing repo — confirm by printing the token length first, then re-check.
|
||||
|
||||
## Other notes
|
||||
|
||||
- `git credential fill` fails headless here (`could not read Username … terminal prompts disabled`); read the store file directly instead.
|
||||
- Never print the token: mask when inspecting the file —
|
||||
`sed -E 's#(://[^:]+:)[^@]+@#\1REDACTED@#' ~/.config/secrets/gitea`
|
||||
- Issue/PR numbers are **per repo**: `dank/dankvoice` #8–#11 (refactor, hardening, AbortError fix, menu defects) are unrelated to `dank/command-center` #8–#11 (SDLC, trunk-law). Read the body from the repo the work targets before referencing an issue in a commit message.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: gitignore-trailing-slash-symlink-contamination
|
||||
description: "Trailing-slash gitignore rules don't match symlinks, so worktree dependency symlinks get committed; detect with git ls-tree mode 120000, fix with git rm --cached."
|
||||
---
|
||||
|
||||
A `.gitignore` entry written with a **trailing slash matches directories only**. A symlink of the same name is not a directory, so it slips past the rule, and `git add -A` in a `git worktree` checkout silently **tracks** it. Cost measured 2026-10-19: a machine-specific symlink (`node_modules -> /home/blank/Projects/dankvoice/node_modules`) landed in `main` with a merged PR.
|
||||
|
||||
Why worktrees are the trigger: a worktree has no `node_modules` (or other installed-deps dir) of its own, so the cheap way to make build/test work there is a symlink into the primary checkout. With `node_modules/` ignored, nothing complains.
|
||||
|
||||
## Detect
|
||||
|
||||
```bash
|
||||
git ls-tree HEAD -- node_modules # mode 120000 == tracked symlink
|
||||
git ls-files -s | awk '$1==120000' # every tracked symlink in the index
|
||||
```
|
||||
|
||||
## Fix (branch, never on main directly)
|
||||
|
||||
```bash
|
||||
git rm -q --cached node_modules # untrack; leaves the file on disk
|
||||
```
|
||||
|
||||
plus relax the ignore rule to the slash-free form, with the reason inline so nobody re-adds the slash:
|
||||
|
||||
```
|
||||
# Trailing slash would only match directories: a node_modules symlink (the
|
||||
# shape `git worktree` checkouts end up with) slips past it and gets tracked.
|
||||
node_modules
|
||||
```
|
||||
|
||||
Then `git check-ignore -v node_modules` must print the rule, and the suite runs after (the symlink still resolves, it is just ignored).
|
||||
|
||||
## Prevent
|
||||
|
||||
In a fresh worktree, create the deps symlink **after** confirming the ignore rule has no trailing slash:
|
||||
|
||||
```bash
|
||||
git check-ignore -v node_modules || sed -i 's|^node_modules/$|node_modules|' .gitignore
|
||||
```
|
||||
|
||||
Same class for any ignored-by-convention dir that a worktree might provide as a link: `node_modules`, `vendor`, `.venv`, `target`, `dist` when built artifacts are ignored.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: land-branch-without-checkout
|
||||
description: "Open a PR from a shared worktree when workspace policy blocks git checkout/restore: push a named branch ref from HEAD without switching branches."
|
||||
---
|
||||
|
||||
# Landing a branch when `git checkout` is policy-blocked
|
||||
|
||||
Shared-workspace policy blocks `git checkout` and `git restore` (working-tree
|
||||
switches could strand concurrent agents' uncommitted work). You still must ship
|
||||
the commit on the current worktree branch as a named `chore|feat|fix/<issue>-<slug>`
|
||||
branch. Never work around the block — go around it via refs.
|
||||
|
||||
## Sequence
|
||||
|
||||
```bash
|
||||
# 1. commit only your files (atomic); leave other dirty paths alone
|
||||
git add <your-files> && git commit -m "..."
|
||||
|
||||
# 2. branch ref at HEAD — no working-tree change, stays on wt/<...>
|
||||
git branch <type>/<issue>-<slug> HEAD
|
||||
git push origin <type>/<issue>-<slug> # pre-push hooks still run
|
||||
|
||||
# 3. PR via tracker API with head = pushed ref (see gitea-pr-api-cycle / gh)
|
||||
```
|
||||
|
||||
- The worktree stays checked out on `wt/<timestamp>`; the new branch exists
|
||||
only as the pushed ref (plus a local ref that merges cleanly later).
|
||||
- `-u` upstream tracking on the wt branch is misleading — skip it; set
|
||||
`git config branch.<name>.remote origin` + `.merge` only if you will
|
||||
later check the branch out.
|
||||
- Dirty regenerated fixtures left behind (e.g. `gen:sample` PDFs rewritten
|
||||
by setup/gate runs): `git restore` is also blocked — do NOT edit-forward
|
||||
to revert them; report them. They are regenerated on every bootstrap, so
|
||||
dirt is expected and harmless.
|
||||
|
||||
## Verify
|
||||
|
||||
- `git ls-remote --heads origin <branch>` prints the same sha as `git rev-parse HEAD`.
|
||||
- PR API returns 201 with `head.sha` equal to that sha.
|
||||
|
||||
## Do NOT
|
||||
|
||||
- `git checkout -b`, `git stash`, `git restore`, `git reset --hard` here —
|
||||
all working-tree mutations the policy can block, and stashes in a shared
|
||||
checkout are a cross-agent footgun.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: land-merged-pr-with-blocked-tree-ops
|
||||
description: "Use when a merged PR must land in a checkout whose working tree is dirty and workspace policy blocks git switch/restore/stash/reset-to-commit: prove identity, commit forward, merge, verify."
|
||||
---
|
||||
|
||||
# Landing a merged PR into a checkout with blocked tree operations
|
||||
|
||||
Shared-workspace policy can block `git switch`, `git checkout <branch>`, `git restore`, `git stash`, and `git reset <commit>` ("uncommitted changes other concurrent agents rely on"). A `git pull --ff-only` then aborts with *"Your local changes to the following files would be overwritten by merge"* — **even when the local content is byte-identical to the incoming commit**, because git compares the index, not intent. Observed twice on 2026-10-19 (dank/command-center, dank/dankvoice).
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Prove identity first — never assume it.**
|
||||
`git fetch && git diff --stat origin/main -- <the dirty paths>` → **empty output** means the working tree already equals the merged result, so committing it forward destroys nothing. If it is not empty, stop: there is real local work; do not proceed.
|
||||
2. **Commit forward** the identical paths (message says exactly that, e.g. *"land merged #N content; policy blocks restore/stash, so the identical copies are committed rather than discarded"*). Hooks still run.
|
||||
3. **Merge, don't ff:** `git merge --no-edit origin/main` (ort resolves identical changes on both sides cleanly). `--ff-only` is impossible once step 2 created a commit.
|
||||
4. **Untracked-collision case:** if the abort says *"Please move or remove them before you merge"*, an untracked local file occupies a path the merge adds. `mv` it aside, merge, then `cmp` the restored file against the copy — identical → drop the copy (if the copy came from `.AGENTTRASH`, delete it and `rmdir`, since the file is now tracked).
|
||||
5. **Verify:** repo lint/tests, `git status --porcelain -- <paths>` empty, and `git rev-parse --short HEAD origin/main` equal **after a fresh `git fetch`**.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `git merge … | tail -3` masks the exit status — `tail` always exits 0 and an `&&` chain then continues past an aborted merge. Run the merge unmasked, or check `${PIPESTATUS[0]}`.
|
||||
- Immediately after a push, `git rev-list --count main..origin/main` can report phantom commits-behind from a stale remote-tracking snapshot. Re-`fetch` and compare `rev-parse` before believing any drift.
|
||||
- Do not push redundant commits into a *deployment* checkout without checking the repo's tier law: machine/live tiers commit straight to `main`, human/agent prose goes by PR.
|
||||
|
||||
## Never
|
||||
|
||||
No `--no-verify`, no `reset --hard`, no `checkout -- <path>`, no "temporary" stash: those are exactly the operations the policy blocks because they discard other agents' work.
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
name: llamacpp-adapter-swap
|
||||
description: "Add or roll back a GGUF LoRA adapter on a dank-fleet llama.cpp endpoint (e.g. frank-zealot): provenance census, fork-flag gate, fetcher+env+deployer surface, seam tests, and the behavioural before/after probe that is the only real proof."
|
||||
---
|
||||
|
||||
# Applying a GGUF LoRA adapter to a fleet endpoint
|
||||
|
||||
An adapter is **payload, not shape**: it changes what the endpoint answers and nothing in
|
||||
the KV/budget equation. Fleet law still applies — no benchmarks unless Frank orders one,
|
||||
rollback is one env line, `./tests/run.sh` before commit, deployer health gate + e2e gate.
|
||||
|
||||
## 1. Census the artifact BEFORE writing any code
|
||||
|
||||
```bash
|
||||
curl -sS "https://huggingface.co/api/models/<repo>?blobs=true" # size + LFS oid (None => plain git object)
|
||||
curl -sSL -o /tmp/a.gguf "https://huggingface.co/<repo>/resolve/main/<file>" && sha256sum /tmp/a.gguf
|
||||
```
|
||||
|
||||
Parse the GGUF KV section (15 lines of struct.unpack) and read **provenance**: abliteration
|
||||
adapters carry `prism.abliterate.base_gguf` / `base_sha256`, `adapter.type`, `sites`,
|
||||
`target_layers`, `target_embedding`. Compare the declared base against the file the endpoint
|
||||
actually serves (its sha is in the endpoint's own fetcher script). A mismatch is normal for
|
||||
sibling quantizations — the loader matches **tensor names, not hashes** — but it must be
|
||||
stated in the docs, because it means the card's measured numbers are not this endpoint's
|
||||
numbers.
|
||||
|
||||
## 2. Gate the runtime, from the binary's own `--help`
|
||||
|
||||
On the host: `/home/dank/GitPackages/<build>/llama-server --help | grep -iE 'lora|adapter'`.
|
||||
Confirm `--lora-scaled FNAME:SCALE` exists (and `--lora-init-without-apply` + `POST
|
||||
/lora-adapters` if a runtime toggle is ever wanted). Vendor forks gate adapters by metadata
|
||||
version too — e.g. PrismML builds older than `prism-b10777` refuse packs with `unsupported
|
||||
prism.hadamard.version: 2`. Read the card for the build it verified on and put that in the
|
||||
fetcher header.
|
||||
|
||||
## 3. Surface (five files, copy the mmproj convention)
|
||||
|
||||
- `scripts/<svc>.env`: `LORA_FILE=<path>` + `LORA_SCALE=1`; document that **`LORA_FILE=` (empty)**
|
||||
is the rollback and `LORA_SCALE=0` is the inert-but-present mode. Use `${VAR=default}` (no
|
||||
colon) so an explicitly empty value survives — same reason `MMPROJ_FILE`/`PRESENCE_PENALTY` do.
|
||||
- `scripts/serve-<svc>.sh`: `EXTRA_ARGS+=(--lora-scaled "${LORA_FILE}:${LORA_SCALE}")` — **one
|
||||
argv token**, so a scale can never arrive without its file; refuse a missing file by name
|
||||
(`Restart=always` turns an unfinished fetch into a crash loop that reads like a runtime fault).
|
||||
- `scripts/download-<svc>-<adapter>.sh`: copy the sibling fetcher (`-fsSL --fail-with-body`, byte
|
||||
count, sha256, `--verify-only`, idempotent). No LFS oid => pin the content sha. Header carries
|
||||
the provenance finding.
|
||||
- `scripts/deploy-remote.sh`: `LLM_<SVC>_LORA="${LLM_<SVC>_LORA:-<path>}"` + a check whose error
|
||||
names **this** fetcher, not the payload's.
|
||||
- `scripts/deploy-plan.py` needs nothing: fetchers are host-side state and select no service.
|
||||
|
||||
## 4. Seam tests (argv is the contract)
|
||||
|
||||
Stub-engine harness: create the adapter stub from the **shipped env's basename**, emit
|
||||
`LORA_FILE`/`LORA_SCALE` in the generated test env, then assert on output only —
|
||||
`--lora-scaled` value equals `<path>:<scale>`; a distinctive `LORA_SCALE` (0, 0.75) changes the
|
||||
token; `LORA_FILE=` yields **absence** of the flag (`! grep -qxF -- '--lora-scaled'`); a missing
|
||||
file exits non-zero naming that path; extend the env↔deployer anti-drift pair list.
|
||||
|
||||
## 5. Deploy and prove it behaviourally
|
||||
|
||||
Capture the **before** state on the live endpoint first, then: merge → on host `git pull` +
|
||||
run the fetcher → `./scripts/deploy.sh <host> --service <svc>`.
|
||||
|
||||
```bash
|
||||
sudo systemctl show <svc> -p DropInPaths --value # must be empty (bench-drop-in law)
|
||||
sudo tr '\0' ' ' < /proc/$(sudo systemctl show <svc> -p MainPID --value)/cmdline
|
||||
sudo journalctl -u <svc> --since <boot> | grep -iE 'adapter|lora' # llama_adapter_lora_init_impl prints prism.* keys
|
||||
```
|
||||
|
||||
Refusal probe (the only proof of an abliteration adapter): same prompts, `temperature 0.1`,
|
||||
**`max_tokens >= 2048`** — on thinking models a short budget is consumed by
|
||||
`reasoning_content` and `content` comes back empty with `finish_reason: length`, which reads
|
||||
as a refusal that never happened. Classify with a marker regex and report refusal-vs-complied;
|
||||
do not paste generated harmful content into chat. Also run the standing trio: exact-phrase
|
||||
chat, `tool_calls` with valid JSON, and a >=256 px solid-colour vision probe. A residual
|
||||
`⚠️ Important Disclaimer` heading after the swap is style, not the refusal circuit.
|
||||
|
||||
## Docs
|
||||
|
||||
Endpoint page (`serverAppConfigs-docs/LLM/<svc>.md`): Adapter row, Fetch row, Flags row, one
|
||||
Era row whose rollback is the env line, and a measured before/after paragraph. Laws worth
|
||||
adding: provenance is metadata not a gate; an `token_embd` rank would crash, so an adapter is
|
||||
not inert; the runtime pin may gain a second reason. Never restate the card's benchmark table.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: monolith-split-under-dist-pin
|
||||
description: "Use when cutting a monolithic source file into modules behind a black-box test suite whose committed build artifact (dist) must stay byte-identical to src; one verified commit per cut, red-proof against the previous bundle. Not for feature work or doc-only edits."
|
||||
---
|
||||
|
||||
# Monolith split under a committed-bundle pin
|
||||
|
||||
Procedure that landed a 1354-line `index.ts` as 9 modules + 301 lines of wiring across 9 commits with zero red commits. Assumes: black-box test suite (tests drive the built bundle, not internals) and a pin test asserting the committed `dist/` equals a fresh build of `src/`.
|
||||
|
||||
## Per-cut loop (one commit per cut, always in this order)
|
||||
|
||||
1. `bun run build` (or project equivalent) **before** staging — commit `dist/` in the same commit as the moved `src/`, otherwise the pin test fails ("committed dist is byte-identical to a fresh build") and you must amend. Amending is fine only while the branch is unpushed.
|
||||
2. `tsc --noEmit`, then run the full suite async (`bats tests/ > /tmp/x.log 2>&1; grep -E '^1\.\.'`).
|
||||
3. **Never write to `src/` while the suite runs.** `setup()` rebuilds the bundle from `src/`, so a concurrent edit attributes someone else's half-written state to your run. Read-only prep (planning the next cut's exact line ranges) is the correct thing to do while waiting.
|
||||
4. Gate green → next cut. Gate red → fix, rebuild, amend, re-gate.
|
||||
|
||||
## Cutting mechanics
|
||||
|
||||
- Cut leaf/pure modules first (grammar, transcript rules, URL resolution, pure segmenter), then stateful ones (config, IO, pipeline, UI). Never let a module import `index` — if the new module needs something from `index`, that thing is a leaf and must move too (this is what forced `segment.ts` out early).
|
||||
- Extract in **verbatim** bodies: read the exact ranges, write them into the new file, then delete the same ranges. Only module docs, export keywords, and injected dependencies change. Keeps each commit reviewable as a move.
|
||||
- Edit tools that require "displayed lines" from the latest read snapshot: re-read the exact ranges under the current tag, then issue one edit call with all ops listed **bottom-up** (original line numbers throughout). Adjacent ranges merge into one `CUT a=b`; leaving a doubled blank line is the usual scar — one extra `CUT` next turn.
|
||||
- Mutable config must reach a long-lived object as **getters**, not a construction-time snapshot: menu handlers mutate `cfg` after construction, and a snapshot freezes the settings the UI exists to change.
|
||||
- After moving code, grep the old file for the moved names: stale call sites compile under bundlers that skip typechecking (`stop()` vs `speaker.stop()` was caught only by `tsc`, not by `bun build`).
|
||||
|
||||
## Proving a new assertion is red-capable (before trusting it green)
|
||||
|
||||
Build the old bundle and point the test harness at it — no stash, no worktree juggling:
|
||||
|
||||
```bash
|
||||
git show HEAD:dist/index.js > /tmp/old-dist.js # bundle is self-contained
|
||||
# rebuild the throwaway harness by extracting the driver the suite writes
|
||||
sed -n "<heredoc-body-range>p" tests/behavior.bats > "$W/driver.ts"
|
||||
DV_EXT=/tmp/old-dist.js <same env as the test> bun "$W/driver.ts" <scenario> # expect RED
|
||||
DV_EXT="$PWD/dist/index.js" <same env> bun "$W/driver.ts" <scenario> # expect GREEN
|
||||
```
|
||||
|
||||
Copy the project's real player/server stub verbatim — a simplified stub (e.g. one that skips writing the log file the driver counts) prints a false `PLAYS 0` and looks like a failing fix. Pin the scratch dir to a fixed path; `ls -dt /tmp/tmp.*` picks up another session's directory.
|
||||
|
||||
## Black-box test hygiene learned the hard way
|
||||
|
||||
- A test that counts artifacts in the **shared** temp directory observes every other process on the box (live agent sessions synthesize into the same `dankvoice-*` dirs). Give the run a private `TMPDIR="$WORK/scratch-tmp"` and assert there; the assertion's sensitivity is proven by the fact it caught a real leftover while scoped globally.
|
||||
- Let preseeds reference the ephemeral fixture port with a placeholder (`__PORT__`) substituted by the driver, so a test can preseed a URL pointing at its own fixture server.
|
||||
- Refactor-only commits get **no new test**: re-asserting unchanged labels/order after a commit whose point is that they are unchanged is a tautology; the existing suite is the gate. New tests belong to behavior changes, red-first.
|
||||
- Tests must observe **output**, never internals: assert `PLAYS`/`CALLS`/`CALL <url>`/`NOTIFY <text>`, not that a variable was set.
|
||||
|
||||
## Docs after a split (same branch, own commit)
|
||||
|
||||
Move each explanation to the file that now owns it (knob list → the module that reads env; state-machine prose → the module that implements it) and leave the entry point a one-line-per-module map. Then grep for pointers into the old file — a README saying "see header of `src/index.ts`" becomes a lie the moment the knobs move.
|
||||
@@ -22,7 +22,7 @@ description: "Locate, edit, validate, and commit niri compositor config on Frank
|
||||
- Preset list order = cycle order for `Mod+R` (`switch-preset-column-width`); `Mod+Shift+R` cycles backwards. WITH custom preset lists the cycle contains ONLY the configured presets — no implicit full/maximize tail (verified: [0.25, 0.5] cycles period 2). Maximize = separate `Mod+F` `maximize-column`. Built-in defaults without the node: 1/3, 1/2, 2/3.
|
||||
- WITHOUT `preset-window-heights`, `switch-preset-window-height` FALLS BACK to the width presets — user sees width-fractions as heights; define the node explicitly to stop it.
|
||||
- Ground truth for node placement: the binary's embedded template — `grep -a -B4 -A8 "preset-column-widths {" $(which niri)`.
|
||||
- Window outlines: base geometry (`width`, `off`) lives in `cfg/layout.kdl` `focus-ring`/`border`; COLORS effectively live in `noctalia.kdl` (see include-order law). `border { off }` + focus-ring active = the visible outline.
|
||||
- Window outlines: base geometry (`width`, `off`) lives in `cfg/layout.kdl` `focus-ring`/`border`; COLORS effectively live in `noctalia.kdl` (see include-order law). `border { off }` + focus-ring active = the visible outline. Frank's look since 2026-10-18: amber-orange `#ff8c00`, width 4 (`active`+`urgent` in noctalia.kdl; tab-indicator/insert-hint/recent-windows stay red `#f43841` by choice).
|
||||
|
||||
## Workflow
|
||||
1. Edit the live file under `~/.config/niri/cfg/` (symlinked = repo file; except `noctalia.kdl`, edit live then `cp` to mirror).
|
||||
|
||||
@@ -1,50 +1,52 @@
|
||||
---
|
||||
name: noctalia-bar-layout-edit
|
||||
description: "Edit/verify Noctalia v5 bar layout and clock widgets (multi-timezone instances, capsule groups); GUI→repo propagation chain, settings.toml symlink audit, and clobber-recovery via dangling stashes on Frank's dotfiles workstations"
|
||||
description: "Edit/verify Noctalia v5 bar layout and clock widgets (multi-timezone instances, capsule groups); GUI-repo propagation chain, settings.toml symlink audit, and stash-clobber recovery on Frank's dotfiles workstations"
|
||||
---
|
||||
|
||||
# Noctalia v5 bar layout edit
|
||||
|
||||
Bar widget layout on Frank's noctalia v5 workstations is GUI-override state, not config.toml.
|
||||
|
||||
## Where
|
||||
- Layout file: `~/.local/state/noctalia/settings.toml` → `[bar.<name>]` keys `start` / `center` / `end` (arrays of widget ids; `[widget.<id>]` subtables hold per-widget settings).
|
||||
- On Frank's boxes this file MUST be a **symlink into the command-center repo** (e.g. `~/.omp/dotfiles/noctalia/.local/state/noctalia/settings.toml`) — edit the repo file, not the link path (atomic-rename writes through the link path would replace the symlink).
|
||||
- Layout file: `~/.local/state/noctalia/settings.toml` -> `[bar.<name>]` keys `start` / `center` / `end` (arrays of widget ids; `[widget.<id>]` subtables hold per-widget settings).
|
||||
- On Frank's boxes this file MUST be a **symlink into the command-center repo** (e.g. `~/.omp/dotfiles/noctalia/.local/state/noctalia/settings.toml`) - edit the repo file, not the link path (atomic-rename writes through the link path would replace the symlink).
|
||||
- Layer order: built-in defaults < `~/.config/noctalia/config.toml` (+`[include]`) < `settings.toml` (wins). Noctalia writes through symlinks and keeps them.
|
||||
- Defaults when a key is absent: `start = ["launcher","wallpaper","workspaces"]`, `center = ["clock"]`.
|
||||
- **Noctalia rewrites this file itself** (GUI actions + normalizes keys/format, e.g. strips `{:%...}` braces, reorders widget lists). Always re-read the exact lines immediately before editing; stale line anchors WILL drift between reads.
|
||||
- Defaults when a key is absent: `start = ["launcher","wallpaper","workspaces"]`, `center = ["clock"]`, `end = ["notifications","clipboard","network","bluetooth","tray","battery","session","control-center","clock"]`.
|
||||
|
||||
## GUI→repo propagation (checkout-is-deployment)
|
||||
- `~/.config/noctalia/config.toml` is symlinked into the repo → GUI settings writes land in the repo working tree automatically.
|
||||
- `~/.local/state/noctalia/settings.toml` can SILENTLY become a plain file (observed 2026-10-17; hardlink/symlink wiring decays) → GUI bar-layout changes then diverge from the repo with no signal. AUDIT: `readlink ~/.local/state/noctalia/settings.toml` must print the repo path; if it's a plain file and repo copy is content-identical, re-arm: `ln -sfn <repo-path> <live-path>` + `noctalia config validate`.
|
||||
- Landing in the working tree ≠ propagated: other boxes only see it after commit + push to Gitea main. `live-sync.path` (PR #15; DirectoryNotEmpty on `dotfiles/`) commits dotfiles-tier writes within ~2 s once deployed — until it is active on a box, the hourly heartbeat leaves up to 59 min of exposure, and on 2026-10-18 exactly that window let a trunk-sync stash destroy months of GUI layout work. Until verified active (`systemctl --user status live-sync.path`), commit GUI layout edits the same session yourself. Noctalia's runtime churn (wallpaper path, nightlight, panel positions) rides along; normal.
|
||||
- niri has NO settings GUI; niri "GUI changes" don't exist, and `niri msg output ...` runtime overrides are not persisted. niri cfg mirror↔live links also decay (all inodes decoupled once) → audit by CONTENT: `diff` every `~/.config/niri/cfg/*.kdl` against its `dotfiles/zz-blankpad/niri/...` mirror, never trust links.
|
||||
## Clobber window and recovery (2026-10-18 incident law)
|
||||
- GUI-built layout is **working-tree-only until the hourly live-sync heartbeat commits it**. A merge/sync session that "clears the dirty tree" can `git stash` those edits, and later stash operations can destroy the entries - customizations then vanish from the live bar with the repo copy stale.
|
||||
- Never blame the GUI or noctalia first: diff live file vs `HEAD`, check heartbeat commits (`git log -- dotfiles/noctalia/.local/state/noctalia/settings.toml`), then the stash chain.
|
||||
- Destroyed stashes are recoverable: `git fsck --lost-found` lists dangling commits; `git show <dangling>:dotfiles/noctalia/.local/state/noctalia/settings.toml` dumps each candidate; pick by `[bar.default]` richness (capsule groups, extra clock instances). Validate a candidate in a sandboxed `XDG_*_HOME` before touching the live file.
|
||||
- After recovery, commit the file immediately so the window closes.
|
||||
|
||||
## If the bar got wiped (recovery)
|
||||
- The working-tree file is often the ONLY copy of GUI work; a `git checkout`/stash/merge clobber is survivable because git objects outlive stash entries: `git fsck --lost-found`, inspect `git log -1 --format='%ci %s' <dangling>` (newest `WIP on` stash commits hold the last good settings.toml), dump with `git show <c>:dotfiles/noctalia/.local/state/noctalia/settings.toml`.
|
||||
- Before restoring: back up the clobbered file to `.AGENTTRASH/`; validate the candidate in a sandbox (`XDG_CONFIG_HOME=… XDG_STATE_HOME=… noctalia config validate` — plugin-widget "unrecognized type" warnings there are offline-CLI blindness, not blockers); restore via `cp` onto the repo path (cp writes through the symlink; `mv` would replace it); prove visually with `noctalia msg screenshot-fullscreen`; commit immediately.
|
||||
- Cross-check the recovered layout against Frank's old screenshots in `~/Pictures/screenshot_*.png` (crop top ~7% with magick) — they are ground truth for what the bar should look like.
|
||||
## Widgets (v5 census, verified 2026-10-17)
|
||||
- Full roster in `settings.toml` widget_order plus docs: activity, backlinks, battery, bluetooth, brief, bright, cat, cheatsheet, chart, clipby, clipboard, control-center, cpu, controls, cava (now playing), disk, gpu, help, keyboard, language, launcher, mediaplayer, memory, mic, network, night-light, noctalia-palette, notification-history, notifications, niri-colors, power-section, power-profile, screenrecord, scratchpad, screenshot, session, shell (custom shell command), slowouts, slowouts-visible (per-output bar assignment), speedtest, swayosd-*, taskbar, tray, voiceline, volume, wallpaper, weather, workspaces.
|
||||
- `clock` widget: `format` (chrono strftime), `schema` (12/24), `multi_line`, `show_seconds`, `popup`, and **timezone** = IANA name. Multiple clock widget instances are supported: ids like `clock`, `clock-utc`, `clock-sea` each get a `[widget.<id>]` subtable -> one capsule per instance. This is how multi-timezone (Seattle work + UTC dev) lands: add `clock` + `clock-utc` to `end` (or `center`), set `timezone = "Etc/UTC"` on the utc one.
|
||||
- `cat` widget: shows random ASCII art, `rows` setting (4-10).
|
||||
- `shell` widget: `command`, `run interval` (seconds), `icon`, `terminal`; output lands in its capsule popup - can wrap any CLI (hyprstatus-style).
|
||||
- `taskbar` widget: shows window buttons **only on outputs where `niri-bridged` is enabled** (noctalia-launcher enables it). Buttons render **icon-only** (app icon from icon theme; fallback = first letter of title); the window title string is NOT shown in the bar. Hover tooltip carries title + app_name. Per-widget settings: `show_icons`, `show_actions`, `disable_middle_click`, `preferred_app_names` (map app_name -> icon for missing icons).
|
||||
- Widget ids usable in `[bar.*].start|center|end` are restricted to "bar widgets" (`is_bar_widget` allowlist in `bar.rs`); non-bar widget ids are silently dropped at load. `group:<id>` entries render capsule groups.
|
||||
- Capsule groups: `[bar.<name>.capsule_groups]` id = `["widget",...]`.
|
||||
- Niri layout law for bars: window width must stay **odd** (strut = (W-1)/2); bars reserve via layer-shell, niri's reserved area = layer surface size, so bar height changes need no niri edit.
|
||||
|
||||
## Apply/verify loop (no restart, no reload IPC)
|
||||
1. Edit the repo `settings.toml`.
|
||||
2. `noctalia config validate` (checks merged layers; also `noctalia config export [full] | yq -p toml` to inspect effective values).
|
||||
3. Both layers are file-watched → hot-reload. Prove visually: `noctalia msg screenshot-fullscreen` saves a PNG to `~/Pictures`; read the image and inspect the bar.
|
||||
4. **Screenshot race:** the `ok` return precedes the PNG write. Never chain `noctalia msg screenshot-fullscreen && magick "$(ls -t ~/Pictures/screenshot_*.png | head -1)" ...` in one command — you crop the PREVIOUS capture. List `ls -t` in a separate later call and crop the named file.
|
||||
5. Crop for reading: `magick <png> -crop 100%x8%+0+0 +repage /tmp/strip.png`, then read the strip; cross-check clocks against `date -u` / `date`.
|
||||
## Bar styles (GUI-visible, settings.toml keys)
|
||||
- `[bar.*]`: `thickness`, `font_family`, `font_weight`, `font_size`, `opacity`, `border_width`, `border_offset`, `rounded`, `paddings`, `icon_ratio`, `follow_layout`, `output`, `exclusive`, `layer` (`bottom` = below window layer; `overlay` = above windows), `position` (`top`/`bottom`).
|
||||
- Per-monitor bars: add a `[bar.<name>]` with `output = "<monitor>"` and `follow_layout = true` (niri) -> per-workspace strut follows tall windows on that monitor. This is the mechanism for "bar shrinks when a window is tall" (the bar is a layer-shell strut, NOT a niri layer rule).
|
||||
- The GUI "Bar" page is the reference list; anything absent from settings.toml is a built-in default.
|
||||
|
||||
## Clock widgets
|
||||
- `[widget.clock]` settings: `format`, `vertical_format`, `tooltip_format`, `timezone` (IANA id, e.g. `"UTC"`, `""` = system local). Tokens are strftime-compatible (`%H` 24h, `%I:%M %p` 12h, `%F` ISO date); `{:%...}` brace fields optional; `\n` = second line. Docs: docs.noctalia.dev → bar/widgets/clock + configuration/date-format-tokens.
|
||||
- **One timezone per widget; multiple instances allowed:** any `[widget.<id>]` needs `type = "clock"` for custom ids (bare `clock` id infers it). Each instance converts independently — e.g. local 12h + `timezone = "UTC"` 24h side by side.
|
||||
- Placement: add ids to `[bar.<name>]` start/center/end, or to a capsule group: `[[bar.default.capsule_group]]` with `id = "g1"` + `members = [...]`, referenced from `start` as `"group:g1"` (member order = left-to-right inside the capsule).
|
||||
- Local date + local 12h fit in ONE widget (same tz); the UTC 24h clock must be a second widget.
|
||||
- Expect the date widget (local) and UTC clock to disagree on the day after ~5 PM Seattle — that's correct UTC, not a bug; offer `%F` inside the UTC widget if it irks.
|
||||
## Slowouts (window-height-aware bars/panels)
|
||||
- `slowouts` = hidden engine (not a bar capsule): `[slowouts]` keys `enabled`, `duration`, `easing`, `trigger_offset`; per-monitor `[slowouts.<monitor-name>]` overrides (e.g. `DP-1`) with the same keys plus `edge`.
|
||||
- `position = "top"` + `exclusive = false` + slowouts on the monitor -> panel hides when a window touches its edge, re-reveals otherwise. `slowouts-visible` = toggle capsule.
|
||||
- **Slowouts needs niri workspace-autoresize ON** (niri `workspace-autoresize` = "per-workspace" in cfg/layout.kdl `output ... { workspace-autoresize per-workspace }`): noctalia computes hide from workspace height < panel bottom edge; uniform-strut layouts never trigger it.
|
||||
- Per-output bars: slowouts config keys are monitor names, so one `[bar.<x>]` with `output` + `exclusive=false` + `layer=overlay` per slowout monitor is the supported shape.
|
||||
|
||||
## Taskbar recipes
|
||||
- Move window indicator beside workspaces: put `"taskbar"` in `start` after `"workspaces"`.
|
||||
- Combine workspaces+windows into ONE widget: `[widget.taskbar] group_by_workspace = true` — per-workspace capsules with disc labels (focused/occupied/empty/urgent colors); `workspace_group_content = "icons"|"count"|"dots"` (count/dots: left-click activates the workspace); `hide_empty_workspaces`, `show_workspace_label`, `minimal`, `workspace_label_placement = corner|centered|inside`. Requires compositor per-window workspace assignment (niri: yes; LabWC: no → stays flat).
|
||||
## Layout modes and window title decorations
|
||||
- `[layout]`: `mode` (`tiling`|`floating`|`monocle`), `pseudotiled`, `focus_bar` (window-title bar at top, center, layer-shell; style via `[ui.controls...]`/bar-ish settings), `window_follows_cursor`, `gaps`, `border_width`, `rounded`, `blur`, `opacity`, `borderless_csd`, `show_native_decoration`.
|
||||
- `focus_bar` is the noctalia analog of niri's drawn window title, but ONE global bar (no per-window arrows). True per-window title tags = niri `draw-with-border` on window rules; noctalia adds no per-window decoration (it's a bar app + portal, composites nothing).
|
||||
- Floating mode: right-click desktop / `SUPER+Shift+F` / toggle widget; widgets `floating-directories`, `floating-files`, `pins` (sticky notes), `canvas`, `sketchpad`, `post-its`. Float windows snap to screen quarters (`SUPER+Ctrl+Alt+Arrows`) and tile into current layout (`SUPER+Shift+T`). `[layout.floating.files]` = icon/view/sort/preview size.
|
||||
- `[bar.*].window_display` = `disabled`|`icons`|`text`|`combined` - per-window chips in the bar strip (niri only; this is what niri-taskbar could not do). Values: `disabled`|`icons`|`centered`|`inside`. Requires compositor per-window workspace assignment (niri: yes; LabWC: no -> stays flat).
|
||||
- The `workspaces` widget itself has no window display; styles are regular/minimal/focus_hint only.
|
||||
|
||||
## Law
|
||||
- Full widget/setting tables: https://docs.noctalia.dev/noctalia/bar/ and .../bar/widgets/<name>/ . Commit bar changes atomically to the command-center repo (checkout is the deployment).
|
||||
- **Ask before applying.** Frank asks "how do I X" questions about the live bar; answer with the exact snippet + file, apply ONLY on explicit go. A settings.toml edit is a live-deployment change, not a demo. (Exception: restoring Frank's own layout after data loss, on his ask.)
|
||||
- Git `M` on settings.toml usually carries noctalia's own runtime churn (wallpaper path, nightlight, panel positions) alongside your edit — mention it when committing.
|
||||
- **Ask before applying.** Frank asks "how do I X" questions about the live bar; answer with the exact snippet + file, apply ONLY on explicit go. A settings.toml edit is a live-deployment change, not a demo.
|
||||
- Git `M` on settings.toml usually carries noctalia's own runtime churn (wallpaper path, nightlight, panel positions) alongside your edit - mention it when committing.
|
||||
- Right after building layout in the GUI, commit settings.toml (or trigger `scripts/live-sync.sh`) before any branch/merge work: the uncommitted-GUI-edit window is exactly where the 2026-10-18 clobber happened.
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: omp-bundle-implementation-port
|
||||
description: "Extract and port a working implementation from omp's minified dist/cli.js bundle (copy-not-invent requests); anchors, python slicing, regex-constant sweep, trace-driven test prediction."
|
||||
---
|
||||
|
||||
# Porting built-in omp behavior out of the minified bundle
|
||||
|
||||
Use when Frank says "copy the omp implementation / work off known-working code" — i.e. port a built-in feature (vocalizer segmentation, retry logic, formatting rules) into our own file instead of hand-rolling a heuristic. For merely *auditing behavior claims*, use `omp-bundle-claim-audit`; this skill is extraction-for-porting.
|
||||
|
||||
Bundle: `~/.bun/install/global/node_modules/@oh-my-pi/pi-coding-agent/dist/cli.js` (minified; single-letter names, private `#x` fields survive).
|
||||
|
||||
## Extraction recipe
|
||||
|
||||
1. **Find the setting/entry strings first.** `grep -o 'ce({id:"<area>[^"]*"[^}]\{0,200\}' cli.js` — setting descriptions state semantics verbatim (e.g. `speech.enhanced` = "Rewrite assistant output into natural spoken prose with the tiny/smol model"). Descriptions are quotable ground truth.
|
||||
2. **Anchor to code with a distinctive token**, not guesses: a method name (`pushDelta`), a state init, or a string. Then slice generously with python (bash heredoc), NOT grep -o which truncates:
|
||||
```python
|
||||
src = open('cli.js').read()
|
||||
i = src.index('pushDelta'); print(src[i-1500:i+3000])
|
||||
```
|
||||
3. **Follow the constructor**: pipeline classes instantiate collaborators there (`this.#t ??= new fHe` → `fHe` is the real worker; `#g()` returned `{speakable:new fHe, pending:[]…}`). Dump `class fHe{` +4000 chars.
|
||||
4. **Regex/helper sweep**: minified code declares shared constants in a trailing `var a,b,c,…` list and initializes them elsewhere (`Xyr=`, `Pwc=function`). Loop the names with `re.search(rf'{name}=', src)` and print ~400 chars each — the actual boundary/cleanup rules live in these one-liners.
|
||||
5. Write slices to `/tmp/*.txt` and `read` them; don't eyeball megabyte greps.
|
||||
|
||||
## Port rules
|
||||
|
||||
- Rename fields/methods to readable names but keep logic **1:1** — the value is behavioral fidelity (Frank's "known-working set of code").
|
||||
- Private `#x` fields port cleanly: TS classes in Bun extensions support `#`.
|
||||
- Global-regex `.lastIndex=0` before each scan loop — minified code relies on it; keep it.
|
||||
|
||||
## Verification shortcut
|
||||
|
||||
Before running tests, **hand-trace** each test input through the ported rules (boundary indices, state transitions) and predict exact segment counts/strings; then run — predicted-vs-actual mismatch localizes port errors faster than test output alone. Worked example (2026-10-19, dankvoice PR #26): traced `"First stop here. Second one goes!!! Really? Third ... done"` → predicted 3 segments incl. `"Third ... done"` merged via flush (10-char boundary < 24-char started-min) → bats green first run.
|
||||
|
||||
## Known anchors (2026-10-19 bundle)
|
||||
|
||||
- Speak-aloud vocalizer pipeline: class `I6o` (gating on `speech.enabled`, suspend counter, enhancer, `#g()` per-message state).
|
||||
- Utterance segmenter: class `fHe` + helpers `Pwc` (line-head classify), `Iwc`/`_wc`/`b6o`/`w6o` (boundary search), `Cwc`+`y6o` (cleanup: links→label, URL→host, path→basename). Regex consts: `Xyr` sentence-end runs, `pHe` weak breaks, `mwc` abbreviations, `hwc` undecided line-heads, `g6o` horizontal rules.
|
||||
- Port already living in `agent/extensions/dankvoice.ts` (`SpeechSegmenter`) — when the bundle updates, re-extract `fHe` and diff against it before assuming our copy still matches.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
name: omp-extension-event-probe
|
||||
description: "Prove which lifecycle events the omp harness actually dispatches to extensions, with a throwaway timestamping probe plugin, instead of inferring delivery from side effects."
|
||||
---
|
||||
|
||||
# Prove harness→extension event delivery with a probe plugin
|
||||
|
||||
Use when an extension handler seems never to fire, or when you must know *when* an
|
||||
event arrives relative to another (mid-run vs settle). Never infer dispatch from
|
||||
the feature's side effects: async work in the handler (LLM rewrites, synthesis,
|
||||
network) lags seconds and destroys the timing signal.
|
||||
|
||||
## Shape (no build step needed)
|
||||
|
||||
`/tmp/probe/package.json` — the installer validates the declared entry on disk:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "event-probe",
|
||||
"version": "0.0.0",
|
||||
"type": "module",
|
||||
"exports": "./index.js",
|
||||
"pi": { "extensions": ["./index.js"] }
|
||||
}
|
||||
```
|
||||
|
||||
`/tmp/probe/index.js` — append arrival times, not counts:
|
||||
|
||||
```js
|
||||
import { appendFileSync } from "node:fs";
|
||||
const LOG = "/tmp/probe.log";
|
||||
const stamp = (line) => appendFileSync(LOG, `${new Date().toISOString().slice(11, 23)} ${line}\n`);
|
||||
export default function probe(pi) {
|
||||
pi.on("agent_start", () => stamp("agent_start"));
|
||||
pi.on("turn_end", (e) => stamp(`turn_end turn=${e.turnIndex} toolResults=${(e.toolResults ?? []).length}`));
|
||||
pi.on("agent_end", (e) => stamp(`agent_end messages=${(e.messages ?? []).length} willContinue=${String(e.willContinue)}`));
|
||||
}
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
1. `cd ~/.omp/plugins && omp plugin link /tmp/probe` (link, never edit
|
||||
`node_modules`), then `omp plugin doctor` → expect a `✔ plugin:event-probe` line.
|
||||
2. Launch a **fresh** TUI session in a pty (`omp --cwd /tmp/probe-run`, `pty: true`).
|
||||
Extensions load at startup only — an already-running session never sees the probe.
|
||||
3. PTY gotcha: input sent before the splash clears is swallowed. Read the screen
|
||||
first, then submit text and finish with a separate write containing a bare
|
||||
`\r` (LF alone left the text sitting in the composer).
|
||||
4. Drive a task with ≥2 tool rounds so turns and settles separate in time
|
||||
(sequential reads work and need no approval; ask for steps "strictly one
|
||||
after the other" or the model parallelises them into one turn).
|
||||
5. Wait for a terminal line with an async bounded loop (`grep -q agent_end`,
|
||||
`sleep 1` inside one backgrounded bash call) — never poll across turns.
|
||||
|
||||
## Read
|
||||
|
||||
`turn_end` timestamps strictly earlier than `agent_end` ⇒ per-turn delivery
|
||||
works. `willContinue=true` on `agent_end` marks a non-terminal settle (harness
|
||||
maps `isTerminal === false` to it) — handlers that `return` on it never speak.
|
||||
|
||||
## Teardown (mandatory)
|
||||
|
||||
`omp plugin uninstall event-probe`; `omp plugin doctor`; delete `/tmp/probe*`.
|
||||
The link writes into `~/.omp/plugins/package.json` (tracked, live tier) —
|
||||
uninstall restores it; confirm with `grep` for the probe name returning nothing.
|
||||
|
||||
## Ground truth read for free
|
||||
|
||||
Event union + payloads: `~/.omp/plugins/node_modules/@oh-my-pi/pi-coding-agent/dist/types/extensibility/shared-events.d.ts`;
|
||||
the session→extension mapping lives in `src/extensibility/extensions/lifecycle-mirror.ts`
|
||||
(`extensionEventFromSessionEvent`). Extension-level `turn_end` carries
|
||||
`turnIndex`/`message`/`toolResults` but **no `willContinue`** — that field exists
|
||||
only on `agent_end`, so "is this the last turn" cannot be decided from `turn_end`.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: omp-extension-live-tui-verify
|
||||
description: "Verify an omp extension's slash commands and event hooks against a real TUI session: pty submit semantics, observable side-effect instrument, transient-toast trap, agent_end willContinue law, PipeWire output triage."
|
||||
---
|
||||
|
||||
Verify that a `.ts` file in `agent/extensions/` actually works inside the interactive TUI — the black-box bats seam cannot see harness-side wiring (event names, arg passing, mode gating), and it will pass beside a dead hook.
|
||||
|
||||
Distinct from `omp-tui-live-visual-verify` (config/theme rendering proven by screenshot): here the instrument is **observable side effects**, not pixels, because extension notifications are transient toasts that vanish before the pty buffer is read.
|
||||
|
||||
## 1. Launch the exact build under test
|
||||
|
||||
```bash
|
||||
omp --no-extensions --no-session -e /abs/path/agent/extensions/<name>.ts
|
||||
```
|
||||
as a **pty service** (`pty: true`, `ready: {"log": "╭|❯|Tip:|─", "timeout": 60}`).
|
||||
`--no-extensions` + explicit `-e` proves the file loads by path and nothing else interferes. Neutral cwd (`/tmp`) so repo state cannot color the run.
|
||||
|
||||
## 2. Submit input correctly (the costly gotcha)
|
||||
|
||||
`write proc://<id>` with a trailing `\n` does **not** submit — the text just sits in the composer.
|
||||
|
||||
- **Works:** two writes — first `"/cmd args"` (no terminator), then a **separate** write whose whole content is `\r`.
|
||||
- **Broken:** `"/cmd args\r"` in one write. It still submits, but the trailing `\r` arrives inside the command args. Symptom: `unknown voice 'narrator1\r'`, an unexpected extra utterance, or a chat message whose transcript line ends in a literal `\r`. That is a harness artifact — never "fix" it by stripping control characters in the extension (`.trim()` already covers what a keyboard can produce).
|
||||
|
||||
## 3. Instrument with a stub, never with ears or eyes
|
||||
|
||||
Point the extension's player/handler at a script that appends evidence:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
echo "$1" >> /tmp/stub.log; cp "$1" "/tmp/out-$$.wav"; exit 0
|
||||
```
|
||||
Run the TUI with that stub in the env the extension reads (`DANKVOICE_PLAYER=…` style). Then every assertion is a file: `wc -l` of the log (how many times it fired), `ffprobe -show_entries format=duration` (what it produced), `ls -d <tmp-glob>` (that cleanup ran).
|
||||
|
||||
Sequence per behavior: command → CR → short `sleep` → count. **Count before and after** each command; the delta is the only honest proof a subcommand stayed silent.
|
||||
|
||||
## 4. Discriminate "fired twice" from "fired once per turn"
|
||||
|
||||
Stray CR writes can create extra empty turns, each with its own `agent_end` — two utterances for one intended prompt is usually **two turns**, not a duplicate speak. Confirm by counting assistant turns in the transcript (`read proc://<id>`), and by matching each clip's duration to its own answer's length.
|
||||
|
||||
## 5. Event-hook contract traps
|
||||
|
||||
- `agent_end` payload (`dist/types/extensibility/shared-events.d.ts`): `{ type, messages: AgentMessage[], willContinue? }`. **`willContinue: true` means the harness already scheduled a continuation** (auto-retry / empty-stop retry) — a subscriber that narrates or notifies there speaks an answer the model is about to redo. Gate on it.
|
||||
- Read the real event/payload shape from `~/.bun/install/global/node_modules/@oh-my-pi/pi-coding-agent/dist/types/…`; grep `dist/cli.js` for the literal event names to confirm they exist.
|
||||
- A driver test that *calls* the handler by hand proves nothing about emission — that is exactly the gap this live check closes.
|
||||
|
||||
## 6. Audio output goes silent while every command reports success
|
||||
|
||||
`pw-play` exits 0 and prints `stream drained` even when nobody hears anything. Triage order:
|
||||
|
||||
```bash
|
||||
wpctl status # Sinks, the `*` default, and live Streams
|
||||
wpctl get-volume @DEFAULT_AUDIO_SINK@
|
||||
```
|
||||
- A **`Default Configured Devices` entry naming a `bluez_output.*` node that is not in the Sinks list** = remembered-but-offline Bluetooth: the classic silent-route cause.
|
||||
- Confirm the stream really links: a short clip can vanish before inspection, so use a multi-second clip and read `wpctl status` mid-play, or target explicitly: `pw-play --target <node-id> <file>` (node ids from `wpctl status`; `wpctl inspect <id>` gives `node.name`).
|
||||
- Prove the synth path independently of the player: `curl -o x.wav …` then `file x.wav` (expect `RIFF … WAVE … PCM`).
|
||||
|
||||
## 7. Land it
|
||||
|
||||
Live result belongs in the commit message, the reference page, and the work log — and the guard the live check uncovered gets its own behavior test (assert observable output only: fired count, payload content, silence).
|
||||
|
||||
## Cleanup
|
||||
|
||||
`/exit` or `proc://<id>/kill` the service; move stub scripts, logs, and clips to `~/.AGENTTRASH/<date>-<topic>/` — never `rm`. Kill background jobs that keep playing to the operator's speakers.
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: omp-extension-stale-module-audit
|
||||
description: "Diagnose recurring omp extension handler timeouts/silence when the on-disk fix is already deployed: harness has no extension hot-reload, so running sessions freeze the module at session start (zombie modules)."
|
||||
---
|
||||
|
||||
Use when a fix for an omp extension is merged/deployed yet the SAME errors persist ("handler timed out after 30000ms", silence, wrong behavior) in running sessions.
|
||||
|
||||
**Core fact:** the harness loads extension `.ts` modules at session start and never reloads them (no `/reload` for extensions; bundle `reload` hits are LSP/MCP/browser only). A session older than the fix keeps executing stale code regardless of the file on disk.
|
||||
|
||||
**Procedure:**
|
||||
1. Census the errors from logs, not guesses — timeout lines carry the culprit pids:
|
||||
`grep -h "handler timed out" ~/.omp/logs/omp.<today>*.log | python3 -c 'import sys,json; rows=[json.loads(l) for l in sys.stdin]; rows.sort(key=lambda r:r["timestamp"]); [print(r["timestamp"], r["event"], r["pid"]) for r in rows]'`
|
||||
(`~/.omp/logs/omp.<date>.<pid>.log` — the pid in the filename is the session.)
|
||||
2. Session start times vs deployed file versions:
|
||||
`ps -eo pid,lstart,args | grep '[o]mp'`
|
||||
`git log --format='%h %ad %s' --date=format:'%m-%d %H:%M' -- <extension path>`
|
||||
`git reflog --format='%gd %ad %gs' --date=format:'%m-%d %H:%M' | grep merge` (live-sync pull times = when the file content changed).
|
||||
Any session whose start time precedes the fix's arrival has the stale module.
|
||||
3. Confirm the stale revision actually contains the bad pattern — read what that session loaded, not HEAD:
|
||||
`git show <old-rev>:<path> | grep -n 'await speak\|async (event'`
|
||||
(e.g. an `async agent_end` awaiting a >30 s chain is exactly what the harness's per-handler `Kzc` 30 s race logs; fire-and-forget `void` is fine — detached promises are NOT tracked by the dispatch window.)
|
||||
4. Optional empirical proof when static analysis says "impossible": probe extensions through the real harness:
|
||||
`omp -p --no-extensions -e /tmp/probe.ts "Reply OK"` with a probe that logs handler start/return and `void`s a 40 s detached promise + spawns a long child. Fast exit + no timeout log ⇒ detachment is untracked ⇒ blame stale modules, not the harness.
|
||||
5. Remediation: restart the affected sessions (kill the listed pids or their terminals — the operator session itself may be one of the zombies; say so explicitly). There is no other fix.
|
||||
|
||||
**Gotchas:**
|
||||
- Per-handler attribution in the log is accurate per dispatch; don't blame other extensions before checking the pid's start time.
|
||||
- Endpoints healthy + silence persisting usually means TWO stacked bugs (e.g. stale awaiting module in old sessions AND a mute bug in current code) — diagnose each pid cohort separately.
|
||||
- Extension chain code must use self-owned `AbortSignal.timeout(...)`, never a ctx-scoped signal: harness dispatch signals die when the dispatch window closes.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: omp-lifecycle-event-selection
|
||||
description: "Pick the right omp harness lifecycle event for narration/telemetry: agent_end is a settle, turn_end waits for tool results, message_end is earliest with content, after_provider_response carries no content."
|
||||
---
|
||||
|
||||
# Choosing an omp lifecycle event (narration, telemetry, hooks)
|
||||
|
||||
Re-derived twice from the installed typings/sources (`@oh-my-pi/pi-coding-agent`, `@oh-my-pi/pi-agent-core`) while fixing dankvoice. Read this before wiring a handler; verify against the installed version's `src/extensibility/extensions/types.ts` and `pi-agent-core/src/agent-loop.ts` when precision matters.
|
||||
|
||||
| event | payload you can use | when it fires | traps |
|
||||
|---|---|---|---|
|
||||
| `after_provider_response` | `status`, `headers`, `requestId`, `metadata` **only** | response headers arrive, **before the stream body is consumed** | No message content at all: useless for anything that speaks or summarizes text. Right for latency telemetry, non-200/abort detection, `requestId` correlation. |
|
||||
| `message_end` | full `message` (assistant, user, tool-result, gate-stop, error) | the moment that message is built (`agent-loop.ts`, assistant case) | Earliest event carrying assistant text. Fires for non-assistant messages too → gate `message.role === "assistant"`. Payload is cloned for extensions. |
|
||||
| `turn_end` | `turnIndex`, `message`, `toolResults` | after that turn's **tool results** come back | Stale by the tool duration: a 14 s tool delays a `turn_end` trigger 14 s. Extension event has **no `willContinue`** (the `turnIndex` is numbered by the lifecycle mirror), so you cannot tell the terminal turn from here. |
|
||||
| `agent_end` | `messages`, `willContinue`, `awaitingAsyncWork` | once per user prompt, when the loop settles | `willContinue: true` = the harness scheduled a continuation (auto-retry / empty-stop), not a user-visible settle — narrating it speaks an answer the model is about to redo. Silence-prone on long-running sessions. |
|
||||
| `agent_start` | — | run begin | Use to reset per-run dedupe state, so a later answer with identical text is not swallowed. |
|
||||
|
||||
## Narration cadence pattern (what dankvoice ships)
|
||||
|
||||
- `turn_end` (or `message_end` if the tool-time lag matters) for mid-run commentary; `agent_end` only for the settled answer.
|
||||
- Drop a mid-run trigger while audio/IO is busy — never queue, never interrupt.
|
||||
- Dedupe the terminal text per run (`agent_start` resets), because the last turn and the settle carry the same string.
|
||||
- Skip `ctx.mode === "print" | "json"` so benches stay silent.
|
||||
|
||||
## Prove delivery, don't infer it
|
||||
|
||||
Audio/UI timing cannot show which events arrive. Link a throwaway extension that appends timestamps per event and run one real session: `omp plugin link /tmp/probe` (package.json with `"pi": { "extensions": ["./index.js"] }`), prompt a task with a slow tool, read the log, then `omp plugin uninstall probe`. Confirmed live 2026-10-19: extensions receive `agent_start`, `turn_end`, `agent_end`; `turn_end` landed seconds before `agent_end`.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: omp-local-artifact-path
|
||||
description: Resolve a local:// artifact to a real filesystem path (for scripts/curl) without guessing the session-slug directory.
|
||||
---
|
||||
|
||||
`local://<name>.md` artifacts live in a per-cwd, per-session directory whose name you cannot guess. Three invented paths failed before this worked (2026-10-19).
|
||||
|
||||
Actual layout:
|
||||
|
||||
```
|
||||
~/.omp/agent/sessions/<cwd-slug>/<timestamp>Z_<session-id>/local/<name>.md
|
||||
```
|
||||
|
||||
`<cwd-slug>` is the working directory with `/` replaced by `-`, **keeping the leading dash** — `/home/blank/.omp/wt/wt-20261019-125509-7c8a99d` under `/home/blank` becomes `-.omp-wt-wt-20261019-125509-7c8a99d`. Dropping the leading `-.` is the mistake that produces `FileNotFoundError`.
|
||||
|
||||
## Get the real filesystem path
|
||||
|
||||
Ask the tool, don't reconstruct it — a ranged read of the artifact reports the resolved path in its header:
|
||||
|
||||
```
|
||||
read local://pr-body-msgend.md:1-1 -> [~/.omp/agent/sessions/-.../local/pr-body-msgend.md#TAG]
|
||||
```
|
||||
|
||||
Or locate it by basename (needs enough depth; `maxdepth 5` misses it — the file sits 6–7 levels down):
|
||||
|
||||
```bash
|
||||
f=$(find ~/.omp/agent/sessions -maxdepth 7 -name 'pr-body-msgend.md' | head -1)
|
||||
cp "$f" /tmp/prbody.md # stable path for the python/curl step
|
||||
```
|
||||
|
||||
Copy to `/tmp` before handing the content to a script: one cheap command, and every later tool call stops depending on session-scoped paths.
|
||||
|
||||
## Related lookups in the same directory tree
|
||||
|
||||
- Session id for the current cwd — `~/.omp/agent/history.db` (prompts table has `session_id`, `cwd`; the column is `prompt`, **not** `text`, and raw `?q=` queries cannot be combined with a table selector or pagination):
|
||||
```
|
||||
read history.db?q=SELECT session_id, cwd, substr(prompt,1,50) AS p FROM history WHERE cwd LIKE '%wt-20261019%' ORDER BY rowid DESC LIMIT 3
|
||||
```
|
||||
Use the real id in issue/PR attribution footers; never reuse an id remembered from a compacted transcript.
|
||||
- Transcript dir for the same session sits beside `local/`.
|
||||
@@ -1,12 +1,14 @@
|
||||
---
|
||||
name: omp-models-yml-runner-api-placement
|
||||
description: Add or edit a non-chat (tts/image/embedding) model row in ~/.omp/agent/models.yml without disabling every custom provider
|
||||
description: "Add or edit a non-chat (tts/image/embedding) model row in ~/.omp/agent/models.yml, wire the speech role, and diagnose a failing speech device without blaming a healthy endpoint"
|
||||
---
|
||||
|
||||
# Scope
|
||||
|
||||
`~/.omp/agent/models.yml` (or `models.yaml`) on a workstation: adding a **runner**
|
||||
model row (TTS / image / embedding / rerank / video) alongside chat providers.
|
||||
model row (TTS / image / embedding / rerank / video) alongside chat providers,
|
||||
flipping `config.yml` `modelRoles` at it, and diagnosing a speech device that
|
||||
refuses to speak.
|
||||
|
||||
# The landmine
|
||||
|
||||
@@ -63,9 +65,52 @@ yielding.
|
||||
# Speech transport contract (`openai-speech`)
|
||||
|
||||
Body is `{model, input, response_format, voice?, speed?}`; `voice` is forwarded
|
||||
only when explicitly set; `sample_rate`/`bit_rate` **throw**
|
||||
**only** when the caller passed `voice_id`; `sample_rate`/`bit_rate` **throw**
|
||||
(`openai-speech does not support sampleRate`). Codec comes from the
|
||||
`output_path` suffix; 15 000-character cap. An unknown `voice` name is a server
|
||||
error, not a fallback — and `auth: none` makes a dead endpoint read as
|
||||
*available*, so point `modelRoles.speech` at such a provider only after it has
|
||||
answered a real synthesis.
|
||||
|
||||
# Three speech-role laws (bundle-read + measured 2026-10-18)
|
||||
|
||||
1. **No setting carries a voice for `openai-speech`.** The transport builds the
|
||||
body with `voice` from the call's `voice_id` and nothing else.
|
||||
`tts.localVoice` is the *local Kokoro backend's* key; `speech.voice` is the
|
||||
TUI's **speak-aloud** voice, an enum of Kokoro names that never reaches an
|
||||
HTTP request (setting it to a pack name breaks the vocalizer and does nothing
|
||||
for synthesis — read `cfgSpeechVoice` in `dist/cli.js` before believing
|
||||
otherwise). So a voice-less call is the harness's **normal** shape: a
|
||||
voice-required endpoint needs a server-side default (fleet answer:
|
||||
`DEFAULT_VOICE` in `scripts/frank-tts.env`, journal marks the line
|
||||
`(default)`, unknown names still 400 with the list).
|
||||
2. **An explicit role has no fallback candidate.** `modelRoles.speech` set to one
|
||||
model yields exactly one candidate, so a dead endpoint or an unresolvable
|
||||
selector fails with *"No available speech model matches the speech role"*
|
||||
instead of degrading to Kokoro. Loud-by-design; do not "fix" it by adding a
|
||||
fallback in the client.
|
||||
3. **`models.yml` loads at process start; `config.yml` is read live.** A session
|
||||
older than the new row fails exactly like a dead endpoint while a fresh
|
||||
process succeeds in the same minute.
|
||||
|
||||
# Diagnosing "speech device won't speak" — in this order
|
||||
|
||||
1. `curl` the endpoint directly with a real voice name. 200 + audio ⇒ the service
|
||||
is fine and the fault is harness-side; stop touching the server.
|
||||
2. Control test in the *same* session: point `modelRoles.speech` at
|
||||
`local/kokoro` and call the device. It works ⇒ `config.yml` is live-read and
|
||||
the runner selector or its voice argument is the problem, not the endpoint.
|
||||
Revert the role immediately after.
|
||||
3. Distinguish stale registry from dead endpoint with a fresh process:
|
||||
`omp -p --no-session --no-extensions --no-skills --no-rules --tools tts --cwd /tmp
|
||||
'Call the tts tool exactly once with text="…" output_path=/tmp/x.wav
|
||||
voice_id=<name> — then reply with the tool result line.'`
|
||||
Succeeds ⇒ the long-lived session predates the `models.yml` row; restart it.
|
||||
4. Cross-check server-side, never trust the client's echo: the façade/access log
|
||||
line and the engine's own ledger must carry the same byte count the tool
|
||||
printed. (`model=…` in a tool result is what the client *requested*; a server
|
||||
line is the fact.)
|
||||
5. For harness-behaviour claims, read the bundle rather than reasoning from the
|
||||
tool description: `/home/blank/.bun/install/global/node_modules/@oh-my-pi/pi-coding-agent/dist/cli.js`,
|
||||
e.g. `grep -o -E '.{300}response_format:[a-zA-Z$]+.{250}'` for the speech body
|
||||
builder, `grep -o -E 'cfgSpeechVoice:.{0,400}'` for what a settings key feeds.
|
||||
|
||||
@@ -1,27 +0,0 @@
|
||||
---
|
||||
name: one-decision-per-ask-question
|
||||
description: "Use when putting decisions to Frank via the ask tool or a grilling round — exactly one work item per question, rounds may group many single-item questions."
|
||||
---
|
||||
|
||||
# One Decision, One Work Item, One Question
|
||||
|
||||
Friction-earned rule (corrected twice on 2026-10-18, during the voice/TTS planning).
|
||||
|
||||
## Rules
|
||||
|
||||
- Each question addresses **exactly one work item**. A round may contain many questions covering different work items; a question may not.
|
||||
- Never merge two artefacts into one option because they share an implementation dependency. Sharing an endpoint, a service, or a build step is NOT grounds for one question. (`frank-tts` serving both a podcast and an audiobook generator produced exactly this bug.)
|
||||
- Never offer something Frank did not ask for inside another question's option text. If an idea is mine, it gets its own question or stays out of the plan; it never rides along as "…and also X" in an option label or preview.
|
||||
- Do not derive option wording from my own scaffolding (todo text, my table row labels, my runbook headings). Derive it from his stated goal.
|
||||
- Mark the recommended option; a bare "yes" must accept it.
|
||||
- Facts are mine to fetch (subagent/scout), never a question for him. Only decisions go to him.
|
||||
- A question whose answer depends on an unsettled fact or another open question belongs to a **later round**, not this one.
|
||||
|
||||
## Anti-patterns that triggered this
|
||||
|
||||
- One option: "Audiobook/podcast factory — open-notebook + EPUB→narration builder." Two products, one shared dependency, zero requests for the second one. He answered "I never asked for audiobook generation."
|
||||
- One question: "Is the deliverable the podcast, or the platform (harness voice + OpenWebUI + alerts)?" Four work items in one question. Rejected: "You're lumping multiple asks into one question. Ask me per item."
|
||||
|
||||
## Check before sending
|
||||
|
||||
For each question: can I name the single artefact it decides? If two names come to mind, split it.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: open-notebook-provision-model-rest
|
||||
description: "Provision or repair an open-notebook model (LLM/TTS/embedding) over its REST API on Dankclaw: credential→model→defaults order, the defaults-store-the-ID trap, the /api/embed 500-that-succeeds, and the two-note ranking proof for embeddings."
|
||||
---
|
||||
|
||||
# open-notebook model provisioning over REST
|
||||
|
||||
Use when adding/repairing a model row (embedding, LLM, TTS) in open-notebook on Dankclaw, when semantic
|
||||
search is dead, or when an embed error appears in the worker log. Config lives in the **database**
|
||||
(encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`) — no repo file mirrors it, REST or the UI are the only doors.
|
||||
|
||||
## Access (never put the bearer in argv)
|
||||
|
||||
The API bearer IS `OPEN_NOTEBOOK_PASSWORD`; the API publishes on loopback `127.0.0.1:5056` only, so run
|
||||
from Dankclaw over ssh. `ssh dankclaw 'curl -H "Authorization: Bearer …"'` leaks the secret into
|
||||
`/proc/<pid>/cmdline` — use a mode-600 curl config:
|
||||
|
||||
```bash
|
||||
PW=$(grep -m1 '^OPEN_NOTEBOOK_PASSWORD=' /home/dank/Containers/open-notebook/env | cut -d= -f2- | tr -d '"')
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: Bearer %s"\n' "$PW" > "$cfg"
|
||||
B=http://127.0.0.1:5056 # not 5055: seerr holds the host's 5055
|
||||
```
|
||||
|
||||
## Order, and the address rule
|
||||
|
||||
`POST /api/credentials` → `POST /api/models` (`credential: <credential id>`) → `PUT /api/models/defaults`
|
||||
→ any profile that names a model (speaker `voice_model`, episode `speaker_config`/`outline_llm`).
|
||||
|
||||
- Credential provider for any in-house OpenAI-compatible server: **`openai_compatible`** (not `openai`),
|
||||
with a dummy `api_key` when the upstream does not authenticate.
|
||||
- The app's URL guard refuses base URLs resolving in `169.254.0.0/16`, so `host.containers.internal`
|
||||
cannot be stored; a **container name on `dankclaw_default`** (e.g. `http://frank-dragoon-halogen:8731/v1`)
|
||||
or another machine's RFC1918 IP both pass.
|
||||
|
||||
## The two traps that eat the most time
|
||||
|
||||
1. **`PUT /api/models/defaults` takes the model ID, never the name.** `"default_embedding_model":
|
||||
"qwen3-embedding-0.6b"` is accepted and echoed back, then every job dies in the worker with
|
||||
`Model with ID qwen3-embedding-0.6b not found`. Use the row's `model:…` id (GET defaults first, change
|
||||
one key, PUT the whole object back — the other keys in that response are ids and must be preserved).
|
||||
2. **`POST /api/embed` answers 500 *after* the work succeeds.** Upstream `EmbedResponse.command_id`
|
||||
declares `str` and the worker returns a `RecordID`, so FastAPI raises in serialization only. The truth is
|
||||
`podman logs open-notebook | grep "Successfully embedded"`, never the HTTP code. Note saves
|
||||
(`POST /api/notes`) submit `embed_note` themselves; `/api/embed` is only for backfills.
|
||||
|
||||
## Proof it actually works
|
||||
|
||||
- Per row: `POST /api/models/<id>/test` → `Embedding dimensions: 1024` (embeddings) or a real completion.
|
||||
- Embeddings, discriminating test: create **two** unrelated notes, then
|
||||
`POST /api/search` `{"query": "...", "type": "vector", "search_notes": true, "limit": 5}` — the enum is
|
||||
`text|vector` (not `semantic`) and the score field is `similarity`. Each query must rank its own note
|
||||
high and the other low (measured 0.659/0.292 vs 0.906/0.318). One note proves nothing: `total_count` is 1
|
||||
either way. Delete the probe notes afterwards (`DELETE /api/notes/<id>`).
|
||||
- Confirm which engine served it: `podman logs <engine container> | grep "POST /v1/embeddings"`.
|
||||
|
||||
Owner pages: `serverAppConfigs-docs/Dankclaw/Open Notebook.md` (app-side shapes), and the endpoint's own
|
||||
page for whatever serves the model.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: openwebui-tts-chain-audit
|
||||
description: Audit/repair OpenWebUI TTS on Dankclaw when cloned voices get rejected by a 29-name Kokoro enum or synthesis is silent; rootless DB + endpoint probes
|
||||
---
|
||||
|
||||
Audit OpenWebUI's TTS/STT wiring on Dankclaw when a voice is rejected, synthesis is silent, or someone asks "is the cloned voice wired correctly". Not for model-picker/ACL issues (openwebui-picker-acl-repair owns those).
|
||||
|
||||
## Shape of the stack
|
||||
- Container `open-webui` runs **rootless as `dank`** on Dankclaw: `ssh dankclaw 'podman exec open-webui ...'` (NO sudo: `sudo podman` sees an empty world). Host port 3005.
|
||||
- Server-side config = `webui.db` `config` table (admin), per-user overrides = `user.settings` JSON column. Settings changed via UI persist per-user; admin page persists to `config`.
|
||||
- Cloned voices live behind the frank-tts façade `:7772` (`/v1/audio/speech`, `/v1/audio/voices`); voices resolve to `/home/dank/voices/<name>/reference.wav`.
|
||||
|
||||
## The trap (2026-10-18 law)
|
||||
`Error: Voice "narrator1" not found. Should be one of: af_heart, ...` (29 names) is **OpenWebUI's bundled in-browser Kokoro** (`browser-kokoro`) rejecting client-side — frank-tts never sees the request. Signature: container logs show zero `audio/speech` hits. Fix = the personal dropdown Settings -> Audio -> TTS Engine -> Default (empty string), not any deploy.
|
||||
|
||||
## Probe sequence (read-only, safe)
|
||||
1. Admin truth:
|
||||
`podman exec open-webui python -c "import sqlite3; c=sqlite3.connect('/app/backend/data/webui.db'); [print(k,'=',v) for k,v in sorted(dict(c.execute('SELECT key,value FROM config')).items()) if 'audio' in k or 'tts' in k.lower()]"`
|
||||
Expect: `audio.tts.engine=openai`, `audio.tts.openai.api_base_url=http://host.containers.internal:7772/v1`, `audio.tts.model=frank-tts`, `audio.tts.voice=<cloned name>`.
|
||||
2. User overrides — a user pinned to `browser-kokoro` beats admin config. CAUTION: `user.settings` can be SQL NULL or the JSON string `"null"`:
|
||||
`... for uid,s in c.execute('SELECT id,settings FROM user'): d=json.loads(s) if s else None; d=d if isinstance(d,dict) else {}; hits={k:v for k,v in d.items() if 'audio' in k or 'tts' in k.lower()} ...`
|
||||
3. Endpoint truth: `curl -s http://127.0.0.1:7772/v1/audio/voices` (roster), then POST the exact OpenWebUI body `{"model":"frank-tts","input":"probe","voice":"narrator1","response_format":"mp3","speed":1.0}` to `/v1/audio/speech` — expect `200 audio/mpeg`.
|
||||
4. Contrast logic: server-path failure leaves a container log line AND a façade error; browser-kokoro failure leaves neither.
|
||||
|
||||
## Law
|
||||
- Full contract + diagnosis live in serverAppConfigs docs (`Dankclaw/OpenWebUI.md` § Text-to-speech, `Dankclaw/Frank TTS.md` troubleshooting). Docs are the source; this skill is just the probe route.
|
||||
- Rootless gotcha generalizes: any Dankclaw service probe through podman goes WITHOUT sudo unless the unit is a system quadlet.
|
||||
@@ -1,68 +1,34 @@
|
||||
---
|
||||
name: parallel-worktree-path-hazard
|
||||
description: "\"Use when several agents edit the same repo in different git worktrees at once, or when an edit/read may have landed in the wrong checkout: which path forms bind where, how to detect with git -C status, and the repair protocol (cp + git show HEAD blob write-back) because git checkout/restore is policy-blocked in a shared workspace.\""
|
||||
description: "Use when several agents edit the same repo in different git worktrees at once, or when an edit lands in the wrong checkout: relative tool paths resolve to the session cwd, not the shell cwd."
|
||||
---
|
||||
|
||||
# Concurrent-worktree path hazard (shared repo, many agents)
|
||||
Use when several agents (or you across two checkouts) edit the same repo in different git worktrees at once, or when an edit/query lands in the wrong checkout.
|
||||
|
||||
Measured 2026-10-18 on `serverAppConfigs` with two agents live at once. The
|
||||
failure is **silent**: the tool reports success, the diff looks right, and the
|
||||
change lives in a *different* worktree.
|
||||
## The silent failure: relative tool paths resolve to the SESSION cwd
|
||||
|
||||
## Which path form binds where
|
||||
- **Parent/top-level session**: a **relative** `path` resolves against the
|
||||
session CWD — its own worktree — never against the worktree named in the
|
||||
prompt. This is how a change meant for `wt-tts-trial-docs` landed in
|
||||
`wt-20261017-…`.
|
||||
- **Subagent**: an **absolute** path to its own assigned worktree can resolve
|
||||
into the parent's worktree when the same *relative* path exists there
|
||||
(`edit` twice onto `dankclaw/.env.example`). Its `write` with the same
|
||||
absolute path was fine — the bug is `edit`/`read` path resolution, not `write`.
|
||||
`read` / `edit` / `write` / `grep` resolve a **relative** `path` against the **session working directory** — never against the `cd` inside a neighboring `bash` call, and not against a `../other-worktree/...` prefix either (that gets normalized back into the session tree). So after `cd ../wt-B && grep …` you can `read path="docs/page.md"` and get **wt-A's** copy.
|
||||
|
||||
Rules that hold:
|
||||
1. A subagent passes paths as `../<its-worktree>/<rel>` (relative to the parent
|
||||
worktree) for `read` **and** `edit`; the parent passes **absolute** paths to
|
||||
any worktree that is not its own CWD.
|
||||
2. After every edit batch, prove the landing site, do not trust the tool's echo:
|
||||
`git -C <intended-worktree> status --short` must list the file. A file you
|
||||
edited that is missing from that list went somewhere else.
|
||||
3. Keep each work item on its own branch/worktree, and keep ONE integration
|
||||
owner (the parent) for shared files — `scripts/deploy-remote.sh`,
|
||||
`scripts/deploy-plan.py`, `tests/`, `serverAppConfigs-docs/index.md`,
|
||||
`AGENTS.md` are the collision hot spots in this repo. Never rebase a
|
||||
sibling's branch.
|
||||
Why it survives "I just read it to check": the returned path is printed workspace-relative and the file name is identical in every checkout, so the output looks like a correct read of the tree you meant. Only content that differs between the checkouts betrays it.
|
||||
|
||||
## Repair protocol (no `git checkout`, no `git restore`)
|
||||
`git checkout -- <path>` and `git restore <path>` are **policy-blocked** in a
|
||||
shared workspace ("overwriting working-tree files … discards changes other
|
||||
concurrent agents rely on"). Unstaging (`git reset [<path>]`,
|
||||
`git restore --staged <path>`) is allowed but does not undo content. So:
|
||||
Rules:
|
||||
- **Absolute paths for every tool call that targets a worktree other than the session's.** No exceptions for quick reads.
|
||||
- Confirm the checkout from content you know differs (a heading, a status line), not from the echoed path.
|
||||
- A `read` error like "Did you mean .AGENTTRASH/<name>" is a signal the relative path resolved into the session tree — re-issue absolute.
|
||||
- Same trap for `bash`: it starts each call in the session cwd, so `cd` twice per call or use absolute paths there too.
|
||||
|
||||
```bash
|
||||
F="path/with spaces if any"
|
||||
# 1. move the change to the worktree that should own it (both at the same base commit)
|
||||
cp "<wrong-wt>/$F" "<right-wt>/$F"
|
||||
# 2. undo in the wrong worktree by writing the committed blob back
|
||||
git -C "<wrong-wt>" show "HEAD:$F" | tee "<wrong-wt>/$F" > /dev/null
|
||||
# 3. prove both sides
|
||||
md5sum <(git -C "<wrong-wt>" show "HEAD:$F") "<wrong-wt>/$F" "<right-wt>/$F"
|
||||
git -C "<wrong-wt>" status --short # must be empty for that file
|
||||
git -C "<right-wt>" diff --stat # must show the intended delta
|
||||
```
|
||||
## Landing a change when the session worktree is stale
|
||||
|
||||
Never fix it blind: if the wrong worktree had its own uncommitted edits to that
|
||||
file, `cp`/blob-write-back would destroy them — check
|
||||
`git -C <wrong-wt> status --short` FIRST and reconcile content, don't overwrite.
|
||||
`git diff --name-only HEAD origin/main` first. If main moved a file you are editing, your edit's base is wrong even though the file "looks fine" — do not commit on the stale base.
|
||||
|
||||
## Tidy after a cancelled sibling
|
||||
A killed subagent can leave a worktree plus an empty branch. Removing them is
|
||||
safe only when verified empty: `git -C <main> log --oneline -3 <branch>` (no
|
||||
unique commits) and `git -C <wt> status --short` (clean), then
|
||||
`git worktree remove <wt>` + `git branch -D <branch>`. An orphan worktree at a
|
||||
stale commit is the same class of trap as a leftover bench drop-in: the next
|
||||
agent reads it as unfinished work.
|
||||
1. `git worktree add ../wt-clean -b <type>/<slug> origin/main`; copy the new files in.
|
||||
2. Re-apply the edit against the clean tree's version **with absolute paths**, then read it back absolutely to prove which tree changed.
|
||||
3. Run the suite inside the clean worktree; commit there.
|
||||
4. Remove your own duplicate edit from the session worktree by editing it away (`CUT`/`PUT`). `git checkout -- <path>` and branch switches are policy-blocked in shared workspaces and should stay blocked — they discard edits other agents rely on.
|
||||
5. Never carry the same uncommitted edit in two worktrees at once; if you must keep both trees, keep a task that reconciles them.
|
||||
|
||||
## Report it
|
||||
File the misresolution to `xd://report_issue` (false positives are fine) and
|
||||
record the incident in the repo worklog — a repair nobody wrote down looks like
|
||||
a hijack later.
|
||||
## Multi-agent hazards in one shared worktree
|
||||
|
||||
- Another agent can stage, commit, or check out files under you; re-read a file immediately before editing it if any tool result hinted at intervening change.
|
||||
- Untracked scratch you drop in a shared tree (reports, notes, `.AGENTTRASH/` entries) is visible to everyone and can be committed by them. Put throwaway artifacts outside the checkout.
|
||||
- Never `git clean -f`, `rm -rf`, or branch switching to "get a clean state" in a tree another agent is using.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
name: plugin-git-install-committed-artifact
|
||||
description: "Use when shipping or landing an omp/pi plugin (omp plugin install/upgrade git): install validation fails on a missing build artifact, a stale bun git-mirror cache is reused, the live bundle needs proof it is the merged build, or upgrade says \"not installed\"."
|
||||
---
|
||||
|
||||
# Plugin install from git: committed artifact + fresh mirror
|
||||
|
||||
`omp plugin install git:<url>` (and `pi install git:`) **clone the repo and validate the manifest's declared extension entry WITHOUT running a build step**. Two failure classes seen live (dank/dankvoice, 2026-10-19):
|
||||
|
||||
## 1. Declared entry missing from the clone
|
||||
|
||||
Symptom: `Plugin <name> extension validation failed: ./dist/index.js: declared extension entry not found on disk`.
|
||||
|
||||
Cause: build output (`dist/`) is gitignored; the manifest points at it; the clone has only `src/`.
|
||||
|
||||
Fix — ship the artifact:
|
||||
- Remove `dist/` from `.gitignore`, `bun run build`, commit `dist/`.
|
||||
- Prove build determinism first: build twice, `cmp` the outputs (bun esm builds are byte-reproducible without sourcemaps).
|
||||
- Pin the contract in a small bats suite so it can't rot:
|
||||
1. the manifest's declared entry is tracked in git (`git ls-files -- "${entry#./}"` non-empty — this is the installer's exact contract, not a config assertion);
|
||||
2. committed bytes equal a fresh build (`bun run build` then `git diff --name-only -- dist` is empty → red on drift).
|
||||
|
||||
## 2. Stale bun git-mirror cache
|
||||
|
||||
Symptom: fix pushed, retry **still** fails identically, and the retry is suspiciously fast (seconds).
|
||||
|
||||
Cause: bun keeps bare mirrors in `~/.bun/install/cache/<hash>.git` and may reuse them at an old commit — even after the fix is pushed and the default branch is correct.
|
||||
|
||||
Diagnose + fix:
|
||||
```bash
|
||||
find ~/.bun/install/cache -maxdepth 2 -iname "*<repo>*"
|
||||
git --git-dir ~/.bun/install/cache/<hash>.git log --oneline -3 --all # stale HEAD?
|
||||
rm -rf ~/.bun/install/cache/<hash>.git # it is a cache; safe
|
||||
```
|
||||
Prove the fetch path independently first (also validates credentials for private Gitea repos):
|
||||
```bash
|
||||
GIT_TERMINAL_PROMPT=0 git clone -q <url> /tmp/x && git -C /tmp/x log --oneline -1 && ls /tmp/x/dist
|
||||
```
|
||||
|
||||
## Order of investigation (don't skip)
|
||||
1. Gitea default branch correct? (`GET /api/v1/repos/<o>/<r>` with token — private repos 404/None unauthenticated.)
|
||||
2. Plain unattended clone contains the entry? If yes, the installer's view is cached → purge mirror.
|
||||
3. Only then suspect the installer itself (validation code lives in the minified bundle; `grep -oE '.{600}<error string>' "$bunfile"` on `readlink -f $(which omp)` shows the exact check).
|
||||
|
||||
## Landing a merged PR on the live box
|
||||
|
||||
After the merge and the `--ff-only` on the dev checkout (`ls -ld node_modules` after landing anything that could have touched it):
|
||||
|
||||
- **`omp plugin upgrade <name>` takes the PACKAGE name, not the install alias.** `omp plugin upgrade dankvoice` fails `dankvoice is not installed` while the live copy sits in `~/.omp/plugins/node_modules/omp-dankvoice/`. Read the directory name (or `package.json` `"name"`) and upgrade that.
|
||||
- Then close the chain of custody with one command — the deployed bytes are the merged bytes:
|
||||
```bash
|
||||
cmp ~/Projects/<repo>/dist/index.js ~/.omp/plugins/node_modules/<pkg>/dist/index.js
|
||||
```
|
||||
If only docs/tests changed, `dist/` is untouched and this should already match; if `src/` changed, the plugin must be upgraded first or this fails with `char <n>` — which is a real finding (stale live bundle), not a test bug.
|
||||
- The session that did the merge keeps the OLD bundle in memory: it is still running pre-merge code. Never conclude "the change does not work" from the merging session.
|
||||
|
||||
## After install — proof without a TUI
|
||||
- `omp plugin doctor` → per-plugin ✔ line.
|
||||
- Audible/behavioral chain when TUI input is unusable (update-banner modal freezes pty submits): run `omp --mode=rpc --no-session` with the feature env armed (dankvoice: `DANKVOICE_AUTO=1`, gate is print/json-only) and watch for objective side effects, e.g. PipeWire: `pw-dump | grep -q "say-<ms>.wav"` proves synth+playback end-to-end.
|
||||
- When pty submits DO work (no modal), the strongest instrument is a stub player plus a clip count, not ears or pixels: `DANKVOICE_PLAYER=/tmp/stub.sh` where the stub appends `$1` to a log and exits 0. Launch the DEPLOYED bundle by path (`omp --no-extensions --no-session -e ~/.omp/plugins/node_modules/<pkg>/dist/index.js`) in a neutral cwd with `DANKVOICE_CONFIG` pointed at a scratch file so the operator's settings are untouched.
|
||||
- **Count clips per narration EVENT, not per turn.** One turn can fire several narrations (dankvoice: `message_end` carries prose+thinking, `agent_end` the settled answer; the dedupe only skips byte-identical text, so thinking-on means both speak). A "too many clips" result is usually correct behaviour — check the harness transcript for the turn count before believing a bug. The player deletes each temp wav, so the log line count is the only surviving evidence: park it in `~/.AGENTTRASH/<date>-<topic>/` if it becomes evidence for an issue.
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: red-verify-without-stash
|
||||
description: "Prove a regression test is red-capable against pre-fix code in a shared worktree where git stash/checkout are policy-blocked, via git-show blob swap with md5-verified restore."
|
||||
---
|
||||
|
||||
# Red-verifying tests without `git stash`
|
||||
|
||||
## When
|
||||
You fixed a file and added tests that should have failed before the fix, but workspace policy blocks `git stash` (and `git checkout -- file`) in shared worktrees (e.g. `~/.omp` shared-workspace policy). Needed to prove red-capture instead of asserting it.
|
||||
|
||||
## Recipe
|
||||
One `bash` call — swap the fixed file for the HEAD (pre-fix) blob, run only the fix-relevant tests, restore, verify byte-identity:
|
||||
|
||||
```bash
|
||||
cd "$WT"
|
||||
md5sum path/to/fixed.file > /tmp/file-hash
|
||||
cp path/to/fixed.file /tmp/fixed.file.bak
|
||||
git show HEAD:path/to/fixed.file > path/to/fixed.file # pre-fix content; new tests now exercise old code
|
||||
bats -f "test name A|test name B" tests/suite.bats # expect: not ok for every fix-relevant test
|
||||
cp /tmp/fixed.file.bak path/to/fixed.file
|
||||
md5sum -c /tmp/file-hash && echo RESTORED
|
||||
```
|
||||
|
||||
## Notes
|
||||
- `git show HEAD:path > path` truncates-and-refills the file in place; safe because you hold the md5 and a backup copy, and the restore check proves the fixed bytes are back.
|
||||
- Run only the fix-relevant tests via the runner's name filter (`bats -f` regex); a full suite wastes time and old tests pass on both versions.
|
||||
- Expect exactly the tests that pin the fix's behavior to fail pre-fix; tests unrelated to the change (pure coverage of old behavior) pass on both versions — don't count them as red evidence.
|
||||
- `bats` pipelines mask exit codes without `pipefail`; judge by the `ok`/`not ok` lines, not the shell status.
|
||||
- For non-test proof of endpoint behavior (live A/B of an HTTP request body), replay the exact body with `curl` against the live endpoint instead of speculating; keep both timings.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: ryzenai-generic-failure-loader-order
|
||||
description: "Debug AMD Ryzen AI OGA \"Exception during initialization: Generic Failure\" on Linux XDNA2 — the LD_LIBRARY_PATH-vs-activate ordering trap, its proof instruments, and the OGA 0.14 run contract."
|
||||
---
|
||||
|
||||
Symptom (any `onnxruntime_genai` run on a Ryzen AI NPU, container or bare):
|
||||
|
||||
```
|
||||
[E:onnxruntime:, inference_session.cc:2811 operator()] Exception during initialization: Generic Failure
|
||||
```
|
||||
|
||||
Fails ~2 ms after `og.Model()`. No `openat` of `/dev/accel/accel0`, **nothing in the kernel log**, no XRT log (the INI logger is not built into AMD's shipped XRT), no core dump. It is NOT a platform, model-export, or driver-ABI failure.
|
||||
|
||||
## Root cause
|
||||
|
||||
`source <venv>/bin/activate` from `install_ryzen_ai.sh` **prepends** to `LD_LIBRARY_PATH`:
|
||||
|
||||
```
|
||||
site-packages/flexml/flexml_extras/lib : site-packages/onnxruntime/capi : site-packages/voe/lib
|
||||
```
|
||||
|
||||
`voe/lib` is the **compiler-side** bundle: it ships its own `libxrt_coreutil.so.2` and `libryzen_mm.so.1`. Set the path *before* activating and those copies win, XRT's XDNA shim binds against the wrong coreutil, `xrt_core::shim_loader::shim_loader()` → `get_userpf_device()` → `xrt::device(0)` throws, `ryzenai::Allocator::Alloc` throws next, and ORT flattens both into the bare string above.
|
||||
|
||||
**Rule: activate first, THEN set `LD_LIBRARY_PATH`, with `/opt/xilinx/xrt/lib` first.** Passing the var via `podman -e` and sourcing activate afterwards in the same `bash -lc` is the broken shape — that is exactly how a runner script differs from a hand-typed probe, and it is why diagnostics pass while the runner fails. Mounting XRT is not enough either: the tree must be **on** the path.
|
||||
|
||||
## Proof instruments, cheapest first
|
||||
|
||||
1. `ldd <deployment>/lib/libonnxruntime_providers_ryzenai.so | grep -E "libxrt_coreutil|libryzen_mm"` — the resolved path settles it alone: `/opt/xilinx/xrt/lib` = good, `.../voe/lib` = this bug.
|
||||
2. `XILINX_XRT=/opt/xilinx/xrt /opt/xilinx/xrt/bin/xrt-smi examine` → `RyzenAI-npu5 | aie2p | 6x8` + `NPU Firmware Version`. Needs `libboost_filesystem1.83.0`, `libboost_program_options1.83.0`, `libuuid1` (26.04 names; the 24.04 docs' 1.74.0 does not exist). A "No such file or directory" for a file that exists = missing interpreter/dep, not a missing binary.
|
||||
3. `strace -f -e trace=openat` — `/dev/accel/accel0` never opened ⇒ failure is pre-device ⇒ loader-level.
|
||||
4. `gdb -q -batch` with `catch throw` + `bt 10` loop → names `shim_loader` / `Allocator::Alloc`.
|
||||
|
||||
Known dead ends (one run each, then stop): `--ulimit memlock`; XRT INI verbosity; `DD_LOG_LEVEL` / `RYZENAI_TRACE` / `RYZENAI_EP_TRACING` (print nothing before the throw); byte-scanning XRT `.so` for PCI IDs (ambiguous); libboost presence for the OGA path (only `xrt-smi` needs it); flag-by-flag container matrices (`--memory`, `--security-opt`, `--name`, extra mounts — all innocent).
|
||||
|
||||
## Verified facts on this stack
|
||||
|
||||
- Vendor XRT 2.25.37 userspace (`dpkg-deb -x`, never dpkg-installed) works against Ubuntu's in-tree `amdxdna` 0.7.0; no DKMS module needed. PCI `1022:17f0` rev 11 = XDNA2, reported as `aie2p 6x8`.
|
||||
- AMD's `quicktest` passing does **not** prove NPU use (it can silently run on CPU); trust `xrt-smi` + logits, not "it didn't crash".
|
||||
- `/dev/accel/accel0` is `root:render 0660`; rootless podman cannot read it (userns drops the gid; `keep-groups`, `keep-id:gid=990`, `--group-add render` all fail) → rootful container, or a udev rule like the GPU nodes'.
|
||||
- OGA 0.14 (RAI 1.8) contract: `GeneratorParams` has only `set_search_options` (no `input_ids`, no `model.generate`); feed with `Generator.append_tokens(ids)`, loop `generate_next_token()`, read `get_sequence(0)`; `max_length` counts **prompt + generated**. Full Fusion (`*_npu_4K`) caps input+output at 4096.
|
||||
- Validate content, not just exit code: `np.argsort(logits[0])[-8:]`. A real logits tensor after `append_tokens` proves the graph executed (prefill is folded into step 1, so a sub-millisecond first step + a 1-token sequence means it did not); if argmax is the sequence terminator (`
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
name: ryzenai-npu-oga-linux-run
|
||||
description: "Run an AMD vendor-compiled Ryzen AI (XDNA2) model — OGA/onnxruntime-genai — on a Linux Ryzen AI host with the in-tree amdxdna driver, without installing the vendor driver or touching the host driver; includes the triage for ORT \"Exception during initialization: Generic Failure\"."
|
||||
---
|
||||
|
||||
Run a vendor-compiled Ryzen AI (XDNA2) model on Linux with the **in-tree** amdxdna driver: no vendor kernel module, no `dpkg -i`, host driver untouched.
|
||||
|
||||
## Preconditions
|
||||
- Host: Strix Point / Strix Halo / Krackan (PCI `1022:17f0`), in-tree `amdxdna` bound to the NPU function (`readlink /sys/bus/pci/devices/<bdf>/driver` → `amdxdna`), `/dev/accel/accel0` present.
|
||||
- `ryzen_ai-<ver>.tgz` installed into a venv (`install_ryzen_ai.sh -a yes -p <venv>`; it hard-codes `python3.12` — put a uv-managed 3.12 **bin dir** on PATH, never a symlink: a symlinked shim breaks venv creation with `No module named 'encodings'`).
|
||||
- Model artifact MUST match the RAI version: `<name>_rai_<ver>_npu_<ctx>`. Other versions load and emit garbage.
|
||||
- Vendor XRT userspace: `dpkg-deb -x` the debs from `RAI_<ver>_Linux_NPU_XRT.zip` (base + npu + `xrt_plugin…amdxdna.deb` — the plugin is what ships `libxrt_driver_xdna.so.*`) into a host dir, bind-mount it at `/opt/xilinx/xrt`.
|
||||
|
||||
## Container run line (the part that is easy to get wrong)
|
||||
```
|
||||
--device /dev/accel/accel0 # root:render 0660 → rootful, or a world-writable udev rule
|
||||
-v <xrt-root>/opt/xilinx/xrt:/opt/xilinx/xrt:ro
|
||||
-e RYZEN_AI_INSTALLATION_PATH=<venv> -e XILINX_XRT=/opt/xilinx/xrt
|
||||
-e LD_LIBRARY_PATH=/opt/xilinx/xrt/lib:<venv>/onnxruntime/lib:<venv>/deployment/lib
|
||||
# cwd MUST be <venv>/deployment/lib — the RyzenAI EP is dlopen'd by bare name and the
|
||||
# only place ORT looks is the working directory.
|
||||
# Activate the venv explicitly; a login shell (`bash -lc`) re-sources the profile and
|
||||
# can put a numpy-less base python first on PATH.
|
||||
```
|
||||
|
||||
## LAW: mounting XRT is not enough — `/opt/xilinx/xrt/lib` must be ON `LD_LIBRARY_PATH`
|
||||
The RyzenAI EP reaches the XDNA shim through XRT's own loader. With the tree bind-mounted
|
||||
but off the loader path, `xrt::device(0)` throws in `xrt_core::shim_loader`, the RMM
|
||||
allocator fails next, and ORT flattens all of it into
|
||||
|
||||
```
|
||||
[E:onnxruntime:, inference_session.cc:2811] Exception during initialization: Generic Failure
|
||||
```
|
||||
|
||||
Signature that distinguishes this from a real device/driver fault: fails in ~2 ms, **never
|
||||
opens `/dev/accel/accel0`**, nothing in `dmesg`/`journalctl -k`, no XRT log (the INI logger
|
||||
is not built into the vendor build — `XRT_INI_PATH` + `[Common] verbosity` writes nothing).
|
||||
|
||||
## Triage recipe for "Generic Failure" (in this order — each step is one call)
|
||||
1. `strace -f -e trace=openat` → does it open `/dev/accel/accel0`? If **no**, the failure is pre-device: loader/shim, not the driver. Also note the last file it opens (`genai_config.json` = config parsed, then provider setup fails).
|
||||
2. `gdb -q -batch -x cmds --args python driver.py` with `catch throw` + `bt 10` + `continue`. Frames `xrt_core::shim_loader::shim_loader` / `get_userpf_device` / `xrt::device::device` then `ryzenai::Allocator::Alloc` ⇒ the shim never loaded ⇒ fix `LD_LIBRARY_PATH`.
|
||||
3. Prove it with a two-arm run in ONE container (toggle only the suspect) before writing the law down. Both arms passing ⇒ that variable was never the cause.
|
||||
4. Ground truth on device visibility: `xrt-smi examine` → `RyzenAI-npu5 | aie2p | 6x8` + `NPU Firmware Version`. If xrt-smi dies on `libboost_filesystem.so.1.83.0`, install `libboost_filesystem1.83.0`+`libboost_program_options1.83.0` — **boost is only needed by xrt-smi, not by the OGA/EP path** (verified by purging it and still loading the model).
|
||||
5. `voe` in site-packages is a *compiler* toolkit (passes/patterns), not a runtime — swapping `LD_LIBRARY_PATH` to `voe/lib` fixes nothing.
|
||||
6. Byte-scanning XRT `.so` files for supported PCI IDs is inconclusive (2-byte ID patterns collide constantly) — don't bother.
|
||||
|
||||
## Distrust "it ran" on this stack
|
||||
- Vendor `quicktest` can print success while computing on CPU (missing peano lib ⇒ silent CPU fallback). Prove NPU residency per run: sample `/sys/class/accel/accel0/device/mem_usage` (needs root) during generation, or read the EP's `Actually running on NPU` line.
|
||||
- Never report request #1 after a boot as a rate — first pass includes graph/init cost.
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
name: ryzenai-npu-oga-linux-spike
|
||||
description: "Run an AMD-precompiled AMD Ryzen AI (XDNA2) ONNX-GenAI model on a Linux Strix Halo host from public artifacts: staging, rootless NPU device-access wall, container recipe, and the NuTTS-air speech-token contract (65,536-code vocab, 50 codes/s real-time line)."
|
||||
---
|
||||
|
||||
Run an AMD-precompiled XDNA2 model (OGA / onnxruntime-genai) on a Linux Strix Halo host (Dankclaw-class) without installing the vendor stack on the host.
|
||||
|
||||
## Where the bits actually live (both public, no AMD account)
|
||||
|
||||
- Runtime tarball: `https://download.amd.com/opendownload/RyzenAI/1.8.0/ryzen_ai-1.8.0.tgz` (~7.28 GiB). Verify with a ranged GET — first bytes `1f 8b 08`.
|
||||
- NPU driver/XRT zip: `https://download.amd.com/opendownload/RyzenAI/Driver/RAI_1.8_Linux_NPU_XRT.zip` (90 MB).
|
||||
- Dead ends (404, do not re-probe): `download.amd.com/driver/linux/RAI_1.8_Linux_NPU_XRT.zip`, `.../opendownload/RyzenAI/ryzen_ai-1.8.0.tgz`, `www.xilinx.com/bin/public/openDownload?filename=...` (301 → 404). The login-walled EULA *form* is not the gate; the `opendownload` tree is the real file host.
|
||||
|
||||
Tarball has 30 top-level entries: cp312/linux_x86_64 wheels (`onnxruntime_genai_ryzenai-0.14.0`, `onnxruntime_vitisai-1.27.0`, `vitis_mllib`, `flexml*`, `device_essentials_strx`, `model_generate`, `voe-1.8.0`), plus `deployment.tar.gz` (contains `lib/libonnxruntime-genai.so`, `lib/libonnxruntime_providers_ryzenai.so`, AIE compiler libs), `onnxruntime.tar.gz`, `quicktest.tar.gz` (VitisAI-EP install validator), `LLM.tar.gz` (text-only OGA examples: `run_model.py`, `tokenizer_factory.py`), `install_ryzen_ai.sh`. `voe-4.0-linux_x86_64.tar.gz` is only `vaip_config.json` — no TTS xclbins. **AMD ships no TTS runner and no TTS example.**
|
||||
|
||||
## Host rules
|
||||
|
||||
- **Never `dpkg -i` the vendor XRT.** `dpkg-deb -x <deb> <dir>` and bind-mount `<dir>/opt/xilinx/xrt` at the same path in the container. The plugin deb would install a DKMS `amdxdna` that replaces the distro in-tree driver; the in-tree driver on Ubuntu 26.04 is `amdxdna 0.7.0` and already loads `amdnpu/17f0_11/npu_7.sbin` at boot (`dmesg | grep amdxdna`).
|
||||
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render 0660`): the host gid never enters the userns gid map, so `--device` yields `nobody:nogroup`. Verified failing 2026-10-19: `--group-add keep-groups`, `--userns=keep-id`, `--userns=keep-id:gid=990` (process gets gid 990, node still unreadable), `--group-add render`, `--group-add gid:990` (both error). Options: run rootful (`sudo podman`, separate storage → rebuild the image rootful, ~1 min), or add a world-writable udev rule for `accel0` (the fleet already does exactly this for `kfd`/`renderD*` in `99-comfyui-gpu-access.rules`, whose header explains why: systemd user-unit spawns carry no supplementary groups).
|
||||
- **Rootless image build trap:** the vendor installer's `tar -xf` preserves recorded uids (95777 etc.) and aborts under `set -e` with `Cannot change ownership ... Invalid argument`, silently skipping the `deployment`/`onnxruntime` extraction. Patch it in the Containerfile: `sed -i 's/tar -xf/tar --no-same-owner -xf/g' install_ryzen_ai.sh`.
|
||||
- Installer usage: `./install_ryzen_ai.sh -a yes -p /opt/rai -c /opt/rai` (python **3.12** required; base `python:3.12-slim`; needs `linux-libc-dev zip pciutils patch libgomp1`). It builds a venv from the local wheels, reinstalls `onnxruntime-vitisai` then `--no-deps onnxruntime-genai-ryzenai`, installs `requirements_extras.txt` (transformers 4.57.6), patches `bin/activate` (sets `RYZEN_AI_INSTALLATION_PATH=$VIRTUAL_ENV` + `LD_LIBRARY_PATH` into site-packages), and untars `LLM/deployment/onnxruntime/voe/quicktest` into the cpp path.
|
||||
- `xrt-smi` in a container needs `libboost-filesystem1.83.0` + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names) and `apt update` first; boost is **only** needed by `xrt-smi`, never by the OGA/EP path.
|
||||
|
||||
## HF artifact staging
|
||||
|
||||
- Gated (401 on API, 403 on git) for a plain account: `neuphonic/neucodec` (the **only** NeuCodec encoder), `neuphonic/neucodec-onnx-decoder`, `neuphonic/neucodec-onnx-decoder-int8`, `neuphonic/neutts-air`. `amd/neutts-air_rai_1.8.0_npu_4K` is public. Public mirrors of the codec exist (`eugenehp/neucodec` has the full 1.16 GB `pytorch_model.bin`) — treat a mirror of a gated release as a licensing decision for the operator, not a workaround to take silently.
|
||||
- Gated-repo loader trap: `HF_HUB_OFFLINE=1` + `HF_HOME` pointed at a cache laid out as `hub/models--<org>--<repo>/snapshots/<rev>/…` works, but `NeuCodec._from_pretrained` **asserts the repo id** is in `["neuphonic/neucodec", "neuphonic/distill-neucodec"]` — a local directory path is refused outright, so pass the repo id and satisfy it from the cache.
|
||||
- `github.com/neuphonic/neutts` is public and ships `samples/{dave,emily,greta,jo}.{pt,txt,wav}` = **pre-encoded reference codes + transcripts** → a spike needs zero gated bytes.
|
||||
- Auth without leaking the token into `ps`: build a 600-perm credential file with a shell builtin and point git at it —
|
||||
`umask 077; printf 'https://oauth2:%s@huggingface.co\n' "$(cat ~/.cache/huggingface/token)" > ~/.git-credentials-hf`
|
||||
then `git -c credential.helper="store --file ~/.git-credentials-hf" clone ...`. Never `git clone https://oauth2:$TOKEN@...`.
|
||||
|
||||
## NuTTS-air on XDNA2 (the model-side contract)
|
||||
|
||||
- The package is an OGA model dir: `genai_config.json` (`type: qwen2`, 24 layers, hidden 896, 14 heads / 2 KV, ctx 32768, `vocab_size` **217,652**, `provider_options: [{"RyzenAI": {"hybrid_opt_token_backend": "npu"}}]`), plus `dd_metastate_*` AIE artifacts and `model.onnx` + `model.onnx.data`.
|
||||
- **Speech tokens live in the text vocab, and the codebook is 65,536 — not 10,001.** `tokenizer.json` `added_tokens` holds exactly 65,536 `<|speech_N|>` entries, ids **151,671–217,206** contiguous, over a base BPE vocab of 151,643; specials are 151,644–151,670 (`<|SPEECH_GENERATION_END|>` = 151,670). The vendor template maps a codec index straight to its id: `"".join(f"<|speech_{i}|>" for i in ref_codes)`. An encoded reference really does contain codes in the 60k range, so a 10k assumption silently truncates the space.
|
||||
- The graph's own `eos_token_id` is `[151645, 151643]` (the **chat turn** stop) and `search.eos_token_ids` is unset — pass 151,670 as an extra eos or speech ends after one token.
|
||||
- **Real-time line = 50 speech codes/s** — NeuCodec `sample_rate 24000 / hop_length 480`, measured exactly (49.96–50.00 across three 20–25 s references). Judge any NPU/iGPU TTS number against 50. (Qwen3-TTS, the fleet's deployed arm, is 12.5 codes/s — do not conflate the two bars.)
|
||||
- Reference encode is cheap and cacheable: `librosa.load(sr=16000, mono=True)` → `[1,1,T]` float tensor → `codec.encode_code(audio_or_path=…)` (mirrors `NeuTTS.encode_reference`); CPU encoder RTF 0.28–0.38 on 8 threads. `neucodec` needs `torchtune` (`RotaryPositionalEmbeddings`) — install both `--no-deps --target <dir>` so a second torch never shadows the venv's.
|
||||
- Build the prompt by *importing* the vendor function, never by copying the template:
|
||||
`sys.path.insert(0, <neutts repo>); from neutts.neutts import NeuTTS`, then
|
||||
`NeuTTS._apply_chat_template(SimpleNamespace(tokenizer=hf_tok, input_format="text"), ref_codes, ref_text, text, None)` — works on a duck-typed `self`, so no backbone/codec load. Template: `user\n<|TEXT_REPLACE|>\nassistant\n<|SPEECH_REPLACE|>`; splice `TEXT_PROMPT_START + encode(f"{ref_text} {text}") + TEXT_PROMPT_END` at TEXT_REPLACE, then truncate at SPEECH_REPLACE and append `[SPEECH_GENERATION_START] + encode("".join(f"<|speech_{i}|>"))`.
|
||||
- Voice identity = reference codes + transcript (request-time conditioning). No trained artifact exists to preserve; packs holding only `reference.wav` require the gated encoder once, offline — **and** a transcript, which a voice pack does not carry (`manifest.json` holds provenance/crop geometry only) and a host without ASR cannot produce.
|
||||
- Full-Fusion input ceiling: 4096 tokens **total**, input + output.
|
||||
|
||||
## Status caveat (2026-10-19)
|
||||
|
||||
Staged artifacts, image build, device-access matrix, prompt contract, reference pre-encode:
|
||||
**done and reproducible.** Token generation on the NPU: **NO-GO, and the loader is the
|
||||
wall.** The device is healthy (`xrt-smi validate` PASSED incl. `throughput`,
|
||||
`RyzenAI-npu5 / aie2p / 6×8`, `pyxrt.device(0)` constructs), and `strace` shows the
|
||||
failing load reading `genai_config.json`, opening `model.onnx`, loading
|
||||
`libxrt_driver_xdna.so.2` and opening `/dev/accel/accel0` with successful GEM ioctls —
|
||||
then the RyzenAI EP **declines the graph before touching any `dd_metastate_*`**, silently:
|
||||
`GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`,
|
||||
`set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT` all emit
|
||||
zero lines. Hardlinked mirror dirs of the model tree were ruled out as shadowing. One run
|
||||
loaded and produced finite logits; byte-identical runs never reproduced it. **No token rate
|
||||
was ever measured, so quote none**, and remember `quicktest` success ≠ NPU residency (that
|
||||
build falls back to CPU silently — prove residency via
|
||||
`/sys/class/accel/accel0/device/mem_usage`).
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
name: ryzenai-oga-model-dir-and-eos-override
|
||||
description: "Debug OGA on XDNA2 when the RyzenAI EP never claims the If-rooted graph (shadow model dirs, bind-mounted configs, missing device), or a TTS/LLM graph stops on the wrong eos token."
|
||||
---
|
||||
|
||||
# OGA on XDNA2: graph not claimed, or generation stops at once
|
||||
|
||||
Two independent failure families in the AMD `onnxruntime-genai` (OGA) + RyzenAI EP stack on
|
||||
Linux XDNA2 (Strix Halo / Ryzen AI Max). Diagnose them in this order — they are separate
|
||||
gates and each has its own instrument.
|
||||
|
||||
## Gate 0 — is the NPU path alive at all?
|
||||
|
||||
Run the vendor's own instrument before touching anything else:
|
||||
|
||||
```bash
|
||||
cd /opt/rai/quicktest && RYZEN_AI_INSTALLATION_PATH=/opt/rai python quicktest.py
|
||||
# want: "Setting environment for STX/KRK" ... "Session successfully initialized" ... "Test Finished"
|
||||
```
|
||||
|
||||
It finds the device by shelling out to `lspci -nn` and grepping `1022:17f0`, so it also
|
||||
proves `lspci` is present in the container. Pass ⇒ the in-tree `amdxdna` driver and the
|
||||
extracted userspace pair correctly; stop blaming the host.
|
||||
|
||||
Separate **XRT enumeration** from **OGA's device probe** — they are not the same check:
|
||||
|
||||
```python
|
||||
import sys; sys.path.insert(0, "/opt/xilinx/xrt/python")
|
||||
import pyxrt; dev = pyxrt.device(0) # constructs => XRT sees the device
|
||||
```
|
||||
|
||||
`pyxrt.device(0)` succeeding while OGA says `No RyzenAI devices detected` means the
|
||||
refusal is OGA's own predicate, inside `libonnxruntime-genai.so` (the string lives there,
|
||||
next to `RyzenAI`, `max_length_for_kv_cache`, `library_path`). EP-side env levers worth
|
||||
pulling: `RYZENAI_EP_TRACING`, `RYZENAI_EP_PERFORMANCE_COUNTERS`, `RYZENAI_LLM_ASYNC_INIT`,
|
||||
`RYZENAI_LLM_PROFILE_INIT`.
|
||||
|
||||
## Gate 1 — `invalid model … Node (If) cond_full`
|
||||
|
||||
This is **ONNX Runtime verifying the raw graph**, which happens when no provider claimed
|
||||
it: the EP registered nothing and the CPU provider was left holding an `If` whose branches
|
||||
mismatch. It is never a corrupted model.
|
||||
|
||||
```bash
|
||||
LD_DEBUG=files LD_DEBUG_OUTPUT=/tmp/ld python loadprobe.py
|
||||
grep -a "calling init:.*ryzenai\|undefined symbol" /tmp/ld.*
|
||||
# observed: libonnxruntime_providers_ryzenai.so loaded, then
|
||||
# symbol lookup error: undefined symbol: GetProvider (fatal)
|
||||
```
|
||||
|
||||
`GetProvider` is not a dynamic symbol in the ORT cores, the shared-provider stub, or the
|
||||
EP (`nm -D` shows neither defines nor wants it), so the failure is registration, not an
|
||||
ELF mismatch. Confirm the cores are identical before chasing versions: both shipped
|
||||
`libonnxruntime.so.1.27.0` copies hash equal.
|
||||
|
||||
Separate the two gates with an explicit global-scope load before importing OGA:
|
||||
|
||||
```python
|
||||
import ctypes
|
||||
ctypes.CDLL("/opt/rai/deployment/lib/libonnxruntime_providers_ryzenai.so",
|
||||
mode=ctypes.RTLD_GLOBAL)
|
||||
import onnxruntime_genai as og
|
||||
og.Model(MODEL) # registration fixed => error MOVES to "No RyzenAI devices detected"
|
||||
```
|
||||
|
||||
An error that **moves** proves you cleared one gate and exposes the next. Silence in the
|
||||
first form means the provider list was empty.
|
||||
|
||||
### Dead ends (measured, do not retry)
|
||||
|
||||
- **Shadow model dirs never work.** Hardlink mirror, symlink mirror, and even a
|
||||
read-only single-file bind mount of a patched `genai_config.json` over the export's own
|
||||
`genai_config.json` all produce the same `If` rejection. The graph is only claimable
|
||||
when loaded from the export's own directory, and there is no config-path override in
|
||||
`0.14.0-dev`: the only env names in the shipped binaries are `GENAI_LOG_ORT_LIB`,
|
||||
`GENAI_ORT_VERBOSE_LOGGING`, `OG_ORT_LIB`. Carrying a config change therefore means
|
||||
either editing the download in place or a **single-file bind mount over the real dir**
|
||||
(`-v file.json:/spike/<model>/genai_config.json:ro`) — and if the EP still refuses, the
|
||||
gate is not the config.
|
||||
- **Bisect the image before believing "environment drift".** Run the same load probe
|
||||
against every local tag; three tags failing identically rules the image out.
|
||||
- **Rule out host state with instruments, not memory:** `fuser -v /dev/accel/accel0`,
|
||||
`dmesg | grep -i xdna`, `stat -c "%i %y"` on the model files, `df -h`, `free -g`.
|
||||
- `sysfs` class is `accel`, not `misc`: glob `/sys/class/accel*/*`. An empty
|
||||
`/sys/class/misc/accel*` is your glob, not a missing device.
|
||||
- `GENAI_ORT_VERBOSE_LOGGING=1` and `og.set_log_options(...)` produce nothing useful in
|
||||
this build; `set_log_options` is a pybind builtin (no `inspect.signature`) and rejects
|
||||
`severity` as an int. Don't burn turns trying to make it talk — use `LD_DEBUG`.
|
||||
- `strace` is absent from the RAI image and `apt` may have no mirror: `LD_DEBUG=files` is
|
||||
the loader tracer that needs nothing installed.
|
||||
|
||||
## Gate 2 — generation stops after one token
|
||||
|
||||
OGA reads its stop list from the **model** section: `model.eos_token_id`. Chat exports
|
||||
carry the chat-turn-close id (e.g. 151645) there, so the model emits it first and the
|
||||
speech/tool budget is never spent — while `search` has **no** eos key: writing
|
||||
`search.eos_token_id` is a parse refusal (`Unknown value "eos_token_id"`), not a no-op.
|
||||
|
||||
- Fix: rewrite `model.eos_token_id` to only the task's own end token (TTS: the
|
||||
speech-generation-end id), or pass it through `set_search_options` if the build accepts
|
||||
that key name.
|
||||
- A sub-millisecond `generate_next_token()` **with finite, high-confidence logits** means
|
||||
the graph ran and chose a stop token — it is not evidence of a CPU fallback.
|
||||
- Verify a token-range census against the export's `added_tokens.json`, not against a
|
||||
guessed vocabulary size: NeuCodec codes reach 65,536 entries (ids 151671…217206), and a
|
||||
census bounded at 10k silently miscounts every code as a text token.
|
||||
- A hardlinked run dir is a loaded gun: `open(dst, "w")` on a hardlink truncates the
|
||||
**original** through the shared inode. `os.unlink` the link before writing a patched
|
||||
file. Also skip `.git/` and mirror directories when walking an HF checkout.
|
||||
|
||||
## Rate measurement
|
||||
|
||||
Once a load succeeds, measure at ≥300 steps with a host-side `%CPU` sampler beside the
|
||||
container (`ps -C python -o pid=,pcpu=,rss=` every 2 s): a device-bound decode waits on
|
||||
`accel0` and stays under one core, a silent CPU fallback pegs at least one. Never report
|
||||
request #1 after a boot — graph capture makes it a warmup artifact.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: ryzenai-venv-dependency-triangle
|
||||
description: "Debug the torchao/torchtune/transformers import triangle inside an AMD Ryzen AI (ryzen_ai-*.tgz) venv under torch 2.6, including the masked \"cannot import name\" failures and the port-the-one-symbol escape."
|
||||
---
|
||||
|
||||
# Ryzen AI venv dependency triangle (torch 2.6 era)
|
||||
|
||||
Symptom that starts it all — and it never names the culprit:
|
||||
|
||||
```
|
||||
ImportError: cannot import name 'HubertModel' from 'transformers'
|
||||
```
|
||||
|
||||
The named symbol is healthy. `import torchao` alone reproduces the failure; `import transformers` alone does not. transformers imports its **torchao integration** whenever the real modelling path loads, and the failure is re-raised two frames deep by the lazy loader as a missing name.
|
||||
|
||||
## Instruments
|
||||
|
||||
Never conclude from `grep -c Symbol transformers/__init__.py` — 4.5x generates its lazy `__init__`, so an absent literal proves nothing. Get the real exception:
|
||||
|
||||
```python
|
||||
import traceback, torch, transformers
|
||||
try:
|
||||
__import__("transformers.models.hubert.modeling_hubert")
|
||||
except Exception:
|
||||
traceback.print_exc()
|
||||
```
|
||||
|
||||
That prints the truth, e.g. `AttributeError: module 'torchao.quantization' has no attribute 'Float8WeightOnlyConfig'` from `transformers/modeling_layers.py`.
|
||||
|
||||
Also probe the order matrix (`import X; from transformers import Y` for each suspect): only the poisoned order fails, which isolates the culprit without reading vendor code.
|
||||
|
||||
## The triangle (measured 2026-10-19, ryzen_ai 1.8.0, torch 2.6.0+cpu, transformers 4.57.6)
|
||||
|
||||
| Option | Result |
|
||||
|---|---|
|
||||
| torchao absent | `from transformers import HubertModel` -> OK |
|
||||
| torchao 0.7.0+cpu | `ImportError: Float8WeightOnlyConfig` (masked as missing symbol) |
|
||||
| torchao 0.8.0+cpu | imports (warns `No module named 'triton'`), still lacks `Float8WeightOnlyConfig` |
|
||||
| torchao >= 0.9 | has the symbol, requires torch >= 2.7 -> impossible in this venv |
|
||||
| torchtune, ANY version (0.6.1 … 0.1.1) | guard `torchao not installed`, or `pyarrow requires NumPy 2.0 or newer, found 1.26.4` |
|
||||
|
||||
Two independent dead ends: torchtune's package init pulls `datasets` -> **the pyarrow AMD ships in its own venv refuses that venv's own numpy 1.26.4**. So `import torchtune` is structurally dead, and torchao (the only thing that satisfies its guard) breaks transformers. **Never "just upgrade" torch/numpy: the vendor's ONNX runtime and torch build are what the NPU EP is loaded against.**
|
||||
|
||||
## Escape: port the one symbol, pin both absent
|
||||
|
||||
Ask what the third-party package is actually used for before fighting the resolver. In `neucodec` it was one symbol (`RotaryPositionalEmbeddings`, ~40 lines, copied verbatim from `torchtune/modules/position_embeddings.py`, BSD-3) used by two files. Apply it in the image with a transform that **asserts** each import line exists, so a vendor update fails the build instead of shipping a half-patched package:
|
||||
|
||||
```dockerfile
|
||||
COPY patches/rope_local.py /opt/patch/rope_local.py
|
||||
RUN python - <<'PY'
|
||||
import pathlib
|
||||
from importlib.util import find_spec
|
||||
pkg = pathlib.Path(find_spec("neucodec").origin).parent # find_spec: importing it is what is impossible yet
|
||||
(pkg / "rope_local.py").write_text(pathlib.Path("/opt/patch/rope_local.py").read_text())
|
||||
old, new = "from torchtune.modules import RotaryPositionalEmbeddings", "from .rope_local import RotaryPositionalEmbeddings"
|
||||
for name in ("bs_roformer5.py", "codec_decoder_vocos.py"):
|
||||
p = pkg / name; s = p.read_text()
|
||||
assert old in s, name + ": import line moved - re-read the vendor source"
|
||||
p.write_text(s.replace(old, new))
|
||||
PY
|
||||
```
|
||||
|
||||
Gate both absences at build time so nobody re-adds them and inherits a three-week-later mystery:
|
||||
|
||||
```dockerfile
|
||||
RUN ... import-check \
|
||||
&& python -c "import importlib.util as u; assert u.find_spec('torchao') is None; assert u.find_spec('torchtune') is None"
|
||||
```
|
||||
|
||||
Prove a ported numeric kernel with the vendor's own fixtures, not with faith: `neucodec` ships `samples/jo.wav` *and* its own encoding `samples/jo.pt`, so re-encoding the WAV and comparing integer codes is a pass/fail correctness test of the RoPE port.
|
||||
|
||||
## Rules of thumb
|
||||
|
||||
- `--no-deps` for anything adjacent to torch/onnxruntime/numpy; check the vendor pins by reading them out of the venv (`python -c "import torch; print(torch.__version__)"`), never from docs.
|
||||
- A `--dry-run` install that resolves without touching torch/numpy/onnxruntime is the licence to install *with* deps.
|
||||
- `uv pip install --target DIR` / a nested venv does **not** layer over a parent venv: a venv created from `/opt/rai/bin/python` resolves "system site-packages" to the base interpreter, and uv's resolver ignores `.pth` paths and will pull a second numpy that breaks the vendor C extensions.
|
||||
- Ephemeral `podman run --rm` probes discard everything you installed in them — build a throwaway tag (truncate the Containerfile before the failing step) instead of re-hypothesising.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: sigterm-kill-test-stub
|
||||
description: "Pin \"code kills its child process\" behavior with a bash stub player: make the stub SIGTERM-reactive via sleep&wait, log KILLED via trap; diagnose deferred-trap failures."
|
||||
---
|
||||
|
||||
Use when a test must prove production code sends SIGTERM to a spawned child (player, worker, server stub) and the test asserts the kill happened — or when such a test mysteriously fails with `KILLED` missing and wall time equal to the stub's sleep duration.
|
||||
|
||||
## The trap-during-sleep gotcha
|
||||
|
||||
A bash stub that sleeps in the **foreground** does NOT run its TERM trap when signalled: bash defers traps until the current foreground command completes. The stub dies (or logs) only after the sleep finishes — long after the test read the evidence file.
|
||||
|
||||
Symptoms:
|
||||
- Assertion on a `KILLED` marker line fails although production code demonstrably calls `child.kill("SIGTERM")`.
|
||||
- Test wall time equals the stub's full sleep (the kill "had no effect").
|
||||
|
||||
## Fix: reactive stub skeleton
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$STUB_LOG" # call record
|
||||
trap 'printf "KILLED\n" >> "$STUB_LOG"; exit 143' TERM # BEFORE the sleep
|
||||
for arg in "$@"; do case "$arg" in *.wav) cp "$arg" "$COPY" ;; esac; done
|
||||
sleep "${STUB_HOLD:-0.1}" & # background sleep…
|
||||
wait $! # …wait is interruptible
|
||||
printf 'DONE\n' >> "$STUB_LOG"
|
||||
```
|
||||
|
||||
- `sleep N & wait $!` is the idiom: a signal makes builtin `wait` return immediately, the trap runs at once, `exit` inside the trap prevents fallthrough to `DONE`.
|
||||
- Tune duration per test via the env knob (`STUB_HOLD=5`) so the child is provably still alive when the kill is triggered.
|
||||
- Put the trap BEFORE any long command; a trap registered after the sleep never guards it.
|
||||
|
||||
## Harness wiring (black-box driver pattern)
|
||||
|
||||
- Scenario: start the long work → settle (>spawn latency, <hold) → trigger the killer (shutdown event, second invocation, `/stop`) → settle → read log.
|
||||
- Expose the stub log lines on the driver's stdout (`LOG <line>` per line) so tests grep output, not files: black-box seam stays output-only.
|
||||
- Assert `KILLED` present; for interrupt-semantics also count (`kills -eq 1`) and total call lines (`PLAYS 3` = call, KILLED, call).
|
||||
|
||||
## Orphan hygiene
|
||||
|
||||
Second child still sleeping when the driver exits: spawn with `stdio: "ignore"` so the orphan holds no capture pipe — it SIGPIPEs/quiets at teardown; do not lengthen tests waiting for it.
|
||||
|
||||
## Verify before trusting
|
||||
|
||||
Run the pair separately: kill-tests green while a byte-equality test (`cp` of the payload before sleep) stays green — the reactive rewrite must not drop the copy step (it sits between trap and sleep).
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: skills-cli-retroactive-store-wiring
|
||||
description: "Retroactively register hand-copied skills with the vercel-labs skills CLI (global store + lockfile), wire omp via the agents provider (no repo symlinks), and extract authored overrides without twins"
|
||||
---
|
||||
|
||||
# Skills CLI retroactive store wiring (omp tiers)
|
||||
|
||||
Retrofit provenance for hand-copied skills and wire them for omp on Dank workstations. Executed 2026-10-18 on blankpad; doctrine doc: `omp-docs/tools/vercel-labs-skills/skill-store-wiring.md` (repo).
|
||||
|
||||
## Tier law (final design — do NOT reintroduce per-skill symlinks in the repo)
|
||||
| Tier | Path | Owner |
|
||||
|---|---|---|
|
||||
| Store | `~/.agents/skills/` + lockfile `~/.agents/.skill-lock.json` (v3) | `skills` CLI |
|
||||
| Authored | `<repo>/agent/skills/` real dirs | Frank, branch → PR |
|
||||
| Managed | `~/.omp/agent/managed-skills/` | `manage_skill` only |
|
||||
|
||||
omp has NO agent id in the skills CLI (79-agent registry in `dist/cli.mjs`; `-a pi` writes `~/.agents/skills` too — pi detector keys on `~/.pi/agent`). omp reads the store DIRECTLY via its `agents` provider (`skills.enableAgentsUser`, default true; check with `read cfg://skills`). Precedence: authored (native, 100) > skillshare > ... > agents provider > managed (5). Identical copies collapse by realpath; differing same-named copies survive namespaced `agents/<name>` with a startup warning — so when authoring an override, REMOVE the store twin (`npx -y skills remove <name> -g`; prunes lockfile too) or `skills update -g` resurrects a twin.
|
||||
|
||||
## Procedure (retroactive registration)
|
||||
1. List the repo/live skill dirs to register; identify true sources (per-skill — e.g. `find-skills` is `vercel-labs/skills`, not `mattpocock/skills`).
|
||||
2. Drift check FIRST, per skill, before anything moves: `diff -rq <live>/<name> ~/.agents/skills/<name>` after canonical install; abort-and-report on drift (byte-identical = clean cutover).
|
||||
3. Canonical install (scratch dir with a package.json; global scope): `npx -y skills add <owner>/<repo> --all -g` (+ per extra source `npx -y skills add vercel-labs/skills --skill find-skills --agent universal -g`). Artifacts: store dirs + `~/.agents/.skill-lock.json`; verify `jq '.skills|length'`.
|
||||
4. Authored extraction (only for skills the user will customize): copy dir into repo `agent/skills/<name>/`, add an "Authored override" note after frontmatter, then `npx -y skills remove <name> -g` and confirm `jq '.skills|has("<name>")' == false`.
|
||||
5. Clear repo copies of store-managed skills (repo records authored overrides ONLY). Live cutover after merge: move old dirs to /tmp snapshot, `git merge --ff-only origin/main` in the live checkout.
|
||||
6. `skills ls` from a repo containing `agent/skills` mislabels skills as Eve project skills (path collision) — attribution noise, ignore.
|
||||
7. Fresh-box recipe (three commands, no wiring): add `--all -g`, add find-skills, `remove grilling -g`.
|
||||
|
||||
## Gotchas
|
||||
- Eve/Codex `agents/openai.yaml` files ride along from upstream; omp ignores them — prune from authored dirs (they get swept off disk elsewhere, leaving phantom ` D` in git status).
|
||||
- `skills remove` prunes only global scope if `-g` given; check lockfile after.
|
||||
- Mid-run doc readers (subagents) may see pre-cutover trees — send them a correction message, don't let stale advice ship in issue bodies.
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: strix-halo-npu-readiness-census
|
||||
description: "Engine-agnostic census of whether an AMD Strix Halo / Ryzen AI host can run anything on its XDNA2 NPU: driver bind, firmware, device perms, IOMMU, distro XRT, missing AIE compiler, fabric clock, RAM headroom — plus the strace test that separates a dead device from a runtime that declines the graph."
|
||||
---
|
||||
|
||||
Use when asked "can we run <engine/model> on the Strix Halo / Ryzen AI NPU?" for a runtime that is **not** halogen (halogen has its own gate + vendor catalog: use `halogen-npu-feasibility-census` for that). This answers only: is the *host* able, and what is missing.
|
||||
|
||||
## Why it is usually the wrong door first
|
||||
|
||||
XDNA2 executes **precompiled AIE partition programs** loaded through XRT → `amdxdna`. There is no generic tensor dispatch: an engine built on ggml/llama.cpp (or any GGUF loader) cannot "enable" the NPU — the port must be a re-export + vendor compile, not a flag. So pair this census with the accelerator the same engine *can* select by build flag (Vulkan/HIP on the gfx1151 iGPU) before proposing NPU work. On Dankclaw the iGPU flag was ours to flip all along (`-DGGML_VULKAN=OFF` written out in the fleet's own engine Containerfile) — check the engine's own build file before treating the NPU as the only hardware door.
|
||||
|
||||
## Census (all read-only, over ssh from the operator box)
|
||||
|
||||
```bash
|
||||
ssh <host> '
|
||||
cat /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l # iommu=pt required; groups==0 => IOMMU off => NPU dead
|
||||
lsmod | grep -i amdxdna # in-tree driver
|
||||
sudo dmesg | grep -iE "xdna|npu" | tail # "Initialized amdxdna_accel_driver <ver> for 0000:c6:00.1"
|
||||
lspci -nn | grep -iE "npu|processing" # Strix Halo = 1022:17f0
|
||||
ls /sys/bus/pci/drivers/amdxdna # must list the BDF
|
||||
ls -l /dev/accel # accel0, root:render 660
|
||||
id -nG <deploy-user> | tr " " "\n" | grep -E "render|video" # rootless podman needs render, else --device fails at create
|
||||
find /usr/lib/firmware -iname "*npu*" | head # firmware is /usr/lib/firmware/amdnpu/<devid>_<rev>/npu.sbin*.zst — match the PCI rev
|
||||
dpkg -l | grep -iE "xrt|xdna"; ls /opt/xilinx/xrt/lib 2>&1 # distro libxrt-npu2 = XDNA2 runtime; vendor /opt/xilinx/xrt usually absent
|
||||
apt-cache policy libxrt2 libxrt-npu2 | grep Candidate
|
||||
apt-cache search onnxruntime | head # distro ONNX RT exists; the Ryzen AI EP does not
|
||||
python3 -c "import onnxruntime" 2>&1 | tail -1 # absence of the EP + of ryzenai-compile is the real blocker
|
||||
for c in /sys/class/drm/card*/device; do echo "-- $c"; cat $c/power_dpm_force_performance_level 2>&1; cat $c/pp_dpm_fclk 2>&1 | tr "\n" " "; echo; done
|
||||
nproc; free -g | head -2 # RAM headroom + thread count the engine will pick
|
||||
'
|
||||
```
|
||||
|
||||
## Reading it
|
||||
|
||||
- Driver bound + `accel0` + firmware dir + `iommu=pt` ⇒ **kernel side is ready**; do not re-litigate "is the NPU disabled". A missing device node is *not* the modern failure mode.
|
||||
- `auto` on `pp_dpm_fclk` / `power_dpm_force_performance_level` ⇒ fabric clock **unheld**. GPU+NPU concurrently with a moving fclk hangs these hosts and corrupts NPU results; any NPU plan must pin the top fclk step (or run in a container holding it) first.
|
||||
- No `ryzenai-compile`/ONNX Ryzen AI EP ⇒ nothing on the host can produce a partition program. That is the enablement delta, and it is a toolchain install + a model-specific export/quantize/compile project, not a config change.
|
||||
- `free` headroom matters: NPU partitions want a few GB of host RAM each and can log benign `Failed to alloc dev bo memory, ret -28` under pressure.
|
||||
|
||||
## Device healthy, load still fails? Classify before blaming the hardware
|
||||
|
||||
Two different failures, one call apart. Run the load under `strace -f -e trace=openat`:
|
||||
|
||||
- **Never opens `/dev/accel/accel0`** ⇒ pre-device: loader/shim/env (see `ryzenai-npu-oga-linux-run` — the `Generic Failure` in ~2 ms is this class).
|
||||
- **Opens `/dev/accel/accel0`, GEM ioctls succeed, then the EP declines** ⇒ post-device: the *runtime* rejects the graph. The device is innocent; stop the hardware investigation and fix the last file it opened (`model.onnx` = graph rejected; `dd_metastate_*` = partition artifact problem).
|
||||
|
||||
Ground truth that the device itself is sound, inside the container: `xrt-smi examine` (expect `RyzenAI-npu5 | aie2p | 6x8` + firmware version) then `xrt-smi validate` — `PASSED` incl. `throughput`. Needs `libboost-filesystem1.83.0` + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`); boost is required by `xrt-smi` only, never by the OGA/EP path. Vendor `quicktest` printing success does **not** prove NPU residency (that build falls back to CPU silently) — sample `/sys/class/accel/accel0/device/mem_usage` during generation instead.
|
||||
|
||||
When the vendor runtime declines silently, exhaust its own levers and then say so: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT` all produced zero lines on the 2026-10-19 NuTTS-Air decline. Rule out shadowed artifacts by moving hardlinked mirror copies out of the scanned tree (glob-based sibling scans) — if the failure is byte-identical afterwards, that theory is dead. **If a load ever succeeded once and the byte-identical command never reproduces it (files pristine by inode+mtime), that is a NO-GO, not a lead: no rate may be quoted for a run that cannot be repeated.**
|
||||
|
||||
## Laws / traps
|
||||
|
||||
- Read state only. Never probe or load the NPU beside a live inference service — stop + mask the service first (fleet benching policy); this host class has a precedent of a concurrent GPU/NPU experiment needing a power cycle.
|
||||
- `sudo podman ...` finds nothing (per-user rootless storage); run podman as the deploy user.
|
||||
- `/dev/accel/accel0` existing does not mean a context/PASID can be created; with IOMMU off the failure is at context creation, not at the node.
|
||||
- Rootless containers cannot read `accel0` (`root:render 0660`) by any documented flag combination — rootful, or a world-writable udev rule.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: tts-endpoint-parameter-capability-audit
|
||||
description: "Determine which request parameters a fleet TTS endpoint actually honors by probing façade and engine separately, with a nondeterminism control, instead of trusting docs or byte diffs."
|
||||
---
|
||||
|
||||
Use when asked whether a TTS (or other OpenAI-compatible shim) endpoint supports a parameter — mood, tone, instructions, speed, seed — or when a request "seems to" change the audio. Docs and single-request comparisons both lie; audit the layers.
|
||||
|
||||
## Why the naive checks fail
|
||||
- **Byte-size diffs are meaningless.** Synthesis is nondeterministic: three byte-identical requests to frank-tts came back 107564 / 134444 / 111404 bytes (~±15 % length). "The mood file differs from the baseline" proves nothing.
|
||||
- **HTTP 200 proves nothing.** A shim that rebuilds its own payload answers 200 to `speed:"banana"` because it never looks at the field.
|
||||
- **A façade can drop fields the engine supports**, so testing only the published port under-reports capability.
|
||||
|
||||
## Procedure
|
||||
1. **Read the shim's parse block** (which request keys it reads) — not its docstring:
|
||||
`ssh -o BatchMode=yes dankclaw 'podman exec frank-tts grep -nE "body\.get|payload = |json\.loads" /app/app.py'`
|
||||
Whatever keys appear there (plus what it forwards in its own `payload = {...}` dict) is the real published contract.
|
||||
2. **Enumerate the engine's key table from the binary** (works with no `strings` in the image):
|
||||
`podman exec <engine> sh -c 'grep -aoE ".{60}instructions.{60}" /app/tts-server | head -2'`
|
||||
Adjacent tokens come out as a run — that run IS the accepted-key list (`...instructions response_formatspeedseedmax_new_tokensrepetition_penalty subtalker_top_k subtalker_temperature ...`). Also `podman exec <engine> /app/tts-server --help`, and `podman inspect <c> --format "{{json .Config.Entrypoint}}"` to find the binary.
|
||||
3. **Reach the engine directly** when its port is unpublished: `TTS_UPSTREAM` in `podman inspect <shim>` gives the real in-network address (`http://frank-tts-engine:8880`, not `host.containers.internal`). No curl/python on the host? Pipe a script into the container that has one: `podman exec -i frank-tts python3 - <<'PY' ... PY`.
|
||||
4. **Control for nondeterminism before claiming an effect**: repeat the identical request twice as the control; only compare conditions separated by far more than the jitter, and prefer an error response (a 400 naming the unsupported flag) as the decisive evidence.
|
||||
|
||||
## Findings for the dank fleet (verify before trusting — they move)
|
||||
- Published façade `:7772` reads only `model`, `input`, `voice`/`voice_id`, `response_format`; it rebuilds the payload, so `speed` and every unknown field are dropped before the engine.
|
||||
- Engine (`frank-tts-engine`, qwentts.cpp, qwen-talker-1.7b-**base**, Vulkan) accepts `instructions`, `speed`, `seed`, `max_new_tokens`, `repetition_penalty`, `subtalker_top_k/_temperature/_top_p`, but answers `400 "--instruct is not supported for base models"` for `instructions` on the base checkpoint. `seed` does not make output reproducible.
|
||||
- Therefore tone/mood must be produced in the **text layer** (prompt directives), not the TTS request; and `speed` needs a two-line façade fix if tempo control is ever wanted.
|
||||
|
||||
## Verification habits
|
||||
- POST with a wrong *type* (`speed:"banana"`) to learn whether anything validates, and with a bogus `voice` to see the error shape (`available_voices` came from the engine, not the shim).
|
||||
- Plain GET of `/v1/audio/voices` for the roster.
|
||||
- Read-only probes of the fleet are always allowed; synthesis requests are transient GPU work, keep them to a handful.
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: typescript-fleet-repo-bootstrap
|
||||
description: "Initialize a TypeScript monorepo (Bun workspaces + Turborepo + Biome + Express/Vite + quadlet dev DB + Storybook/Playwright gates) on Frank's fleet, with the traps that cost hours at franksign init."
|
||||
---
|
||||
|
||||
# TypeScript fleet repo bootstrap
|
||||
|
||||
Bootstrap a fleet-style TS monorepo the way franksign was initialized (2026-10-20):
|
||||
Bun workspaces + Turborepo + Biome + Express 5 on Node + Vite/React SPA + quadlet
|
||||
dev Postgres + Storybook-as-component-test + prod-shaped Playwright e2e, gated by a
|
||||
single `tests/run.sh` seam runner (serverAppConfigs pattern).
|
||||
|
||||
## Shape
|
||||
|
||||
```
|
||||
apps/ homepage (zero-JS), web (Vite+React SPA), api (Express, single origin: /, /app, /api, /health)
|
||||
packages/ domain libs (each: src/*.ts, tests colocated, tsc -p tsconfig.build.json)
|
||||
templates/ data the API serves
|
||||
tests/ run.sh + seams/*.bats + e2e/*.spec.ts + lib/ guards (.mjs)
|
||||
infra/ quadlet/*.container + setup-dev.sh (idempotent bootstrap)
|
||||
<repo>-docs/ OKF v0.2 KB: index.md, log.md, read-only/ (agent-off-limits)
|
||||
AGENTS.md git law (trunk-based, --no-ff, explicit merge OK) + TDD tiers + docs law
|
||||
HANDOFF.md at repo base for worktree-based future sessions
|
||||
```
|
||||
|
||||
## Non-obvious traps (all measured on Frank's box, Node 26 / bun 1.4.3 / playwright 1.64)
|
||||
|
||||
- **Playwright `testDir` resolves against process CWD, not the config file.**
|
||||
Pin the seam to invoke playwright with `cd` into the spec dir.
|
||||
- **A `testMatch` regex with a leading `^` silently matches nothing** — Playwright
|
||||
tests it against prefixed paths; "No tests found", zero diagnostics. Anchor only
|
||||
the trailing `$`. And keep raw `/` out of the pattern if a guard script parses
|
||||
the config literals.
|
||||
- **`tsc` with `noEmit: false` but no `rootDir`/`outDir` emits `.js` strays into
|
||||
`src/`** while package `main` points at `dist/` — the strays resurface on every
|
||||
build. Pin `rootDir: "src", outDir: "dist", declaration: true` in every
|
||||
tsconfig.build.json.
|
||||
- **`bunx <tool>` ignores the workspace install** and pulls its own version (observed
|
||||
with biome and playwright). Git hooks MUST call `./node_modules/.bin/<tool>`.
|
||||
- **Node 26 rejects `--env-file-if-set`** (no "if-set" variant on that build);
|
||||
use `--env-file=<file>` — a shell-set env var still wins over file values, so
|
||||
`PORT=3100 node --env-file=.env dist/cli.js` overrides cleanly.
|
||||
- **Quadlet resolves `EnvironmentFile=` relative to the UNIT directory**, not the
|
||||
repo: `setup-dev.sh` must link the env file beside the unit in
|
||||
`~/.config/containers/systemd/`. Generated quadlet units cannot be
|
||||
`systemctl --user enable --now`'d — the generator's `default.target.wants`
|
||||
symlink IS the enablement; plain `start` is enough.
|
||||
- **`gitignore` rule `.env.*` swallows `.env.example`** — add the `!.env.example`
|
||||
negation when you write the first ignore file.
|
||||
- **Gitea 1.27.3 API create-repo with a JSON body can answer 422 `[Name]: Required`**
|
||||
even when the field is present; probe `/api/v1/repos/<owner>/<name>` first — it
|
||||
may already exist (empty = ready for `git push -u origin main`).
|
||||
- **Biome 2.x**: `biome migrate` needs `--write` (without it, it only prints the
|
||||
plan) and migrates `rules.recommended` → `rules.preset`; `check --write` skips
|
||||
`useTemplate`/`noNonNullAssertion` (unsafe) — prefer rewriting `!` assertions to
|
||||
named lookups so `--unsafe` never touches tests.
|
||||
- **Grafana-grade gate law** (from serverAppConfigs): every e2e spec must be routed
|
||||
by a `projects[].testMatch` or it silently never runs — ship a scaffold-guard
|
||||
script that fails the seam on an orphan spec.
|
||||
|
||||
## Gate wiring
|
||||
|
||||
`tests/run.sh` (bats seams: build, unit, docs-lint, homepage budget, quadlet,
|
||||
storybook build, e2e) ← pre-push hook; pre-commit = secret-regex scan on staged
|
||||
diff + `./node_modules/.bin/biome check` on staged code. e2e seam boots the
|
||||
prod-shaped stack (built assets + Express + quadlet DB on a dedicated port,
|
||||
health-gated with `curl` before Playwright starts).
|
||||
|
||||
## Session close-out
|
||||
|
||||
log.md entry with `(cwd · session)` footer; HANDOFF.md updated; origin on Gitea
|
||||
owner `dank`; PR bodies carry the attribution footer.
|
||||
@@ -0,0 +1,187 @@
|
||||
---
|
||||
name: vault-one-fact-one-owner-dedupe
|
||||
description: "\"Collapse per-payload/per-attempt vault pages into one owner page per endpoint without losing a fact: port-before-cut orphan census (corpus must exclude the file under test), doubled-page repair, trash-aware tools, delegated-write verification, honest per-area line accounting, index regeneration, and the merge-resolution order.\""
|
||||
---
|
||||
|
||||
# One fact, one owner: dedupe the vault without losing a fact
|
||||
|
||||
Use when a repo has accumulated one page per attempt/era (runbooks, bench logs, swap
|
||||
narratives) and the law is **one page owns one thing**. Applies to
|
||||
`serverAppConfigs-docs/` (Obsidian vault, `AGENTS.md` → *Documentation Policy*), but the
|
||||
mechanics are general.
|
||||
|
||||
## 1. Decide ownership before touching files
|
||||
|
||||
- Endpoint/service page owns: current shape, **Eras** table (window | payload | runtime |
|
||||
shape | replaced because | rollback), **Laws**, wire contract, files.
|
||||
- Technique page owns the mechanism and the arithmetic — **no per-endpoint throughput
|
||||
table or boot ledger**.
|
||||
- `log.md` owns incidents and laws only. A swap is an Eras row, never a log entry.
|
||||
- `AGENTS.md` carries operator-critical numbers + cross-border laws; everything else links.
|
||||
- A **retired** endpoint still gets one locator page: era windows quoted verbatim from each
|
||||
era page's own banner, no numbers restated, and any date conflict recorded as an open
|
||||
question instead of silently resolved. Never pick a winner between two primary sources.
|
||||
|
||||
## 2. Single-writer rule (the mistake that costs the most)
|
||||
|
||||
Facts get **ported into the owner page before they are cut** anywhere else. So while any
|
||||
subagent may port facts, only ONE writer may touch the owner pages. Spawn children with an
|
||||
explicit `DO NOT EDIT <owner pages>` and require a `needsPort` list in their report instead
|
||||
of letting them append. Send the narrowing message early — a child already editing them has
|
||||
to be reconciled by hand.
|
||||
|
||||
## 3. Port-before-cut census (proves nothing was lost)
|
||||
|
||||
For every deleted line, extract distinctive keys and check them against the vault minus the
|
||||
file being rewritten. Any line with **no** surviving key is an orphan → port, or accept
|
||||
deliberately and say so. Use `scripts/port-census.py <new-dir> <git-HEAD:path|file>` (numbers
|
||||
with ≥ 4 int digits, decimals, flags, paths, backticked spans, `--`, ALL-CAPS, or ≥ 2 hyphens).
|
||||
|
||||
Three calibration traps:
|
||||
- **Prose words are not facts.** A hyphen or trailing period inside a word (`long-standing`,
|
||||
`deployed:`) was counted as a tracked token while real identifiers hid behind it; that noise
|
||||
buried 300 real lines under 979 fake ones. Strip punctuation, require fact shape.
|
||||
- **The corpus decides the question.** Corpus = the new file only → "what did this rewrite
|
||||
drop?" Corpus = whole vault → "is the dropped fact orphaned repo-wide?" Ask one at a time.
|
||||
- **The corpus MUST exclude the file under test, and you must assert it.** A census pointed at
|
||||
the directory that *contains* the page proves the page is held by itself: ten era pages
|
||||
reported `0 sole-record lines`, which reads as "pure duplication, trash it". With the file
|
||||
excluded, the same pages held **14–63 sole-record lines each** — campaign ledgers that exist
|
||||
nowhere else. A tool's own docstring promise ("excluding it") is not a check: print the
|
||||
corpus size, assert the target is absent, and treat `0 unheld` on a long page as a tool bug
|
||||
until proven otherwise. A mechanical census is a scout: its verdict is a lead, not a finding.
|
||||
|
||||
Also grep `AGENTS.md` separately — a fact it carries is not an orphan.
|
||||
|
||||
## 4. When a page is the same page twice
|
||||
|
||||
Signature: a page whose `wc -l` is ~2× its `HEAD` size, with a **second frontmatter block in
|
||||
the body** whose closing `---` is glued onto the end of a body/timestamp line. Cause chain: a
|
||||
frontmatter rewrite that re-reads the file it just wrote (so its own output becomes the body),
|
||||
then a fence-repair pass keyed on `line == "---"` that cannot see a glued fence and inserts a
|
||||
fence mid-page, so the next page-tail becomes a new document.
|
||||
|
||||
Un-double rule that worked: find the first frontmatter's close (accepting a glued fence), then
|
||||
the first `type:` line after it that has more keys (`title|description|tags|timestamp`) behind
|
||||
it — that is the second copy, and a stray `type:` in a code block has no key run. Keep the copy
|
||||
whose body is a **superset** (compare after stripping the dropped copy's own frontmatter),
|
||||
un-glue the fence, and prove `0` census losses before writing. Refuse to guess: if the halves
|
||||
differ materially, stop.
|
||||
|
||||
## 5. `.AGENTTRASH` is not vault content
|
||||
|
||||
Every tool that walks the vault must exclude it — **both the scan set and the resolution
|
||||
index**. The link checker resolving links *into* the trash turns "live page points at a
|
||||
deleted page" into a false PASS, which is worse than a false alarm. The `docs_completeness`
|
||||
seam's `find` must carry `-not -path '*/.AGENTTRASH/*'` in both walks, or a trashed page
|
||||
demands an index row and a trashed page can satisfy the docs requirement for a live service.
|
||||
|
||||
Trash is gitignored; move with plain `mv` into the dated dir. `git mv` into an ignored path
|
||||
stages a rename the next agent cannot see (`git restore --staged <src> <dst>`). Never `rm`,
|
||||
never `git clean -f`.
|
||||
|
||||
## 6. Rewire + regenerate indexes mechanically
|
||||
|
||||
```bash
|
||||
grep -rnoE "\[\[(page-a|page-b|...)\]\]" --include="*.md" serverAppConfigs-docs | grep -v AGENTTRASH
|
||||
grep -rnoE "\((page-a|page-b|...)\.md\)" --include="*.md" serverAppConfigs-docs | grep -v AGENTTRASH
|
||||
```
|
||||
|
||||
Both wikilinks and markdown path refs must be repointed (Research `See Also` lists and bench
|
||||
pages are where path refs hide). Then **regenerate** index rows with a script, never by hand.
|
||||
Generator bugs to check for: presence tested as a **bare stem substring** — a page whose name
|
||||
prefixes another's (`frank-medium` inside `frank-medium-rocmfpx`) then silently never gets a
|
||||
row (test the wikilink form `[[<stem>]]`); and **no dead-row sweep**, so deleting a page leaves
|
||||
an index row that fails the link check — the generator must drop rows for absent pages too.
|
||||
|
||||
## 7. Enforce the law with a seam
|
||||
|
||||
`tests/docs_completeness.bats`: derive the live set instead of hardcoding it — an endpoint is
|
||||
live when `scripts/` holds **both** `<name>.env` and `<name>.service`; fail if any
|
||||
`<name>-*.md` page exists, and guard against a vacuous derivation (`derived -ge 3`). A
|
||||
re-created era page then cannot pass tests. For the length hint, prefer **derived** exemptions
|
||||
(live endpoint, or a `retired-endpoint` tag: a campaign ledger that exists nowhere else is
|
||||
owned by the page that measured it) plus one documented per-page opt-out; never let a size
|
||||
ceiling force a ledger to be split across two owners.
|
||||
|
||||
Splitting an over-long page to satisfy a ceiling is usually line-NEGATIVE: one measured split
|
||||
turned a 700-line page into 488 + 306 across five files (+23 for the folder net of stub
|
||||
deletions), because frontmatter blocks, pointer sections and connective prose are charged per
|
||||
file while the facts move for free. Roll the split back, carry `okf-exempt: length` with the
|
||||
reason, and keep the stub pages deleted.
|
||||
|
||||
## 8. Edit-tool hazard (costs the most time)
|
||||
|
||||
Body rows beginning with `-` or `+` are op markers. The documented `+-` escape **writes a
|
||||
literal `+`** — do not use it. Safe rewrites inside an existing bullet: keep the `- ` opener
|
||||
line untouched and PUT only the indented continuation lines, or convert the note to a nested
|
||||
bullet whose body lines start with two/four spaces. If a wrapped `[[Two Word Page]]` shows up
|
||||
as an unresolved link, the target was hard-wrapped across lines — unwrap it.
|
||||
|
||||
A `PUT` that spans a multi-line statement can silently drop neighbouring assignments
|
||||
(auto-repaired boundary echo); after editing a constants block, `grep -n "^[A-Z_]* = "` and
|
||||
run a syntax probe before executing.
|
||||
|
||||
## 9. Delegated writes: verify bytes, never narrative
|
||||
|
||||
Measured in one campaign, all four failure modes live:
|
||||
- A subagent killed by the runtime limit yielded a 495-line owner page, a verified era table
|
||||
and a byte census — **the file did not exist** and its table was self-referential.
|
||||
- Another wrote its output to disk and died without reporting, leaving a rewritten `log.md`;
|
||||
the census showed it had dropped laws held nowhere else.
|
||||
- Verdict tables claimed "0 sole records / no contradictions" while missing the contradiction
|
||||
sitting in their own evidence row, and invented bullet IDs — one renamed a bullet to the
|
||||
**opposite** claim.
|
||||
- A scout called accurate docs "WRONG" because the docs said `wip: "down"`.
|
||||
|
||||
Rules: after ANY delegated write run `wc`/`ls` + the orphan census yourself; re-grep every
|
||||
claimed holder before cutting a line; treat `cancelled`/timed-out yields as unverified no
|
||||
matter how concrete they read; and re-read the source bullet before trusting a table's ID.
|
||||
|
||||
## 10. Merge order (a long-lived branch under a moving main)
|
||||
|
||||
1. Commit your work in atomic groups **before** merging (a dirty tree cannot merge cleanly).
|
||||
2. Inspect the incoming commits: `git log --oneline --stat HEAD..origin/main`. If they modify
|
||||
a page you deleted, **port their additions into the owner page**, then merge and resolve
|
||||
the modify/delete as `git rm`.
|
||||
3. Resolve a content block with `write conflict://<id>` (`@ours` / `@theirs` / explicit text),
|
||||
then **`git add` the file** — resolving content does not stage it.
|
||||
4. **Re-probe live state after the merge, before you write any number.** A campaign that
|
||||
reported fleet shapes at the start was made stale by a mid-campaign reslice on main: probe
|
||||
`/slots` and `tr '\0' '\n' < /proc/$(pidof <engine>)/cmdline` again at the end, and let the
|
||||
live answer beat both your earlier probe and the docs.
|
||||
5. `git * --no-verify` is forbidden here. Sweep dead paths in `AGENTS.md`
|
||||
(`grep -oE "[A-Za-z0-9._-]+\.md" AGENTS.md | sort -u`, then `find` each; hard-wrapped names
|
||||
are false positives — confirm with `ls`).
|
||||
|
||||
## 11. Verify, open, land
|
||||
|
||||
- `./tests/run.sh` (full suite) + `python3 scripts/check-vault-links.py serverAppConfigs-docs`
|
||||
→ must print `0 unresolved links`, and the OKF linter `0 violations`.
|
||||
- Env trims: prove value-neutrality mechanically (`git show HEAD:scripts/<svc>.env | grep -E
|
||||
'^[A-Z_]+=' | sort` vs working copy, diff empty). **`scripts/<svc>.env` is a deploy-planner
|
||||
path**: a comment-only trim makes the next service-scoped deploy restart a healthy endpoint.
|
||||
- Open the PR through the API with a 600-perm curl config (managed skill
|
||||
`gitea-api-pr-operations`), end the body with the `cwd:` / `session:` footer, then
|
||||
`xdg-open <PR-URL>` the instant it exists. Never merge without Frank's current explicit OK.
|
||||
Push with an explicit refspec (`HEAD:refs/heads/<pr-branch>`) — a worktree's own branch name
|
||||
is not the PR branch, and `git push origin HEAD` opens a stray branch.
|
||||
- Docs-only change ⇒ **no deploy**. Fast-forward the deploy checkout only
|
||||
(`git -C ~/Projects/serverAppConfig merge --ff-only origin/main`).
|
||||
- Say what was NOT done, in the PR body: soft lint tallies left (missing `## Citations`),
|
||||
conflicts left standing, repo config that was never installed on a host.
|
||||
|
||||
## 12. Report line counts per area, never one net
|
||||
|
||||
"Compaction" measured against the merge-base **per area**: docs / `scripts/` / `AGENTS.md`.
|
||||
A single headline number hides that tooling and prose move in opposite directions — one
|
||||
campaign reported "−1,849 vault lines" while the vault was flat and the growth was the two
|
||||
enforcement scripts that made the law mechanical. Expect a mature one-owner vault to land
|
||||
**docs flat**: the reductions available are deletions of *restatement* (duplicated pages,
|
||||
transient deploy chatter in `log.md`, `AGENTS.md` bullets whose numbers an endpoint page owns)
|
||||
and they are roughly cancelled by the cost of the citations the verification pass adds. When
|
||||
asked to shrink further, show the menu — drop the tools, drop the citations, or accept content
|
||||
loss — instead of inventing cuts. `log.md` compaction rule that survived census: drop a line
|
||||
only when it matches a transient shape (`/health`, `deployed`, `restart`, `systemctl start`)
|
||||
**and** every fact token in it is held by an owner page; a token-held line can still be the
|
||||
only record of *when* an incident happened.
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792441535971
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792441746823
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792443092036
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792440454430
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792532762570
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792368375152,
|
||||
"lastConsolidatedAt": 1792439707623,
|
||||
"lastResult": {
|
||||
"at": 1791599865775,
|
||||
"sessions": 1,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792359704968
|
||||
"lastConsolidatedAt": 1792526055797
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792359739617
|
||||
"lastConsolidatedAt": 1792469747088
|
||||
}
|
||||
|
||||
+59
-107
@@ -1,118 +1,70 @@
|
||||
## Runtime Stack
|
||||
## Subagent routing (harness law, 2026-10-18)
|
||||
|
||||
**llama.cpp** is the inference runtime for all `frank-zealot`-class endpoints. The `frank-zealot` service runs llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build, under a root systemd unit.
|
||||
**No subagent class may route to `frank-archon`** — the operator session lives there and
|
||||
the endpoint runs 3 sequences (`--max-num-seqs 3`), so `task.maxConcurrency + 1` children
|
||||
eat the parent's own KV slots. Violation cost: three doc children + parent = 4 sequences
|
||||
on 3 slots, children ran 2 h 27 m / 2 h 45 m, the interactive session starved.
|
||||
Routing and bounds live in `~/.omp/agent/config.yml` (`task:` block — the owner; verified
|
||||
with `read cfg://task/agentModelOverrides`): heavy writers (`task`, `scout`) → frank-dragoon
|
||||
(2 slots, idle, slow card ⇒ slice work smaller), bursty readers (`reviewer`, `sonic`,
|
||||
`designer`, `librarian`) → frank-zealot (2 slots, also the compaction hub and every
|
||||
tiny/smol/plan/commit/memory role). Caps: `maxConcurrency: 2`, `maxRuntimeMs: 1800000`,
|
||||
`softRequestBudget: 120`, `maxRecursionDepth: 1` (children never spawn children).
|
||||
`cfg://.../save` approvals auto-deny after a hardcoded 10 s dialog — after Frank has
|
||||
approved a change, edit `config.yml` directly and validate the YAML; a running session
|
||||
picks global-config edits up at once. Full procedure: managed skill
|
||||
`bounding-subagent-runtime`.
|
||||
|
||||
## Model Inventory
|
||||
## Documentation law (2026-10-18)
|
||||
|
||||
### frank-zealot (Dankclaw: 192.168.0.143:7771)
|
||||
**One fact, one owner.** The `serverAppConfigs` repo owns every fleet fact;
|
||||
`AGENTS.md` restates only what a deploy gate or the harness needs and otherwise
|
||||
links. `serverAppConfigs-docs/` (Obsidian vault) carries the lore: one page per
|
||||
endpoint (`[[frank-archon]]`, `[[frank-dragoon]]`, `[[frank-zealot]]`) holding
|
||||
Current shape, **Eras** (one row per payload swap: dates, payload, runtime,
|
||||
shape, why replaced, rollback), Laws, wire contract, files. `log.md` records
|
||||
incidents and laws only — no per-swap ledgers, no bench tables. A retired
|
||||
payload is an Eras row, never a page of its own. `scripts/<service>.env` beats
|
||||
any page: where they disagree, the page is the bug.
|
||||
|
||||
**Ternary-Bonsai-2-27B-PQ2_0.gguf** (prism-ml/Ternary-Bonsai-2-27B-gguf)
|
||||
## Model swaps (2026-10-18)
|
||||
|
||||
- Runtime: llama.cpp
|
||||
- Format: PQ2_0 quantized GGUF
|
||||
- 27B parameter class
|
||||
- Vision: native BF16 mmproj projector for image input
|
||||
- MTP: OFF (no `nextn` tensors in checkpoint)
|
||||
- Context: 256k per-slot window (`--ctx-size 401408 --parallel 2` = 200,704 tokens effective)
|
||||
- Batch: 4096 / ubatch 2048 (CUDA optimum)
|
||||
- Log verbosity: 4
|
||||
- Ports: 7771 on Dankclaw
|
||||
Swap the model, don't instrument the swap: edit the trio
|
||||
(`scripts/<svc>.env` + serve script + unit) → `./tests/run.sh` → PR →
|
||||
`deploy.sh <host> --service <svc>` → `/health` + one chat / tool-call / vision
|
||||
probe → done. **No benchmarks, A/Bs, ladders, or acceptance tables unless
|
||||
Frank explicitly asks for data.** If a payload misbehaves, revert the commit and
|
||||
redeploy. When a bench IS ordered, the GPU-safety procedure (stop + mask, no
|
||||
scratch ports, never `systemctl disable`) is `AGENTS.md` → *Model swaps*
|
||||
and *Deployment completeness*.
|
||||
|
||||
### frank-dragoon (Dankclaw: 192.168.0.143:7770)
|
||||
## Where facts live
|
||||
|
||||
**Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated** (`.hgn` format)
|
||||
|Need|Owner|
|
||||
|---|---|
|
||||
|Topology, ports, payload, shape|repo `AGENTS.md` → Fleet topology; `scripts/<svc>.env` is the authority|
|
||||
|Endpoint lore, eras, laws, rollback|vault `serverAppConfigs-docs/LLM/<endpoint>.md`|
|
||||
|Harness model entries (`contextWindow`, `maxTokens`, `reasoningContentField`, `modelRoles`)|`~/.omp/agent/models.yml`|
|
||||
|Incidents and operational laws|vault `serverAppConfigs-docs/log.md`|
|
||||
|Service/infra pages|vault `Dankclaw/`, `Dankserver/`, `Infrastructure/`, `Architecture/`|
|
||||
|
||||
- Runtime: halogen-flash-server 0.16.2 (digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`)
|
||||
- Format: v2 `.hgn` (flash-server exclusive)
|
||||
- MTP head: INSIDE checkpoint (`HALOGEN_MTP_HEAD` unset)
|
||||
- Vision: ON (v2 tower loads beside checkpoint)
|
||||
- Base: abliterated (27B-era was censored finetune)
|
||||
- Slots: 2 × 262,144 ctx (KV pool 524,288 positions)
|
||||
- Server sampler: 1.0/0.95/20
|
||||
- Prompt cache: ON by default
|
||||
Models.yml resolution rules that are harness-local: a default model resolves as
|
||||
a `defaultProvider` + `defaultModel` pair; a subagent override is a bare model
|
||||
`id` and resolves only when unique across providers; two ports on one host need
|
||||
two provider blocks — the file cannot express "same host, different port".
|
||||
|
||||
### frank-archon (Dankserver: 192.168.0.235:7770)
|
||||
There is **no audio input anywhere in the fleet** (gone with the Gemma4
|
||||
retirement, 2026-10-01). All three chat endpoints take text + images.
|
||||
|
||||
**StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe** (`.rad` format)
|
||||
## Retired laws
|
||||
|
||||
- Runtime: radiance 1.0.13 (digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`)
|
||||
- Format: fp8-KV `.rad` (121,969,901,568 B = 116,322 MiB)
|
||||
- MTP head: MTP depth 2, MNBT 2048, Froggeric v22.5 template
|
||||
- Vision: 27-layer Qwen-VL tower native
|
||||
- KV: 3 × 200,000 fp8-KV (`--max-num-seqs 3`)
|
||||
- 30,720 MiB of 30,969 MiB claimable
|
||||
- 44 GiB/rank pinned host pool, PLE pinned to RAM
|
||||
- Base-alignment: censored (abliterated family lives on frank-dragoon)
|
||||
|
||||
## Resolution Rules
|
||||
|
||||
Official model ids are StarCraft unit names, on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: `frank-archon` → `192.168.0.235:7770`, `frank-dragoon` → `192.168.0.143:7770`, `frank-zealot` → `192.168.0.143:7771`.
|
||||
|
||||
The default model resolves as a `defaultProvider` + `defaultModel` pair (provider must serve the model); a subagent override uses a bare model `id`, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".
|
||||
|
||||
When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 **all three endpoints accept images** (`input: [text, image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap, while `frank-archon` takes them natively and `frank-zealot` through its BF16 mmproj.
|
||||
|
||||
Reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — `models.yml` sets `reasoningContentField` per endpoint to match.
|
||||
|
||||
There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01.
|
||||
|
||||
## Boot Persistence & Self-Healing (MANDATORY)
|
||||
|
||||
Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from `compose.yaml` or a manual `podman run` dies silently at reboot — rootless Podman does not auto-start bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. Never report such a service as "up".
|
||||
|
||||
Unit requirements:
|
||||
|
||||
- **Boot persistence** — `[Install] WantedBy=default.target`, installed to `~/.config/containers/systemd/<name>.container`, enabled in the user manager, with linger on for the deploy user (`loginctl show-user dank -p Linger` → `Linger=yes`).
|
||||
- **Self-healing** — `Restart=always` (or `on-failure`) with a short `RestartSec`; quadlet health check where the service has one.
|
||||
- **Repo provenance** — the `.container` file lives in this repo (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.
|
||||
|
||||
Before declaring a service deployed: `systemctl --user is-enabled <name>` → `enabled`, `is-active` → `active`, and the container answers its health check.
|
||||
|
||||
A running container with no unit is a gap: write the unit, test it, deploy it.
|
||||
|
||||
This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under `dankclaw/observability/quadlet/` and their compose definitions were removed — one owner per container name. The stack is **configured and delivering as of 2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, loopback :9093) → host-side Signal relay (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as `host.containers.internal`; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
|
||||
|
||||
Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
|
||||
## Benchmarking Policy (MANDATORY)
|
||||
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench requires asking first.
|
||||
|
||||
Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.
|
||||
|
||||
A gufo-era law: **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
|
||||
|
||||
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real** text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently returns nothing. Generate **at least 128 tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise and the run-to-run spread swamps every real difference.
|
||||
|
||||
This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
|
||||
`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon`
|
||||
- Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot`
|
||||
- Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2):
|
||||
`sudo systemctl stop frank-archon && sudo systemctl mask frank-archon`
|
||||
2. Bench.
|
||||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in `/etc/systemd/system/` is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking.
|
||||
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
|
||||
|
||||
**CRITICAL:** never `systemctl disable` a system unit on these hosts — it deletes the unit file from `/etc/systemd/system/`, not just the symlink. `mask` symlinks to `/dev/null` and is safe/reversible.
|
||||
|
||||
## Subagent Lifecycle (MANDATORY)
|
||||
|
||||
A subagent that appears stuck must have an explicit recovery path defined before the subagent is considered deployed. "Stuck" includes: silent completion halt, repeated identical responses, or extended latency without token emission.
|
||||
|
||||
Recovery requirements:
|
||||
|
||||
- **Explicit timeout** — a hard timeout (≤30s) on the subagent's response channel, with a defined action (retry, fallback, or escalate).
|
||||
- **State checkpoint** — the subagent maintains checkpoints at decision boundaries, allowing resumption from a known good state.
|
||||
- **Health signal** — the subagent publishes a heartbeat (≤5s interval) that triggers the recovery path when missed.
|
||||
|
||||
A subagent without these three properties is a gap: write the timeout, add the checkpoint, wire the heartbeat, then deploy.
|
||||
|
||||
This rule exists because subagent hangs were observed in early deployments (2026-09-xx: silent completion halt; 2026-09-xx: repeated identical responses). The deployment script must verify the heartbeat channel is live before reporting "deployed".
|
||||
|
||||
Before declaring a subagent deployed: verify the heartbeat endpoint returns within the timeout window, and confirm the recovery path is wired (retry/fallback/escalation logic exists in the codepath).
|
||||
|
||||
A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.
|
||||
- The old "Benchmarking Policy (MANDATORY)" wording demanded a bench harness per
|
||||
payload; superseded 2026-10-18 by the swap law above.
|
||||
- A "Subagent Lifecycle (MANDATORY)" rule (heartbeat ≤5 s, checkpoints, hard
|
||||
timeout) was recorded here 2026-09 and **never implemented anywhere**; it
|
||||
described nothing that exists and is deleted 2026-10-18. Do not re-add it
|
||||
without a real implementation.
|
||||
- The fleet model inventory that used to live in this memory file drifted out of
|
||||
date within hours (it still claimed Bonsai-2-**27B** at `--ctx-size 401408`
|
||||
and a dragoon 524,288-position pool). Memory files do not get deploys — the
|
||||
repo does. That is why it is pointers now.
|
||||
|
||||
@@ -1,7 +1,24 @@
|
||||
## Voice Generation
|
||||
|
||||
- Custom voice source material must exclude the first 10 seconds of any audio sample file used to generate voices; this is a project-wide invariant across all source files, not a per-file exception.
|
||||
- Voice selector valid options: af_heart, af_alloy, af_aoede, af_bella, af_jessica, af_kore, af_nicole, af_nova, af_river, af_sarah, af_sky, am_adam, am_echo, am_eric, am_fenrir, am_liam, am_michael, am_onyx, am_puck, am_santa, bf_emma, bf_isabella, bm_george, bm_lewis, bf_alice, bf_lily, bm_daniel, bm_fable.
|
||||
|
||||
## Input Capabilities
|
||||
|
||||
All endpoints accept text input with optional image input: `frank-dragoon` returned to vision with the flash-server v2 swap, while `frank-archon` takes them natively and `frank-zealot` through its BF16 mmproj.
|
||||
|
||||
There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01.
|
||||
|
||||
## Voice Usage Notes
|
||||
|
||||
- Voice generator output does not affect token counting or inference costs — all voices are rendered post-inference.
|
||||
- Voice quality varies by source sample: exclude the first 10 seconds of any audio sample file used to generate voices; this is a project-wide invariant across all source files, not a per-file exception.
|
||||
- Voice selection is client-side only; the server does not generate or return audio data.
|
||||
|
||||
## Load Management
|
||||
|
||||
- Subagent settings must prevent frank-archon from being overwhelmed; concurrent request handling prioritizes endpoint stability over throughput.
|
||||
|
||||
## Podcast
|
||||
|
||||
- Podcast functionality is sourced from open-notebook; this is a project decision established before this document was created.
|
||||
- Podcast functionality is sourced from open-notebook; this is a project decision established before this document was created.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792370846394,
|
||||
"lastConsolidatedAt": 1792400853634,
|
||||
"lastResult": {
|
||||
"at": 1792370515419,
|
||||
"at": 1792393886960,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
|
||||
@@ -1,2 +1,3 @@
|
||||
## top bar
|
||||
- Preserve top bar customizations; customizations lost or obliterated on subsequent runs
|
||||
- Todo list updates as tasks change, not waiting for entire bill of tasks to finish
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792370776395,
|
||||
"lastConsolidatedAt": 1792381281460,
|
||||
"lastResult": {
|
||||
"at": 1792370420509,
|
||||
"at": 1792381281460,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
## window outlines
|
||||
- Window outline color is amber, and outline thickness is thickened
|
||||
- Window outline color is amber, and outline thickness is thickened
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792381246498
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
# Output behavior
|
||||
|
||||
Output must not be shaken off during presentation. Output remains anchored until explicitly confirmed.
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792439324791,
|
||||
"lastResult": {
|
||||
"at": 1792398531282,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792381487662
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792393561866,
|
||||
"lastResult": {
|
||||
"at": 1792381870670,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792388836682
|
||||
}
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"v":1,"kind":"product_decision","statement":"Limit on zealot set to 3, dragoon set to 2, archon set to 3","source":"explicit_user","evidence":"make the limit on zealot 3, dragoon 2, archon 3","friction":{"corrective":false,"regression":false,"subtle":false},"sessionId":"01a15302-f12d-77d8-9a33-be31a1a05528","ts":1792394470419}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792394427081
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792439315327,
|
||||
"lastResult": {
|
||||
"at": 1792439315327,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792439612626,
|
||||
"lastResult": {
|
||||
"at": 1792439612626,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792398333838
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792524715562,
|
||||
"lastResult": {
|
||||
"at": 1792433353188,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user