diff --git a/agent/managed-skills/radiance-calibration-tap-shape/SKILL.md b/agent/managed-skills/radiance-calibration-tap-shape/SKILL.md new file mode 100644 index 0000000..20a0ff1 --- /dev/null +++ b/agent/managed-skills/radiance-calibration-tap-shape/SKILL.md @@ -0,0 +1,71 @@ +--- +name: radiance-calibration-tap-shape +description: "Run and measure a radiance GPTQ calibration tap on a disk-tiered BF16 rig — n-gram tier, step sized to the document, tap marker gates, and reading the real rate from /stats instead of the log." +--- + +# Running and measuring a radiance calibration tap + +A calibration tap (`RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS`) boots a BF16 reference container +on the serving cards under `--weights-disk-tier` and prefills a corpus until the pooled token floor +is reached, writing per-expert Grams for GPTQ. It is the one campaign phase whose cost is dominated +by **disk read volume**, not by compute, and the shape that serves production is the wrong shape for +it. + +## The four things that decide the tap's wall time + +1. **N-gram table tier.** On the BF16 payload the table is tens of GiB and `--ngram-placement ram` + PINS it. Beside a full host pool that leaves a few GiB of RAM and the boot dies at + `startup stopped: out of memory` while the same pool and slot count had already booted the + reference recording. Calib only needs the routed weights prefilled, so pass + `--ngram-placement disk` for calib and keep `ram` for the KL modes — their numbers are only + comparable at an identical shape. +2. **Step ≥ document length.** A document cut into N steps re-streams each layer's routed expert + union N times from the container. With `--max-num-batched-tokens` below the document length the + tap runs several times slower than with one wide step, and the KV pool does not change at all + (`kv_tokens_total` is unaffected). This is legal for calib and NOT for a scoring boot: scoring + numbers are only comparable at one shape, so the step is calib's own knob. +3. **KV pool, not slots.** When the pool holds barely one document, extra `--max-num-seqs` only + queue (and preempt) requests; raising slots does not raise throughput. +4. **The token floor is pooled, not per round.** One corpus pass may be smaller than the floor, so a + round that seals no bytes is NORMAL — a driver that treats a flat round as failure aborts one + round early. Fail on a tap that *loses* bytes; let "never sealed" fail at max rounds. + +## Never push a tap that belongs to no rig + +A tap dir is a directory; production is also an endpoint on the same port. Without proof, a pusher +will happily feed millions of tokens into whatever answers and record nothing. + +- The rig writes a marker into the tap dir at boot (`container=`, `image=`, `pid=`), retires it + when its boot exits non-zero, and the pusher refuses unless the marker exists, names the + container it targets, and that container is running. +- The window script stops on a failed phase instead of falling through its health wait, and banks a + large reference artifact only when both phases exit 0 and the tap holds files. + +## Measuring the rate (do not read it off the log) + +The campaign log is `tee`'d and block-buffered, so its request count lags by minutes. + +```bash +curl -s http://127.0.0.1:7770/stats | python3 -c 'import json,sys; d=json.load(sys.stdin); \ + print({k: d[k] for k in ("uptime_s","prefill_tps","steps_prefill","total_requests","preemptions","kv_tokens_total")})' +``` + +`prefill_tps` is CUMULATIVE over uptime — `tokens ≈ prefill_tps × uptime_s`, so take two samples a +couple of minutes apart and difference them. `steps_prefill / total_requests` proves whether a +document is one step or many. Discard the first minutes after boot (warmup and the first requests +are not the steady state). + +## Sizing the tap before paying for it + +Probe one 5-minute sample at the candidate step, compute the ratio against the inherited shape, and +choose the floor from the measured rate — not from an earlier campaign's runtime on a different +corpus and shape. Then set `RADIANCE_CALIB_TOKENS` (via the runner's env knob) and launch ONE run; +if the probe already beat the bar, that probe run IS the real run and needs no restart. + +## Deliverables that make it unattended + +- a gated window script: boot → health with a hard timeout → push → stop → restore the service → + bank the reference artifact only on success; +- a pusher that is marker-gated and progress-aware; +- a ledger line recording the shape used and the target chosen, so the next session does not + re-derive it. diff --git a/agent/managed-skills/radiance-disk-tier-throughput-census/SKILL.md b/agent/managed-skills/radiance-disk-tier-throughput-census/SKILL.md new file mode 100644 index 0000000..123d40c --- /dev/null +++ b/agent/managed-skills/radiance-disk-tier-throughput-census/SKILL.md @@ -0,0 +1,68 @@ +--- +name: radiance-disk-tier-throughput-census +description: "Measure a radiance rig's real prefill rate and prove whether the bottleneck is disk, CPU or shape — for calibration taps, KLD recording or score boots; use when a rate looks wrong, an ETA must be quoted, or someone asks about CPU allocation." +--- + +# Census a radiance rig's throughput without lying to yourself + +Ask three questions in order: what is the rate, what is the binding resource, and does the shape +explain it. Every number below comes from a kernel or engine counter — never from a gauge. + +## 1. The rate: counters, never `prefill_tps` + +`/stats` on :7770 exposes `prefill_tps`, and it is a **rolling gauge**. Reading 69.5 → 149.1 across +120 s and multiplying by `uptime_s` produced a bogus "320 tok/s / 10.4 h" verdict that had to be +retracted (measured 2026-10-29). Derive the rate from counters over a ≥240 s window: + +```bash +python3 - <<'PY' +import json, time, urllib.request +def stats(): return json.load(urllib.request.urlopen("http://127.0.0.1:7770/stats", timeout=10)) +a = stats(); t0 = time.time(); time.sleep(240); b = stats(); t1 = time.time(); dt = t1 - t0 +docs = b["total_requests"] - a["total_requests"] +steps = b["steps_prefill"] - a["steps_prefill"] +MEAN = 4238.0 # tokens/doc, from the corpus: total_tokens / documents +print("docs %d in %.0fs -> %.0f tok/s | steps/doc %.2f" % (docs, dt, docs * MEAN / dt, steps / docs)) +PY +``` + +`steps/doc` is the read-amplification signal: a document cut into N prefill steps re-streams each +layer's routed expert union N times when the weights are on the disk tier, so N > 1 is leaked time. + +## 2. The binding resource: diskstats and the idle/iowait split + +```bash +# per-device read bandwidth (parent device, not the partition) +awk '{print $3, $6*512}' /proc/diskstats | grep -E "^nvme" +# host busy = total - idle - iowait (summing /proc/stat's first 8 fields WITHOUT subtracting is +# the classic error: the sum already covers idle, so it reads ~800% on an 8-core box) +``` + +Then split CPU by process: sample `/proc//stat` fields 14+15 (utimes + stimes) twice. + +Two hard readings that decide the argument: + +- `radiance` compute at ~0.26 cores while the disk delivers ~3.2 GB/s and `iowait` is ~29 % of the + box ⇒ **disk-bound**. More cores or a `--cpuset-cpus` pin cannot add bytes; pinning only caps it. +- A container with `cpu_quota=0` and an empty `CpusetCpus` (check `podman inspect -f + '{{.HostConfig.CpuQuota}} {{.HostConfig.CpusetCpus}}'`) is already unrestricted — "allocate more + CPU" is a no-op, and "allocate 6 of 8" is a ceiling. + +Sanity relation: tok/s × (GiB streamed per document ÷ tokens per document) should match the measured +device bandwidth. If it does not, the shape, not the hardware, is the story. + +## 3. The shape: what may legally be widened + +- `kldref` and `score` must keep production's `--max-num-batched-tokens` and `--max-num-seqs`: their + KL numbers are only comparable across boots at an identical shape (the audit law). +- A **calibration tap** is exempt — its output is layer Hessians, nothing compares it. Give it a step + that covers a whole document (`Q38_CALIB_MNBT` ≥ the corpus window) and read `steps/doc`: on a + 7,168-token corpus, 2,048 → 8,192 moved 2.5 → 1.00 steps/doc and 94 → 113 tok/s (**1.2×**, not the + 4× the step-count ratio suggests — total bytes are unchanged, so do not promise more). +- The KV pool, not the step, bounds concurrency: when `kv_tokens_total` ≈ one document, requests queue + and preempt; extra slots buy nothing. + +## Reporting + +Quote tok/s, steps/doc, device GB/s and the busy/iowait split together, plus the shape each number +was taken at. A rate without its shape is not a measurement; a gauge without its window is not a rate. diff --git a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json index a73f82f..5212cc2 100644 --- a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json +++ b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1793316653836, + "lastConsolidatedAt": 1793320373858, "lastResult": { "at": 1793315963987, "sessions": 1, diff --git a/dotfiles/noctalia/.local/state/noctalia/settings.toml b/dotfiles/noctalia/.local/state/noctalia/settings.toml index ed28f59..007ee20 100644 --- a/dotfiles/noctalia/.local/state/noctalia/settings.toml +++ b/dotfiles/noctalia/.local/state/noctalia/settings.toml @@ -279,22 +279,22 @@ transition = [ "fade" ] interval_seconds = 1200 [wallpaper.default] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [wallpaper.last] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [wallpaper.monitors.DP-1] path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [wallpaper.monitors.DP-2] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [wallpaper.monitors.HDMI-A-1] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [wallpaper.monitors.eDP-1] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [weather] unit = "imperial"