Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4df5a28ba6 | ||
|
|
03bf48956f | ||
|
|
4d76c6038e | ||
|
|
3eb1ada55b |
No files matched your search
+1
-1
@@ -96,7 +96,7 @@ compaction:
|
|||||||
experimentalContextManagement: false
|
experimentalContextManagement: false
|
||||||
interruptMode: immediate
|
interruptMode: immediate
|
||||||
defaultThinkingLevel: xhigh
|
defaultThinkingLevel: xhigh
|
||||||
hideThinkingBlock: false
|
hideThinkingBlock: true
|
||||||
proseOnlyThinking: false
|
proseOnlyThinking: false
|
||||||
terminal:
|
terminal:
|
||||||
showProgress: true
|
showProgress: true
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
name: live-checkout-landing-chain
|
||||||
|
description: "Prove whether a merged change is actually deployed in ~/.omp: walk origin/main vs the checked-out main commit vs the worktree, then fast-forward in place — git refuses fetch into a checked-out branch, and a running harness caches its config."
|
||||||
|
---
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
The user says a merged/committed change is "not there" (`/switch` can't see a model, a doc is missing, a config edit is invisible) while the repo work itself is green. In a repo where **the checkout is the deployment** (`dank/command-center` at `~/.omp`), "merged" and "deployed" are different commits — measure both before believing either claim.
|
||||||
|
|
||||||
|
## The ladder — read all four levels, in order, before answering
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.omp && git rev-parse HEAD origin/main # 1+2: deployment vs forge
|
||||||
|
git ls-tree --name-only HEAD # is the file even in HEAD?
|
||||||
|
git worktree list # 3: which worktree am I in vs main?
|
||||||
|
```
|
||||||
|
|
||||||
|
1. `origin/main` — the forge truth (`gh`/`tea` absent; Gitea API or a fetch).
|
||||||
|
2. `/home/blank/.omp` HEAD — **what the running agent actually reads.** Merged PR at `3eb1ada` while this sat at `53025a6` is the whole bug.
|
||||||
|
3. The feature/session worktree — where the edit was authored; its files can be correct while main is stale, which is why the author sees the change and the user does not.
|
||||||
|
4. The running process — `models.yml` / `config.yml` are read at startup and cached for the process lifetime; an open session needs a restart even after level 2 is correct.
|
||||||
|
|
||||||
|
Belt-and-braces when the answer hinges on file contents, not just commit ids: parse the live file (`python3 -c "import yaml; ..."` for `agent/models.yml`, or grep the doc) rather than inferring from `git log`.
|
||||||
|
|
||||||
|
## Landing main without touching the tree
|
||||||
|
|
||||||
|
`git fetch origin main:main` **fails**: `fatal: refusing to fetch into branch 'refs/heads/main' checked out at '/home/blank/.omp'`. The fetch-into-ref trick only works for branches that are *not* checked out anywhere.
|
||||||
|
|
||||||
|
For a checked-out main, prove then fast-forward:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git merge-base --is-ancestor main origin/main && echo "ff possible"
|
||||||
|
git merge --ff-only origin/main # no rebase — Frank dislikes history rewrites
|
||||||
|
```
|
||||||
|
|
||||||
|
`--ff-only` is the safe verb on a dirty main checkout: tracked-file commits (the live-sync tier) and runtime dirt (`agent/memories/*/state.json`, `dotfiles/**`) survive untouched, and it cannot rewrite history. Confirm the dirt is still there afterwards — if it vanished, something else clobbered the tree.
|
||||||
|
|
||||||
|
## The heartbeat is not instant
|
||||||
|
|
||||||
|
`scripts/live-sync.sh` (systemd user timer, hourly) is what normally advances main. Check its cadence rather than assuming it ran:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl --user status live-sync.timer --no-pager # "Trigger: …" = next beat
|
||||||
|
systemctl --user status live-sync.service --no-pager # "since …" = last run
|
||||||
|
```
|
||||||
|
|
||||||
|
A merge made minutes after the last beat waits until the next one; fast-forwarding by hand is legitimate (idempotent with the script) and is the answer when the user wants it now. Never `--no-verify`, never force-push.
|
||||||
|
|
||||||
|
## Do not confuse with
|
||||||
|
|
||||||
|
- `omp-worktree-path-gotchas` — edits landing in the *wrong* worktree (cwd drift), not a stale main.
|
||||||
|
- `dotfiles-deployment-truth-audit` — classifies which files are tracked vs runtime state.
|
||||||
|
- `serverappconfigs-vault-page-registration` / `gitea-merge-form-encoded` — the PR mechanics upstream of this question.
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
---
|
||||||
|
name: omp-provider-row-sources
|
||||||
|
description: "Add a provider/model row to ~/.omp/agent/models.yml by reading the harness's own catalog tables (provider name, api, window, cost, compat) instead of guessing or web-searching, then prove it with a scratch-HOME call."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Scope
|
||||||
|
|
||||||
|
Adding (or correcting) a provider/model row in `~/.omp/agent/models.yml` for a
|
||||||
|
provider the harness may already know: an upstream vendor API (DeepSeek, OpenAI,
|
||||||
|
Anthropic, Moonshot, …) or a new local endpoint. The rule this skill exists for:
|
||||||
|
**almost every field you are about to research on the web already exists in the
|
||||||
|
installed bundle** — research there first, and the web only confirms it.
|
||||||
|
|
||||||
|
Canonical install root (resolve the symlink first if the repo lives under a
|
||||||
|
worktree): `/home/blank/.bun/install/global/node_modules/@oh-my-pi/`.
|
||||||
|
|
||||||
|
# The three tables that own the facts
|
||||||
|
|
||||||
|
| You want | Read | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| The legal **provider name**, its URL marker, and which quirks the harness gates on it | `pi-catalog/src/hosts.ts` (minified in `dist/cli.js` as the `VIn` map; consumed by `pi-catalog/src/compat/resolve.ts`) | The provider key is a **lookup key, not a display name**. e.g. `deepseekDirect: { providers: ["deepseek"], urlMarkers: ["api.deepseek.com"] }`, `anthropic: { providers: ["anthropic"], urlMarkers: ["api.anthropic.com"] }`. A label matching neither loses the behaviour set silently (the `max_tokens` field name, the thinking `extraBody`, the thinking-disable mode). |
|
||||||
|
| Model rows: ids, `contextWindow`, `maxTokens`, `cost` rates, per-model `compat` | `pi-catalog/src/provider-models/<api>.ts` — `openai-compat.ts`, `anthropic-messages.ts`, …; factory helpers `openAiCompletionsDescriptor(...)` / `anthropicMessagesDescriptor(...)` | Rows are fetched from the provider's models-dev key, so the bundle carries the same numbers the vendor publishes (and often rates the vendor page hides). The descriptor's `compat:` block is the authored truth for that provider's transport shape — copy it, never simplify it. |
|
||||||
|
| The **schema** itself (which keys are legal on a provider vs a model, and which are required) | `pi-coding-agent/src/config/models-config-schema-bundle.ts` (and the identical `dist/` copy) | Provider level and model level have *different* field sets. Cheaper and more correct than reading the minified bundle. |
|
||||||
|
|
||||||
|
Also in `pi-coding-agent/src/config/`: `model-registry.ts` (how `models.yml`
|
||||||
|
merges onto the built-in catalog — `#mergeCustomModels` → `mergeByModelKey`),
|
||||||
|
`resolve-config-value.ts` (how `apiKey` values resolve).
|
||||||
|
|
||||||
|
# Two traps this table pays for
|
||||||
|
|
||||||
|
- **Provider-level `api` only accepts chat/judge transports.** Naming a runner API
|
||||||
|
(`openai-speech`, `openai-images`, `openai-embeddings`) at provider level fails
|
||||||
|
whole-file validation and disables *every* custom provider, not just yours
|
||||||
|
(`omp models` prints "models.yml validation failed", lists only built-ins).
|
||||||
|
Runner api belongs on the **model** entry; see the managed skill
|
||||||
|
`omp-models-yml-runner-api-placement`.
|
||||||
|
- **A `models.yml` entry is an overlay, not a replacement.** Adding one model to a
|
||||||
|
provider the catalog already ships does not remove the provider's other rows —
|
||||||
|
`omp models <provider>` will show yours (yours wins on shared fields, including
|
||||||
|
`cost` and `contextWindow`) plus the catalog rows. Same-id rows patch with the
|
||||||
|
`replace` transport policy; a row that omits `contextWindow` inherits the
|
||||||
|
bundled row's window and its policy tiers.
|
||||||
|
|
||||||
|
# Secret handling (non-prose tracked file)
|
||||||
|
|
||||||
|
Never inline the credential. `apiKey` resolution order is: leading `!` → command
|
||||||
|
(`stdout` trimmed, cached for the process lifetime) → exact env-var name → literal.
|
||||||
|
So write `apiKey: "!cat ~/.config/secrets/<name>_api_key"` with mode 600.
|
||||||
|
|
||||||
|
- A bare `sk-…` trips the pre-commit **shape** gate (every file type).
|
||||||
|
- An unquoted `!` is a YAML tag — quote it.
|
||||||
|
- A quoted command value also dodges the *assignment* gate: the gate's value
|
||||||
|
charset rejects `~` and spaces, so only a literal key matches.
|
||||||
|
|
||||||
|
# Procedure
|
||||||
|
|
||||||
|
1. Read the three tables above for the provider you are adding. Take the exact
|
||||||
|
provider name, base URL, api, and compat from them; take `contextWindow` /
|
||||||
|
`maxTokens` / `cost` from the model row and cite the source in a comment.
|
||||||
|
2. Write the secret to `~/.config/secrets/<name>_api_key`, `chmod 600`.
|
||||||
|
3. Edit `models.yml`; for a repo-tracked config, follow that repo's own
|
||||||
|
branch/log/index rules (in `dank/command-center`: branch → issue → PR).
|
||||||
|
4. Verify with a **scratch HOME**, because `models.yml` loads at process start
|
||||||
|
and a live session keeps the old registry:
|
||||||
|
```bash
|
||||||
|
mkdir -p /tmp/ompx/.omp/agent /tmp/ompx/.config
|
||||||
|
cp agent/models.yml agent/config.yml /tmp/ompx/.omp/agent/
|
||||||
|
ln -sfn ~/.config/secrets /tmp/ompx/.config/secrets # ~ in a !cmd value = $HOME
|
||||||
|
HOME=/tmp/ompx omp models --no-extensions | grep -c 'validation failed' # must be 0
|
||||||
|
HOME=/tmp/ompx omp --model '<provider>/<id>:off' --tools bash --no-session \
|
||||||
|
--no-skills --no-rules --no-extensions --no-pty --no-lsp --no-title \
|
||||||
|
--cwd /tmp -p 'Reply with exactly the word OK and nothing else.'
|
||||||
|
```
|
||||||
|
Then the **negative control**, the step that separates a real auth success from
|
||||||
|
a cached/built-in answer: `sed` the `baseUrl` to a dead port in the scratch copy
|
||||||
|
and repeat — it must hang or error, not answer. Then `rm -rf /tmp/ompx` (it
|
||||||
|
holds endpoint URLs).
|
||||||
|
|
||||||
|
# Voice notes
|
||||||
|
|
||||||
|
- Prefer the bundled schema file over grepping `dist/cli.js`; grep the bundle only
|
||||||
|
when the source and the bundle disagree, or for a runtime default the schema does
|
||||||
|
not state (e.g. `reasoningDisableMode` per provider family).
|
||||||
|
- `cost` is optional in the schema. Omitting it makes the status line report `$0`
|
||||||
|
forever; one rate per direction is all the schema models, so a peak/off-peak
|
||||||
|
vendor table can only carry the peak value.
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
name: quant-pipeline-adversarial-review
|
||||||
|
description: "Review or audit the serverAppConfigs quantization campaign pipeline (convert-calib-kldref-score plus corpus builder/rechunker/pusher, fp8 stacker) for defects that silently waste GPU-hours or produce a number that is not the claimed number."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Adversarial review of the quant campaign pipeline
|
||||||
|
|
||||||
|
Friction-earned 2026-10-29 across `scripts/convert-qwen38-w5nl64.sh`,
|
||||||
|
`run-qwen38-bf16.sh`, `make-calib-corpus.py`, `rechunk-kld-corpus.py`,
|
||||||
|
`calib-push.py`, `stack-qwen38-official-fp8.py`. Every class below was a real
|
||||||
|
defect in that tree. Review code, not vibes: quote the line, name the cost.
|
||||||
|
|
||||||
|
## Files in scope (one campaign = this set)
|
||||||
|
|
||||||
|
| Stage | File | Owns |
|
||||||
|
|---|---|---|
|
||||||
|
| corpus | `make-calib-corpus.py` | calib + kld JSONL, manifest |
|
||||||
|
| corpus | `rechunk-kld-corpus.py` | windows that fit the container's KV window |
|
||||||
|
| convert | `convert-<model>.sh` | plan → artifact per arm |
|
||||||
|
| tape | `run-qwen38-bf16.sh` | the rig's argv, mount, trap, gates |
|
||||||
|
| drive | `calib-push.py` | prefill traffic into the tap |
|
||||||
|
| fp8 path | `stack-qwen38-official-fp8.py` | expert stacking + PLE merge |
|
||||||
|
|
||||||
|
## Defect classes to hunt, in cost order
|
||||||
|
|
||||||
|
1. **Claimed identity that is only a length check.** A tool that writes TEXT back
|
||||||
|
out (`tok.decode(window)`) and asserts only counts is not proving token
|
||||||
|
identity — decode→encode can merge/split. Fix: re-encode each window and
|
||||||
|
compare id lists; refuse on mismatch. Same shape: an assertion on "N tokens
|
||||||
|
in, N tokens out" that counts windows rather than re-encoded text.
|
||||||
|
2. **A loop that cannot finish and cannot notice.** Drivers that re-push a corpus
|
||||||
|
`--max-rounds` times without measuring the sink. Compare the tap/output dir's
|
||||||
|
byte total per round and abort on zero growth; abort on zero successes; abort
|
||||||
|
on zero prompts. Without this the loss is silent for hours.
|
||||||
|
3. **`set -u` argument holes.** `--flag) VAR="$2"; shift 2` dies with "unbound
|
||||||
|
variable" when the flag is last. Guard `[ "$#" -ge 2 ]` in every such arm.
|
||||||
|
4. **Restore paths that leave the GPU holder alive.** `--rm` only fires when
|
||||||
|
podman itself exits, so a killed run leaves conmon holding the cards while
|
||||||
|
`systemctl is-active` says `active`. `restore()` MUST `podman rm -f "$C"`
|
||||||
|
before starting production. Prove with `podman ps` after a `kill`.
|
||||||
|
5. **Gates that run after the expensive phase.** A leftover container name, an
|
||||||
|
existing output, a non-writable tap dir, an unset figure — check them before
|
||||||
|
the plan/boot, not when podman/sed fails mid-run.
|
||||||
|
6. **Zero-exit with no artifact.** After `rc=0`, assert the output file exists
|
||||||
|
before `stat`/arithmetic on it.
|
||||||
|
7. **`assert` in a verifier.** `python -O` deletes it silently. Use `raise`.
|
||||||
|
8. **Warning where the code's own assumption breaks.** e.g. a 128-lane pack fed
|
||||||
|
an HD that is not a multiple of 128 — warning + continue writes a wrong table.
|
||||||
|
9. **Re-tokenization in loops.** A pool cycled 40× re-tokenizing each fragment
|
||||||
|
per pass; and a manifest recomputing a total by re-encoding the whole corpus.
|
||||||
|
Cache counts once, accumulate during generation.
|
||||||
|
10. **Substring tests standing in for key tests.** `"content" in line` before
|
||||||
|
`json.loads(line)["content"]` crashes on any list-shaped `.json`.
|
||||||
|
11. **Path guards compared as strings.** `A/../A/x` defeats an equality check;
|
||||||
|
add `realpath -m` on both sides.
|
||||||
|
12. **Shellcheck's honest false positive**: single-quoted `$CALIB` passed to a
|
||||||
|
converter that expands it from the container env is correct (SC2016). Say so
|
||||||
|
in the PR; do not "fix" it.
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
1. Inventory: `wc -l` the scripts + recipes; read each **completely** (ranges,
|
||||||
|
not whole files, but every line). Recipes carry the numbers scripts assert.
|
||||||
|
2. Read the tests that pin the argv (`tests/qwen38_reference.bats`'s `SHAPE`
|
||||||
|
fixture, `tests/qwen38_campaign.bats`) — they are the contract you must not
|
||||||
|
break, and the first place a stale number hides.
|
||||||
|
3. Mechanical gates:
|
||||||
|
```bash
|
||||||
|
python3 -m py_compile scripts/*.py
|
||||||
|
bash -n scripts/<each>.sh
|
||||||
|
shellcheck -S warning scripts/<each>.sh # never pass .py to it
|
||||||
|
./tests/run.sh
|
||||||
|
```
|
||||||
|
4. Fix the real ones only; keep diffs surgical; re-run the suite after the batch.
|
||||||
|
5. Smoke the changed path for real — smoke beats the suite:
|
||||||
|
```bash
|
||||||
|
# run a local script on the server's data via the corpus venv, no checkout needed
|
||||||
|
ssh dankserver '/home/dank/venvs/radcorpus/bin/python - --corpus <corpus> --out /tmp/<x> --force' \
|
||||||
|
< scripts/rechunk-kld-corpus.py
|
||||||
|
```
|
||||||
|
Import-level helpers can be exercised with
|
||||||
|
`importlib.util.spec_from_file_location` (e.g. a new `tap_bytes()`: empty→0,
|
||||||
|
one file→its size, absent→0).
|
||||||
|
6. Stacked PRs: a review commit goes on its own branch off the open one; a diff
|
||||||
|
that includes the parent commit is a stacked PR, not a mistake — say which PR
|
||||||
|
merges first in the body.
|
||||||
|
|
||||||
|
## Report
|
||||||
|
|
||||||
|
One table of defect → cost → fix. Name the failure the defect produced in
|
||||||
|
practice (hours wasted, wrong number), because that is what makes it reviewable.
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
---
|
||||||
|
name: radiance-calibration-tap-shape
|
||||||
|
description: "Run and measure a radiance GPTQ calibration tap on a disk-tiered BF16 rig — n-gram tier, step sized to the document, tap marker gates, and reading the real rate from /stats instead of the log."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Running and measuring a radiance calibration tap
|
||||||
|
|
||||||
|
A calibration tap (`RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS`) boots a BF16 reference container
|
||||||
|
on the serving cards under `--weights-disk-tier` and prefills a corpus until the pooled token floor
|
||||||
|
is reached, writing per-expert Grams for GPTQ. It is the one campaign phase whose cost is dominated
|
||||||
|
by **disk read volume**, not by compute, and the shape that serves production is the wrong shape for
|
||||||
|
it.
|
||||||
|
|
||||||
|
## The four things that decide the tap's wall time
|
||||||
|
|
||||||
|
1. **N-gram table tier.** On the BF16 payload the table is tens of GiB and `--ngram-placement ram`
|
||||||
|
PINS it. Beside a full host pool that leaves a few GiB of RAM and the boot dies at
|
||||||
|
`startup stopped: out of memory` while the same pool and slot count had already booted the
|
||||||
|
reference recording. Calib only needs the routed weights prefilled, so pass
|
||||||
|
`--ngram-placement disk` for calib and keep `ram` for the KL modes — their numbers are only
|
||||||
|
comparable at an identical shape.
|
||||||
|
2. **Step ≥ document length.** A document cut into N steps re-streams each layer's routed expert
|
||||||
|
union N times from the container. With `--max-num-batched-tokens` below the document length the
|
||||||
|
tap runs several times slower than with one wide step, and the KV pool does not change at all
|
||||||
|
(`kv_tokens_total` is unaffected). This is legal for calib and NOT for a scoring boot: scoring
|
||||||
|
numbers are only comparable at one shape, so the step is calib's own knob.
|
||||||
|
3. **KV pool, not slots.** When the pool holds barely one document, extra `--max-num-seqs` only
|
||||||
|
queue (and preempt) requests; raising slots does not raise throughput.
|
||||||
|
4. **The token floor is pooled, not per round.** One corpus pass may be smaller than the floor, so a
|
||||||
|
round that seals no bytes is NORMAL — a driver that treats a flat round as failure aborts one
|
||||||
|
round early. Fail on a tap that *loses* bytes; let "never sealed" fail at max rounds.
|
||||||
|
|
||||||
|
## Never push a tap that belongs to no rig
|
||||||
|
|
||||||
|
A tap dir is a directory; production is also an endpoint on the same port. Without proof, a pusher
|
||||||
|
will happily feed millions of tokens into whatever answers and record nothing.
|
||||||
|
|
||||||
|
- The rig writes a marker into the tap dir at boot (`container=`, `image=`, `pid=`), retires it
|
||||||
|
when its boot exits non-zero, and the pusher refuses unless the marker exists, names the
|
||||||
|
container it targets, and that container is running.
|
||||||
|
- The window script stops on a failed phase instead of falling through its health wait, and banks a
|
||||||
|
large reference artifact only when both phases exit 0 and the tap holds files.
|
||||||
|
|
||||||
|
## Measuring the rate (do not read it off the log)
|
||||||
|
|
||||||
|
The campaign log is `tee`'d and block-buffered, so its request count lags by minutes.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://127.0.0.1:7770/stats | python3 -c 'import json,sys; d=json.load(sys.stdin); \
|
||||||
|
print({k: d[k] for k in ("uptime_s","prefill_tps","steps_prefill","total_requests","preemptions","kv_tokens_total")})'
|
||||||
|
```
|
||||||
|
|
||||||
|
`prefill_tps` is CUMULATIVE over uptime — `tokens ≈ prefill_tps × uptime_s`, so take two samples a
|
||||||
|
couple of minutes apart and difference them. `steps_prefill / total_requests` proves whether a
|
||||||
|
document is one step or many. Discard the first minutes after boot (warmup and the first requests
|
||||||
|
are not the steady state).
|
||||||
|
|
||||||
|
## Sizing the tap before paying for it
|
||||||
|
|
||||||
|
Probe one 5-minute sample at the candidate step, compute the ratio against the inherited shape, and
|
||||||
|
choose the floor from the measured rate — not from an earlier campaign's runtime on a different
|
||||||
|
corpus and shape. Then set `RADIANCE_CALIB_TOKENS` (via the runner's env knob) and launch ONE run;
|
||||||
|
if the probe already beat the bar, that probe run IS the real run and needs no restart.
|
||||||
|
|
||||||
|
## Deliverables that make it unattended
|
||||||
|
|
||||||
|
- a gated window script: boot → health with a hard timeout → push → stop → restore the service →
|
||||||
|
bank the reference artifact only on success;
|
||||||
|
- a pusher that is marker-gated and progress-aware;
|
||||||
|
- a ledger line recording the shape used and the target chosen, so the next session does not
|
||||||
|
re-derive it.
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
---
|
||||||
|
name: radiance-disk-tier-throughput-census
|
||||||
|
description: "Measure a radiance rig's real prefill rate and prove whether the bottleneck is disk, CPU or shape — for calibration taps, KLD recording or score boots; use when a rate looks wrong, an ETA must be quoted, or someone asks about CPU allocation."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Census a radiance rig's throughput without lying to yourself
|
||||||
|
|
||||||
|
Ask three questions in order: what is the rate, what is the binding resource, and does the shape
|
||||||
|
explain it. Every number below comes from a kernel or engine counter — never from a gauge.
|
||||||
|
|
||||||
|
## 1. The rate: counters, never `prefill_tps`
|
||||||
|
|
||||||
|
`/stats` on :7770 exposes `prefill_tps`, and it is a **rolling gauge**. Reading 69.5 → 149.1 across
|
||||||
|
120 s and multiplying by `uptime_s` produced a bogus "320 tok/s / 10.4 h" verdict that had to be
|
||||||
|
retracted (measured 2026-10-29). Derive the rate from counters over a ≥240 s window:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 - <<'PY'
|
||||||
|
import json, time, urllib.request
|
||||||
|
def stats(): return json.load(urllib.request.urlopen("http://127.0.0.1:7770/stats", timeout=10))
|
||||||
|
a = stats(); t0 = time.time(); time.sleep(240); b = stats(); t1 = time.time(); dt = t1 - t0
|
||||||
|
docs = b["total_requests"] - a["total_requests"]
|
||||||
|
steps = b["steps_prefill"] - a["steps_prefill"]
|
||||||
|
MEAN = 4238.0 # tokens/doc, from the corpus: total_tokens / documents
|
||||||
|
print("docs %d in %.0fs -> %.0f tok/s | steps/doc %.2f" % (docs, dt, docs * MEAN / dt, steps / docs))
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
|
||||||
|
`steps/doc` is the read-amplification signal: a document cut into N prefill steps re-streams each
|
||||||
|
layer's routed expert union N times when the weights are on the disk tier, so N > 1 is leaked time.
|
||||||
|
|
||||||
|
## 2. The binding resource: diskstats and the idle/iowait split
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# per-device read bandwidth (parent device, not the partition)
|
||||||
|
awk '{print $3, $6*512}' /proc/diskstats | grep -E "^nvme"
|
||||||
|
# host busy = total - idle - iowait (summing /proc/stat's first 8 fields WITHOUT subtracting is
|
||||||
|
# the classic error: the sum already covers idle, so it reads ~800% on an 8-core box)
|
||||||
|
```
|
||||||
|
|
||||||
|
Then split CPU by process: sample `/proc/<pid>/stat` fields 14+15 (utimes + stimes) twice.
|
||||||
|
|
||||||
|
Two hard readings that decide the argument:
|
||||||
|
|
||||||
|
- `radiance` compute at ~0.26 cores while the disk delivers ~3.2 GB/s and `iowait` is ~29 % of the
|
||||||
|
box ⇒ **disk-bound**. More cores or a `--cpuset-cpus` pin cannot add bytes; pinning only caps it.
|
||||||
|
- A container with `cpu_quota=0` and an empty `CpusetCpus` (check `podman inspect -f
|
||||||
|
'{{.HostConfig.CpuQuota}} {{.HostConfig.CpusetCpus}}'`) is already unrestricted — "allocate more
|
||||||
|
CPU" is a no-op, and "allocate 6 of 8" is a ceiling.
|
||||||
|
|
||||||
|
Sanity relation: tok/s × (GiB streamed per document ÷ tokens per document) should match the measured
|
||||||
|
device bandwidth. If it does not, the shape, not the hardware, is the story.
|
||||||
|
|
||||||
|
## 3. The shape: what may legally be widened
|
||||||
|
|
||||||
|
- `kldref` and `score` must keep production's `--max-num-batched-tokens` and `--max-num-seqs`: their
|
||||||
|
KL numbers are only comparable across boots at an identical shape (the audit law).
|
||||||
|
- A **calibration tap** is exempt — its output is layer Hessians, nothing compares it. Give it a step
|
||||||
|
that covers a whole document (`Q38_CALIB_MNBT` ≥ the corpus window) and read `steps/doc`: on a
|
||||||
|
7,168-token corpus, 2,048 → 8,192 moved 2.5 → 1.00 steps/doc and 94 → 113 tok/s (**1.2×**, not the
|
||||||
|
4× the step-count ratio suggests — total bytes are unchanged, so do not promise more).
|
||||||
|
- The KV pool, not the step, bounds concurrency: when `kv_tokens_total` ≈ one document, requests queue
|
||||||
|
and preempt; extra slots buy nothing.
|
||||||
|
|
||||||
|
## Reporting
|
||||||
|
|
||||||
|
Quote tok/s, steps/doc, device GB/s and the busy/iowait split together, plus the shape each number
|
||||||
|
was taken at. A rate without its shape is not a measurement; a gauge without its window is not a rate.
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
---
|
||||||
|
name: script-refactor-parity-vs-previous-revision
|
||||||
|
description: "Prove a refactor of sibling data-pipeline scripts is behavior-identical: run the pre-refactor revision from git HEAD beside the new one on the host that owns the venv/data, compare parsed records rather than bytes, and bootstrap the shared module so file, runpy and container paths all import it."
|
||||||
|
---
|
||||||
|
|
||||||
|
Refactoring a pipeline of sibling scripts (corpus builders, re-chunkers, pushers) whose only
|
||||||
|
tests are smokes. The test suite cannot see a moved record boundary, so parity has to be
|
||||||
|
*measured against the previous revision*.
|
||||||
|
|
||||||
|
## 1. Keep the old revision runnable
|
||||||
|
|
||||||
|
Never trust the diff. Materialize both sides where the data lives (the host with the venv,
|
||||||
|
tokenizer, parquet, corpus):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git show HEAD:scripts/<tool>.py > /tmp/qc-old/<tool>.py # pre-refactor
|
||||||
|
cp scripts/<tool>.py scripts/<newmodule>.py /tmp/qc-new/ # post, module beside it
|
||||||
|
```
|
||||||
|
|
||||||
|
`ssh host 'cat > /tmp/...'` with the bytes on stdin is enough; no repo on the target needed.
|
||||||
|
Run both with **identical seeds and flags**, into separate output dirs.
|
||||||
|
|
||||||
|
## 2. Compare records, not bytes — unless the escaping is unchanged
|
||||||
|
|
||||||
|
- If the refactor funnels writes through one helper, that helper's `ensure_ascii` choice moves
|
||||||
|
every non-ASCII character. A byte diff then reports a false regression while the rows are
|
||||||
|
identical.
|
||||||
|
- Prove it correctly: parse both sides and compare in order —
|
||||||
|
|
||||||
|
```python
|
||||||
|
a = [json.loads(l) for l in open(old) if l.strip()]
|
||||||
|
b = [json.loads(l) for l in open(new) if l.strip()]
|
||||||
|
print(a == b, hashlib.sha256(json.dumps(a, sort_keys=True).encode()).hexdigest()[:16])
|
||||||
|
```
|
||||||
|
|
||||||
|
- Where the refactor does not touch serialization (a tool that already wrote literal UTF-8),
|
||||||
|
demand the stronger proof: `cmp` + md5 equality of the artifact.
|
||||||
|
- Order matters: shuffled output with an unchanged seed proves the RNG draw sequence survived
|
||||||
|
the hoist of nested closures. Compare the *sorted* file only as a second check — permutation
|
||||||
|
equality alone hides a reordered `rng.choice`.
|
||||||
|
- Hashing the record set with `sort_keys=True` also catches dict-key drift in manifests.
|
||||||
|
|
||||||
|
## 3. Sibling-module bootstrap that survives every invocation
|
||||||
|
|
||||||
|
A new shared module must import when the tool is run three ways: `python3 scripts/tool.py`,
|
||||||
|
`runpy.run_path('/work/tool.py')` from a container mount (`-v scripts:/work:ro`), and from any
|
||||||
|
cwd. Standard preamble:
|
||||||
|
|
||||||
|
```python
|
||||||
|
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
|
||||||
|
if _HERE not in sys.path:
|
||||||
|
sys.path.insert(0, _HERE)
|
||||||
|
import <module> # noqa: E402 - sibling module, same directory
|
||||||
|
```
|
||||||
|
|
||||||
|
Piping the script through `python3 -` has no `__file__` and defeats this — run tools by path and
|
||||||
|
say so in the module docstring. Verify both forms explicitly before claiming it works.
|
||||||
|
|
||||||
|
## 4. Unit-test the refusals without the heavy dependency
|
||||||
|
|
||||||
|
Hoist validation into a function so it can be driven with a stub, in-process:
|
||||||
|
|
||||||
|
```python
|
||||||
|
spec = importlib.util.spec_from_file_location("<module>", ".../<module>.py")
|
||||||
|
m = importlib.util.module_from_spec(spec); sys.modules["<module>"] = m; spec.loader.exec_module(m)
|
||||||
|
```
|
||||||
|
|
||||||
|
Registering the module in `sys.modules` under its real name *before* loading the dependent
|
||||||
|
script is required, or the script's own `import <module>` re-imports it and your stub is invisible.
|
||||||
|
A stub object with `encode`/`decode` that mangles ids past a threshold exercises a
|
||||||
|
"token identity" check in one page of output, no tokenizer file needed.
|
||||||
|
|
||||||
|
## 5. Gate order
|
||||||
|
|
||||||
|
`py_compile` every touched file → run the tool's `--help` through both invocation forms →
|
||||||
|
old-vs-new parity on the host → suite → commit. A suite that enumerates shell seams will not
|
||||||
|
notice a corpus-tool regression; the parity run is the only evidence there.
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
---
|
||||||
|
name: sftp-client-unreachable-triage
|
||||||
|
description: "Diagnose \"SFTP/SSH from box A to box B doesn't work\" when box A is unreachable from the fleet: exonerate the server with a log+key census, then discriminate sleep vs firewall vs name-resolution vs missing key from outside."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Triage a client-side SFTP/SSH failure (client box unreachable)
|
||||||
|
|
||||||
|
Use when Frank reports "FileZilla/SFTP/scp from <client> to <server> isn't working" and the
|
||||||
|
client box is not reachable from the fleet. Produces a ranked, evidence-backed verdict
|
||||||
|
without access to the client. **Read-only probes only**; no restarts, no config writes.
|
||||||
|
|
||||||
|
## Order of operations (each step can end the investigation)
|
||||||
|
|
||||||
|
### 1. Server-side exoneration — gather before theorising
|
||||||
|
```bash
|
||||||
|
ssh -o BatchMode=yes <server> 'sudo -n sshd -T | grep -Ei "^(port|listenaddress|pubkeyauth|passwordauth|kbdinteractive|allowusers|subsystem|authorizedkeysfile)"'
|
||||||
|
ssh -o BatchMode=yes <server> 'ls -l /usr/lib/openssh/sftp-server; sudo -n ls -l /etc/ssh/ssh_host_*key.pub' # host-key mtime = known-hosts churn check
|
||||||
|
ssh -o BatchMode=yes <server> 'sudo -n iptables -S | head; sudo -n nft list ruleset | head -20' # INPUT ACCEPT = no firewall
|
||||||
|
```
|
||||||
|
Host keys unchanged for years ⇒ FileZilla cannot be tripping a host-key warning.
|
||||||
|
|
||||||
|
### 2. Censuses that turn "maybe" into "never" (the payload of this skill)
|
||||||
|
```bash
|
||||||
|
# keys trusted by the server, by comment
|
||||||
|
ssh <server> 'ssh-keygen -lf ~/.ssh/authorized_keys'
|
||||||
|
|
||||||
|
# every IP that ever authenticated, and with which key
|
||||||
|
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-90 days" \
|
||||||
|
| grep -oE "Accepted publickey for [a-z]+ from [0-9.]+" | awk "{print \$NF}" | sort | uniq -c'
|
||||||
|
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-90 days" \
|
||||||
|
| grep -oE "ED25519 SHA256:[A-Za-z0-9+/=]+" | sort | uniq -c'
|
||||||
|
|
||||||
|
# did the client IP EVER appear, anywhere?
|
||||||
|
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-45 days" | grep -c "<client-ip>"'
|
||||||
|
ssh <server> 'sudo -n zgrep -h "<client-ip>" /var/log/auth.log* | grep -v sudo | tail'
|
||||||
|
```
|
||||||
|
Count 0 across journal **and** rotated logs ⇒ no TCP ever established ⇒ the failure is
|
||||||
|
DNS, client egress, or a sleeping host — **not** credentials. Say so explicitly, because
|
||||||
|
"permission denied" and "connection timed out" have different fixes.
|
||||||
|
|
||||||
|
### 3. Is the client even at the IP everyone believes? (DHCP drift)
|
||||||
|
```bash
|
||||||
|
avahi-resolve -n <Client>.local # mDNS truth
|
||||||
|
ip neigh | grep -E '\.(old|new) ' # FAILED vs REACHABLE
|
||||||
|
nmap -sn -n 192.168.0.0/24 # works unprivileged; arp-scan needs root
|
||||||
|
```
|
||||||
|
macOS **Private Wi-Fi Address** rotation changes the MAC ⇒ new DHCP lease ⇒ every
|
||||||
|
hardcoded IP for that box rots. Check both old and new addresses for mentions in
|
||||||
|
`/var/log/auth.log*` and `journalctl`.
|
||||||
|
|
||||||
|
### 4. Discriminate sleep vs firewall (ARP/mDNS alive, everything else dead)
|
||||||
|
- ARP replies + live mDNS but **zero** ICMP and **zero** TCP on a spread of ports
|
||||||
|
(22, 80, 445, 548, 5900, 8080, 5000, 7000, 3283) from **multiple** hosts including a
|
||||||
|
wired one: probe from a wired box on the same L2 — that also rules out Wi-Fi AP isolation.
|
||||||
|
- Same result ⇒ either the host is asleep (Bonjour sleep proxy / AP answers ARP+mDNS on its
|
||||||
|
behalf) **or** a host packet filter (macOS "Block all incoming", LuLu/Little Snitch,
|
||||||
|
ProtonVPN kill switch) is dropping it.
|
||||||
|
- **Do not overclaim which.** State both, and note the built-in macOS firewall only blocks
|
||||||
|
*inbound* — so if Frank is actively using that Mac, a built-in-firewall verdict cannot
|
||||||
|
explain an *outbound* SFTP failure; a third-party filter/VPN or DNS is then the cause.
|
||||||
|
|
||||||
|
### 5. Name resolution (the silent failure)
|
||||||
|
```bash
|
||||||
|
getent hosts <server> <server>.local <server>.<domain>
|
||||||
|
dig +short <server> @<pihole-ip>
|
||||||
|
```
|
||||||
|
Traps seen on this fleet:
|
||||||
|
- Bare `dankserver` resolves **nowhere** — no Pi-hole custom record, gateway DNS refuses
|
||||||
|
client queries. Only mDNS `<host>.local`, which may advertise a **link-local IPv6 only**.
|
||||||
|
- `dankserver.franksdank.cloud` resolves to **Caddy (192.168.0.143)**, not the SSH box —
|
||||||
|
a wildcard vhost. Never suggest a `*.franksdank.cloud` name for SSH/SFTP.
|
||||||
|
- **FileZilla ignores `~/.ssh/config`** — `Host dankserver` aliases do not apply; the Host
|
||||||
|
field needs a literal IP that resolves.
|
||||||
|
|
||||||
|
### 6. Fix recipe handed to the user (client side)
|
||||||
|
```bash
|
||||||
|
ping -c2 <server-ip> && nc -vz <server-ip> 22
|
||||||
|
ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519 -C "<client-name>"
|
||||||
|
ssh-copy-id -i ~/.ssh/id_ed25519.pub dank@<server-ip>
|
||||||
|
sftp -i ~/.ssh/id_ed25519 dank@<server-ip>
|
||||||
|
```
|
||||||
|
FileZilla: Protocol **SFTP**, Host literal IP, Port 22, Logon Type **Key file** →
|
||||||
|
`/Users/dank/.ssh/id_ed25519` (or Normal + password). The "Encryption" dropdown is FTP-only
|
||||||
|
noise for SFTP sites.
|
||||||
|
|
||||||
|
### 7. Report collateral you proved, don't guess it
|
||||||
|
Dead client IP ⇒ grep the fleet for hardcoded references and *observe* the breakage:
|
||||||
|
```bash
|
||||||
|
grep -rn --exclude-dir=.git "<old-ip>" ~/Projects/serverAppConfig | wc -l
|
||||||
|
curl -sk -o /dev/null -w '%{http_code}\n' --resolve "radarr.franksdank.cloud:443:<caddy-ip>" https://radarr.franksdank.cloud/
|
||||||
|
```
|
||||||
|
(`-H Host:` alone gives `000` — curl sends the IP as TLS SNI and the handshake fails. Use
|
||||||
|
`--resolve` to keep SNI correct before concluding anything about a vhost.)
|
||||||
|
|
||||||
|
## Laws
|
||||||
|
- Zero packets at the server beats any credential theory; log census first, key census second.
|
||||||
|
- ARP/mDNS answering is **not** proof a host is awake; the AP or a sleep proxy can answer.
|
||||||
|
- Probe from a wired host before blaming Wi-Fi isolation.
|
||||||
|
- A rotating private Wi-Fi MAC will re-break every hardcoded IP: recommend a DHCP
|
||||||
|
reservation as the fix, not 40 file edits.
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"v": 1,
|
"v": 1,
|
||||||
"lastConsolidatedAt": 1793308587833,
|
"lastConsolidatedAt": 1793314235443,
|
||||||
"lastResult": {
|
"lastResult": {
|
||||||
"at": 1791599865775,
|
"at": 1791599865775,
|
||||||
"sessions": 1,
|
"sessions": 1,
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
{
|
{
|
||||||
"v": 1,
|
"v": 1,
|
||||||
"lastConsolidatedAt": 1793215693649
|
"lastConsolidatedAt": 1793310987675
|
||||||
}
|
}
|
||||||
@@ -1,7 +1,5 @@
|
|||||||
## Product Quality Standards
|
## Product Quality Standards
|
||||||
|
|
||||||
Quant quality must hold regardless of implementation choice.
|
|
||||||
|
|
||||||
## Behavioral Updates
|
## Behavioral Updates
|
||||||
|
|
||||||
Hourly updates on a deterministic timer.
|
Merge any fixes immediately.
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
{
|
{
|
||||||
"v": 1,
|
"v": 1,
|
||||||
"lastConsolidatedAt": 1793309257223,
|
"lastConsolidatedAt": 1793320373858,
|
||||||
"lastResult": {
|
"lastResult": {
|
||||||
"at": 1793218183375,
|
"at": 1793315963987,
|
||||||
"sessions": 1,
|
"sessions": 1,
|
||||||
"deltas": 1,
|
"deltas": 1,
|
||||||
"model": "frank-zealot"
|
"model": "frank-zealot"
|
||||||
|
|||||||
Whitespace-only changes.
@@ -0,0 +1,4 @@
|
|||||||
|
## Models
|
||||||
|
|
||||||
|
- Models settings include /switch to switch to it
|
||||||
|
- hdmi monitor to the left and the other to the right of eDP-1
|
||||||
@@ -1,4 +1,10 @@
|
|||||||
{
|
{
|
||||||
"v": 1,
|
"v": 1,
|
||||||
"lastConsolidatedAt": 1793309241992
|
"lastConsolidatedAt": 1793312076890,
|
||||||
|
"lastResult": {
|
||||||
|
"at": 1793312076890,
|
||||||
|
"sessions": 1,
|
||||||
|
"deltas": 1,
|
||||||
|
"model": "frank-zealot"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Whitespace-only changes.
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"v": 1,
|
||||||
|
"lastConsolidatedAt": 1793310990286
|
||||||
|
}
|
||||||
Whitespace-only changes.
Whitespace-only changes.
@@ -0,0 +1,10 @@
|
|||||||
|
{
|
||||||
|
"v": 1,
|
||||||
|
"lastConsolidatedAt": 1793315256444,
|
||||||
|
"lastResult": {
|
||||||
|
"at": 1793314927653,
|
||||||
|
"sessions": 1,
|
||||||
|
"deltas": 1,
|
||||||
|
"model": "frank-zealot"
|
||||||
|
}
|
||||||
|
}
|
||||||
Whitespace-only changes.
@@ -107,11 +107,11 @@ widget_order = [
|
|||||||
[lockscreen_widgets.widget."lockscreen-login-box@DP-2"]
|
[lockscreen_widgets.widget."lockscreen-login-box@DP-2"]
|
||||||
box_height = 196.0
|
box_height = 196.0
|
||||||
box_width = 810.0
|
box_width = 810.0
|
||||||
cx = 1719.999755859375
|
cx = 599.9998779296875
|
||||||
cy = 1221.60009765625
|
cy = 1628.8001708984375
|
||||||
output = "DP-2"
|
output = "DP-2"
|
||||||
placement_height = 1440.0
|
placement_height = 1920.0
|
||||||
placement_width = 3440.0
|
placement_width = 1200.0
|
||||||
rotation = 0.0
|
rotation = 0.0
|
||||||
type = "login_box"
|
type = "login_box"
|
||||||
|
|
||||||
@@ -279,13 +279,13 @@ transition = [ "fade" ]
|
|||||||
interval_seconds = 1200
|
interval_seconds = 1200
|
||||||
|
|
||||||
[wallpaper.default]
|
[wallpaper.default]
|
||||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||||
|
|
||||||
[wallpaper.last]
|
[wallpaper.last]
|
||||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||||
|
|
||||||
[wallpaper.monitors.DP-1]
|
[wallpaper.monitors.DP-1]
|
||||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||||
|
|
||||||
[wallpaper.monitors.DP-2]
|
[wallpaper.monitors.DP-2]
|
||||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||||
@@ -294,7 +294,7 @@ transition = [ "fade" ]
|
|||||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||||
|
|
||||||
[wallpaper.monitors.eDP-1]
|
[wallpaper.monitors.eDP-1]
|
||||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||||
|
|
||||||
[weather]
|
[weather]
|
||||||
unit = "imperial"
|
unit = "imperial"
|
||||||
|
|||||||
Reference in new issue
Block a user