Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
59920f849d | ||
|
|
bfd6a6340e | ||
|
|
4849f46eb2 | ||
|
|
289f74062a | ||
|
|
d7bd05398d | ||
|
|
004962ffce | ||
|
|
56fc899763 | ||
|
|
35ceae6d7b | ||
|
|
655a39ffb6 |
No files matched your search
+18
-3
@@ -54,13 +54,27 @@ Suggest these tools or languages as possible solutions to problems if they are a
|
|||||||
|
|
||||||
### 1. Maintain your projects source of truth.
|
### 1. Maintain your projects source of truth.
|
||||||
|
|
||||||
The knowledge base at `docs/` is the source of truth for your project. Your training is out of date. Default to the documentation and the code before you assume anything. Maintain a work log of actions taken and their result. Refer to the work log to remember the progress of your tasks.
|
Documentation exists for ONE reason: **point of use**. A page earns its bytes only if it helps you perform a task at the moment you perform it — the shape to deploy, the flag that must be passed, the failure mode to recognise, the rollback to run. It is NOT a museum, NOT a lab notebook, and NOT a place to keep track of numbers.
|
||||||
|
|
||||||
|
**THE PURPOSE TEST — apply it before you write, and again before you keep:**
|
||||||
|
- Would an agent or Frank, doing a real task tomorrow, read this line and act differently because of it? If no: **delete it.**
|
||||||
|
- Is this a measured number (t/s, tokens, ms, MiB, %, bench ladder, acceptance rate) for a model, runtime, or hardware configuration that is NOT currently deployed and NOT currently being compared? Then it is **dead weight — destroy it**, do not archive it, do not "preserve it in an Eras row", do not cite it. AI advances make such a figure wrong within weeks, and a wrong number in a runbook is worse than no number: it gets believed.
|
||||||
|
- "It is the only copy of that fact" is NOT a reason to keep it. Sole custody of a useless fact is still useless. Losing information is an acceptable — often correct — outcome; do not hesitate to lose it.
|
||||||
|
- What survives the test: live configuration and its exact numbers, the procedure that must be followed, the law/causal story that prevents a repeat failure, security posture, and the interface contract a consumer depends on. Qualitative laws outlive the measurement that taught them — keep the law, drop the numbers.
|
||||||
|
- Never let a tool's "this is duplicated / this is the only record" census decide what to keep. That is a question of USEFULNESS, which only the operator answers.
|
||||||
|
|
||||||
1. Read the relevant `index.md` before you touch a topic.
|
1. Read the relevant `index.md` before you touch a topic.
|
||||||
2. Follow `omp-docs/references/okf-spec.md` for every doc you create or edit.
|
2. Follow `omp-docs/references/okf-spec.md` for every doc you create or edit.
|
||||||
3. Read the target file completely before you edit it.
|
3. Read the target file completely before you edit it.
|
||||||
4. Always update the documentation when you've completed a task.
|
4. Always update the documentation when you've completed a task — and when updating means deleting a page or a paragraph, delete it.
|
||||||
5. Compact the work log when necessary.
|
5. Compact the work log when necessary; a work log holds laws and incidents, never bookkeeping.
|
||||||
|
|
||||||
|
#### THE WORK LOG IS ONE FILE.
|
||||||
|
The work log is a single page in the knowledge base: `omp-docs/log.md` in this repo, or whichever change-log page the repo's docs index names. Write the entry when the change lands — in the same commit as the change — and read that page to recover where things stand before starting. It follows its own entry law: a dated entry is what changed plus the durable lesson, a few lines.
|
||||||
|
|
||||||
|
- NEVER create a second log: not per directory, not per project, feature, branch or run, not `NOTES.md` / `PROGRESS.md` / `STATUS.md` / `worklog/`, not a dated diary. Logging in fifty places is not logging — the entry is never where the next reader looks, and each copy re-dates the same event on its own authority. In `dank/command-center` the KB linter reports any `log.md` that is not the bundle root's.
|
||||||
|
- A log holds laws and incidents, never bookkeeping: no per-file inventories, no run-by-run numbers, no transcripts. A number in a log still has to pass the purpose test above.
|
||||||
|
- If a knowledge base has no log page, add one at its root and list it in the index. That is the fix — never a log somewhere else.
|
||||||
|
|
||||||
#### USE PROGRESSIVE DISCLOSURE.
|
#### USE PROGRESSIVE DISCLOSURE.
|
||||||
Always ensure that all produced documentation is laid out in files following WIKI conventions: start with a top level README.md index file that provides a brief, concise introduction to the item being documented, and then SPLIT OFF individual aspects, features, or arguments into their own interlinked pages so that future agents can read the documentation in a piecewise fashion, only taking in the details they need. Where relevant, crosslink any created documentation with other relevant documentation. This structure may also be nested, if necessary.
|
Always ensure that all produced documentation is laid out in files following WIKI conventions: start with a top level README.md index file that provides a brief, concise introduction to the item being documented, and then SPLIT OFF individual aspects, features, or arguments into their own interlinked pages so that future agents can read the documentation in a piecewise fashion, only taking in the details they need. Where relevant, crosslink any created documentation with other relevant documentation. This structure may also be nested, if necessary.
|
||||||
@@ -90,6 +104,7 @@ Never assume the user reports correct information. He is human, and humans make
|
|||||||
- Before starting new work, `git pull` from the remote, and start a new branch in a separate worktree using `master` or `main` as the base branch.
|
- Before starting new work, `git pull` from the remote, and start a new branch in a separate worktree using `master` or `main` as the base branch.
|
||||||
- Commit atomically: after every change that forms a complete functional unit, commit. Do so early and often.
|
- Commit atomically: after every change that forms a complete functional unit, commit. Do so early and often.
|
||||||
- When the work is complete, ask the user if they would like to open a pull request.
|
- When the work is complete, ask the user if they would like to open a pull request.
|
||||||
|
- **Report a pull request as link AND title, every time.** A bare `#54` is not a reference: numbers are per-repository, so a number alone does not say which work, in which repo, is being reported. Give the PR's web URL on that forge — on the Dankclaw Gitea that is `https://gitea.franksdank.cloud/<owner>/<repo>/pulls/N` — together with the PR's title. Applies everywhere a PR is mentioned: chat replies, status summaries, issue and commit bodies, alerts. Another forge (GitHub, GitLab, …): the equivalent URL its web UI serves, same pairing.
|
||||||
|
|
||||||
### 5. Test Driven Development
|
### 5. Test Driven Development
|
||||||
|
|
||||||
|
|||||||
@@ -16,6 +16,4 @@ These files define the structural conventions that govern the agent system itsel
|
|||||||
* [OMP Status Line Customization](statusline.md) — How presets, composer shapes, and segment arrays actually interact; the fleet's tuned bar. (moved from /pi/, 2026-10-19)
|
* [OMP Status Line Customization](statusline.md) — How presets, composer shapes, and segment arrays actually interact; the fleet's tuned bar. (moved from /pi/, 2026-10-19)
|
||||||
* [OMP Subagent Limits & Fan-out Policy](subagent-limits.md) — Live caps: concurrency 2, no nested spawns (the `maxRecursionDepth: 0` trap), per-endpoint in-flight ceilings, isolation policy. (moved from /pi/, 2026-10-19)
|
* [OMP Subagent Limits & Fan-out Policy](subagent-limits.md) — Live caps: concurrency 2, no nested spawns (the `maxRecursionDepth: 0` trap), per-endpoint in-flight ceilings, isolation policy. (moved from /pi/, 2026-10-19)
|
||||||
|
|
||||||
## Change Tracking
|
**Change log:** the bundle keeps exactly one, one level up — [Bundle Change Log](../log.md). A directory here never has its own.
|
||||||
|
|
||||||
* [Bundle Change Log](log.md) — Chronological update history.
|
|
||||||
@@ -1,21 +0,0 @@
|
|||||||
---
|
|
||||||
type: changelog
|
|
||||||
title: Core Bundle Change Log
|
|
||||||
description: Change log for the core documentation directory — tracks migrations and structural changes.
|
|
||||||
tags:
|
|
||||||
- core
|
|
||||||
- changelog
|
|
||||||
- okf
|
|
||||||
timestamp: 2026-07-18
|
|
||||||
---
|
|
||||||
|
|
||||||
# Core Bundle Change Log
|
|
||||||
|
|
||||||
## 2026-07-18
|
|
||||||
- **Migration**: Converted `index.md` (root) to `core/index-doc.md` — moved root bundle index into core domain
|
|
||||||
- **Migration**: Converted `agents-md-format.md` (root) to `core/agents-md-format.md` — agents-md format belongs in core
|
|
||||||
- **Creation**: Established `core/` directory with `index.md` as the domain entry point
|
|
||||||
- **Creation**: This change log for tracking core directory modifications
|
|
||||||
|
|
||||||
## 2026-07-15
|
|
||||||
- **Initialization**: OKF v0.1 conversion of the entire docs bundle — frontmatter applied, structure established, first `log.md` entries created
|
|
||||||
@@ -16,6 +16,50 @@ lesson, a few lines. Full incident narratives are recoverable from git
|
|||||||
(`git log --follow omp-docs/log.md`); knowledge that outlives an entry lives
|
(`git log --follow omp-docs/log.md`); knowledge that outlives an entry lives
|
||||||
on its concept page, linked here. Never re-expand an entry in place.
|
on its concept page, linked here. Never re-expand an entry in place.
|
||||||
|
|
||||||
|
This page is the bundle's **only** work log: one file, at the root. Never start a
|
||||||
|
second one in a subdirectory or under another name — `scripts/kb-lint.sh` reports any
|
||||||
|
`log.md` that is not this one.
|
||||||
|
|
||||||
|
## 2026-10-20 — one work log per bundle
|
||||||
|
|
||||||
|
The bundle had two: this page and `core/log.md`, created "for tracking core
|
||||||
|
directory modifications" — two entries of pure migration bookkeeping (file moved,
|
||||||
|
directory created, log created), and a date for the OKF conversion that neither this
|
||||||
|
page nor git agrees with: 2026-07-15 there, 2026-01-15 here, while `git log` dates the
|
||||||
|
OKF rebuild 2026-08-28. `core/index.md` was still pointing readers at it, which is how
|
||||||
|
a stale page stays alive. Trashed to `omp-docs/core/.AGENTTRASH/log.md` — its content
|
||||||
|
is recoverable from git, which is the entire argument against a second log.
|
||||||
|
|
||||||
|
Laws, now written down in [references/okf-spec.md](references/okf-spec.md) and
|
||||||
|
[references/okf-maintenance.md](references/okf-maintenance.md), and gated: a bundle has
|
||||||
|
ONE work log, `log.md`, at its root, and `scripts/kb-lint.sh` reports any `log.md` that
|
||||||
|
is not the root's — so the sprawl that made logging useless once cannot restart
|
||||||
|
quietly. The agent-side duty is in `agent/APPEND_SYSTEM.md` § 1: log here, read here,
|
||||||
|
never start another.
|
||||||
|
|
||||||
|
## 2026-10-20 — two instruments rescued from `$HOME` script sediment
|
||||||
|
|
||||||
|
`$HOME` had accumulated 123 loose scripts at its root (archived to
|
||||||
|
`.AGENTTRASH/home-scripts-2026-10-20`, ~1.8 GiB of stray audio consolidated into
|
||||||
|
`~/audiosamples`). Two were durable instruments and now live in the repo with
|
||||||
|
black-box tests and pages: [tools/remote-job](tools/remote-job.md) and
|
||||||
|
[tools/pcie-probe](tools/pcie-probe.md). Refs #53. Laws learned:
|
||||||
|
|
||||||
|
- **A readiness pattern in a log is not evidence of completion.** `ssh host 'cmd |
|
||||||
|
tail -f log | sed -n /ready/q'` fails three independent ways — the broker aborts
|
||||||
|
the long connection, the pattern can match a header printed before the work
|
||||||
|
starts, and `sed -n /x/q` exits the moment it matches, tearing down the job it was
|
||||||
|
watching. The only honest signal is a sentinel the job writes last, read over a
|
||||||
|
short-lived connection.
|
||||||
|
- **Never benchmark beside a live service, and never let a tool decide to stop one.**
|
||||||
|
The guard refuses before anything is built; unloading a production endpoint is a
|
||||||
|
human's call. A number taken as noise beside serving traffic is both disruptive and
|
||||||
|
wrong.
|
||||||
|
- **An instrument that segfaults is not an instrument.** Two of the three shipped
|
||||||
|
`.hip` probes called `hipGetDeviceProperties`, which crashes on this ROCm build
|
||||||
|
while `hipGetDeviceCount` works fine; device identity now comes from sysfs, and the
|
||||||
|
trap is recorded in the sources so it is not reintroduced.
|
||||||
|
|
||||||
## 2026-10-20 — the secret gate was reading the wrong files, and one credential had already escaped
|
## 2026-10-20 — the secret gate was reading the wrong files, and one credential had already escaped
|
||||||
|
|
||||||
Found while deciding what else belongs in `dotfiles/`. (1) The gate whitelisted
|
Found while deciding what else belongs in `dotfiles/`. (1) The gate whitelisted
|
||||||
|
|||||||
@@ -21,32 +21,46 @@ Knowledge bundles rot. It is the nature of knowledge to attract cobwebs, much li
|
|||||||
|
|
||||||
## Change Tracking with `log.md`
|
## Change Tracking with `log.md`
|
||||||
|
|
||||||
Every OKF bundle has a `log.md` file. It is not optional. It is the bundle's memory.
|
Every OKF bundle has a `log.md`, and it is the bundle's **only** work log: one file,
|
||||||
|
at the bundle root. A subdirectory never gets its own — not `core/log.md`, not
|
||||||
|
`notes.md`, not a per-project, per-branch or per-run diary. Sprawl is not caution:
|
||||||
|
with fifty logs the entry is never where the next agent looks, and each copy re-dates
|
||||||
|
the same event on its own authority. `scripts/kb-lint.sh` reports any `log.md` that is
|
||||||
|
not the root's.
|
||||||
|
|
||||||
**Format:** Date-grouped entries, newest first. One line per change with timestamp, file path, and what changed.
|
**Format:** one dated section per day, newest first; a few lines per change — what
|
||||||
|
happened, and the durable lesson. It is not a file manifest.
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
# Bundle Change Log
|
# Bundle Change Log
|
||||||
|
|
||||||
## 2026-01-15
|
## 2026-01-15 — the gate was reading the wrong files
|
||||||
- `/docs/ghostty.md` — Updated keybind table after v1.2 release
|
|
||||||
- `/docs/pi-permission-system/` — Added `policy.md` and `audit.md`
|
A gate that gets an empty file list reports success. Law: a check MUST fail closed;
|
||||||
- `/docs/index.md` — Synced with directory contents
|
a silently-passing gate is worse than none, because it buys the belief of safety.
|
||||||
|
Details: [secret gate](../tools/index.md).
|
||||||
```
|
```
|
||||||
|
|
||||||
**Rules:**
|
**Rules:**
|
||||||
- **Newest entries first.** Putting old entries at the top inverts the timeline and makes it impossible to see what just happened.
|
- **Newest entries first.** Old entries at the top invert the timeline.
|
||||||
- **One line per change.** A single line is sufficient. The git history is there if someone wants the gory details.
|
- **What changed + the durable lesson, a few lines.** Full narratives are recoverable
|
||||||
- **Group by date.** Batch all changes from the same day together.
|
from git (`git log --follow log.md`); never re-expand one in place.
|
||||||
- **Always date-stamp the section.** Use `## YYYY-MM-DD`.
|
- **Laws and incidents, never bookkeeping.** No per-file inventories, no run-by-run
|
||||||
- **Mention `index.md` whenever you change a bundle.** If you added or removed a file, update the index and note it in the log.
|
numbers, no transcripts. Measured numbers still have to pass the purpose test.
|
||||||
|
- **Link, do not duplicate.** Knowledge that outlives the entry belongs on its concept
|
||||||
|
page; the entry points there.
|
||||||
|
- **One dated section per day** (`## YYYY-MM-DD`), and mention `index.md` whenever a
|
||||||
|
file is added or removed.
|
||||||
|
|
||||||
### Common Mistakes
|
### Common Mistakes
|
||||||
|
|
||||||
- ❌ Writing `"Updated ghostty.md"` — Updated it with what? Be specific.
|
- ❌ `Updated ghostty.md` — updated it with what, and what did it teach?
|
||||||
- ❌ Forgetting to write to `log.md` — This is the equivalent of doing surgery without keeping a record.
|
- ❌ Forgetting the entry — the change lands undocumented and the next agent rediscovers
|
||||||
- ❌ Using relative paths — Use absolute paths (`/docs/ghostty.md`).
|
the failure the hard way.
|
||||||
- ❌ Leaving stale entries from manual edits — The log is the single source of truth.
|
- ❌ Starting a second log in a subdirectory — the sprawl that makes logging useless.
|
||||||
|
- ❌ Invented absolute paths in links — links resolve relative to the page, and the
|
||||||
|
linter checks them.
|
||||||
|
- ❌ Re-expanding an old entry in place — the entry is a pointer; git is the transcript.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -84,6 +98,14 @@ OKF is built on the principle that a file is one concept.
|
|||||||
- ❌ Don't create a file called `misc.md` — `misc` is a graveyard, not a category.
|
- ❌ Don't create a file called `misc.md` — `misc` is a graveyard, not a category.
|
||||||
- ❌ Don't use `# Other` or `# Additional Notes` sections — these are taping things to the wall.
|
- ❌ Don't use `# Other` or `# Additional Notes` sections — these are taping things to the wall.
|
||||||
|
|
||||||
|
## Quarantine, Not Deletion
|
||||||
|
|
||||||
|
A page is never deleted outright: it moves to `.AGENTTRASH/` beside where it lived and
|
||||||
|
waits for a pruning session with Frank present. The quarantine is not part of the
|
||||||
|
bundle — `scripts/kb-lint.sh` and `scripts/kb-coverage.py` skip any path containing
|
||||||
|
`.AGENTTRASH`. That exemption is deliberate: a gate that goes green only once the
|
||||||
|
quarantined copy is gone is a gate that deletes history.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Auto-Generation Strategies
|
## Auto-Generation Strategies
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ An OKF knowledge bundle is a directory tree. Every concept is one file. The file
|
|||||||
```
|
```
|
||||||
my-bundle/
|
my-bundle/
|
||||||
├── index.md # Optional directory listing
|
├── index.md # Optional directory listing
|
||||||
├── log.md # Optional update history
|
├── log.md # The bundle's ONE update history — root only
|
||||||
├── concept-one.md # A concept at the root level
|
├── concept-one.md # A concept at the root level
|
||||||
└── subdirectory/
|
└── subdirectory/
|
||||||
├── index.md # Optional subdirectory listing
|
├── index.md # Optional subdirectory listing
|
||||||
@@ -55,7 +55,13 @@ my-bundle/
|
|||||||
| Filename | Purpose |
|
| Filename | Purpose |
|
||||||
|----------|---------|
|
|----------|---------|
|
||||||
| `index.md` | Directory listing for progressive disclosure |
|
| `index.md` | Directory listing for progressive disclosure |
|
||||||
| `log.md` | Chronological update history |
|
| `log.md` | Chronological update history — **exactly one, at the bundle root** |
|
||||||
|
|
||||||
|
**One work log per bundle.** `log.md` is reserved, and it is reserved for a single
|
||||||
|
file: the bundle root's. A subdirectory never keeps its own change log — a bundle
|
||||||
|
that logs in fifty places has no log, because the entry is never where the next
|
||||||
|
reader looks and every copy re-dates the same event. `scripts/kb-lint.sh` reports
|
||||||
|
any `log.md` that is not the root's.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -20,6 +20,8 @@ Configuration guides and reference material for installed software. Each tool ha
|
|||||||
* [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation.
|
* [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation.
|
||||||
* [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management.
|
* [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management.
|
||||||
* [packages](packages.md) — `packages/*.pkgs` desired state and the two tools that keep a machine and its manifest agreeing: `install.sh` installs the delta, `pkg-inventory.sh` reports what the machine has outgrown.
|
* [packages](packages.md) — `packages/*.pkgs` desired state and the two tools that keep a machine and its manifest agreeing: `install.sh` installs the delta, `pkg-inventory.sh` reports what the machine has outgrown.
|
||||||
|
* [pcie-probe](pcie-probe.md) — HIP probes that separate PCIe link bandwidth from the engine's read path at the expert-slab size, and the refuse-by-default guard against benchmarking beside a live service.
|
||||||
|
* [remote-job](remote-job.md) — Detached job runner for `ssh` work: completion sentinels on the host replace the `tail -f | sed -n /ready/q` pattern that reported success on jobs that never finished.
|
||||||
* [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures.
|
* [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures.
|
||||||
|
|
||||||
## Adding a New Tool
|
## Adding a New Tool
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: pcie-probe — link or read path
|
||||||
|
description: HIP instruments that separate PCIe link bandwidth from the engine's read path at the real expert-slab size, plus the refuse-by-default guard that stops a benchmark from running beside a live service.
|
||||||
|
tags:
|
||||||
|
- gpu
|
||||||
|
- rocm
|
||||||
|
- rdna4
|
||||||
|
- benchmarking
|
||||||
|
- dankserver
|
||||||
|
verified: 2026-10-20
|
||||||
|
timestamp: 2026-10-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# pcie-probe
|
||||||
|
|
||||||
|
`scripts/pcie-probe.sh` answers one question, which comes up every time MoE
|
||||||
|
expert offload is discussed on the R9700 box
|
||||||
|
([dankserver](../system/dankserver.md)): **is the stall the PCIe link, or is it
|
||||||
|
the engine's read path?** The two look identical from the outside — decode is
|
||||||
|
slow while weights stream in — and they have opposite fixes.
|
||||||
|
|
||||||
|
## Instruments
|
||||||
|
|
||||||
|
Sources in `scripts/probes/`, built with the host `hipcc`:
|
||||||
|
|
||||||
|
| Probe | Question it answers |
|
||||||
|
|-------|---------------------|
|
||||||
|
| `hip_smoke.hip` | is host ROCm usable from a plain process at all (count, alloc, copy) |
|
||||||
|
| `pcie_bw.hip` | host→device bandwidth at the engine's exact slab size, pinned vs pageable |
|
||||||
|
| `pcie_probe.hip` | the size sweep — where throughput collapses as transfers shrink |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/pcie-probe.sh # guard, build, topology, measure
|
||||||
|
scripts/pcie-probe.sh --device 0 --device 1
|
||||||
|
scripts/pcie-probe.sh --build-only # compile, never touch the cards
|
||||||
|
```
|
||||||
|
|
||||||
|
## The law: never benchmark beside a live service
|
||||||
|
|
||||||
|
The guard runs **before anything is built or run** and refuses when the GPU is
|
||||||
|
occupied — `gpu_busy_percent` above `PCIE_PROBE_MAX_BUSY` (default 5 %) or any
|
||||||
|
process holding `/dev/kfd`. Refusal is exit `3`, and it stops nothing.
|
||||||
|
|
||||||
|
Auto-stopping a production endpoint is not this tool's decision to make, and a
|
||||||
|
measurement taken as noise beside serving traffic is worthless anyway: it is
|
||||||
|
both disruptive and wrong. Take the service down deliberately, or pass `--force`
|
||||||
|
and mean it. `tests/pcie-probe.bats` pins that the refusal happens before the
|
||||||
|
build step, so a contended box cannot be measured by accident.
|
||||||
|
|
||||||
|
## Reading the numbers
|
||||||
|
|
||||||
|
The reference is the **link**, printed as a topology block from sysfs
|
||||||
|
(`current_link_width` / `current_link_speed` against `max_*`): a card trained to
|
||||||
|
fewer lanes or a lower generation than it is wired for caps everything
|
||||||
|
downstream, and no engine change will recover it.
|
||||||
|
|
||||||
|
Then the sweep table, whose first row is the engine's actual expert-slab size —
|
||||||
|
throughput at *that* size is what matters, not the big-transfer figure:
|
||||||
|
|
||||||
|
- Small transfers reach the trained-link ceiling → the link is healthy, the
|
||||||
|
engine's read path (fetch granularity, pipelining, pinning) is the limit.
|
||||||
|
- Small transfers stall far below a large-transfer figure → the wall is the
|
||||||
|
link or the IOMMU path, not the engine.
|
||||||
|
- Pinned versus pageable columns apart means pinning is load-bearing; together
|
||||||
|
means the copies are already pinned by someone else.
|
||||||
|
|
||||||
|
Measure one device at a time (`--device`). A run that lets the runtime pick
|
||||||
|
several devices cannot attribute bandwidth to a card at all.
|
||||||
|
|
||||||
|
## The trap carried in the sources
|
||||||
|
|
||||||
|
`hipGetDeviceProperties` **segfaults** on the ROCm build these probes were
|
||||||
|
written against, while `hipGetDeviceCount` works fine and reports every device.
|
||||||
|
None of the three sources calls it; device identity comes from sysfs instead.
|
||||||
|
The setvbuf call in `hip_smoke.hip` is a fossil of discovering that — an
|
||||||
|
unflushed stdout hid the crash point for an afternoon. Keep both facts intact
|
||||||
|
when touching these files.
|
||||||
|
|
||||||
|
## Exit status and seams
|
||||||
|
|
||||||
|
`0` measured · `2` usage or no sources · `3` refused, GPU busy · `4` no HIP
|
||||||
|
toolchain or build failed. A client box without the SDK exits `4` loudly rather
|
||||||
|
than reporting an empty success; the probes belong on the box with the cards.
|
||||||
|
|
||||||
|
Test seams (all documented in the script header): `PCIE_PROBE_SRC`,
|
||||||
|
`PCIE_PROBE_BUILD`, `PCIE_PROBE_SYSFS`, `PCIE_PROBE_KFD`, `PCIE_PROBE_HIPCC`,
|
||||||
|
`PCIE_PROBE_MAX_BUSY` — the sysfs/kfd paths are what make a refusal testable on
|
||||||
|
a machine with no cards, which is the only honest way to test a refusal.
|
||||||
|
|
||||||
|
Measured campaign numbers are not kept here; they belong with the endpoint they
|
||||||
|
were taken on, in the serverAppConfigs LLM runbook vault.
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: remote-job — detached jobs that outlive the session
|
||||||
|
description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished.
|
||||||
|
tags:
|
||||||
|
- ssh
|
||||||
|
- fleet
|
||||||
|
- automation
|
||||||
|
- verification
|
||||||
|
verified: 2026-10-20
|
||||||
|
timestamp: 2026-10-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# remote-job
|
||||||
|
|
||||||
|
`scripts/remote-job.sh` starts a command **detached on the host**, and reports
|
||||||
|
what actually happened from files on disk. It exists because
|
||||||
|
`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the
|
||||||
|
worktree campaign.
|
||||||
|
|
||||||
|
## Why the pipe lies
|
||||||
|
|
||||||
|
Three independent failure modes, all of them structural:
|
||||||
|
|
||||||
|
- **The broker kills long-lived ssh.** A job longer than the connection's
|
||||||
|
patience dies mid-run; the caller sees a closed pipe, not a failure.
|
||||||
|
- **A readiness pattern can match a log header.** `sed -n /building/q` fires on
|
||||||
|
a banner printed before the work starts — the job is at line 1 and the caller
|
||||||
|
already moved on.
|
||||||
|
- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is
|
||||||
|
genuinely ready-shaped, the pipeline tears down the remote command it was
|
||||||
|
watching. Result: green report, dead job.
|
||||||
|
|
||||||
|
The only reliable signal is a **completion sentinel written by the job itself**,
|
||||||
|
read afterwards by a short-lived connection. That is what this is.
|
||||||
|
|
||||||
|
## Contract
|
||||||
|
|
||||||
|
```bash
|
||||||
|
remote-job.sh start NAME [--] CMD... # launch detached, return at once
|
||||||
|
remote-job.sh status [NAME] # one line per job; exit 0 only if all DONE
|
||||||
|
remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines
|
||||||
|
remote-job.sh out NAME [N] # tail a job's output
|
||||||
|
remote-job.sh tail [NAME] # follow one job, or every job at once
|
||||||
|
remote-job.sh watch # refreshing status, for a spare pane
|
||||||
|
```
|
||||||
|
|
||||||
|
Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+
|
||||||
|
stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**).
|
||||||
|
|
||||||
|
Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or
|
||||||
|
timed out · `2` usage. The `rc=` column is the job's own code — a job that
|
||||||
|
failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and
|
||||||
|
"did it work" are different questions a supervisor must be able to ask
|
||||||
|
separately.
|
||||||
|
|
||||||
|
Written for the case where the job **restarts the endpoint the agent itself is
|
||||||
|
running on**: the ssh connection dies, the session may die, and the answer is
|
||||||
|
still sitting in `$JOBS` when a session comes back.
|
||||||
|
|
||||||
|
## Use over ssh
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
|
||||||
|
bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
|
||||||
|
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect
|
||||||
|
```
|
||||||
|
|
||||||
|
Each ssh call is short, so there is nothing there to abort, and the second call
|
||||||
|
can be made from a fresh session on a machine that rebooted underneath it.
|
||||||
|
|
||||||
|
## Failure modes
|
||||||
|
|
||||||
|
| Event | Result |
|
||||||
|
|-------|--------|
|
||||||
|
| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it |
|
||||||
|
| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE |
|
||||||
|
| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status |
|
||||||
|
| Two jobs, same NAME | the second replaces the first's state files |
|
||||||
|
| Job dir missing | created on first use |
|
||||||
|
|
||||||
|
Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel
|
||||||
|
ordering — never internals).
|
||||||
+13
-1
@@ -4,7 +4,15 @@ import re, sys
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
kb = Path(sys.argv[1])
|
kb = Path(sys.argv[1])
|
||||||
issues = []
|
issues = []
|
||||||
dirs = sorted({p.parent for p in kb.rglob("*.md")})
|
|
||||||
|
|
||||||
|
def kept(p):
|
||||||
|
# `.AGENTTRASH/` is quarantine, not bundle content: a gate that flags it pushes an
|
||||||
|
# agent to delete a page that is only waiting for a pruning session with the operator.
|
||||||
|
return ".AGENTTRASH" not in p.parts
|
||||||
|
|
||||||
|
|
||||||
|
dirs = sorted({p.parent for p in kb.rglob("*.md") if kept(p)})
|
||||||
for d in dirs:
|
for d in dirs:
|
||||||
idx = d / "index.md"
|
idx = d / "index.md"
|
||||||
if not idx.is_file() and d != kb:
|
if not idx.is_file() and d != kb:
|
||||||
@@ -21,12 +29,16 @@ for d in dirs:
|
|||||||
# wikilinks too
|
# wikilinks too
|
||||||
wl_names = {m.group(1).strip().rsplit("/", 1)[-1] for m in re.finditer(r"\[\[([^\]|#]+)", text)}
|
wl_names = {m.group(1).strip().rsplit("/", 1)[-1] for m in re.finditer(r"\[\[([^\]|#]+)", text)}
|
||||||
for sib in sorted(d.iterdir()):
|
for sib in sorted(d.iterdir()):
|
||||||
|
if sib.name == ".AGENTTRASH":
|
||||||
|
continue
|
||||||
if sib.suffix != ".md" or sib.name == "index.md":
|
if sib.suffix != ".md" or sib.name == "index.md":
|
||||||
continue
|
continue
|
||||||
if sib.resolve() not in linked and sib.stem not in wl_names:
|
if sib.resolve() not in linked and sib.stem not in wl_names:
|
||||||
issues.append(f"{d.relative_to(kb)}/index.md: does not link {sib.name}")
|
issues.append(f"{d.relative_to(kb)}/index.md: does not link {sib.name}")
|
||||||
# subdirs of d linked as their index?
|
# subdirs of d linked as their index?
|
||||||
for sub in sorted(d.iterdir()):
|
for sub in sorted(d.iterdir()):
|
||||||
|
if sub.name == ".AGENTTRASH":
|
||||||
|
continue
|
||||||
if sub.is_dir() and (sub / "index.md").is_file():
|
if sub.is_dir() and (sub / "index.md").is_file():
|
||||||
subidx = (sub / "index.md").resolve()
|
subidx = (sub / "index.md").resolve()
|
||||||
dirref = sub.resolve()
|
dirref = sub.resolve()
|
||||||
|
|||||||
+22
-2
@@ -12,7 +12,15 @@ from pathlib import Path
|
|||||||
kb = Path(sys.argv[1])
|
kb = Path(sys.argv[1])
|
||||||
violations = []
|
violations = []
|
||||||
|
|
||||||
for p in sorted(kb.rglob("*.md")):
|
def pages():
|
||||||
|
"""Markdown pages that belong to the bundle. `.AGENTTRASH/` is a quarantine, not
|
||||||
|
content: linting a deleted-but-not-yet-pruned page would demand its destruction to
|
||||||
|
make the gate go green, which is the wrong incentive."""
|
||||||
|
for p in kb.rglob("*.md"):
|
||||||
|
if ".AGENTTRASH" not in p.parts:
|
||||||
|
yield p
|
||||||
|
|
||||||
|
for p in pages():
|
||||||
rel = p.relative_to(kb)
|
rel = p.relative_to(kb)
|
||||||
text = p.read_text(errors="replace")
|
text = p.read_text(errors="replace")
|
||||||
lines = text.splitlines()
|
lines = text.splitlines()
|
||||||
@@ -38,9 +46,21 @@ for p in sorted(kb.rglob("*.md")):
|
|||||||
if not isinstance(fm, dict) or not str(fm.get("type", "")).strip():
|
if not isinstance(fm, dict) or not str(fm.get("type", "")).strip():
|
||||||
violations.append(f"{rel}: frontmatter lacks required 'type:'")
|
violations.append(f"{rel}: frontmatter lacks required 'type:'")
|
||||||
|
|
||||||
|
|
||||||
|
# One work log per bundle, at the bundle root. A `log.md` is a reserved name, but
|
||||||
|
# reserved does not mean "allowed in every directory": per-directory logs are how a
|
||||||
|
# bundle ends up with fifty of them, the entry never where the next agent looks, and
|
||||||
|
# each copy dating the same event differently. A bundle with no log at all is left to
|
||||||
|
# the maintenance page to nag about; this gate's job is sprawl.
|
||||||
|
logs = sorted(str(p.relative_to(kb)) for p in pages() if p.name == "log.md")
|
||||||
|
if len(logs) > 1 or (logs and logs[0] != "log.md"):
|
||||||
|
violations.append(
|
||||||
|
"a bundle has one work log, log.md, at the bundle root; found: " + ", ".join(logs)
|
||||||
|
)
|
||||||
|
|
||||||
# link integrity: fenced code blocks hold example paths, not live links — strip first.
|
# link integrity: fenced code blocks hold example paths, not live links — strip first.
|
||||||
# Vendored spec is exempt: its example paths are illustrations, not live links.
|
# Vendored spec is exempt: its example paths are illustrations, not live links.
|
||||||
for p in sorted(kb.rglob("*.md")):
|
for p in pages():
|
||||||
if str(p.relative_to(kb)) == "references/okf-spec.md":
|
if str(p.relative_to(kb)) == "references/okf-spec.md":
|
||||||
continue
|
continue
|
||||||
text = re.sub(r"^```[^\n]*$^```[^\n]*$", "", p.read_text(errors="replace"), flags=re.M)
|
text = re.sub(r"^```[^\n]*$^```[^\n]*$", "", p.read_text(errors="replace"), flags=re.M)
|
||||||
|
|||||||
Executable
+126
@@ -0,0 +1,126 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/pcie-probe.sh — is the decode bottleneck the LINK, or the engine's read path?
|
||||||
|
#
|
||||||
|
# Instruments live in scripts/probes/: hip_smoke.hip answers "is the host ROCm usable from a plain
|
||||||
|
# process", pcie_bw.hip measures host→device bandwidth at the engine's real expert-slab size, and
|
||||||
|
# pcie_probe.hip sweeps sizes so a stall can be attributed to the link rather than to the engine.
|
||||||
|
# Written during the RDNA4 expert-offload campaign, kept because the same cards are in Dankserver.
|
||||||
|
#
|
||||||
|
# LAW — never benchmark beside a live service. The guard below runs BEFORE anything is built or run,
|
||||||
|
# and it stops nothing: putting a production endpoint down is not this tool's decision to make. When
|
||||||
|
# you have deliberately decided that contention is acceptable, say so with --force.
|
||||||
|
#
|
||||||
|
# Trap carried in the sources: hipGetDeviceProperties SEGFAULTS on this ROCm build even though
|
||||||
|
# hipGetDeviceCount works fine and reports every device. Nothing here calls it. Keep it that way.
|
||||||
|
#
|
||||||
|
# usage: pcie-probe.sh [--device N]... [--build-only] [--force]
|
||||||
|
#
|
||||||
|
# --device N measure this device; repeat for each card. Without it the probes run once with the
|
||||||
|
# environment's own device selection — one shared run cannot attribute bandwidth.
|
||||||
|
# --build-only compile and stop; never touches the cards.
|
||||||
|
# --force measure even though the guard says the GPU is busy.
|
||||||
|
#
|
||||||
|
# Environment (defaults in parentheses) — the sysfs and kfd paths are overridable so the guard can
|
||||||
|
# be tested on a machine with no cards, which is the only honest way to test a refusal:
|
||||||
|
# PCIE_PROBE_SRC probe sources (<repo>/scripts/probes)
|
||||||
|
# PCIE_PROBE_BUILD build output ($HOME/.cache/pcie-probe)
|
||||||
|
# PCIE_PROBE_SYSFS drm sysfs root (/sys/class/drm)
|
||||||
|
# PCIE_PROBE_KFD KFD device node (/dev/kfd)
|
||||||
|
# PCIE_PROBE_HIPCC compiler (hipcc)
|
||||||
|
# PCIE_PROBE_MAX_BUSY refusal threshold, percent (5)
|
||||||
|
#
|
||||||
|
# Exit status: 0 measured · 2 usage or missing sources · 3 REFUSED, GPU busy · 4 no toolchain or
|
||||||
|
# build failed.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
SRC="${PCIE_PROBE_SRC:-$(dirname "$(realpath "$0")")/probes}"
|
||||||
|
BUILD="${PCIE_PROBE_BUILD:-$HOME/.cache/pcie-probe}"
|
||||||
|
SYSFS="${PCIE_PROBE_SYSFS:-/sys/class/drm}"
|
||||||
|
KFD="${PCIE_PROBE_KFD:-/dev/kfd}"
|
||||||
|
HIPCC="${PCIE_PROBE_HIPCC:-hipcc}"
|
||||||
|
MAX_BUSY="${PCIE_PROBE_MAX_BUSY:-5}"
|
||||||
|
|
||||||
|
force=0
|
||||||
|
build_only=0
|
||||||
|
devices=()
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--device) [[ $# -ge 2 ]] || { echo "pcie-probe: --device needs an argument" >&2; exit 2; }
|
||||||
|
devices+=("$2"); shift 2 ;;
|
||||||
|
--build-only) build_only=1; shift ;;
|
||||||
|
--force) force=1; shift ;;
|
||||||
|
*) echo "usage: pcie-probe.sh [--device N]... [--build-only] [--force]" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
mapfile -t sources < <(find "$SRC" -maxdepth 1 -name '*.hip' 2>/dev/null | sort)
|
||||||
|
if (( ${#sources[@]} == 0 )); then
|
||||||
|
echo "pcie-probe: no .hip probe sources in $SRC" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- the guard, before anything touches the card --------------------------------
|
||||||
|
if (( ! force )); then
|
||||||
|
busy=""
|
||||||
|
if [[ -e "$KFD" ]] && command -v fuser > /dev/null 2>&1; then
|
||||||
|
holders="$(fuser "$KFD" 2>/dev/null | tr -s '[:space:]' ' ')"
|
||||||
|
[[ -n "${holders// /}" ]] && busy="KFD in use by:${holders}"
|
||||||
|
fi
|
||||||
|
for card in "$SYSFS"/card*; do
|
||||||
|
[[ -f "$card/device/gpu_busy_percent" ]] || continue
|
||||||
|
pct="$(cat "$card/device/gpu_busy_percent" 2>/dev/null)"
|
||||||
|
[[ "$pct" =~ ^[0-9]+$ ]] || continue
|
||||||
|
(( pct > MAX_BUSY )) && busy="${busy:+$busy; }$(basename "$card") at ${pct}%"
|
||||||
|
done
|
||||||
|
if [[ -n "$busy" ]]; then
|
||||||
|
printf 'pcie-probe: REFUSED — GPU busy (%s)\n' "$busy" >&2
|
||||||
|
printf ' never benchmark beside a live service. Take the service down deliberately,\n' >&2
|
||||||
|
printf ' or pass --force if you have decided the contention is acceptable.\n' >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- build ----------------------------------------------------------------------
|
||||||
|
if ! command -v "$HIPCC" > /dev/null 2>&1; then
|
||||||
|
printf 'pcie-probe: no HIP toolchain (%s) — run this on the box with the cards, not here\n' "$HIPCC" >&2
|
||||||
|
exit 4
|
||||||
|
fi
|
||||||
|
mkdir -p "$BUILD"
|
||||||
|
for src in "${sources[@]}"; do
|
||||||
|
bin="$BUILD/$(basename "${src%.hip}")"
|
||||||
|
if ! "$HIPCC" -O2 -o "$bin" "$src" 2>&1 | tail -3; then
|
||||||
|
printf 'pcie-probe: build failed for %s\n' "$(basename "$src")" >&2
|
||||||
|
exit 4
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if (( build_only )); then
|
||||||
|
printf 'pcie-probe: built %s probe(s) into %s\n' "${#sources[@]}" "$BUILD"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- topology: the link the numbers must be read against ------------------------
|
||||||
|
printf 'pcie-probe host=%s hipcc=%s probes=%s\n' "$(hostname)" "$HIPCC" "${#sources[@]}"
|
||||||
|
for card in "$SYSFS"/card*; do
|
||||||
|
[[ -f "$card/device/current_link_width" ]] || continue
|
||||||
|
printf ' link %-9s current x%s @ %s max x%s @ %s\n' "$(basename "$card")" \
|
||||||
|
"$(cat "$card/device/current_link_width")" \
|
||||||
|
"$(cat "$card/device/current_link_speed" 2>/dev/null || echo '?')" \
|
||||||
|
"$(cat "$card/device/max_link_width" 2>/dev/null || echo '?')" \
|
||||||
|
"$(cat "$card/device/max_link_speed" 2>/dev/null || echo '?')"
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---- measure, per device --------------------------------------------------------
|
||||||
|
(( ${#devices[@]} > 0 )) || devices=(default)
|
||||||
|
for d in "${devices[@]}"; do
|
||||||
|
if [[ $d == default ]]; then
|
||||||
|
printf '\n########## engine device selection ##########\n'
|
||||||
|
else
|
||||||
|
printf '\n########## device %s ##########\n' "$d"
|
||||||
|
fi
|
||||||
|
for name in hip_smoke pcie_bw pcie_probe; do
|
||||||
|
bin="$BUILD/$name"
|
||||||
|
[[ -x "$bin" ]] || continue
|
||||||
|
if [[ $d == default ]]; then "$bin"; else HIP_VISIBLE_DEVICES="$d" "$bin" "$d"; fi
|
||||||
|
done
|
||||||
|
done
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
// Minimal HIP smoke test: is the host ROCm usable from a plain process?
|
||||||
|
#include <cstdio>
|
||||||
|
#include <cstring>
|
||||||
|
#include <hip/hip_runtime.h>
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
int n = -1;
|
||||||
|
hipError_t e = hipGetDeviceCount(&n);
|
||||||
|
printf("hipGetDeviceCount -> %d (%s), count=%d\n",
|
||||||
|
(int)e, hipGetErrorString(e), n);
|
||||||
|
if (e != hipSuccess || n <= 0) return 1;
|
||||||
|
setvbuf(stdout, NULL, _IONBF, 0); /* an earlier crash lost the buffer, hiding the crash point */
|
||||||
|
// Device properties are deliberately NOT queried here: hipGetDeviceProperties segfaults on the
|
||||||
|
// ROCm build this probe was written for, even though hipGetDeviceCount above works fine. Card
|
||||||
|
// identity comes from sysfs anyway — scripts/pcie-probe.sh prints the link topology.
|
||||||
|
|
||||||
|
const size_t bytes = 2459136; /* 2.344 MiB: the engine's expert slab */
|
||||||
|
void *dev = NULL, *host = NULL;
|
||||||
|
if (hipMalloc(&dev, bytes) != hipSuccess) { printf("hipMalloc failed\n"); return 1; }
|
||||||
|
if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) {
|
||||||
|
printf("hipHostMalloc failed\n"); return 1;
|
||||||
|
}
|
||||||
|
memset(host, 0x5a, bytes);
|
||||||
|
|
||||||
|
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
|
||||||
|
hipDeviceSynchronize();
|
||||||
|
hipEventRecord(a, 0);
|
||||||
|
const int iters = 200;
|
||||||
|
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
|
||||||
|
hipEventRecord(b, 0); hipEventSynchronize(b);
|
||||||
|
float ms = 0; hipEventElapsedTime(&ms, a, b);
|
||||||
|
printf("\n %.3f MiB x %d iters in %.2f ms -> %.2f GB/s H2D (pinned)\n",
|
||||||
|
bytes/1048576.0, iters, ms, (double)bytes*iters/(ms/1000.0)/1e9);
|
||||||
|
printf(" Gen5 x8 ceiling = 31.5 GB/s\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
// H2D bandwidth at the engine's expert-slab size.
|
||||||
|
// Deliberately avoids hipGetDeviceProperties: it segfaults on this ROCm build
|
||||||
|
// (hipGetDeviceCount works fine and reports 3 devices).
|
||||||
|
#include <cstdio>
|
||||||
|
#include <cstring>
|
||||||
|
#include <cstdlib>
|
||||||
|
#include <hip/hip_runtime.h>
|
||||||
|
|
||||||
|
static double one(size_t bytes, bool pinned, int iters) {
|
||||||
|
void *dev = NULL, *host = NULL;
|
||||||
|
if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
|
||||||
|
if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
|
||||||
|
else { host = malloc(bytes); }
|
||||||
|
memset(host, 0x5a, bytes);
|
||||||
|
|
||||||
|
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
|
||||||
|
for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
|
||||||
|
hipDeviceSynchronize();
|
||||||
|
|
||||||
|
hipEventRecord(a, 0);
|
||||||
|
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
|
||||||
|
hipEventRecord(b, 0); hipEventSynchronize(b);
|
||||||
|
float ms = 0; hipEventElapsedTime(&ms, a, b);
|
||||||
|
double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
|
||||||
|
|
||||||
|
if (!pinned) free(host); else hipHostFree(host);
|
||||||
|
hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
|
||||||
|
return gb;
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(int argc, char **argv) {
|
||||||
|
setvbuf(stdout, NULL, _IONBF, 0);
|
||||||
|
int dev = (argc > 1) ? atoi(argv[1]) : 0;
|
||||||
|
if (hipSetDevice(dev) != hipSuccess) { printf("setDevice(%d) failed\n", dev); return 1; }
|
||||||
|
printf("device %d selected\n", dev);
|
||||||
|
|
||||||
|
size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
|
||||||
|
printf(" %10s | %7s | %7s | %7s\n", "MiB", "pinned", "pagin.", "xfer us");
|
||||||
|
printf(" %s\n", "-------------------------------------------");
|
||||||
|
for (int i = 0; i < 6; i++) {
|
||||||
|
size_t b = sz[i];
|
||||||
|
int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
|
||||||
|
double p = one(b, true, iters);
|
||||||
|
double g = one(b, false, iters);
|
||||||
|
// single-transfer latency
|
||||||
|
void *dv, *hv;
|
||||||
|
if (hipMalloc(&dv, b) == hipSuccess && hipHostMalloc(&hv, b, hipHostMallocDefault) == hipSuccess) {
|
||||||
|
memset(hv, 0x5a, b);
|
||||||
|
hipEvent_t a, c; hipEventCreate(&a); hipEventCreate(&c);
|
||||||
|
hipMemcpy(dv, hv, b, hipMemcpyHostToDevice); hipDeviceSynchronize();
|
||||||
|
hipEventRecord(a, 0);
|
||||||
|
hipMemcpy(dv, hv, b, hipMemcpyHostToDevice);
|
||||||
|
hipEventRecord(c, 0); hipEventSynchronize(c);
|
||||||
|
float ms = 0; hipEventElapsedTime(&ms, a, c);
|
||||||
|
printf(" %10.3f | %7.2f | %7.2f | %7.1f\n", b/1048576.0, p, g, ms*1000.0);
|
||||||
|
hipHostFree(hv); hipFree(dv);
|
||||||
|
} else {
|
||||||
|
printf(" %10.3f | %7.2f | %7.2f | (alloc failed)\n", b/1048576.0, p, g);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
printf("\n 2.344 MiB is the engine's exact expert slab size.\n");
|
||||||
|
printf(" Gen5 x8 ceiling = 31.5 GB/s per direction.\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
// HIP adaptation of the pcie-link-capability-probe.
|
||||||
|
// Question: is the decode bottleneck the LINK, or the engine's read PATH?
|
||||||
|
// Measure H2D at the engine's actual expert slab size (2.344 MiB) and across
|
||||||
|
// a size sweep. If 2.344 MiB copies reach ~31 GB/s, the link is healthy and
|
||||||
|
// the engine is leaving bandwidth on the table. If they stall near 10 GB/s,
|
||||||
|
// the link is the wall.
|
||||||
|
//
|
||||||
|
// build: hipcc -O2 -o pcie_probe pcie_probe.hip (scripts/pcie-probe.sh does this and guards the run)
|
||||||
|
#include <cstdio>
|
||||||
|
#include <cstring>
|
||||||
|
#include <cstdlib>
|
||||||
|
#include <hip/hip_runtime.h>
|
||||||
|
|
||||||
|
#define CK(x) do { hipError_t e = (x); if (e != hipSuccess) { \
|
||||||
|
fprintf(stderr, "HIP %s (%s:%d)\n", hipGetErrorString(e), __FILE__, __LINE__); \
|
||||||
|
return 1; } } while (0)
|
||||||
|
|
||||||
|
static double ms_between(hipEvent_t a, hipEvent_t b) {
|
||||||
|
float ms = 0; hipEventElapsedTime(&ms, a, b); return (double)ms;
|
||||||
|
}
|
||||||
|
|
||||||
|
static int bench_size(size_t bytes) {
|
||||||
|
const int iters = bytes >= (1UL<<30) ? 20
|
||||||
|
: bytes >= (256UL<<20) ? 30
|
||||||
|
: bytes >= (16UL<<20) ? 60 : 150;
|
||||||
|
void *dev = NULL, *host = NULL, *pageable = NULL;
|
||||||
|
hipStream_t s = NULL;
|
||||||
|
CK(hipMalloc(&dev, bytes));
|
||||||
|
CK(hipHostMalloc(&host, bytes, hipHostMallocDefault)); // PINNED, like the engine's slabs
|
||||||
|
memset(host, 0x5a, bytes);
|
||||||
|
CK(hipStreamCreate(&s));
|
||||||
|
hipEvent_t a, b; CK(hipEventCreate(&a)); CK(hipEventCreate(&b));
|
||||||
|
|
||||||
|
for (int i = 0; i < 3; i++) CK(hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice));
|
||||||
|
CK(hipDeviceSynchronize());
|
||||||
|
|
||||||
|
// sync copy
|
||||||
|
CK(hipEventRecord(a, 0));
|
||||||
|
for (int i = 0; i < iters; i++) CK(hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice));
|
||||||
|
CK(hipEventRecord(b, 0)); CK(hipEventSynchronize(b));
|
||||||
|
double h2d = (double)bytes * iters / (ms_between(a,b)/1000.0) / 1e9;
|
||||||
|
|
||||||
|
// async copy on a stream (the engine's fill pipeline shape)
|
||||||
|
CK(hipEventRecord(a, 0));
|
||||||
|
for (int i = 0; i < iters; i++) CK(hipMemcpyAsync(dev, host, bytes, hipMemcpyHostToDevice, s));
|
||||||
|
CK(hipEventRecord(b, 0)); CK(hipEventSynchronize(b));
|
||||||
|
double h2da = (double)bytes * iters / (ms_between(a,b)/1000.0) / 1e9;
|
||||||
|
|
||||||
|
// single-transfer latency at this size
|
||||||
|
CK(hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice));
|
||||||
|
CK(hipDeviceSynchronize());
|
||||||
|
CK(hipEventRecord(a, 0));
|
||||||
|
CK(hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice));
|
||||||
|
CK(hipEventRecord(b, 0)); CK(hipEventSynchronize(b));
|
||||||
|
double one_us = ms_between(a,b) * 1000.0;
|
||||||
|
|
||||||
|
// pageable, to see whether pinning is load-bearing here
|
||||||
|
pageable = malloc(bytes); memset(pageable, 0x5a, bytes);
|
||||||
|
CK(hipMemcpy(dev, pageable, bytes, hipMemcpyHostToDevice));
|
||||||
|
CK(hipEventRecord(a, 0));
|
||||||
|
for (int i = 0; i < iters; i++) CK(hipMemcpy(dev, pageable, bytes, hipMemcpyHostToDevice));
|
||||||
|
CK(hipEventRecord(b, 0)); CK(hipEventSynchronize(b));
|
||||||
|
double pag = (double)bytes * iters / (ms_between(a,b)/1000.0) / 1e9;
|
||||||
|
|
||||||
|
printf(" %8.3f MiB | %6.2f | %6.2f | %8.1f us | %6.2f\n",
|
||||||
|
bytes/1048576.0, h2d, h2da, one_us, pag);
|
||||||
|
|
||||||
|
free(pageable); hipHostFree(host); hipFree(dev);
|
||||||
|
hipEventDestroy(a); hipEventDestroy(b); hipStreamDestroy(s);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(int argc, char **argv) {
|
||||||
|
int dev_id = 0;
|
||||||
|
if (argc > 1) dev_id = atoi(argv[1]);
|
||||||
|
// No hipGetDeviceProperties here: it segfaults on this ROCm build while hipGetDeviceCount is
|
||||||
|
// fine, and the link topology these numbers must be read against comes from sysfs, which
|
||||||
|
// scripts/pcie-probe.sh prints. Only the device index is needed to pick what to measure.
|
||||||
|
printf("device %d (PCI bus from sysfs, see the topology block above)\n", dev_id);
|
||||||
|
printf(" size | H2D | async | 1x xfer | pageable [GB/s]\n");
|
||||||
|
printf(" %s\n", "-----------------------------------------------------------");
|
||||||
|
size_t sz[] = {2459136UL, /* 2.344 MiB: the engine's exact expert slab */
|
||||||
|
3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30, 2UL<<30};
|
||||||
|
for (int i = 0; i < 7; i++) if (bench_size(sz[i])) return 1;
|
||||||
|
printf("\nGen5 x8 ceiling = 31.5 GB/s per direction\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
Executable
+115
@@ -0,0 +1,115 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/remote-job.sh — a detached job whose result outlives the session that started it
|
||||||
|
#
|
||||||
|
# Why this exists rather than `ssh host 'cmd | tail -f | sed -n /ready/q'`, which lied about success
|
||||||
|
# repeatedly, for three structural reasons — all of them the pipe's fault: the broker aborts
|
||||||
|
# long-lived ssh; a readiness pattern can match a log *header* before the work has begun; and
|
||||||
|
# `sed -n /x/q` exits the instant it matches, so the supervisor reports green on a job that never
|
||||||
|
# finished.
|
||||||
|
#
|
||||||
|
# Here the job runs detached on the box it is invoked on (over ssh, or locally). It writes NAME.out,
|
||||||
|
# then NAME.rc, then touches NAME.done LAST. Status is then a cheap read of a directory: no
|
||||||
|
# long-lived connection to lose, and the answer survives the session, the terminal, and the agent —
|
||||||
|
# including the case where the job restarts the very service the agent is running on.
|
||||||
|
#
|
||||||
|
# remote-job.sh start NAME [--] CMD... launch detached, return at once
|
||||||
|
# remote-job.sh status [NAME] one line per job
|
||||||
|
# remote-job.sh wait NAME [SECONDS] [N] block until DONE or timeout, then tail N lines
|
||||||
|
# remote-job.sh out NAME [LINES] print the tail of a job's output
|
||||||
|
# remote-job.sh tail [NAME] follow one job, or every job at once
|
||||||
|
# remote-job.sh watch refreshing status, for a spare pane
|
||||||
|
#
|
||||||
|
# Job directory: $REMOTE_JOBS_DIR, default $HOME/jobs. Ship this file to the host and run it there;
|
||||||
|
# nothing about it is box-specific, which is the point.
|
||||||
|
#
|
||||||
|
# Exit status: 0 when every job reported is finished · 1 when something is unfinished or a wait
|
||||||
|
# timed out · 2 usage. A job that FAILED still counts as finished for `status`: "is it done" and
|
||||||
|
# "did it work" are different questions, and the second one is the rc= column.
|
||||||
|
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
JOBS="${REMOTE_JOBS_DIR:-$HOME/jobs}"
|
||||||
|
mkdir -p "$JOBS"
|
||||||
|
|
||||||
|
case "${1:-}" in
|
||||||
|
start)
|
||||||
|
[[ $# -ge 2 ]] || { echo "usage: remote-job.sh start NAME [--] CMD..." >&2; exit 2; }
|
||||||
|
name="$2"; shift 2
|
||||||
|
[[ "${1:-}" == "--" ]] && shift
|
||||||
|
[[ $# -ge 1 ]] || { echo "remote-job: start needs a command for '$name'" >&2; exit 2; }
|
||||||
|
# Clear the previous run entirely: a stale .out would make the new run's line count a lie.
|
||||||
|
rm -f "$JOBS/$name.out" "$JOBS/$name.rc" "$JOBS/$name.done"
|
||||||
|
# setsid leaves the invoking process group, so a dropped ssh or a killed agent takes nothing
|
||||||
|
# down with it. rc lands BEFORE the sentinel: a waiter must never see DONE with no exit code.
|
||||||
|
JOBS="$JOBS" setsid nohup bash -c '
|
||||||
|
name="$1"; shift
|
||||||
|
"$@" > "$JOBS/$name.out" 2>&1
|
||||||
|
echo $? > "$JOBS/$name.rc"
|
||||||
|
touch "$JOBS/$name.done"
|
||||||
|
' _ "$name" "$@" < /dev/null > /dev/null 2>&1 &
|
||||||
|
printf 'started %s\n' "$name"
|
||||||
|
;;
|
||||||
|
|
||||||
|
status)
|
||||||
|
if [[ -n "${2:-}" ]]; then
|
||||||
|
names="$2"
|
||||||
|
else
|
||||||
|
names=$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)
|
||||||
|
fi
|
||||||
|
pending=0
|
||||||
|
for n in $names; do
|
||||||
|
if [[ -f "$JOBS/$n.done" ]]; then
|
||||||
|
printf ' %-20s DONE rc=%s %s lines\n' "$n" \
|
||||||
|
"$(cat "$JOBS/$n.rc" 2>/dev/null)" "$(wc -l < "$JOBS/$n.out" 2>/dev/null | tr -d ' ')"
|
||||||
|
elif [[ -f "$JOBS/$n.out" ]]; then
|
||||||
|
printf ' %-20s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" 2>/dev/null | tr -d ' ')"
|
||||||
|
pending=1
|
||||||
|
else
|
||||||
|
printf ' %-20s PENDING (no output yet)\n' "$n"
|
||||||
|
pending=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [[ -z "$names" ]]; then echo " (no jobs)"; exit 0; fi
|
||||||
|
exit "$pending"
|
||||||
|
;;
|
||||||
|
|
||||||
|
wait)
|
||||||
|
[[ $# -ge 2 ]] || { echo "usage: remote-job.sh wait NAME [SECONDS] [TAIL]" >&2; exit 2; }
|
||||||
|
name="$2"; limit="${3:-3600}"; lines="${4:-25}"; waited=0
|
||||||
|
while [[ ! -e "$JOBS/$name.done" ]] && (( waited < limit )); do
|
||||||
|
sleep 1; waited=$((waited + 1))
|
||||||
|
done
|
||||||
|
if [[ -e "$JOBS/$name.done" ]]; then
|
||||||
|
printf 'DONE %s rc=%s\n' "$name" "$(cat "$JOBS/$name.rc" 2>/dev/null)"; rc=0
|
||||||
|
else
|
||||||
|
printf 'TIMEOUT %s after %ss\n' "$name" "$waited"; rc=1
|
||||||
|
fi
|
||||||
|
tail -n "$lines" "$JOBS/$name.out" 2>/dev/null
|
||||||
|
exit "$rc"
|
||||||
|
;;
|
||||||
|
|
||||||
|
out)
|
||||||
|
[[ $# -ge 2 ]] || { echo "usage: remote-job.sh out NAME [LINES]" >&2; exit 2; }
|
||||||
|
tail -n "${3:-40}" "$JOBS/$2.out" 2>/dev/null
|
||||||
|
;;
|
||||||
|
|
||||||
|
tail)
|
||||||
|
# Multiplexed follow: no NAME means every job at once, so one pane shows the whole harness.
|
||||||
|
if [[ -n "${2:-}" ]]; then exec tail -f "$JOBS/$2.out"; fi
|
||||||
|
exec tail -f "$JOBS"/*.out
|
||||||
|
;;
|
||||||
|
|
||||||
|
watch)
|
||||||
|
while true; do
|
||||||
|
clear 2>/dev/null || true
|
||||||
|
date '+%H:%M:%S'
|
||||||
|
"$0" status
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
;;
|
||||||
|
|
||||||
|
*)
|
||||||
|
echo "usage: remote-job.sh {start|status|wait|out|tail|watch} ..." >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Executable
+49
@@ -0,0 +1,49 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# tests/kb-coverage.bats — black-box tests for scripts/kb-coverage.py
|
||||||
|
#
|
||||||
|
# Contract: exit 0 when every directory of a knowledge bundle has an index.md and each
|
||||||
|
# index links its siblings. Observable surface: stdout lines and exit status.
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
CHECK="$BATS_TEST_DIRNAME/../scripts/kb-coverage.py"
|
||||||
|
KB="$(mktemp -d)/kb"
|
||||||
|
mkdir -p "$KB/sub"
|
||||||
|
printf '# Index\n\n* [Good](good.md)\n* [Sub](sub/)\n' > "$KB/index.md"
|
||||||
|
printf '# Sub\n\n* [Deep](deep.md)\n' > "$KB/sub/index.md"
|
||||||
|
printf 'body\n' > "$KB/good.md"
|
||||||
|
printf 'body\n' > "$KB/sub/deep.md"
|
||||||
|
}
|
||||||
|
|
||||||
|
teardown() { rm -rf "$(dirname "$KB")"; }
|
||||||
|
|
||||||
|
@test "a bundle whose directories index their contents passes" {
|
||||||
|
run python3 "$CHECK" "$KB"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"0 issue(s)"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a directory nobody indexes is reported" {
|
||||||
|
mkdir "$KB/stray"
|
||||||
|
printf 'body\n' > "$KB/stray/thing.md"
|
||||||
|
run python3 "$CHECK" "$KB"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"stray: no index.md"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an index that forgets a sibling page is reported" {
|
||||||
|
printf 'body\n' > "$KB/orphan.md"
|
||||||
|
run python3 "$CHECK" "$KB"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"does not link orphan.md"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the quarantine trash directory is not part of the bundle" {
|
||||||
|
# Pages deleted from the KB wait in .AGENTTRASH/ for a pruning session with Frank.
|
||||||
|
# Requiring an index there, or listing its contents, would push an agent to delete
|
||||||
|
# the quarantined copy just to make the gate green.
|
||||||
|
mkdir -p "$KB/.AGENTTRASH"
|
||||||
|
printf 'retired page\n' > "$KB/.AGENTTRASH/old-page.md"
|
||||||
|
run python3 "$CHECK" "$KB"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"0 issue(s)"* ]]
|
||||||
|
}
|
||||||
@@ -91,6 +91,35 @@ EOF
|
|||||||
[ "$status" -eq 2 ]
|
[ "$status" -eq 2 ]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "a second work log in a subdirectory is reported" {
|
||||||
|
# One bundle, one work log. Per-directory logs are how a bundle ends up with fifty
|
||||||
|
# of them: the entry is never where the next agent looks, and each copy re-dates the
|
||||||
|
# same event on its own.
|
||||||
|
printf -- '---\ntype: changelog\n---\n## 2026-01-01\n- created a log\n' > "$KB/log.md"
|
||||||
|
printf -- '---\ntype: changelog\n---\n## 2026-01-02\n- also a log\n' > "$KB/sub/log.md"
|
||||||
|
run bash "$LINT" "$KB"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"one work log"*"at the bundle root"* ]]
|
||||||
|
[[ "$output" == *"sub/log.md"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a work log that is not at the bundle root is reported" {
|
||||||
|
printf -- '---\ntype: changelog\n---\n## 2026-01-02\n- misplaced log\n' > "$KB/sub/log.md"
|
||||||
|
run bash "$LINT" "$KB"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"one work log"*"at the bundle root"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the quarantine trash directory is not part of the bundle" {
|
||||||
|
# Deleted pages wait in .AGENTTRASH/ for a pruning session with Frank. Holding them
|
||||||
|
# to frontmatter and one-log rules would push an agent to delete history to go green.
|
||||||
|
mkdir -p "$KB/.AGENTTRASH"
|
||||||
|
printf 'no frontmatter, and a second log besides\n' > "$KB/.AGENTTRASH/log.md"
|
||||||
|
run bash "$LINT" "$KB"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"0 violation(s)"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
@test "the shipped knowledge base itself is conformant" {
|
@test "the shipped knowledge base itself is conformant" {
|
||||||
# The tests above exercise the linter against fixtures; nothing else in the
|
# The tests above exercise the linter against fixtures; nothing else in the
|
||||||
# suite ever points it at the real bundle, so a page added without frontmatter
|
# suite ever points it at the real bundle, so a page added without frontmatter
|
||||||
|
|||||||
Executable
+144
@@ -0,0 +1,144 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# tests/pcie-probe.bats — black-box tests for scripts/pcie-probe.sh (Refs #53)
|
||||||
|
#
|
||||||
|
# Contract: measure host→device bandwidth on an RDNA4 box, and refuse to measure while the GPU is
|
||||||
|
# in use. Fleet law: never benchmark beside a live service — so the guard runs BEFORE anything is
|
||||||
|
# built or run, and the tool stops nothing and starts nothing (auto-stopping a production service is
|
||||||
|
# the thing it must not do unasked).
|
||||||
|
#
|
||||||
|
# Exit status: 0 measured · 2 usage · 3 refused (GPU busy) · 4 no toolchain / build failed.
|
||||||
|
#
|
||||||
|
# hipcc and the compiled probes are stubbed: this box has no HIP SDK (it is a client laptop) and a
|
||||||
|
# real measurement belongs on a machine with the cards, beside no running model.
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
TEST_TMP="$(mktemp -d)"
|
||||||
|
SCRIPT="$BATS_TEST_DIRNAME/../scripts/pcie-probe.sh"
|
||||||
|
SRC="$TEST_TMP/src"
|
||||||
|
BUILD="$TEST_TMP/build"
|
||||||
|
SYSFS="$TEST_TMP/drm"
|
||||||
|
BIN="$TEST_TMP/bin"
|
||||||
|
mkdir -p "$SRC" "$BUILD" "$BIN" "$SYSFS/card0/device" "$SYSFS/card1/device"
|
||||||
|
|
||||||
|
# Fixtures stand in for the real sysfs files the driver reads.
|
||||||
|
printf '8\n' > "$SYSFS/card0/device/current_link_width"
|
||||||
|
printf '32.0 GT/s PCIe\n' > "$SYSFS/card0/device/current_link_speed"
|
||||||
|
printf '16\n' > "$SYSFS/card0/device/max_link_width"
|
||||||
|
printf '32.0 GT/s PCIe\n' > "$SYSFS/card0/device/max_link_speed"
|
||||||
|
printf '0\n' > "$SYSFS/card0/device/gpu_busy_percent"
|
||||||
|
printf '8\n' > "$SYSFS/card1/device/current_link_width"
|
||||||
|
printf '32.0 GT/s PCIe\n' > "$SYSFS/card1/device/current_link_speed"
|
||||||
|
printf '16\n' > "$SYSFS/card1/device/max_link_width"
|
||||||
|
printf '32.0 GT/s PCIe\n' > "$SYSFS/card1/device/max_link_speed"
|
||||||
|
printf '0\n' > "$SYSFS/card1/device/gpu_busy_percent"
|
||||||
|
|
||||||
|
# Probe sources: the real .hip files are shipped by the script's own directory, so the fixtures
|
||||||
|
# only need to exist for the build step to be invoked.
|
||||||
|
: > "$SRC/pcie_probe.hip"; : > "$SRC/pcie_bw.hip"; : > "$SRC/hip_smoke.hip"
|
||||||
|
|
||||||
|
# Stub toolchain: records that it was called, and emits a probe binary that records that IT ran.
|
||||||
|
cat > "$BIN/hipcc" <<'STUB'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
[ "${STUB_HIPCC_FAILS:-0}" = 1 ] && { echo "stub: no matching GPU target" >&2; exit 1; }
|
||||||
|
out=""
|
||||||
|
while [ $# -gt 0 ]; do [ "$1" = "-o" ] && out="$2"; shift; done
|
||||||
|
[ -n "$out" ] || exit 2
|
||||||
|
: > "$PCIE_PROBE_BUILD/.built"
|
||||||
|
{ echo '#!/usr/bin/env bash'
|
||||||
|
echo ': > "$PCIE_PROBE_BUILD/.ran"'
|
||||||
|
echo 'echo "PROBE RAN dev=${HIP_VISIBLE_DEVICES:-none} bin=$(basename "$0")"'
|
||||||
|
echo 'echo " slab 2.344 MiB: 31.4 GB/s"'
|
||||||
|
} > "$out"
|
||||||
|
chmod +x "$out"
|
||||||
|
STUB
|
||||||
|
cat > "$BIN/fuser" <<'STUB'
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
[ -n "${STUB_KFD_PIDS:-}" ] && printf '%s\n' $STUB_KFD_PIDS
|
||||||
|
exit 0
|
||||||
|
STUB
|
||||||
|
chmod +x "$BIN/hipcc" "$BIN/fuser"
|
||||||
|
|
||||||
|
export PATH="$BIN:$PATH" PCIE_PROBE_SRC="$SRC" PCIE_PROBE_BUILD="$BUILD" \
|
||||||
|
PCIE_PROBE_SYSFS="$SYSFS" PCIE_PROBE_KFD="$TEST_TMP/kfd" PCIE_PROBE_HIPCC="$BIN/hipcc"
|
||||||
|
: > "$TEST_TMP/kfd"
|
||||||
|
}
|
||||||
|
|
||||||
|
teardown() { rm -rf "$TEST_TMP"; }
|
||||||
|
|
||||||
|
@test "a missing HIP toolchain is a loud failure, not a silent skip" {
|
||||||
|
run env PCIE_PROBE_HIPCC="$TEST_TMP/nope-hipcc" "$SCRIPT"
|
||||||
|
[ "$status" -eq 4 ]
|
||||||
|
[[ "$output" == *"hip"* ]]
|
||||||
|
[ ! -e "$BUILD/.built" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a failed build is reported as a failure" {
|
||||||
|
run env STUB_HIPCC_FAILS=1 "$SCRIPT"
|
||||||
|
[ "$status" -eq 4 ]
|
||||||
|
[[ "$output" == *"build"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a GPU held by another process refuses the measurement before anything is built" {
|
||||||
|
# Order is the contract: build-and-then-ask would already have loaded the card beside a live
|
||||||
|
# service. The .built marker proves the guard ran first.
|
||||||
|
run env STUB_KFD_PIDS="4242" "$SCRIPT"
|
||||||
|
[ "$status" -eq 3 ]
|
||||||
|
[[ "$output" == *"busy"* ]]
|
||||||
|
[ ! -e "$BUILD/.built" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a busy GPU percent refuses too" {
|
||||||
|
printf '91\n' > "$SYSFS/card0/device/gpu_busy_percent"
|
||||||
|
run "$SCRIPT"
|
||||||
|
[ "$status" -eq 3 ]
|
||||||
|
[[ "$output" == *"card0"* ]]
|
||||||
|
[ ! -e "$BUILD/.built" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "--force overrides the guard for a deliberately contended measurement" {
|
||||||
|
run env STUB_KFD_PIDS="4242" "$SCRIPT" --force
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$BUILD/.ran" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an idle GPU builds and runs the probe" {
|
||||||
|
run "$SCRIPT"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$BUILD/.built" ]
|
||||||
|
[ -e "$BUILD/.ran" ]
|
||||||
|
[[ "$output" == *"PROBE RAN"* ]]
|
||||||
|
[[ "$output" == *"31.4 GB/s"* ]] # probe output passes through untouched
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "--build-only compiles without touching the card" {
|
||||||
|
run "$SCRIPT" --build-only
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[ -e "$BUILD/.built" ]
|
||||||
|
[ ! -e "$BUILD/.ran" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the link topology is reported from sysfs, current against max" {
|
||||||
|
run "$SCRIPT"
|
||||||
|
[[ "$output" == *"card0"*"x8"*"32.0 GT/s"* ]]
|
||||||
|
[[ "$output" == *"max x16"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "each requested device is measured separately" {
|
||||||
|
# One shared run cannot attribute bandwidth to a card; the whole point is per-card numbers.
|
||||||
|
run "$SCRIPT" --device 0 --device 1
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"dev=0"* ]]
|
||||||
|
[[ "$output" == *"dev=1"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "no probe sources is a usage-level error, not an empty success" {
|
||||||
|
rm -f "$SRC"/*.hip
|
||||||
|
run "$SCRIPT"
|
||||||
|
[ "$status" -ne 0 ]
|
||||||
|
[[ "$output" == *"probe"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an unknown flag is a usage error" {
|
||||||
|
run "$SCRIPT" --frobnicate
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
}
|
||||||
Executable
+135
@@ -0,0 +1,135 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# tests/remote-job.bats — black-box tests for scripts/remote-job.sh (Refs #53)
|
||||||
|
#
|
||||||
|
# Contract: a job runs detached, writes its output, then its exit code, then touches .done LAST.
|
||||||
|
# Status is then a cheap independent read that survives the session that started the job — the
|
||||||
|
# whole point is replacing `ssh host 'tail -f | sed'`, which reported success on jobs that had not
|
||||||
|
# finished. Exit status answers "is everything finished"; the rc= field answers "did it work".
|
||||||
|
#
|
||||||
|
# Jobs really run here, against a temp job dir. No fake filesystem state is pre-seeded, because a
|
||||||
|
# fixture that pre-creates .done proves nothing about the code that is supposed to create it.
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
TEST_TMP="$(mktemp -d)"
|
||||||
|
SCRIPT="$BATS_TEST_DIRNAME/../scripts/remote-job.sh"
|
||||||
|
export REMOTE_JOBS_DIR="$TEST_TMP/jobs"
|
||||||
|
}
|
||||||
|
|
||||||
|
teardown() {
|
||||||
|
# Jobs are detached on purpose; leave nothing behind that could outlive the test run.
|
||||||
|
pkill -P 1 -f "$TEST_TMP" 2>/dev/null || true
|
||||||
|
rm -rf "$TEST_TMP"
|
||||||
|
}
|
||||||
|
|
||||||
|
# until_done NAME SECONDS — poll instead of sleeping a fixed amount, so the suite is neither slow
|
||||||
|
# nor flaky on a loaded machine.
|
||||||
|
until_done() {
|
||||||
|
local name="$1" limit="${2:-10}" waited=0
|
||||||
|
while [[ ! -e "$REMOTE_JOBS_DIR/$name.done" && $waited -lt $((limit * 10)) ]]; do
|
||||||
|
sleep 0.1; waited=$((waited + 1))
|
||||||
|
done
|
||||||
|
[[ -e "$REMOTE_JOBS_DIR/$name.done" ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "start returns immediately while the job is still running" {
|
||||||
|
run timeout 10 "$SCRIPT" start slow -- sleep 2
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == "started slow" ]]
|
||||||
|
# A 2-second job must not have blocked a start that is supposed to return at once, and while it
|
||||||
|
# is still in flight the report must say unfinished — never DONE.
|
||||||
|
run "$SCRIPT" status slow
|
||||||
|
[[ "$output" == *RUN* || "$output" == *PENDING* ]]
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a successful job is reported DONE with its output" {
|
||||||
|
"$SCRIPT" start ok -- bash -c 'echo marker-4711'
|
||||||
|
until_done ok
|
||||||
|
run "$SCRIPT" status ok
|
||||||
|
[[ "$output" == *"ok"*DONE*rc=0* ]]
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run "$SCRIPT" out ok
|
||||||
|
[[ "$output" == *marker-4711* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "a failing job's exit code survives into the report" {
|
||||||
|
# This is the reason the .rc file exists: a supervisor that cannot see the failure code is the
|
||||||
|
# bug the runner was written to fix.
|
||||||
|
"$SCRIPT" start boom -- bash -c 'echo it failed here >&2; exit 7'
|
||||||
|
until_done boom
|
||||||
|
run "$SCRIPT" status boom
|
||||||
|
[[ "$output" == *"boom"*DONE*rc=7* ]]
|
||||||
|
run "$SCRIPT" out boom
|
||||||
|
[[ "$output" == *"it failed here"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the .done sentinel is never written before the exit code" {
|
||||||
|
# If .done landed first, a waiter could read an empty rc and call the job successful.
|
||||||
|
"$SCRIPT" start order -- bash -c 'echo x'
|
||||||
|
until_done order
|
||||||
|
[ -f "$REMOTE_JOBS_DIR/order.rc" ]
|
||||||
|
[ -s "$REMOTE_JOBS_DIR/order.rc" ]
|
||||||
|
run cat "$REMOTE_JOBS_DIR/order.rc"
|
||||||
|
[[ "$output" =~ ^[0-9]+$ ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "wait prints DONE with the exit code and a tail of the output" {
|
||||||
|
"$SCRIPT" start fin -- bash -c 'echo one; echo two'
|
||||||
|
run "$SCRIPT" wait fin 15
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"DONE fin rc=0"* ]]
|
||||||
|
[[ "$output" == *two* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "wait gives up on its own and says so" {
|
||||||
|
"$SCRIPT" start never -- sleep 30
|
||||||
|
run "$SCRIPT" wait never 1
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[[ "$output" == *TIMEOUT*never* ]]
|
||||||
|
"$SCRIPT" status never | grep -q . # the job is still there, unfinished
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "status distinguishes running from pending from done" {
|
||||||
|
run "$SCRIPT" status
|
||||||
|
[[ "$output" == *"(no jobs)"* ]]
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
|
||||||
|
"$SCRIPT" start later -- sleep 3
|
||||||
|
run "$SCRIPT" status later
|
||||||
|
[[ "$output" == *later* ]]
|
||||||
|
[[ "$output" == *RUN* || "$output" == *PENDING* ]]
|
||||||
|
[ "$status" -eq 1 ] # unfinished: a supervisor must be able to branch on this
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "two jobs in flight keep separate state" {
|
||||||
|
"$SCRIPT" start a -- bash -c 'echo A; exit 0'
|
||||||
|
"$SCRIPT" start b -- bash -c 'echo B; exit 2'
|
||||||
|
until_done a && until_done b
|
||||||
|
run "$SCRIPT" status
|
||||||
|
[[ "$output" == *"a"*DONE*rc=0* ]]
|
||||||
|
[[ "$output" == *"b"*DONE*rc=2* ]]
|
||||||
|
run "$SCRIPT" out a
|
||||||
|
[[ "$output" == A ]]
|
||||||
|
run "$SCRIPT" out b
|
||||||
|
[[ "$output" == B ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "out honours a line limit" {
|
||||||
|
"$SCRIPT" start many -- bash -c 'seq 1 40'
|
||||||
|
until_done many
|
||||||
|
run "$SCRIPT" out many 3
|
||||||
|
[ "$(printf '%s\n' "$output" | grep -c .)" -eq 3 ]
|
||||||
|
[[ "$output" == *38* ]] # the tail, not the head
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an unknown verb is a usage error" {
|
||||||
|
run "$SCRIPT" frobnicate
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *usage* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the job directory is created on first use" {
|
||||||
|
[ ! -d "$REMOTE_JOBS_DIR" ]
|
||||||
|
"$SCRIPT" status
|
||||||
|
[ -d "$REMOTE_JOBS_DIR" ]
|
||||||
|
}
|
||||||
Reference in new issue
Block a user