diff --git a/AGENTS.md b/AGENTS.md index 13b1da8..feda973 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -226,6 +226,13 @@ archive. ## Benchmarking Policy (MANDATORY) +**Benchmarking is NOT part of the model-swap procedure** (2026-10-15). A +model change is verified by driving the live service — boot ledger, +per-request `timings`/`Server-Timing`, one real request per harness role +(`serverAppConfigs-docs/LLM/Model Swap and Rollback.md` §6/§8). No bench +window is scheduled for a swap, ever. The rest of this policy governs +*deliberate* bench windows (flag A/Bs, shape sweeps), which are separate +operations on a live host: **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp @@ -1157,7 +1164,7 @@ history. | Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. | | Deploying a committed change | **`scripts/deploy.sh `** — pushes, pulls ff-only, plans, dispatches, health-checks. | | Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. | -| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. | +| Benchmarking | **Not part of a model swap** — swaps verify on the live service. A deliberate bench window: **stop and mask the service first**, bench, unmask, restart, verify health. | | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | | Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. | diff --git a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md index 9f8b6ca..fc4a9c6 100644 --- a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md +++ b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md @@ -3,7 +3,7 @@ type: reference title: Model Swap and Rollback description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline. tags: [models, swap, gguf, huggingface, rollback, deployment, verification] -timestamp: 2026-10-05T00:00:00-07:00 +timestamp: 2026-10-15T00:00:00-07:00 --- # Model Swap and Rollback @@ -12,6 +12,15 @@ Applies to every endpoint swap on the fleet (frank-zealot / frank-dragoon / frank-archon). Repo law: all config flows through git; the live models are host state under `/home/dank/models//`. +**No bench step (law, 2026-10-15).** A model swap NEVER schedules a bench +window. Performance evidence comes from the deployed service itself: the +engine's boot ledger, per-request timings (`/completion` `timings`, +`Server-Timing`), and one real request per harness role (§6, §8). A +deliberate standalone bench (flag A/B, shape sweep) is OUT OF PROCEDURE — +a separate operation that follows the AGENTS.md Benchmarking Policy +(stop & mask). The env-file hook (§5) stays so those out-of-procedure +arms remain possible. + ## 1. Fit preflight — never deploy a swap without confirming it FITS 1. Get the checkpoint's on-disk/weight size AND precision from the HF card @@ -144,7 +153,8 @@ recoverable, drift is not. ## 8. Close-out (mandatory) 1. Docs: active runbook updated (build recipe, profile, validation numbers - incl. real-workload prefill/decode); superseded runbook gets a RETIRED + incl. real-workload prefill/decode measured on the LIVE service — no + bench window); superseded runbook gets a RETIRED banner and stays as the rollback record; `LLM/index.md` mermaid + active list; AGENTS.md fleet table + runtime section + history chain; work-log entry in `serverAppConfigs-docs/log.md`. @@ -277,5 +287,6 @@ predecessor "stopped" (the port race above). - [[GGUF Compatibility]] — which runtime can load a payload at all - [[Llama Cpp Serving Sizing]] — pricing the shape once it loads - [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic -- [[LLM Bench Windows and Measurement]] — measuring the new deployment +- [[LLM Bench Windows and Measurement]] — out-of-procedure bench windows + only; a new deployment is measured by §6 live validation, not by a bench - [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index f5332ca..84b914c 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1745,3 +1745,22 @@ every restart since the cutover. [[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected (the w4dn8 entry said STAGED, the fp8-native one said ACTIVE). + +## 2026-10-15 — governance: bench removed from the model-change procedure + +- Frank's call: a model swap never carries a bench window. Codified as law: + - `AGENTS.md` Benchmarking Policy opens with "Benchmarking is NOT part of + the model-swap procedure"; swap verification is the live service (boot + ledger, `timings`/`Server-Timing`, one real request per harness role). + The stop-and-mask rule stays for DELIBERATE bench windows (flag A/Bs, + shape sweeps) — it is host protection (2026-09-16 hard-down precedent), + not a swap step. Summary-table row updated to match. + - [[Model Swap and Rollback]]: "No bench step (law, 2026-10-15)" up top; + §8 close-out numbers must be measured on the live service; the Related + link to [[LLM Bench Windows and Measurement]] rescoped to + out-of-procedure windows only. +- Untouched on purpose: the `FRANK_*_ENV_FILE` hook (staged/out-of-procedure + arms), the Fleet Ops Playbook bench-window mechanics (deliberate windows + still need the procedure), `deployment.md` (already says bench work is a + SEPARATE procedure). Managed skill `llamacpp-endpoint-model-swap` verifies + on the live engine — it never had a bench step.