agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: frank-medium-batch-split-retune
|
||||
description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify."
|
||||
---
|
||||
|
||||
# frank-medium batch geometry + split rebalance
|
||||
|
||||
Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the
|
||||
model on frank-medium, or when someone reports that prefill "does not respond" to
|
||||
those knobs. The batch and the tensor split are **coupled**: the compute buffers are
|
||||
not symmetric across the two cards, so any batch change moves the balance point.
|
||||
|
||||
## Order of work (do not skip the second half)
|
||||
|
||||
1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold
|
||||
a shape constant across dozens of configs:
|
||||
`scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation
|
||||
per cell, `-r 3 --no-warmup`.
|
||||
2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by
|
||||
loading the real service with a per-config env override and reading `rocm-smi`. A
|
||||
llama-bench number cannot decide the split: it has no `--parallel`, sizes its own
|
||||
context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
|
||||
3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed
|
||||
point, then **deploy** and verify.
|
||||
|
||||
## 1. The sweep
|
||||
|
||||
```bash
|
||||
# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is
|
||||
# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here
|
||||
sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
--setenv=OUT=/home/dank/models/logs/bench-bub \
|
||||
--setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \
|
||||
/home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv`
|
||||
(appended as it goes, so progress is readable), samples VRAM per cell, and restarts
|
||||
the service from an EXIT trap. A re-run skips cells that already have a `bench.json`.
|
||||
|
||||
Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not
|
||||
track the request, nothing else in the table means anything. The VRAM columns scaling
|
||||
monotonically with ubatch is the independent confirmation.
|
||||
|
||||
Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth):
|
||||
an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046,
|
||||
384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral
|
||||
(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every
|
||||
cell). Test **both** ends of the range: sampling only upward makes a shallow basin look
|
||||
like a plateau.
|
||||
|
||||
Sanity rule: the same ubatch measured under different logical batches should agree to
|
||||
under 1 tok/s. If it does not, the table is noise and the first suspect is a stray
|
||||
GPU user, not the knob.
|
||||
|
||||
## 2. The split, at the deployed shape
|
||||
|
||||
`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env`
|
||||
to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with
|
||||
`FRANK_MEDIUM_ENV_FILE=<temp>`, polls `/health` for `"status":"ok"`, reads
|
||||
`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under
|
||||
`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`.
|
||||
|
||||
**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH
|
||||
cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the
|
||||
process dies — a 180 s sleep fails, and even 600 s has failed while the process is still
|
||||
alive inside the unit. Skipping the gate produces
|
||||
`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from
|
||||
"this split does not fit" and will send you chasing a phantom OOM.
|
||||
|
||||
Free bytes to expect, and the arithmetic behind them:
|
||||
`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs
|
||||
the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB
|
||||
against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one
|
||||
layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the
|
||||
reachable minimum gap is about one DeltaNet layer, never zero.
|
||||
|
||||
2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32;
|
||||
**35,30 @512 -> 6.44 / 6.90 (deployed)**.
|
||||
|
||||
## 3. Seam, deploy, verify
|
||||
|
||||
- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured
|
||||
1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the
|
||||
model reproduces the new point. Check that neighbouring splits still FAIL the balance
|
||||
gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the
|
||||
slack at one DeltaNet layer.
|
||||
- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares
|
||||
summing to the model's 65 blocks, ROCm1 the smaller).
|
||||
- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`.
|
||||
- Verify live: the flags in `/proc/<pid>/cmdline`, `/slots` = 2 x 131072, VRAM matching
|
||||
the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance`
|
||||
lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB
|
||||
lower once the service has served traffic than straight after load.
|
||||
Reference in new issue
Block a user