command-center/agent/managed-skills/frank-medium-batch-split-retune/SKILL.md

name, description

name	description
frank-medium-batch-split-retune	Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify.

frank-medium batch geometry + split rebalance

Use when changing --batch-size, --ubatch-size, --ctx-size, --parallel or the model on frank-medium, or when someone reports that prefill "does not respond" to those knobs. The batch and the tensor split are coupled: the compute buffers are not symmetric across the two cards, so any batch change moves the balance point.

Order of work (do not skip the second half)

1. Sweep the batch geometry with llama-bench, which is the only tool that can hold a shape constant across dozens of configs: scripts/bench-batch-ubatch.sh, defaults -p 65536 -n 128 -d 65536, one invocation per cell, -r 3 --no-warmup.
2. Re-measure the split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading the real service with a per-config env override and reading rocm-smi. A llama-bench number cannot decide the split: it has no --parallel, sizes its own context as n_prompt + n_gen + n_depth, and cannot load the drafter.
3. Re-fit the balance seam in tests/frank_medium_unit.bats on the new deployed point, then deploy and verify.

1. The sweep

# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is
# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here
sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \
     --property=LimitMEMLOCK=infinity --property=User=dank \
     --setenv=OUT=/home/dank/models/logs/bench-bub \
     --setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \
     /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh

The script stops frank-medium, runs one llama-bench per cell into cells.tsv (appended as it goes, so progress is readable), samples VRAM per cell, and restarts the service from an EXIT trap. A re-run skips cells that already have a bench.json.

Read n_ubatch out of each bench.json to prove the knob was applied — if it does not track the request, nothing else in the table means anything. The VRAM columns scaling monotonically with ubatch is the independent confirmation.

Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth): an optimum at ubatch 512 (~1193 tok/s) with losses on both sides (256 -> 1046, 384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral (2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every cell). Test both ends of the range: sampling only upward makes a shallow basin look like a plateau.

Sanity rule: the same ubatch measured under different logical batches should agree to under 1 tok/s. If it does not, the table is noise and the first suspect is a stray GPU user, not the knob.

2. The split, at the deployed shape

/tmp/measure-ub512b.sh (pattern; rewrite per run) copies scripts/frank-medium.env to a temp file with sed overrides, launches scripts/serve-frank-medium.sh with FRANK_MEDIUM_ENV_FILE=<temp>, polls /health for "status":"ok", reads rocm-smi --showmeminfo vram --csv, kills, and repeats per split. Run it under systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank.

The settle gate is mandatory. Between configs, wait until rocm-smi shows BOTH cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the process dies — a 180 s sleep fails, and even 600 s has failed while the process is still alive inside the unit. Skipping the gate produces error loading model: unable to allocate ROCm0 buffer, which is indistinguishable from "this split does not fit" and will send you chasing a phantom OOM.

Free bytes to expect, and the arithmetic behind them: card = weights_share + KV_share + constant + ubatch_term, where the ubatch term costs the drafter-carrying card roughly 1.85x what it costs the other (measured 2.41 GiB against 1.30 GiB going 1536 -> 512). Balance by measurement only: one -ts step is one layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is about one DeltaNet layer, never zero.

2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32; 35,30 @512 -> 6.44 / 6.90 (deployed).

3. Seam, deploy, verify

• Re-fit both ubatch coefficients in tests/frank_medium_unit.bats from the measured 1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the model reproduces the new point. Check that neighbouring splits still FAIL the balance gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the slack at one DeltaNet layer.
• Do not pin a literal split elsewhere: the unit test asserts the shape (two shares summing to the model's 65 blocks, ROCm1 the smaller).
• ./tests/run.sh, commit, ./scripts/deploy.sh dankserver --service frank-medium.
• Verify live: the flags in /proc/<pid>/cmdline, /slots = 2 x 131072, VRAM matching the pre-deploy measurement, a tool call, two concurrent requests, draft acceptance lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB lower once the service has served traffic than straight after load.