---
name: frank-medium-batch-split-retune
description: "Retune frank-medium (Dankserver, llama.cpp layer split across 2x R9700) batch/ubatch geometry and the tensor-split balance: sweep with scripts/bench-batch-ubatch.sh, re-measure the split at the deployed shape behind a VRAM settle gate, re-fit the seam, deploy, verify."
---

# frank-medium batch geometry + split rebalance

Use when changing `--batch-size`, `--ubatch-size`, `--ctx-size`, `--parallel` or the
model on frank-medium, or when someone reports that prefill "does not respond" to
those knobs. The batch and the tensor split are **coupled**: the compute buffers are
not symmetric across the two cards, so any batch change moves the balance point.

## Order of work (do not skip the second half)

1. **Sweep the batch geometry with llama-bench**, which is the only tool that can hold
   a shape constant across dozens of configs:
   `scripts/bench-batch-ubatch.sh`, defaults `-p 65536 -n 128 -d 65536`, one invocation
   per cell, `-r 3 --no-warmup`.
2. **Re-measure the split at the DEPLOYED shape** (262144, two slots, drafter loaded) by
   loading the real service with a per-config env override and reading `rocm-smi`. A
   llama-bench number cannot decide the split: it has no `--parallel`, sizes its own
   context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
3. **Re-fit the balance seam** in `tests/frank_medium_unit.bats` on the new deployed
   point, then **deploy** and verify.

## 1. The sweep

```bash
# from a transient unit, never a shell: the deploy's LimitMEMLOCK=infinity is
# load-bearing for ROCm staging buffers and an SSH shell gets 8 MiB here
sudo systemd-run --unit=fm-bench --collect --pipe --property=Type=oneshot \
     --property=LimitMEMLOCK=infinity --property=User=dank \
     --setenv=OUT=/home/dank/models/logs/bench-bub \
     --setenv=BATCHES=6144 --setenv=UBATCHES="256 384 512 768 1024 1536 2048" \
     /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
```

The script stops `frank-medium`, runs one `llama-bench` per cell into `cells.tsv`
(appended as it goes, so progress is readable), samples VRAM per cell, and restarts
the service from an EXIT trap. A re-run skips cells that already have a `bench.json`.

Read `n_ubatch` out of each `bench.json` to prove the knob was applied — if it does not
track the request, nothing else in the table means anything. The VRAM columns scaling
monotonically with ubatch is the independent confirmation.

Expected shape (2026-09-16, 2x R9700, UD-Q8_K_XL + DFlash2, 64k prefill at 64k depth):
an optimum at **ubatch 512** (~1193 tok/s) with losses on both sides (256 -> 1046,
384 -> 1068, 2048 -> 1077). The logical batch is throughput- and VRAM-neutral
(2048..8192 within 2 %), and decode never moves (15.88 tok/s at depth 65536 on every
cell). Test **both** ends of the range: sampling only upward makes a shallow basin look
like a plateau.

Sanity rule: the same ubatch measured under different logical batches should agree to
under 1 tok/s. If it does not, the table is noise and the first suspect is a stray
GPU user, not the knob.

## 2. The split, at the deployed shape

`/tmp/measure-ub512b.sh` (pattern; rewrite per run) copies `scripts/frank-medium.env`
to a temp file with `sed` overrides, launches `scripts/serve-frank-medium.sh` with
`FRANK_MEDIUM_ENV_FILE=<temp>`, polls `/health` for `"status":"ok"`, reads
`rocm-smi --showmeminfo vram --csv`, kills, and repeats per split. Run it under
`systemd-run --pipe --property=LimitMEMLOCK=infinity --property=User=dank`.

**The settle gate is mandatory.** Between configs, wait until `rocm-smi` shows BOTH
cards under ~1 GiB used before the next load. amdgpu does not return ~26 GiB when the
process dies — a 180 s sleep fails, and even 600 s has failed while the process is still
alive inside the unit. Skipping the gate produces
`error loading model: unable to allocate ROCm0 buffer`, which is indistinguishable from
"this split does not fit" and will send you chasing a phantom OOM.

Free bytes to expect, and the arithmetic behind them:
`card = weights_share + KV_share + constant + ubatch_term`, where the ubatch term costs
the drafter-carrying card roughly **1.85x** what it costs the other (measured 2.41 GiB
against 1.30 GiB going 1536 -> 512). Balance by measurement only: one `-ts` step is one
layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the
reachable minimum gap is about one DeltaNet layer, never zero.

2026-09-16 reference: 36,29 @1536 -> 4.71 / 4.91; 36,29 @512 -> 6.01 / 7.32;
**35,30 @512 -> 6.44 / 6.90 (deployed)**.

## 3. Seam, deploy, verify

- Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` from the measured
  1536 -> 512 deltas at the deployed shape, and re-anchor the per-card constants so the
  model reproduces the new point. Check that neighbouring splits still FAIL the balance
  gate, or the seam has stopped discriminating. Keep the floor at 1.5 GiB/card and the
  slack at one DeltaNet layer.
- Do not pin a literal split elsewhere: the unit test asserts the *shape* (two shares
  summing to the model's 65 blocks, ROCm1 the smaller).
- `./tests/run.sh`, commit, `./scripts/deploy.sh dankserver --service frank-medium`.
- Verify live: the flags in `/proc/<pid>/cmdline`, `/slots` = 2 x 131072, VRAM matching
  the pre-deploy measurement, a tool call, two concurrent requests, `draft acceptance`
  lines on BOTH slots, and 0 ROCm errors in the journal. The VRAM figure sits ~0.2 GiB
  lower once the service has served traffic than straight after load.