frank-medium: 3x batch over 2 slots, and rebalance the split to pay for it

The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card
floor, and the context maximised inside it.

Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric:
ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid
~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78.
One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement:
41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free.

That balance is also what maximises the context -- an unbalanced split would have
to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's
trained window (262144), now split over two slots of 131072. Verified under two
concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2,
n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots
(acceptance 0.38/0.22 and 0.52/0.43).

8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to
allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in
the env's sizing table.

The unit seam now models both cards (share + draft + ubatch-scaled buffers,
calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor
on each, and asserts the two cards stay within 0.5 GiB of each other. Both
assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and
a 1:1 split) rather than pass by construction.

scripts/frank-medium.env

@@ -54,12 +54,14 @@
 #
 # SIZING, measured (rocm-smi, 2026-09-16).  Weights split across the two cards;
 # the drafter AND its KV ride the second.  The loader's compute buffers are part
-# of what is seen below -- they are allocated at load, not per request:
+# of what is seen below -- they are allocated at load, not per request, and they
+# scale with --ubatch-size:
 #
-#   CTX_SIZE  split  GPU 0      GPU 1      headroom on the busier card
-#   131072    1,1    19.89 GB   24.30 GB   9.9 GB / 31 %   (DFlash2)
-#   262144    1,1    24.72 GB   29.14 GB   5.07 GB / 15 %  (DFlash2) <- before
-#   262144    38,27  28.52 GB   30.71 GB   3.50 GB / 10 %  (DSpark)  <- deployed
+#   CTX     split  batch  slots  GPU 0 free  GPU 1 free   note
+#   262144  1,1    2048   1      8.64 GiB    4.43 GiB    DFlash2, day's start
+#   262144  38,27  2048   1      5.49 GiB    3.41 GiB    DSpark, 1 slot
+#   262144  38,27  6144   2      3.78 GiB    0.08 GiB    3x batch is ASYMMETRIC
+#   262144  40,25  6144   2      1.87 GiB    1.86 GiB    <- deployed
 #
 # 262144 is the window the model ships (n_ctx_train) and it fits with room, so
 # the fleet gets the whole thing; the arithmetic that used to cap this endpoint
@@ -67,25 +69,32 @@
 # adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
 # rather than assuming, and prefer CTX_SIZE over --gpu-layers.
 #
-# WHY THE SPLIT IS 38:27.  The draft context is built with `n_ctx =
-# llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so the drafter's KV scales
-# with the target window, and DSpark -- unlike DFlash2, which declared
-# sliding_window 2048 and cost ~40 MiB -- caches all of it: 5 layers x 2 x 8 kv
-# heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144.  At a
-# 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the draft's first graph
-# capture died with `ROCm error: out of memory` (in
-# common_speculative_impl_draft_dflash::process).  Handing three target layers
-# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
+# WHY THE SPLIT IS 40:25, AND WHY IT DEPENDS ON THE BATCH.  Two effects pull in
+# opposite directions:
+#   * THE DRAFT KEEPS ITS OWN FULL-CONTEXT KV ON ROCm1.  The draft context is
+#     built with `n_ctx = llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so
+#     DSpark -- which, unlike DFlash2 (`sliding_window` 2048, ~40 MiB), declares
+#     no window -- caches 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per
+#     position: 5.0 GiB at 262144.  At a 1:1 split that put ROCm1 at 33.9 GB of
+#     34.2 GB and the draft's first graph capture died with `ROCm error: out of
+#     memory` (common_speculative_impl_draft_dflash::process).  Hence 38:27.
+#   * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs
+#     ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output
+#     layers AND the drafter's graphs.  At 38:27 that left ROCm1 at 0.08 GiB
+#     free against ROCm0's 3.78 -- the reason for the rebalance.
+# One -ts step shifts ~1.5 GiB of load between the cards at this batch, so the
+# balance is found by measurement, not arithmetic.  40:25 is the measured
+# balance (a 0.05 GiB gap) and it is also what MAXIMISES the context: with the
+# floor at ~1.5 GiB per card, an unbalanced split would have to buy its
+# headroom out of CTX_SIZE.
 #
-# EQUALITY WAS TRIED AND REJECTED.  39:26 levels the cards more closely (3.86 /
-# 4.69 GiB free -- an 0.83 GiB gap, against 2.08 GiB here), but that server
-# died on its eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults
-# and the engine surfaced `ROCm error: an illegal memory access was
-# encountered` -> ggml_abort, i.e. a fault in the cross-card boundary copy,
-# which is this layer split's hot path.  38:27 has since served the whole e2e
-# gate plus a burst of varied requests with none.  Do not chase an exact
-# balance: take the skew that fits and leave the drafter's card on the safe
-# side of the gap.
+# A FAULT TO WATCH.  A 39:26 split, at 1x batch / 1 slot, died on its eighth
+# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
+# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
+# i.e. a fault in the cross-card boundary copy, this layer split's hot path.  It
+# has not reproduced (38:27 served the whole e2e gate and a burst of requests;
+# 40:25 has served this file's verification battery), but the signature is
+# worth recognising -- it is NOT an OOM, and no amount of headroom prevents it.
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
 # those bare vars; 2026-08-13 incident).  llama.cpp takes --host/--port/--temp
@@ -112,27 +121,31 @@ HIP_VISIBLE_DEVICES=0,1
 DEVICES=ROCm0,ROCm1
 SPLIT_MODE=layer
 
-# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
-# its KV, so it must hold fewer target layers for the cards to end level -- at
-# 1:1 the drafter's card would sit ~9 GiB above the other.  This is the measured
-# point that keeps BOTH cards above a 3 GiB floor at the full window; a tighter
-# balance was tried and rejected (see SIZING above).
-TENSOR_SPLIT=38,27
+# 40:25 -- the measured balance at the deployed batch, and the point that
+# maximises CTX_SIZE: ROCm1 carries the drafter, its full-context KV and the
+# output layers, so it takes fewer target layers.  See SIZING above; one -ts
+# step is worth ~1.5 GiB at this batch, so do not move it without re-measuring.
+TENSOR_SPLIT=40,25
 
 NGL=999
 # Auto-fit has no business resizing a context we set explicitly.
 FIT=off
 FLASH_ATTN=on
 
-# One slot at the model's full configured window (n_ctx_train 262144), which was
-# measured to fit before it was deployed: see SIZING above.  --parallel DIVIDES
-# --ctx-size between slots, so one slot carries the whole window and a second
-# concurrent request queues behind it.
+# The model's full window (n_ctx_train 262144) divided over TWO slots, so each
+# conversation gets 131072 -- verified at load as `n_slots = 2, n_ctx_slot =
+# 131072`.  --parallel DIVIDES --ctx-size, so more slots would shrink every
+# conversation rather than grow the cache.
 CTX_SIZE=262144
-PARALLEL=1
+PARALLEL=2
 
-BATCH_SIZE=2048
-UBATCH_SIZE=512
+# 3x the build's defaults (2048/512).  --ubatch-size is the physical batch the
+# graph is built at and therefore what the compute buffers cost: at 262144 this
+# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does
+# -- which is what forced the rebalance above.  8192/2048 does not fit at all
+# here: its 2020 MiB pp buffer fails to allocate on ROCm1.
+BATCH_SIZE=6144
+UBATCH_SIZE=1536
 
 # Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
 NUM_THREADS=8