---
name: llamacpp-drafter-kv-split-sizing
description: "Size a llama.cpp multi-GPU layer-split deployment that carries a speculative drafter: read the drafter's KV law from its GGUF header, then find the split balance by measurement (the batch asymmetry, the ctx/floor trade, and the OOM-vs-page-fault signatures)."
---

# Sizing a llama.cpp layer-split with a speculative drafter

Use when a multi-GPU llama.cpp deployment carries a draft model (`--spec-draft-model`)
and the question is *where do the layers go and how big can the context be*. The two
effects below decide it, and neither is visible in the flags.

## 1. Read both GGUF headers first

The drafter's own properties set the memory bill. `llama-server` prints almost none of
them, and the loader's ledger is not printed at default verbosity.

Minimal GGUF KV reader (no tooling needed) — write `ggufmeta.py`:

```python
import struct, sys

def rd(path):
    f = open(path, 'rb')
    assert f.read(4) == b'GGUF'
    ver = struct.unpack('<I', f.read(4))[0]
    struct.unpack('<Q', f.read(8))                     # tensor count
    nk = struct.unpack('<Q', f.read(8))[0]             # kv count
    def s():
        n = struct.unpack('<Q', f.read(8))[0]
        return f.read(n).decode('utf-8', 'replace')
    # 0=UINT8 1=INT8 2=UINT16 3=INT16 4=UINT32 5=INT32 6=FLOAT32 7=BOOL
    # 8=STRING 9=ARRAY 10=UINT64 11=INT64 12=FLOAT64
    SCALAR = {0:1, 1:1, 2:2, 3:2, 4:4, 5:4, 6:4, 7:1, 10:8, 11:8, 12:8}
    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
    keys = {}
    for _ in range(nk):
        k = s()
        t = struct.unpack('<I', f.read(4))[0]
        if t == 8:
            v = s()
        elif t in SCALAR:
            v = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
        else:
            et = struct.unpack('<I', f.read(4))[0]     # element type FIRST
            n = struct.unpack('<Q', f.read(8))[0]      # then the count
            if et == 8:
                v = ('arr_str', [s() for _ in range(n)])
            elif et in SCALAR:
                v = ('arr', et, n); f.read(n * SCALAR[et])
            else:
                raise Exception('unsupported arr elem type %d' % et)
        keys[k] = v
    return ver, keys

for p in sys.argv[1:]:
    ver, k = rd(p)
    arch = k.get('general.architecture')
    print('=====', p, 'v%d' % ver, 'arch:', arch)
    for key in sorted(k):
        if key.startswith('tokenizer.') or isinstance(k[key], tuple):
            continue
        if key.startswith(arch + '.') or not key.startswith(('general.', 'quantize.')):
            print(' ', key, '=', k[key])
```

The three format traps, each of which cost a debugging session:

1. **The KV stream has NO alignment.** GGUF v3's alignment (from `general.alignment`,
   default 32) applies only to the *tensor data section* that follows. Do not pad
   between KV pairs.
2. **Array layout is `type(9) -> element_type(u32) -> count(u64)`.** Count-then-element
   is wrong and yields a `MemoryError`/`UnicodeDecodeError` a few pairs later.
3. **The type enum is not the order you expect**: `7=BOOL`, `9=ARRAY`, and the 64-bit
   types are `10/11/12`. Getting `3=INT16` (2 bytes) or `5=INT32` (4 bytes) wrong shifts
   the whole stream.

Fields that matter: `*.block_count`, `*.block_size`, `*.attention.head_count_kv`,
`*.attention.key_length`/`value_length`, `*.attention.sliding_window`,
`*.context_length`.

## 2. The drafter's KV is sized from the TARGET

`common/speculative.cpp` builds the draft context with:

```cpp
// the draft context holds as many tokens per sequence as the target context
cparams.n_ctx = llama_n_ctx(ctx_tgt);
```

There is **no flag** to cap it (the `--spec-draft-*` surface has none). Therefore:

```
draft KV bytes = ctx_tgt x block_count x 2 (K,V) x head_count_kv x key_length x dtype
```

*unless* the drafter declares a sliding window, in which case the window bounds it.

Measured on the same 262144-token target: a sidecar with `sliding_window = 2048` cost
~40 MiB of draft KV; a sidecar of the same 5-layer shape with **no** window cost
**5.0 GiB** (20 KiB/position). Swapping between them invalidated an already-balanced
layer split.

**Failure signature:** the engine loads the whole model, answers `/health`, and dies on
the FIRST draft decode with an allocation error (`ROCm error: out of memory` /
`cudaMalloc failed`) inside `common_speculative_impl_draft_dflash::process`. An
allocation failure *after* a successful load is a sizing failure, not a broken drafter
-- and under a self-healing unit it looks like a crash-loop instead.

## 3. A bigger batch is asymmetric across the split

`--ubatch-size` is the physical batch the graph is built at, so it sets the compute
buffers. Those buffers are **~2x heavier on the card that holds the output layers and
the drafter's graphs** than on the other(s). Measured, 2 GPUs at 262144 ctx:

| ubatch | output/drafter card | other card |
|---|---|---|
| 512 -> 1536 | +~3.3 GiB | +~1.7 GiB |

Consequence: **a split balanced at one batch is unbalanced at another.** A split at
1:1 or a hand-measured skew must be re-measured after any change to `--ubatch-size`,
`--ctx-size` or `--parallel`.

## 4. Find the balance by measurement, not arithmetic

The per-step cost is not derivable from the parameter count (KV lands only on the
attention layers, and buffer placement is per-device). One `-ts` step is worth roughly
1.5 GiB of load at ~6k batch / 262k ctx on 30 GiB-class cards — enough to bracket, not
to solve. Procedure:

1. Copy the env file to a scratch path and run the serve script against it
   (`SERVICE_ENV_FILE=/tmp/test.env bash scripts/serve-<svc>.sh`) rather than editing
   the deployed file. Do it with the unit stopped — one GPU, one process.
2. Bracket: try a step, read `rocm-smi --showmeminfo vram` for both cards after a real
   generation, and step again. Three points is usually enough to land within ~0.1 GiB.
3. Prefer the balanced split, because the busier card's headroom is what bounds the
   context:

   ```
   ctx_extra ~= (headroom_target - measured_free) / (per-position cost on the busier card)
   ```

   where per-position cost = the card's share of the target's KV + the drafter's KV if
   that card carries it. A balanced split therefore MAXIMISES ctx for a given floor.
4. Verify with the full battery before committing: `/health`, `/v1/models`, a real
   completion, a tool call, two concurrent long prompts (one per slot), `slots`
   reporting the expected `n_ctx` per slot, and the per-task `draft acceptance = ...`
   line per slot.

## 5. Keep the two failure signatures apart

| Signature | Meaning |
|---|---|
| loads, `/health` ok, first decode OOMs | sizing: drafter KV, batch buffers, or KV vs ctx (section 2) |
| `SDMA0 ... page fault` + `illegal memory access` + `ggml_abort` | a GPU fault in the cross-card boundary copy — the layer split's hot path. **Not** an OOM; headroom does not prevent it |

The second one is why a rebalance should be validated by a request battery and not just
by `rocm-smi` numbers.

## Ops traps

- A server started over ssh (including under a supervisor/`hub`-style runner) **outlives
  stopping that runner** — the remote process keeps the port, and the next start fails
  with `couldn't bind HTTP server socket`. `pkill -x <exact-binary-name>` first.
- `systemctl start` of a root unit from a non-root ssh session fails with
  `Access denied ... requires interactive authentication`: use sudo.
- An old crash's `ROCm error` lines remain in `journalctl -u <svc>` forever. Check the
  current instance's `ActiveEnterTimestamp`/`NRestarts` before treating a grep count as
  a live fault.