command-center/agent/managed-skills/llamacpp-drafter-kv-split-sizing/SKILL.md

name, description

name	description
llamacpp-drafter-kv-split-sizing	Size a llama.cpp multi-GPU layer-split deployment that carries a speculative drafter: read the drafter's KV law from its GGUF header, then find the split balance by measurement (the batch asymmetry, the ctx/floor trade, and the OOM-vs-page-fault signatures).

Sizing a llama.cpp layer-split with a speculative drafter

Use when a multi-GPU llama.cpp deployment carries a draft model (--spec-draft-model) and the question is where do the layers go and how big can the context be. The two effects below decide it, and neither is visible in the flags.

1. Read both GGUF headers first

The drafter's own properties set the memory bill. llama-server prints almost none of them, and the loader's ledger is not printed at default verbosity.

Minimal GGUF KV reader (no tooling needed) — write ggufmeta.py:

import struct, sys

def rd(path):
    f = open(path, 'rb')
    assert f.read(4) == b'GGUF'
    ver = struct.unpack('<I', f.read(4))[0]
    struct.unpack('<Q', f.read(8))                     # tensor count
    nk = struct.unpack('<Q', f.read(8))[0]             # kv count
    def s():
        n = struct.unpack('<Q', f.read(8))[0]
        return f.read(n).decode('utf-8', 'replace')
    # 0=UINT8 1=INT8 2=UINT16 3=INT16 4=UINT32 5=INT32 6=FLOAT32 7=BOOL
    # 8=STRING 9=ARRAY 10=UINT64 11=INT64 12=FLOAT64
    SCALAR = {0:1, 1:1, 2:2, 3:2, 4:4, 5:4, 6:4, 7:1, 10:8, 11:8, 12:8}
    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
    keys = {}
    for _ in range(nk):
        k = s()
        t = struct.unpack('<I', f.read(4))[0]
        if t == 8:
            v = s()
        elif t in SCALAR:
            v = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
        else:
            et = struct.unpack('<I', f.read(4))[0]     # element type FIRST
            n = struct.unpack('<Q', f.read(8))[0]      # then the count
            if et == 8:
                v = ('arr_str', [s() for _ in range(n)])
            elif et in SCALAR:
                v = ('arr', et, n); f.read(n * SCALAR[et])
            else:
                raise Exception('unsupported arr elem type %d' % et)
        keys[k] = v
    return ver, keys

for p in sys.argv[1:]:
    ver, k = rd(p)
    arch = k.get('general.architecture')
    print('=====', p, 'v%d' % ver, 'arch:', arch)
    for key in sorted(k):
        if key.startswith('tokenizer.') or isinstance(k[key], tuple):
            continue
        if key.startswith(arch + '.') or not key.startswith(('general.', 'quantize.')):
            print(' ', key, '=', k[key])

The three format traps, each of which cost a debugging session:

1. The KV stream has NO alignment. GGUF v3's alignment (from general.alignment, default 32) applies only to the tensor data section that follows. Do not pad between KV pairs.
2. Array layout is type(9) -> element_type(u32) -> count(u64). Count-then-element is wrong and yields a MemoryError/UnicodeDecodeError a few pairs later.
3. The type enum is not the order you expect: 7=BOOL, 9=ARRAY, and the 64-bit types are 10/11/12. Getting 3=INT16 (2 bytes) or 5=INT32 (4 bytes) wrong shifts the whole stream.

Fields that matter: *.block_count, *.block_size, *.attention.head_count_kv, *.attention.key_length/value_length, *.attention.sliding_window, *.context_length.

2. The drafter's KV is sized from the TARGET

common/speculative.cpp builds the draft context with:

// the draft context holds as many tokens per sequence as the target context
cparams.n_ctx = llama_n_ctx(ctx_tgt);

There is no flag to cap it (the --spec-draft-* surface has none). Therefore:

draft KV bytes = ctx_tgt x block_count x 2 (K,V) x head_count_kv x key_length x dtype

unless the drafter declares a sliding window, in which case the window bounds it.

Measured on the same 262144-token target: a sidecar with sliding_window = 2048 cost ~40 MiB of draft KV; a sidecar of the same 5-layer shape with no window cost 5.0 GiB (20 KiB/position). Swapping between them invalidated an already-balanced layer split.

Failure signature: the engine loads the whole model, answers /health, and dies on the FIRST draft decode with an allocation error (ROCm error: out of memory / cudaMalloc failed) inside common_speculative_impl_draft_dflash::process. An allocation failure after a successful load is a sizing failure, not a broken drafter -- and under a self-healing unit it looks like a crash-loop instead.

3. A bigger batch is asymmetric across the split

--ubatch-size is the physical batch the graph is built at, so it sets the compute buffers. Those buffers are ~2x heavier on the card that holds the output layers and the drafter's graphs than on the other(s). Measured, 2 GPUs at 262144 ctx:

ubatch	output/drafter card	other card
512 -> 1536	+~3.3 GiB	+~1.7 GiB

Consequence: a split balanced at one batch is unbalanced at another. A split at 1:1 or a hand-measured skew must be re-measured after any change to --ubatch-size, --ctx-size or --parallel.

4. Find the balance by measurement, not arithmetic

The per-step cost is not derivable from the parameter count (KV lands only on the attention layers, and buffer placement is per-device). One -ts step is worth roughly 1.5 GiB of load at ~6k batch / 262k ctx on 30 GiB-class cards — enough to bracket, not to solve. Procedure:

1. Copy the env file to a scratch path and run the serve script against it (SERVICE_ENV_FILE=/tmp/test.env bash scripts/serve-<svc>.sh) rather than editing the deployed file. Do it with the unit stopped — one GPU, one process.

2. Bracket: try a step, read rocm-smi --showmeminfo vram for both cards after a real generation, and step again. Three points is usually enough to land within ~0.1 GiB.

3. Prefer the balanced split, because the busier card's headroom is what bounds the context:

   ctx_extra ~= (headroom_target - measured_free) / (per-position cost on the busier card)
   

   where per-position cost = the card's share of the target's KV + the drafter's KV if that card carries it. A balanced split therefore MAXIMISES ctx for a given floor.

4. Verify with the full battery before committing: /health, /v1/models, a real completion, a tool call, two concurrent long prompts (one per slot), slots reporting the expected n_ctx per slot, and the per-task draft acceptance = ... line per slot.

5. Keep the two failure signatures apart

Signature	Meaning
loads, /health ok, first decode OOMs	sizing: drafter KV, batch buffers, or KV vs ctx (section 2)
SDMA0 ... page fault + illegal memory access + ggml_abort	a GPU fault in the cross-card boundary copy — the layer split's hot path. Not an OOM; headroom does not prevent it

The second one is why a rebalance should be validated by a request battery and not just by rocm-smi numbers.

Ops traps

• A server started over ssh (including under a supervisor/hub-style runner) outlives stopping that runner — the remote process keeps the port, and the next start fails with couldn't bind HTTP server socket. pkill -x <exact-binary-name> first.
• systemctl start of a root unit from a non-root ssh session fails with Access denied ... requires interactive authentication: use sudo.
• An old crash's ROCm error lines remain in journalctl -u <svc> forever. Check the current instance's ActiveEnterTimestamp/NRestarts before treating a grep count as a live fault.