agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
Frank committed
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,168 @@
|
||||
---
|
||||
name: llamacpp-drafter-kv-split-sizing
|
||||
description: "Size a llama.cpp multi-GPU layer-split deployment that carries a speculative drafter: read the drafter's KV law from its GGUF header, then find the split balance by measurement (the batch asymmetry, the ctx/floor trade, and the OOM-vs-page-fault signatures)."
|
||||
---
|
||||
|
||||
# Sizing a llama.cpp layer-split with a speculative drafter
|
||||
|
||||
Use when a multi-GPU llama.cpp deployment carries a draft model (`--spec-draft-model`)
|
||||
and the question is *where do the layers go and how big can the context be*. The two
|
||||
effects below decide it, and neither is visible in the flags.
|
||||
|
||||
## 1. Read both GGUF headers first
|
||||
|
||||
The drafter's own properties set the memory bill. `llama-server` prints almost none of
|
||||
them, and the loader's ledger is not printed at default verbosity.
|
||||
|
||||
Minimal GGUF KV reader (no tooling needed) — write `ggufmeta.py`:
|
||||
|
||||
```python
|
||||
import struct, sys
|
||||
|
||||
def rd(path):
|
||||
f = open(path, 'rb')
|
||||
assert f.read(4) == b'GGUF'
|
||||
ver = struct.unpack('<I', f.read(4))[0]
|
||||
struct.unpack('<Q', f.read(8)) # tensor count
|
||||
nk = struct.unpack('<Q', f.read(8))[0] # kv count
|
||||
def s():
|
||||
n = struct.unpack('<Q', f.read(8))[0]
|
||||
return f.read(n).decode('utf-8', 'replace')
|
||||
# 0=UINT8 1=INT8 2=UINT16 3=INT16 4=UINT32 5=INT32 6=FLOAT32 7=BOOL
|
||||
# 8=STRING 9=ARRAY 10=UINT64 11=INT64 12=FLOAT64
|
||||
SCALAR = {0:1, 1:1, 2:2, 3:2, 4:4, 5:4, 6:4, 7:1, 10:8, 11:8, 12:8}
|
||||
FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
|
||||
keys = {}
|
||||
for _ in range(nk):
|
||||
k = s()
|
||||
t = struct.unpack('<I', f.read(4))[0]
|
||||
if t == 8:
|
||||
v = s()
|
||||
elif t in SCALAR:
|
||||
v = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
|
||||
else:
|
||||
et = struct.unpack('<I', f.read(4))[0] # element type FIRST
|
||||
n = struct.unpack('<Q', f.read(8))[0] # then the count
|
||||
if et == 8:
|
||||
v = ('arr_str', [s() for _ in range(n)])
|
||||
elif et in SCALAR:
|
||||
v = ('arr', et, n); f.read(n * SCALAR[et])
|
||||
else:
|
||||
raise Exception('unsupported arr elem type %d' % et)
|
||||
keys[k] = v
|
||||
return ver, keys
|
||||
|
||||
for p in sys.argv[1:]:
|
||||
ver, k = rd(p)
|
||||
arch = k.get('general.architecture')
|
||||
print('=====', p, 'v%d' % ver, 'arch:', arch)
|
||||
for key in sorted(k):
|
||||
if key.startswith('tokenizer.') or isinstance(k[key], tuple):
|
||||
continue
|
||||
if key.startswith(arch + '.') or not key.startswith(('general.', 'quantize.')):
|
||||
print(' ', key, '=', k[key])
|
||||
```
|
||||
|
||||
The three format traps, each of which cost a debugging session:
|
||||
|
||||
1. **The KV stream has NO alignment.** GGUF v3's alignment (from `general.alignment`,
|
||||
default 32) applies only to the *tensor data section* that follows. Do not pad
|
||||
between KV pairs.
|
||||
2. **Array layout is `type(9) -> element_type(u32) -> count(u64)`.** Count-then-element
|
||||
is wrong and yields a `MemoryError`/`UnicodeDecodeError` a few pairs later.
|
||||
3. **The type enum is not the order you expect**: `7=BOOL`, `9=ARRAY`, and the 64-bit
|
||||
types are `10/11/12`. Getting `3=INT16` (2 bytes) or `5=INT32` (4 bytes) wrong shifts
|
||||
the whole stream.
|
||||
|
||||
Fields that matter: `*.block_count`, `*.block_size`, `*.attention.head_count_kv`,
|
||||
`*.attention.key_length`/`value_length`, `*.attention.sliding_window`,
|
||||
`*.context_length`.
|
||||
|
||||
## 2. The drafter's KV is sized from the TARGET
|
||||
|
||||
`common/speculative.cpp` builds the draft context with:
|
||||
|
||||
```cpp
|
||||
// the draft context holds as many tokens per sequence as the target context
|
||||
cparams.n_ctx = llama_n_ctx(ctx_tgt);
|
||||
```
|
||||
|
||||
There is **no flag** to cap it (the `--spec-draft-*` surface has none). Therefore:
|
||||
|
||||
```
|
||||
draft KV bytes = ctx_tgt x block_count x 2 (K,V) x head_count_kv x key_length x dtype
|
||||
```
|
||||
|
||||
*unless* the drafter declares a sliding window, in which case the window bounds it.
|
||||
|
||||
Measured on the same 262144-token target: a sidecar with `sliding_window = 2048` cost
|
||||
~40 MiB of draft KV; a sidecar of the same 5-layer shape with **no** window cost
|
||||
**5.0 GiB** (20 KiB/position). Swapping between them invalidated an already-balanced
|
||||
layer split.
|
||||
|
||||
**Failure signature:** the engine loads the whole model, answers `/health`, and dies on
|
||||
the FIRST draft decode with an allocation error (`ROCm error: out of memory` /
|
||||
`cudaMalloc failed`) inside `common_speculative_impl_draft_dflash::process`. An
|
||||
allocation failure *after* a successful load is a sizing failure, not a broken drafter
|
||||
-- and under a self-healing unit it looks like a crash-loop instead.
|
||||
|
||||
## 3. A bigger batch is asymmetric across the split
|
||||
|
||||
`--ubatch-size` is the physical batch the graph is built at, so it sets the compute
|
||||
buffers. Those buffers are **~2x heavier on the card that holds the output layers and
|
||||
the drafter's graphs** than on the other(s). Measured, 2 GPUs at 262144 ctx:
|
||||
|
||||
| ubatch | output/drafter card | other card |
|
||||
|---|---|---|
|
||||
| 512 -> 1536 | +~3.3 GiB | +~1.7 GiB |
|
||||
|
||||
Consequence: **a split balanced at one batch is unbalanced at another.** A split at
|
||||
1:1 or a hand-measured skew must be re-measured after any change to `--ubatch-size`,
|
||||
`--ctx-size` or `--parallel`.
|
||||
|
||||
## 4. Find the balance by measurement, not arithmetic
|
||||
|
||||
The per-step cost is not derivable from the parameter count (KV lands only on the
|
||||
attention layers, and buffer placement is per-device). One `-ts` step is worth roughly
|
||||
1.5 GiB of load at ~6k batch / 262k ctx on 30 GiB-class cards — enough to bracket, not
|
||||
to solve. Procedure:
|
||||
|
||||
1. Copy the env file to a scratch path and run the serve script against it
|
||||
(`SERVICE_ENV_FILE=/tmp/test.env bash scripts/serve-<svc>.sh`) rather than editing
|
||||
the deployed file. Do it with the unit stopped — one GPU, one process.
|
||||
2. Bracket: try a step, read `rocm-smi --showmeminfo vram` for both cards after a real
|
||||
generation, and step again. Three points is usually enough to land within ~0.1 GiB.
|
||||
3. Prefer the balanced split, because the busier card's headroom is what bounds the
|
||||
context:
|
||||
|
||||
```
|
||||
ctx_extra ~= (headroom_target - measured_free) / (per-position cost on the busier card)
|
||||
```
|
||||
|
||||
where per-position cost = the card's share of the target's KV + the drafter's KV if
|
||||
that card carries it. A balanced split therefore MAXIMISES ctx for a given floor.
|
||||
4. Verify with the full battery before committing: `/health`, `/v1/models`, a real
|
||||
completion, a tool call, two concurrent long prompts (one per slot), `slots`
|
||||
reporting the expected `n_ctx` per slot, and the per-task `draft acceptance = ...`
|
||||
line per slot.
|
||||
|
||||
## 5. Keep the two failure signatures apart
|
||||
|
||||
| Signature | Meaning |
|
||||
|---|---|
|
||||
| loads, `/health` ok, first decode OOMs | sizing: drafter KV, batch buffers, or KV vs ctx (section 2) |
|
||||
| `SDMA0 ... page fault` + `illegal memory access` + `ggml_abort` | a GPU fault in the cross-card boundary copy — the layer split's hot path. **Not** an OOM; headroom does not prevent it |
|
||||
|
||||
The second one is why a rebalance should be validated by a request battery and not just
|
||||
by `rocm-smi` numbers.
|
||||
|
||||
## Ops traps
|
||||
|
||||
- A server started over ssh (including under a supervisor/`hub`-style runner) **outlives
|
||||
stopping that runner** — the remote process keeps the port, and the next start fails
|
||||
with `couldn't bind HTTP server socket`. `pkill -x <exact-binary-name>` first.
|
||||
- `systemctl start` of a root unit from a non-root ssh session fails with
|
||||
`Access denied ... requires interactive authentication`: use sudo.
|
||||
- An old crash's `ROCm error` lines remain in `journalctl -u <svc>` forever. Check the
|
||||
current instance's `ActiveEnterTimestamp`/`NRestarts` before treating a grep count as
|
||||
a live fault.
|
||||
Reference in new issue
Block a user