55 lines
5.7 KiB
Markdown
55 lines
5.7 KiB
Markdown
---
|
||
name: gguf-header-vram-fit-analysis
|
||
description: "Decide if a GGUF model fits an LLM endpoint's GPU at a target context window: HTTP range-parse the header for KV geometry and custom quant types, derive the KiB/token rate law, price non-KV from measured boots (own repo + the publisher's benchmark reports), and gate on runtime lock-in (fork, build flags, drafter) before any download."
|
||
---
|
||
|
||
Answer "can model M fit window W on endpoint E (plus vision?)" **before** downloading anything. The GGUF header carries the truth; measured boots are the price list.
|
||
|
||
## 1. Range-parse the GGUF header from the Hub (no download)
|
||
|
||
`GET` the `resolve/main/<file>.gguf` URL with `Range: bytes=0-N` and a `User-Agent: curl/8` header, parse GGUF in Python (`struct`):
|
||
|
||
- Layout: magic `GGUF` (read `buf[0:4]` directly — do NOT `read_string()` it), u32 version, u64 ntensors, u64 nkv, then KV pairs (v3: u64 string lengths; v2: u32), then tensor-info table (name, ndim, dims u64[], type u32, offset u64). Handle EVERY array element type or the parse desyncs mid-metadata.
|
||
- **Metadata is >8 MiB** when vocab is ~250k (tokenizer arrays live in the KV section). Fetch 32–64 MiB.
|
||
- **Per-tensor byte sizes without a downloader**: the data section is in table order — `size(i) = offset(i+1) - offset(i)`, last = `file_size - header_end - offset(last)`. Then `bits/elem = bytes*8/prod(dims)`. That names unknown type ids by their rate (2.125 bpw ⇒ ternary-in-2-bit-slots + fp16 scale/128; 16.0 ⇒ BF16; 32 ⇒ F32) and totals weights per type.
|
||
|
||
Extract: `general.architecture`, `block_count`, `full_attention_interval`, `attention.head_count_kv`, `key_length`/`value_length`, `context_length`, `embedding_length`, `ssm.*`, and **`<arch>.nextn_predict_layers`** — its absence (plus `block_count` = real layers, not +1) means **no in-file MTP head** (MTP is a property of the file, not the flags). Type histogram decides stock vs fork: a type id above stock `GGML_TYPE_COUNT` is fork lock-in — read the fork's `ggml/include/ggml.h` enum to name it.
|
||
|
||
## 2. KV rate law (the wall)
|
||
|
||
For hybrid (qwen35-style) models only the FULL-attention layers bill per token; linear/DeltaNet layers bill a fixed state per SLOT (≈50–60 MiB/slot on 27B-class hybrids — noise vs KV, but it is what lets N slots fit).
|
||
|
||
```
|
||
elems/token/cache = (block_count / full_attention_interval) × head_count_kv × key_length
|
||
B/token = elems × bytesK + elems × bytesV (K and V both)
|
||
bytes/elem: f16=2.0, q8_0=1.0625, q4_0=0.5625, q4_1=0.578125
|
||
```
|
||
|
||
Fleet-verified instances: qwen35-9B (8 full layers) 32 KiB/token f16; qwen35-27B (16 full) 64 KiB/token f16, 34 KiB q8_0, 18 KiB q4_0. A low-bit WEIGHT quant never moves this line — state it explicitly, it is the question people actually ask. `--flash-attn on` is required for quantized V; quantized KV needs a `GGML_CUDA_FA_ALL_QUANTS=ON` build.
|
||
|
||
## 3. Price non-KV from MEASURED boots, never a formula
|
||
|
||
Two anchors beat one:
|
||
- **Own repo**: closest runbook boot (same arch, same card, ideally same mmproj): `non_kv = boot_used − rate × boot_pool`, then swap weights/mmproj/drafter/ubatch-rung deltas.
|
||
- **Publisher**: their published benchmark/server reports (e.g. a `community-benchmarks/<card>.md` or `KV-CACHE.md`) carry peak `nvidia-smi` and a steady-state figure for a named argv — back out their overhead = steady − weights − KV(× their pool) − mmproj. Divergence between the two anchors IS your uncertainty band; report both ends.
|
||
Card/vendor peak-VRAM at bench context also calibrates compute buffers at small ubatch; the fleet's ubatch-rung costs (≈0.3 GiB @1024, ≈0.75 GiB @2048 on a 27B) cover the rest.
|
||
|
||
Card ceiling from settled boots (`used + free`, e.g. 24,125 MiB on the 3090), not the spec sheet. An mmproj bills ~1.5× its file size. Keep ≥1 GiB headroom when ranking arms; tightest proven fleet fit is 489 MiB.
|
||
|
||
## 4. Gates beyond VRAM
|
||
|
||
- **Target window > `context_length`** ⇒ YaRN required (`--rope-scale = target/native`); quality at >1.2–1.5× on aggressive quants is unmeasured — say so.
|
||
- **Pool divides across slots**: `n_ctx_slot = CTX_SIZE / PARALLEL`; the per-slot number is the harness contract, and the endpoint's role (e.g. compaction hub needing ≥170k) picks the geometry.
|
||
- **Fork/custom-quant lock-in checklist** (answer all, cite ids):
|
||
1. which fork/branch, and does the pinned release tag contain the needed fixes — `GET /compare/<tag>...prism` and read `ahead_by` + commit subjects (a tag 5 commits behind contains everything merged earlier);
|
||
2. host CPU traps (e.g. AVX-512 load segfaults crash even with all layers GPU-offloaded);
|
||
3. **does the release workflow pass the build flags the contract needs** — grep the release workflow's cmake args: prebuilts often omit `GGML_CUDA_FA_ALL_QUANTS`, and without it mismatched/unsupported K/V types silently run flash attention on the CPU;
|
||
4. drafter availability (in-file `nextn`, a paired drafter, or nothing) — no drafter means the decode regression is unfixable;
|
||
5. pack choice per architecture (the vendor's throughput table is generation-specific; a community report on the same GPU family outranks it);
|
||
6. behavior/role risk: base model censorship posture, `tool_calls` defects, `reasoning_effort` template gate, prompt-cache `--cache-ram` vs host RAM actually available.
|
||
- Vision: needs `--mmproj`; probe with ≥256 px solid color; only the OpenAI `image_url` content part reaches the model.
|
||
|
||
## 5. Output
|
||
|
||
Table of arms (pack × KV dtype × slot geometry) → KV MiB, total, free vs ceiling, FITS/FITS-tight/NO FIT, with the uncertainty band; then the gate list. State every `[INFERENCE]`/estimate. The boot ledger (verbosity 4), not the arithmetic, is the referee; benching requires stop+mask per fleet policy.
|