command-center/agent/managed-skills/gguf-header-vram-fit-analysis/SKILL.md

name, description

name	description
gguf-header-vram-fit-analysis	Decide if a GGUF model fits an LLM endpoint's GPU at a target context window: HTTP range-parse the header for KV geometry and custom quant types, derive the KiB/token rate law, price non-KV from measured boots (own repo + the publisher's benchmark reports), and gate on runtime lock-in (fork, build flags, drafter) before any download.

Answer "can model M fit window W on endpoint E (plus vision?)" before downloading anything. The GGUF header carries the truth; measured boots are the price list.

1. Range-parse the GGUF header from the Hub (no download)

GET the resolve/main/<file>.gguf URL with Range: bytes=0-N and a User-Agent: curl/8 header, parse GGUF in Python (struct):

• Layout: magic GGUF (read buf[0:4] directly — do NOT read_string() it), u32 version, u64 ntensors, u64 nkv, then KV pairs (v3: u64 string lengths; v2: u32), then tensor-info table (name, ndim, dims u64[], type u32, offset u64). Handle EVERY array element type or the parse desyncs mid-metadata.
• Metadata is >8 MiB when vocab is ~250k (tokenizer arrays live in the KV section). Fetch 32–64 MiB.
• Per-tensor byte sizes without a downloader: the data section is in table order — size(i) = offset(i+1) - offset(i), last = file_size - header_end - offset(last). Then bits/elem = bytes*8/prod(dims). That names unknown type ids by their rate (2.125 bpw ⇒ ternary-in-2-bit-slots + fp16 scale/128; 16.0 ⇒ BF16; 32 ⇒ F32) and totals weights per type.

Extract: general.architecture, block_count, full_attention_interval, attention.head_count_kv, key_length/value_length, context_length, embedding_length, ssm.*, and <arch>.nextn_predict_layers — its absence (plus block_count = real layers, not +1) means no in-file MTP head (MTP is a property of the file, not the flags). Type histogram decides stock vs fork: a type id above stock GGML_TYPE_COUNT is fork lock-in — read the fork's ggml/include/ggml.h enum to name it.

2. KV rate law (the wall)

For hybrid (qwen35-style) models only the FULL-attention layers bill per token; linear/DeltaNet layers bill a fixed state per SLOT (≈50–60 MiB/slot on 27B-class hybrids — noise vs KV, but it is what lets N slots fit).

elems/token/cache = (block_count / full_attention_interval) × head_count_kv × key_length
B/token           = elems × bytesK + elems × bytesV      (K and V both)
bytes/elem: f16=2.0, q8_0=1.0625, q4_0=0.5625, q4_1=0.578125

Fleet-verified instances: qwen35-9B (8 full layers) 32 KiB/token f16; qwen35-27B (16 full) 64 KiB/token f16, 34 KiB q8_0, 18 KiB q4_0. A low-bit WEIGHT quant never moves this line — state it explicitly, it is the question people actually ask. --flash-attn on is required for quantized V; quantized KV needs a GGML_CUDA_FA_ALL_QUANTS=ON build.

3. Price non-KV from MEASURED boots, never a formula

Two anchors beat one:

• Own repo: closest runbook boot (same arch, same card, ideally same mmproj): non_kv = boot_used − rate × boot_pool, then swap weights/mmproj/drafter/ubatch-rung deltas.
• Publisher: their published benchmark/server reports (e.g. a community-benchmarks/<card>.md or KV-CACHE.md) carry peak nvidia-smi and a steady-state figure for a named argv — back out their overhead = steady − weights − KV(× their pool) − mmproj. Divergence between the two anchors IS your uncertainty band; report both ends. Card/vendor peak-VRAM at bench context also calibrates compute buffers at small ubatch; the fleet's ubatch-rung costs (≈0.3 GiB @1024, ≈0.75 GiB @2048 on a 27B) cover the rest.

Card ceiling from settled boots (used + free, e.g. 24,125 MiB on the 3090), not the spec sheet. An mmproj bills ~1.5× its file size. Keep ≥1 GiB headroom when ranking arms; tightest proven fleet fit is 489 MiB.

4. Gates beyond VRAM

• Target window > context_length ⇒ YaRN required (--rope-scale = target/native); quality at >1.2–1.5× on aggressive quants is unmeasured — say so.
• Pool divides across slots: n_ctx_slot = CTX_SIZE / PARALLEL; the per-slot number is the harness contract, and the endpoint's role (e.g. compaction hub needing ≥170k) picks the geometry.
• Fork/custom-quant lock-in checklist (answer all, cite ids):
  1. which fork/branch, and does the pinned release tag contain the needed fixes — GET /compare/<tag>...prism and read ahead_by + commit subjects (a tag 5 commits behind contains everything merged earlier);
  2. host CPU traps (e.g. AVX-512 load segfaults crash even with all layers GPU-offloaded);
  3. does the release workflow pass the build flags the contract needs — grep the release workflow's cmake args: prebuilts often omit GGML_CUDA_FA_ALL_QUANTS, and without it mismatched/unsupported K/V types silently run flash attention on the CPU;
  4. drafter availability (in-file nextn, a paired drafter, or nothing) — no drafter means the decode regression is unfixable;
  5. pack choice per architecture (the vendor's throughput table is generation-specific; a community report on the same GPU family outranks it);
  6. behavior/role risk: base model censorship posture, tool_calls defects, reasoning_effort template gate, prompt-cache --cache-ram vs host RAM actually available.
• Vision: needs --mmproj; probe with ≥256 px solid color; only the OpenAI image_url content part reaches the model.

5. Output

Table of arms (pack × KV dtype × slot geometry) → KV MiB, total, free vs ceiling, FITS/FITS-tight/NO FIT, with the uncertainty band; then the gate list. State every [INFERENCE]/estimate. The boot ledger (verbosity 4), not the arithmetic, is the referee; benching requires stop+mask per fleet policy.