serverAppConfigs/serverAppConfigs-docs/LLM/GGUF Compatibility.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	GGUF Compatibility and Runtime Lock-in	How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.	gguf llama.cpp forks quantization speculative	2026-10-05T00:00:00-07:00

GGUF Compatibility and Runtime Lock-in

The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the header before promising a file to any loader. Stock llama.cpp, Ollama and LM Studio refuse custom types at load (unknown tensor type); a fork-private type means exactly one engine can serve the file. Measured on this fleet repeatedly: Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf is ftype 115 with 311 tensors of type 103 (GGML_TYPE_Q8_0_ROCMFPX) — only charlie12345/ROCmFPX reads it. The full type tables live in Llama Cpp Runtimes and Llama Cpp Code Diff (ggml_type 100–108, ftype families 100–106 / 110–113 / 118).

1. Read the header, do not guess

Parse the GGUF KV section and histogram the tensor types plus general.architecture.file_type — one python read over the first few MiB, no model load. Counting types out of the KV section settles a runtime argument in minutes; repo names and model cards do not.

2. Compare routing variants by tensor mix, not by name

Two files of the same quant can differ in how many tensors stay at plain Q8_0 (measured: 187 vs 2 output-head tensors in an _AGENT variant of the same size). That difference is what moves spec-decode acceptance (0.953 vs 0.911) — the "same size" file is not the same file.

3. Pin the build by commit SHA

A fork that can read the file today can still crash mid-load: the 2026-08-25 ROCmFPX builds read the ftype-115 file and then SIGSEGV in ggml_compute_forward_set_rows. Record the SHA in the build script (scripts/build-rocmfpx.sh pins 36e9acd) and check it out explicitly — never a branch.

4. Header diff method (candidate vs known-good)

Answer "will engine X load this GGUF?" by diffing the candidate's header against a file the engine already loads — never by filenames or model cards; filenames lie about quantization and about MTP/vision heads.

1. Get the bytes: both files, from byte 0, at least 64 MiB. Tokenizer KVs (151k token strings + merges) routinely eat 5–8 MiB, so a small range desynchronises the parser. True size from curl -sIL → x-linked-size (a bare HEAD Content-Length can lie):

   curl -sL -r 0-67108863 -o cand.bin \
     "https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
   ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
   

2. Parse — and lock the parser against the known-good file first. Layout (v2/v3): magic[4] version u32 n_tensors u64 n_kv u64, then KVs, then tensor infos. Strings are u64 length + bytes. The value-type enum is NOT llama.cpp's gguf.h: in the files seen (2026-09-28, unsloth + ajgazin UD GGUFs) type 8 = STRING, 9 = ARRAY; llama.cpp's header has STRING=2, ARRAY=3 and desynchronises immediately. Working table:

   0 u8   1 i8   2 u16  3 i16  4 u32  5 i32  6 f32
   7 bool 8 STR  9 ARRAY(u32 et, u64 n)  10 u64  11 i64  12 f64
   

   Validate before trusting: the parse is right iff the tensor-info block ends somewhere sane and names look real (blk.0.attn_q.weight, token_embd.weight, output_norm.weight). EOF throw = wrong enum or offset — do not work around it.

3. Diff what engines actually bind on: general.architecture; tensor count; tensor name list in order; per-tensor ggml_type and dims; geometry KVs (<arch>.block_count, .embedding_length, .feed_forward_length, .attention.head_count[_kv], .rope.*, .ssm.*, .context_length, .nextn_predict_layers); MTP tensors (nextn in the name); vision tensors (visual, v_ prefixes). A same-family fine-tune quantized by the same tool usually comes out byte-identical in every one of these — that is the answer.

4. Then read the engine's source before calling a delta a blocker. A metadata difference only matters if the engine reads that key (grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/). Worked example (gufo v0.1.1): general.file_type is deliberately not consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and qwen35.attention.recurrent_layers is never read (layer kinds come from full_attention_interval) — both deltas were non-issues the header diff alone would have flagged wrongly.

5. Say what you did not do. Header analysis is strong evidence, not a load. State byte sizes, check free disk before proposing a fetch, and do not download tens of GB unasked — pointing the engine at the file is the definitive test.

Report shape: verdict first (loads / will not / loads-with-caveats), then the side-by-side diff table, then only the load-bearing caveats: text-only vs vision tensors, chat-template differences (matters for llama.cpp, not for engines with compiled-in Jinja), license, disk headroom, and whether the base model is abliterated/uncensored.

5. A/B a fork's spec decode by stripping the whole flag set

--model-draft without --spec-type segfaults at load (measured 2026-10-02, core dump, no diagnostic). A baseline "no-draft" arm must remove --model-draft AND every --spec-draft-* AND --spec-type together — the flags move as a set.

6. Before deleting a retired payload, read the rollback tag's env file

A rollback may need a small companion dir: a drafter living in a directory named after an old quant looks like trash but is load-bearing. Same class as the gufo payload removal of 2026-10-03, which turned that rollback into a re-download.

Related

• Llama Cpp Runtimes — which fork ships which custom types
• Llama Cpp Code Diff — the enum tables and code-level differences

• kingjones-q8agent-rocmfpx — the era that served the ftype-115 file, and frank-dragoon for what runs there now

• Llama Cpp Serving Sizing — pricing what a payload costs once it loads