Frank
327e78b950
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
125 lines
5.9 KiB
Markdown
125 lines
5.9 KiB
Markdown
---
|
||
type: reference
|
||
title: GGUF Compatibility and Runtime Lock-in
|
||
description: "Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can."
|
||
tags: [gguf, llama.cpp, forks, quantization, speculative]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# GGUF Compatibility and Runtime Lock-in
|
||
|
||
**The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the
|
||
header before promising a file to any loader.** Stock llama.cpp, Ollama and LM
|
||
Studio refuse custom types at load (`unknown tensor type`); a fork-private type
|
||
means exactly one engine can serve the file. Measured on this fleet repeatedly:
|
||
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
|
||
(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
|
||
tables live in and
|
||
(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
|
||
|
||
## 1. Read the header, do not guess
|
||
|
||
Parse the GGUF KV section and histogram the tensor types plus
|
||
`general.architecture.file_type` — one python read over the first few MiB, no
|
||
model load. Counting types out of the KV section settles a runtime argument in
|
||
minutes; repo names and model cards do not.
|
||
|
||
## 2. Compare routing variants by tensor mix, not by name
|
||
|
||
Two files of the same quant can differ in how many tensors stay at plain Q8_0
|
||
(measured: 187 vs 2 output-head tensors in an `_AGENT` variant of the same
|
||
size). That difference is what moves spec-decode acceptance (0.953 vs 0.911)
|
||
— the "same size" file is not the same file.
|
||
|
||
## 3. Pin the build by commit SHA
|
||
|
||
A fork that can read the file today can still crash mid-load: the 2026-08-25
|
||
ROCmFPX builds read the ftype-115 file and then SIGSEGV in
|
||
`ggml_compute_forward_set_rows`. Record the SHA in the build script
|
||
(`scripts/build-rocmfpx.sh` pins `36e9acd`) and check it out explicitly — never
|
||
a branch.
|
||
|
||
## 4. Header diff method (candidate vs known-good)
|
||
|
||
Answer "will engine X load this GGUF?" by diffing the candidate's header
|
||
against a file the engine **already loads** — never by filenames or model cards;
|
||
filenames lie about quantization and about MTP/vision heads.
|
||
|
||
1. **Get the bytes**: both files, from byte 0, at least 64 MiB. Tokenizer KVs
|
||
(151k token strings + merges) routinely eat 5–8 MiB, so a small range
|
||
desynchronises the parser. True size from `curl -sIL` → `x-linked-size`
|
||
(a bare HEAD `Content-Length` can lie):
|
||
|
||
```sh
|
||
curl -sL -r 0-67108863 -o cand.bin \
|
||
"https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
|
||
ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
|
||
```
|
||
|
||
2. **Parse — and lock the parser against the known-good file first.** Layout
|
||
(v2/v3): `magic[4]` `version u32` `n_tensors u64` `n_kv u64`, then KVs, then
|
||
tensor infos. Strings are u64 length + bytes. **The value-type enum is NOT
|
||
llama.cpp's `gguf.h`:** in the files seen (2026-09-28, unsloth + ajgazin UD
|
||
GGUFs) type **8 = STRING, 9 = ARRAY**; llama.cpp's header has STRING=2,
|
||
ARRAY=3 and desynchronises immediately. Working table:
|
||
|
||
```
|
||
0 u8 1 i8 2 u16 3 i16 4 u32 5 i32 6 f32
|
||
7 bool 8 STR 9 ARRAY(u32 et, u64 n) 10 u64 11 i64 12 f64
|
||
```
|
||
|
||
Validate before trusting: the parse is right iff the tensor-info block ends
|
||
somewhere sane and names look real (`blk.0.attn_q.weight`
|
||
`token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
|
||
offset — do not work around it.
|
||
|
||
3. **Diff what engines actually bind on**: `general.architecture`; tensor
|
||
count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
|
||
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`
|
||
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`
|
||
`.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
|
||
(`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
|
||
tool usually comes out byte-identical in every one of these — that is the
|
||
answer.
|
||
|
||
4. **Then read the engine's source before calling a delta a blocker.** A
|
||
metadata difference only matters if the engine reads that key
|
||
(`grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/`).
|
||
Worked example (gufo v0.1.1): `general.file_type` is deliberately not
|
||
consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and
|
||
`qwen35.attention.recurrent_layers` is never read (layer kinds come from
|
||
`full_attention_interval`) — both deltas were non-issues the header diff
|
||
alone would have flagged wrongly.
|
||
|
||
5. **Say what you did not do.** Header analysis is strong evidence, not a
|
||
load. State byte sizes, check free disk before proposing a fetch, and do
|
||
not download tens of GB unasked — pointing the engine at the file is the
|
||
definitive test.
|
||
|
||
**Report shape:** verdict first (loads / will not / loads-with-caveats), then
|
||
the side-by-side diff table, then only the load-bearing caveats: text-only vs
|
||
vision tensors, chat-template differences (matters for llama.cpp, not for
|
||
engines with compiled-in Jinja), license, disk headroom, and whether the base
|
||
model is abliterated/uncensored.
|
||
|
||
## 5. A/B a fork's spec decode by stripping the whole flag set
|
||
|
||
`--model-draft` without `--spec-type` segfaults at load (measured 2026-10-02,
|
||
core dump, no diagnostic). A baseline "no-draft" arm must remove `--model-draft`
|
||
AND every `--spec-draft-*` AND `--spec-type` together — the flags move as a set.
|
||
|
||
## 6. Before deleting a retired payload, read the rollback tag's env file
|
||
|
||
A rollback may need a small companion dir: a drafter living in a directory
|
||
named after an old quant looks like trash but is load-bearing. Same class as
|
||
the gufo payload removal of 2026-10-03, which turned that rollback into a
|
||
re-download.
|
||
|
||
## Related
|
||
|
||
- which fork ships which custom types
|
||
- the enum tables and code-level differences
|
||
* the era that served the ftype-115 file, and
|
||
[[frank-dragoon]] for what runs there now
|
||
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|