Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
125 lines
6.0 KiB
Markdown
125 lines
6.0 KiB
Markdown
---
|
||
type: reference
|
||
title: GGUF Compatibility and Runtime Lock-in
|
||
description: "Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can."
|
||
tags: [gguf, llama.cpp, forks, quantization, speculative]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# GGUF Compatibility and Runtime Lock-in
|
||
|
||
**The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the
|
||
header before promising a file to any loader.** Stock llama.cpp, Ollama and LM
|
||
Studio refuse custom types at load (`unknown tensor type`); a fork-private type
|
||
means exactly one engine can serve the file. Measured on this fleet repeatedly:
|
||
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
|
||
(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
|
||
tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
|
||
(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
|
||
|
||
## 1. Read the header, do not guess
|
||
|
||
Parse the GGUF KV section and histogram the tensor types plus
|
||
`general.architecture.file_type` — one python read over the first few MiB, no
|
||
model load. Counting types out of the KV section settles a runtime argument in
|
||
minutes; repo names and model cards do not.
|
||
|
||
## 2. Compare routing variants by tensor mix, not by name
|
||
|
||
Two files of the same quant can differ in how many tensors stay at plain Q8_0
|
||
(measured: 187 vs 2 output-head tensors in an `_AGENT` variant of the same
|
||
size). That difference is what moves spec-decode acceptance (0.953 vs 0.911)
|
||
— the "same size" file is not the same file.
|
||
|
||
## 3. Pin the build by commit SHA
|
||
|
||
A fork that can read the file today can still crash mid-load: the 2026-08-25
|
||
ROCmFPX builds read the ftype-115 file and then SIGSEGV in
|
||
`ggml_compute_forward_set_rows`. Record the SHA in the build script
|
||
(`scripts/build-rocmfpx.sh` pins `36e9acd`) and check it out explicitly — never
|
||
a branch.
|
||
|
||
## 4. Header diff method (candidate vs known-good)
|
||
|
||
Answer "will engine X load this GGUF?" by diffing the candidate's header
|
||
against a file the engine **already loads** — never by filenames or model cards;
|
||
filenames lie about quantization and about MTP/vision heads.
|
||
|
||
1. **Get the bytes**: both files, from byte 0, at least 64 MiB. Tokenizer KVs
|
||
(151k token strings + merges) routinely eat 5–8 MiB, so a small range
|
||
desynchronises the parser. True size from `curl -sIL` → `x-linked-size`
|
||
(a bare HEAD `Content-Length` can lie):
|
||
|
||
```sh
|
||
curl -sL -r 0-67108863 -o cand.bin \
|
||
"https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
|
||
ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
|
||
```
|
||
|
||
2. **Parse — and lock the parser against the known-good file first.** Layout
|
||
(v2/v3): `magic[4]` `version u32` `n_tensors u64` `n_kv u64`, then KVs, then
|
||
tensor infos. Strings are u64 length + bytes. **The value-type enum is NOT
|
||
llama.cpp's `gguf.h`:** in the files seen (2026-09-28, unsloth + ajgazin UD
|
||
GGUFs) type **8 = STRING, 9 = ARRAY**; llama.cpp's header has STRING=2,
|
||
ARRAY=3 and desynchronises immediately. Working table:
|
||
|
||
```
|
||
0 u8 1 i8 2 u16 3 i16 4 u32 5 i32 6 f32
|
||
7 bool 8 STR 9 ARRAY(u32 et, u64 n) 10 u64 11 i64 12 f64
|
||
```
|
||
|
||
Validate before trusting: the parse is right iff the tensor-info block ends
|
||
somewhere sane and names look real (`blk.0.attn_q.weight`,
|
||
`token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
|
||
offset — do not work around it.
|
||
|
||
3. **Diff what engines actually bind on**: `general.architecture`; tensor
|
||
count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
|
||
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
|
||
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
|
||
`.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
|
||
(`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
|
||
tool usually comes out byte-identical in every one of these — that is the
|
||
answer.
|
||
|
||
4. **Then read the engine's source before calling a delta a blocker.** A
|
||
metadata difference only matters if the engine reads that key
|
||
(`grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/`).
|
||
Worked example (gufo v0.1.1): `general.file_type` is deliberately not
|
||
consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and
|
||
`qwen35.attention.recurrent_layers` is never read (layer kinds come from
|
||
`full_attention_interval`) — both deltas were non-issues the header diff
|
||
alone would have flagged wrongly.
|
||
|
||
5. **Say what you did not do.** Header analysis is strong evidence, not a
|
||
load. State byte sizes, check free disk before proposing a fetch, and do
|
||
not download tens of GB unasked — pointing the engine at the file is the
|
||
definitive test.
|
||
|
||
**Report shape:** verdict first (loads / will not / loads-with-caveats), then
|
||
the side-by-side diff table, then only the load-bearing caveats: text-only vs
|
||
vision tensors, chat-template differences (matters for llama.cpp, not for
|
||
engines with compiled-in Jinja), license, disk headroom, and whether the base
|
||
model is abliterated/uncensored.
|
||
|
||
## 5. A/B a fork's spec decode by stripping the whole flag set
|
||
|
||
`--model-draft` without `--spec-type` segfaults at load (measured 2026-10-02,
|
||
core dump, no diagnostic). A baseline "no-draft" arm must remove `--model-draft`
|
||
AND every `--spec-draft-*` AND `--spec-type` together — the flags move as a set.
|
||
|
||
## 6. Before deleting a retired payload, read the rollback tag's env file
|
||
|
||
A rollback may need a small companion dir: a drafter living in a directory
|
||
named after an old quant looks like trash but is load-bearing. Same class as
|
||
the gufo payload removal of 2026-10-03, which turned that rollback into a
|
||
re-download.
|
||
|
||
## Related
|
||
|
||
- [[Llama Cpp Runtimes]] — which fork ships which custom types
|
||
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
|
||
* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and
|
||
[[frank-dragoon]] for what runs there now
|
||
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|