37 lines
5.2 KiB
Markdown
37 lines
5.2 KiB
Markdown
---
|
||
name: fork-locked-gguf-endpoint-trial
|
||
description: "Trial a fork-locked GGUF (custom tensor types: ternary/PQ2_0/PTQ1_0, vendor Hadamard metadata) as a replacement payload on a fleet llama.cpp endpoint: identify the fork, close provenance of local variants, build the right binary, and run one trap-guarded census window that measures VRAM, per-slot window, tool/vision/reasoning channels, and long-context stability."
|
||
---
|
||
|
||
# Fork-locked GGUF trial on a live endpoint
|
||
|
||
Use when a candidate GGUF carries **non-stock tensor types** (custom type ids, vendor metadata such as `prism.hadamard.*`) so mainline llama.cpp refuses it or loads a sibling `F16` file silently with garbage. Pairs with `gguf-header-vram-fit-analysis` (paper pricing); this skill is the *measure-it* half.
|
||
|
||
## 1. Header census decides everything downstream
|
||
Range-parse `resolve/main/<file>.gguf` bytes 0–48 MiB (metadata is ~11 MiB at 250k vocab; GGUF v3 strings are **u64**-length). Extract: `general.architecture`, `block_count`, `full_attention_interval`, `attention.head_count_kv`, `key_length`/`value_length`, `context_length` (target window > native ⇒ YaRN), `<arch>.nextn_predict_layers` (absent ⇒ no drafter, ever), tensor-type histogram, and per-tensor byte sizes from consecutive `offset` deltas ⇒ exact bpw.
|
||
- `KV = full_layers × head_count_kv × key_length` elements/token **per cache**; only full-attention layers bill per token, linear/DeltaNet layers bill a fixed state per slot.
|
||
- Ternary packs shrink weights only. At 262,144 tokens KV (16 full layers, 4×256) = 16.0 GiB f16 / 8.5 GiB q8_0 / 4.5 GiB q4_0 — 3× the 6.71 GiB of weights. **The cache is the wall, never the pack.**
|
||
|
||
## 2. Fork identification and tag ancestry
|
||
- Map custom type id → name from the vendor fork (`ggml/include/ggml.h` on its branch, e.g. `GGML_TYPE_PQ2_0 = 142`, `PTQ1_0 = 143`).
|
||
- `GET /repos/<fork>/compare/<tag>...<branch>` → `ahead_by` + commit subjects tells whether the newest **release** contains a fix, without reading history. Release workflows lie by omission: grep the workflow for the flag you need (e.g. prebuilt CUDA lacked `GGML_CUDA_FA_ALL_QUANTS` → quantized/mixed KV silently runs flash attention on CPU, ~2× slower; vendor issue #267).
|
||
|
||
## 3. Close provenance of payload variants already on disk
|
||
`sha256sum` the local file (7.2 GB ≈ 15 s NVMe), then search the Hub (`/api/models?search=<name>`, `/api/models/<repo>?blobs=true` → sibling `lfs.sha256` + size). Equal size + different sha = re-weighted variant, not a corrupt copy. Known houses: `dealignai/*-CRACK-*` (uncensored), `ManniX-ITA/*-MTP-GGUF` (appended MTP head ≈ +450 MB, restores a drafter the official file lacks). Same size/geometry ⇒ the VRAM verdict carries across variants; behaviour (refusals, tool calls) does not.
|
||
|
||
## 4. Build recipe (Dankclaw, RTX 3090)
|
||
`/usr/local/cuda` does not exist for non-interactive shells; `/usr/bin/nvcc` is a Debian wrapper. Mirror the known-good cache instead of guessing:
|
||
```
|
||
cmake -G Ninja -B build-cuda-prism -DCMAKE_BUILD_TYPE=Release \
|
||
-DCMAKE_CUDA_COMPILER=/usr/bin/nvcc -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
|
||
-DCMAKE_CUDA_ARCHITECTURES=86-real -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||
```
|
||
nvcc 12.4 refuses gcc 15 (`gcc-13`). FA template instances dominate the build (~150/804 in 12 min); park it with `pkill -f "[r]un-<build>"` + `pkill -x ninja` before timing anything — the bracket stops `pkill -f` matching your own ssh cmdline.
|
||
|
||
## 5. One-window census (never two outages)
|
||
Fetch through a committed script (`-fsSL --fail-with-body`, byte count + LFS sha, no resume, no token in argv), then stage `/tmp` throwaways: candidate env fed to the **repo serve script** via its `*_ENV_FILE` override (never edit remote config), and a runner whose `trap restore EXIT` always kills the candidate by pid and `systemctl start` the production unit + health-polls to 200.
|
||
Order inside the window: copy the production log aside (serve scripts truncate it) → stop unit → wait `memory.used < 1500` → boot candidate → `/health`, boot ledger (`KV buffer`, `compute buffer`, `flash`), `nvidia-smi --query-compute-apps`, `/v1/models`, `/props`, `/slots` (per-slot `n_ctx` is the harness contract, `pool/parallel`), chat probe (does thinking land in `reasoning_content` or `content` — a vendor `--reasoning-in-content true` default breaks the harness channel), tool-call probe (name + valid-JSON `arguments`), 512 px solid-colour vision probe, then a ~190k-token `/completion` fill reading `timings`. Kill the server, then `llama-bench` dtype arms (`-c <pool> -np <slots> -ctk/-ctv -b 2048 -ub 2048 -p 4096 -n 128 -r 1`) under a 1 Hz `nvidia-smi` sampler, `peak_gpu_mib=$(sort -n | tail -1)` per arm. Skip `-mmp 1` unless `-mmproj` is also passed.
|
||
|
||
## 6. Gates that outvote the memory table
|
||
Vendor `KNOWN_ISSUES.md` + open issues (quantized-KV resets near 200k, inert ngram/`--cache-reuse` on hybrid memory, malformed `tool_calls`, prompt-cache loss from re-rendered reasoning); base alignment (an uncensored endpoint replaced by a censored base is a fleet-wide capability loss); and which harness roles point at the endpoint (`models.yml` + `modelRoles`) — tool-heavy roles fail loudly on a `tool_calls` defect. Repo seams pinning `-ctk/-ctv` literals must be re-keyed to read the env file, so a retune moves one file, not two.
|