command-center/agent/managed-skills/fork-locked-gguf-endpoint-trial/SKILL.md

name, description

name	description
fork-locked-gguf-endpoint-trial	Trial a fork-locked GGUF (custom tensor types: ternary/PQ2_0/PTQ1_0, vendor Hadamard metadata) as a replacement payload on a fleet llama.cpp endpoint: identify the fork, close provenance of local variants, build the right binary, and run one trap-guarded census window that measures VRAM, per-slot window, tool/vision/reasoning channels, and long-context stability.

Fork-locked GGUF trial on a live endpoint

Use when a candidate GGUF carries non-stock tensor types (custom type ids, vendor metadata such as prism.hadamard.*) so mainline llama.cpp refuses it or loads a sibling F16 file silently with garbage. Pairs with gguf-header-vram-fit-analysis (paper pricing); this skill is the measure-it half.

1. Header census decides everything downstream

Range-parse resolve/main/<file>.gguf bytes 0–48 MiB (metadata is ~11 MiB at 250k vocab; GGUF v3 strings are u64-length). Extract: general.architecture, block_count, full_attention_interval, attention.head_count_kv, key_length/value_length, context_length (target window > native ⇒ YaRN), <arch>.nextn_predict_layers (absent ⇒ no drafter, ever), tensor-type histogram, and per-tensor byte sizes from consecutive offset deltas ⇒ exact bpw.

• KV = full_layers × head_count_kv × key_length elements/token per cache; only full-attention layers bill per token, linear/DeltaNet layers bill a fixed state per slot.
• Ternary packs shrink weights only. At 262,144 tokens KV (16 full layers, 4×256) = 16.0 GiB f16 / 8.5 GiB q8_0 / 4.5 GiB q4_0 — 3× the 6.71 GiB of weights. The cache is the wall, never the pack.

2. Fork identification and tag ancestry

• Map custom type id → name from the vendor fork (ggml/include/ggml.h on its branch, e.g. GGML_TYPE_PQ2_0 = 142, PTQ1_0 = 143).
• GET /repos/<fork>/compare/<tag>...<branch> → ahead_by + commit subjects tells whether the newest release contains a fix, without reading history. Release workflows lie by omission: grep the workflow for the flag you need (e.g. prebuilt CUDA lacked GGML_CUDA_FA_ALL_QUANTS → quantized/mixed KV silently runs flash attention on CPU, ~2× slower; vendor issue #267).

3. Close provenance of payload variants already on disk

sha256sum the local file (7.2 GB ≈ 15 s NVMe), then search the Hub (/api/models?search=<name>, /api/models/<repo>?blobs=true → sibling lfs.sha256 + size). Equal size + different sha = re-weighted variant, not a corrupt copy. Known houses: dealignai/*-CRACK-* (uncensored), ManniX-ITA/*-MTP-GGUF (appended MTP head ≈ +450 MB, restores a drafter the official file lacks). Same size/geometry ⇒ the VRAM verdict carries across variants; behaviour (refusals, tool calls) does not.

4. Build recipe (Dankclaw, RTX 3090)

/usr/local/cuda does not exist for non-interactive shells; /usr/bin/nvcc is a Debian wrapper. Mirror the known-good cache instead of guessing:

cmake -G Ninja -B build-cuda-prism -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_COMPILER=/usr/bin/nvcc -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
  -DCMAKE_CUDA_ARCHITECTURES=86-real -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON

nvcc 12.4 refuses gcc 15 (gcc-13). FA template instances dominate the build (~150/804 in 12 min); park it with pkill -f "[r]un-<build>" + pkill -x ninja before timing anything — the bracket stops pkill -f matching your own ssh cmdline.

5. One-window census (never two outages)

Fetch through a committed script (-fsSL --fail-with-body, byte count + LFS sha, no resume, no token in argv), then stage /tmp throwaways: candidate env fed to the repo serve script via its *_ENV_FILE override (never edit remote config), and a runner whose trap restore EXIT always kills the candidate by pid and systemctl start the production unit + health-polls to 200. Order inside the window: copy the production log aside (serve scripts truncate it) → stop unit → wait memory.used < 1500 → boot candidate → /health, boot ledger (KV buffer, compute buffer, flash), nvidia-smi --query-compute-apps, /v1/models, /props, /slots (per-slot n_ctx is the harness contract, pool/parallel), chat probe (does thinking land in reasoning_content or content — a vendor --reasoning-in-content true default breaks the harness channel), tool-call probe (name + valid-JSON arguments), 512 px solid-colour vision probe, then a ~190k-token /completion fill reading timings. Kill the server, then llama-bench dtype arms (-c <pool> -np <slots> -ctk/-ctv -b 2048 -ub 2048 -p 4096 -n 128 -r 1) under a 1 Hz nvidia-smi sampler, peak_gpu_mib=$(sort -n | tail -1) per arm. Skip -mmp 1 unless -mmproj is also passed.

6. Gates that outvote the memory table

Vendor KNOWN_ISSUES.md + open issues (quantized-KV resets near 200k, inert ngram/--cache-reuse on hybrid memory, malformed tool_calls, prompt-cache loss from re-rendered reasoning); base alignment (an uncensored endpoint replaced by a censored base is a fleet-wide capability loss); and which harness roles point at the endpoint (models.yml + modelRoles) — tool-heavy roles fail loudly on a tool_calls defect. Repo seams pinning -ctk/-ctv literals must be re-keyed to read the env file, so a retune moves one file, not two.