serverAppConfigs/serverAppConfigs-docs/LLM/Quantization Production.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
howto	Quantization Production	Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file.	llm quantization	2026-10-19

Quantization Production — making our own quants, per card and runtime

Research page (2026-10-08). Question: how do WE produce quantizations for the RTX 3090, the R9700 (gfx1201), and Strix Halo (gfx1151); what is native to each runtime; and the state of the named prior-art families (REAP, Heretic, abliteration, HauhauCS K_P "Perfect"). Marked facts: [repo-verified] = evidenced by a file in this repo; [verified] = checked first-hand against the cited primary source today; the rest are survey claims to re-verify before acting.

Runtime-native quant matrix

Card	Runtime	Container	Native weight encodings	Production tooling we own
RTX 3090 (sm_86)	llama.cpp CUDA	GGUF	K-quants (Q4_K_M…Q8_0), I-quants (IQ1_S…IQ4_XS), ML, f16/bf16; any tensor type per-tensor	llama-quantize + ggml-imatrix (built by scripts/build-llama-rdna4.sh targets) [repo-verified]
R9700 (gfx1201)	radiance (StillDeadcode)	.rad	per-tensor encodings from a recipe: GPTQ w4 (w4nl64a8h), bf16, int8, fp8_e4m3, u2, MXFP4/MXFP6 (vLLM-radiance era)	full pipeline: rad-convert + calibration tap + derive-protected-experts.py + kld-gate-swift15.sh [repo-verified] — CPU host-backend, the card is irrelevant to production
R9700 (gfx1201)	vLLM (tcclaviger/R4D)	safetensors	native block-FP8, MXFP4/DSFP4, GPTQ/AWQ, compressed-tensors	GPTQModel/AutoRound are hardware-agnostic torch (CPU-scale); AMD decode kernels are the gate [survey]
Strix Halo (gfx1151)	halogen-flash-server (peonist, closed)	.hgn	vendor-defined (w4b era, v2 checkpoint+ngram+vision files)	NONE public — no .hgn quantizer exists; every .hgn on the fleet was made by peonist/Quat3rnion [repo-verified gap]
Strix Halo (gfx1151)	llama.cpp / gufo / Kairic / ROCmFPX forks	GGUF	K/I quants (gufo UD-* = imatrix mixed profiles), custom types: Q4_0_ROCMFP4_STRIX, Q8_0_ROCMFPX (type 103), Kairic IU4	we built the RUNTIMES (scripts/build-kairic-edge-gfx1151.sh, ROCmFPX pin 36e9acd), never the custom-type quantizers — those files came from agentionai/kingjones [repo-verified]

• The lock is asymmetric: llama.cpp on the 3090 is a CHOICE. radiance demands .rad, halogen demands .hgn (the v2 checkpoint loads in no other engine); sm_86 decodes GGUF, GPTQ/AWQ-Marlin, INT8, weight-only FP8, and EXL3 natively — see the 3090 menu under Per-card playbooks.

Laws that steer everything here:

• Production is hardware-agnostic; decode is not. GPTQ/imatrix/ablation surgery runs in torch/CPU anywhere. The card only picks which ENCODED FORM it can DECODE — hence the one-table-one-encoding law, the GGUF custom-type histogram law (GGUF Compatibility), and the fact that our w4 convert ran 5 h 50 m on the host backend while the GPUs served nobody (scripts/convert-swift15-w4dn8.sh) [repo-verified].
• A custom tensor type locks the file to its fork (ROCmFPX type 103 precedent). K_P quants deliberately avoid that trap (below).

The named families

REAP — Router-weighted Expert Activation Pruning (Cerebras, 2025)

• arXiv 2510.13999; github.com/CerebrasResearch/reap [verified].
• One-shot MoE surgery: score every expert by router gate-values × activation energy, prune the low-utility tail, renumber+export a SMALLER MoE checkpoint (safetensors; GGUF by conversion). Published: GLM-4.5-Air −25 % (82B→~A12B class), GLM-4.7-REAP-268B-A32B-FP8; near-lossless at 25–50 % compression on generative benchmarks, and pruning BEATS expert-merging there [verified from paper/README].
• Fleet relevance: it is the inverse of our protected-experts law. derive-protected-experts.py ranks experts by calibration-gram energy share and keeps the hottest 24/layer at bf16; REAP's saliency is the same statistic and deletes the coldest. REAP the Flash-Next MoE (all 49,152 experts w4, 512/layer) by 25–50 % → resident set ~58–87 GiB → the expert pool fits with room for KV/cache, killing the disk tier. A REAPed checkpoint re-enters our normal pipeline: stage → calibrate → rad-convert → KLD gate. Hardware-agnostic; runs on the host.

Heretic (p-e-w/heretic) — automated abliteration + layer pruning

• github.com/p-e-w/heretic, AGPLv3 [verified]. NOT a quantizer. Fully-automatic censorship removal: directional ablation (refusal direction from PCA of harmful/harmless residuals), per-layer ablation weights, optional LAYER DROPPING (model shrinks), auto-tuned by co-minimizing refusals and KL divergence from the original. Runs on our 3090 (torch, ~24–48 GB working set for 9–27B classes; long batch probes) [verified README; runtime cost survey].
• Output is a normal HF checkpoint (fp16/bf16) → feeds GGUF, safetensors (vLLM), and .rad conversion. Order law: surgery → calibrate → quantize; calibrate GPTQ/imatrix on the SURGICAL output, and keep refusal-shaped tokens out of the calibration corpus.
• Fleet note: our abliterated bases (Quat3rnion v2, HauhauCS Aggressive/Balanced, ukisai Swift1.5) are this family's output; the Quat3rnion repo lore "the abliteration workflow appends, compaction drops dead padding" is exactly Heretic-style layer-editing leaving dead blocks (89.8 → 66.7 GiB r2) [repo-verified lore].

Ablation / abliteration (directional ablation)

• Arditi et al. 2024, arXiv 2406.11717: refusal is mediated by a single direction; orthogonalizing weights against it removes refusals with a rank-1 edit. Open implementations: p-e-w/heretic (automated, above) Artio/abliteration-toolkits, EleutherAI/repeng-style rep-overlap variants [verified (tool existence: heretic; rest survey)].
• Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm weights, so it composes with any expert quantization.
• Fleet status: frank-dragoon runs an abliterated base by Frank's choice (AGENTS: censored-base refusals were the accepted trade of the 27B era) frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.

HauhauCS QX_K_P "Perfect" quants

• Our frank-zealot publisher. Method, from the publisher's own cards [verified]: (1) abliterate + finetune ("Aggressive"/"Balanced") (2) quantize with imatrix calibration on the ABLITERATED weights, (3) K_P = a per-model, per-tensor mixed-precision profile — "model-specific analysis to selectively preserve quality where it matters most", e.g. Q8_K_P = a census of Q8_K + F32 + F16 tensors (FriendliAI's safetensor census of a Q8_K_P shows exactly that 399/353/99 tensor split). Result: +1–2 quality rungs for +5–15 % size. It is a standard GGUF — llama-quantize with per-tensor type overrides (--tensor-type), imatrix on; no fork, no custom types (LM Studio just fails to NAME the mixed file, shows "?").
• Reproducible with stock tooling: ggml-imatrix on the target model + llama-quantize --tensor-type <pattern>=<type> per the model's sensitivity census. Both binaries already build from scripts/build-llama-rdna4.sh [repo-verified].
• Our own ladder precedent: the fleet's UD-Q8_K_XL (unsloth "Dynamic" = same family: imatrix + importance-tiered per-tensor types) and agentionai FAST imatrix ROCmFP4-FAST (we ran the imatrix step ourselves on the v2 joined-PLE file — the one self-made quant in repo history) [repo-verified].

Per-card playbooks

RTX 3090 — the unlocked card (frank-zealot runs llama.cpp BY CHOICE)

The 3090 is the only fleet device with no format lock: radiance needs .rad, halogen needs .hgn, and the v2 checkpoint loads in NO other engine. sm_86 + CUDA is the industry's default target, so the runtime menu is open and every self-produced artifact format has a native decoder here. This makes the 3090 the natural quant R&D bench: produce on the host, DECODE AND SCORE on the 3090 against the fleet's own KLD/perplexity gates.

Runtime menu on sm_86 (verified against each project's support matrix):

Runtime	Native quant on Ampere	Notes
llama.cpp CUDA (deployed)	K/I/ML quants, f16	imatrix + --tensor-type K_P ladder
vLLM CUDA	GPTQ INT4 + AWQ via Marlin (sm_80/86 native), INT8 W8A8 SmoothQuant, FP8 as W8A16 weight-only via Marlin (FP8 W8A8 compute is Ada/Hopper-only)	full OpenAI server, batching
SGLang	GPTQ/AWQ (vLLM kernel lineage); Ampere FP8-via-Marlin tracked upstream	CUDA-first engine
ExLlamaV3	EXL3 trellis (QTIP) 2–8 bpw; runs on Ampere incl. the 3090	the quantizer (convert.py) runs on this same card
TensorRT-LLM	INT4 AWQ/GPTQ (FP16 acts), INT8 SmoothQuant; FP8 and INT4-AWQ-with-FP8-acts need SM≥89, NVFP4 is Blackwell	engine-build workflow

Hardware ceilings that no software unlocks on this card: no FP8 matmul hardware (sm_86 < 89), no FP4 (Blackwell). Ampere's fast lanes are INT4 (Marlin/AWQ/GPTQ), INT8 (SmoothQuant), weight-only FP8, f16.

Playbook A — GGUF ladder (status quo, llama.cpp kept):

1. Surgery on host/3090: abliterate/prune (Heretic) — torch.
2. ggml-imatrix (CUDA build) over a calibration corpus (our /home/dank/corpora style set; ≥1M tokens; the fleet's KLD corpus is a reasonable seed).
3. llama-quantize F16 → target rung; K_P-style --tensor-type overrides for attention/KV/first-layer upgrades; --use-repack for IQ rungs.
4. Validate: perplexity + KLD vs BF16 + the K-quant rank checks.

Playbook B — safetensors ladder (the unlocked option): GPTQModel / AutoRound / AWQ run CPU-side, emit GPTQ/AWQ/FP8 checkpoints that vLLM or SGLang serves on the 3090 with Marlin kernels — same KLD-gate discipline as the radiance pipeline. A REAPed + abliterated Flash-Next cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new hardware; EXL3 (convert.py on the 3090, exllama serve) is the quality-per-bit extreme of the same menu. Caveats before any swap: the serving contract (OpenAI server, reasoning_content, vision mmproj, and the compaction-hub window the endpoint declares in its Current shape — frank-zealot) must be met by the challenger runtime, and the 24 GB card runs ONE server — benching follows the stop-and-mask law.

Worked example — the frank-archon w4dn8 recipe ported to the 3090

(audit 2026-10-08, target Qwen3.6-35B-A3B; sized at that date's hub shape — the window history is frank-zealot's Eras table) The methodology ports; the encoding must be translated. llama.cpp decodes no GPTQ — the recipe re-expresses on GGUF-native terms: imatrix on our corpora (replaces the radiance grams; runs the 65 GiB BF16 on host CPU), IQ4-class experts, hot-expert protection as --tensor-type "=f16" patterns over the derive-the-hot-experts law (statistic: imatrix importances), Q6_K trunk, KLD/perplexity gate at the deployed slot shape. Production stays host-side; the 3090 is only the decode target. KV is cheap on this arch: 10 full layers × 2 KV heads × 256 = 20 KiB/token f16 (3.8 GiB per 200k slot); DeltaNet state is per-slot, tens of MiB. Size verdict (weights ≈ 32.2 B experts + 3 B trunk):

• Archon-faithful (w4 experts + bf16 protected + i8 trunk) ≈ 22 GiB → 27 GiB with KV+buffers → over 24; that shape is 2×3090-class.
• Fits 1×200,704 text-only: IQ4_XS experts (17.4) + Q6_K trunk + q8_0 KV (1.9; GGML_CUDA_FA_ALL_QUANTS=ON build) ≈ 21.5 ✓
• Compaction-hub shape (2×200,704 + mmproj): experts at IQ3_M (14.4) ≈ 24.0 — the exact Swift IQ3_XXS 2-slot edge, MTP precedent exists.
• Literal GPTQ-w4 encoding decodes only via vLLM/SGLang Marlin (Playbook B); compressed-tensors expresses the protected set per-module. Shelf check: HauhauCS publishes this model's K_P ladder (IQ2_M 10.86, IQ4_XS 17.44, Q4_K_M 19.71, Q4_K_P 21.82, Q5_K_P 26.1, Q6_K_P 28.5, Q8_K_P 40.6 GiB + mmproj 0.84) — Q4_K_P IS the protected-tensor blend. The self-made delta: calibration on OUR corpora, our hot-expert law, and a KLD gate scored at the serving shape.

R9700 — frank-archon (radiance)

The full self-quant pipeline already exists end to end: fetch BF16 → record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap, RADIANCE_CALIB_DIR/TOKENS) → derive protected experts (energy-share) → convert-swift15-w4dn8.sh (1.0.13, host backend) → KLD gate (MNBT 2048 law, pool 40960, mean KL ≤ 0.03, top-1 ≥ 95 %). New model = new recipe

• new .so arch plugin; every step is scripted. Prior art to fold in: REAP the expert set BEFORE step 3; ablate BEFORE calibration. Native encodings beyond w4: i8 trunk, fp8 HC/PLE/KV, bf16 protected, u2 draft — all recipe-expressible (see the w4dn8 runbook).

Recipe portability: the w4dn8 shape onto other MoEs (2026-10-08 audit)

The pin 1.0.13 container ships SIX arch plugins (/opt/radiance/share/radiance/architectures/): qwen4exp_fp8 qwen35moe_fp8, qwen35_fp8, qwen35_bf16, llama_fp8 llama_dense_fp8 (census via podman run --rm --entrypoint sh <image> -c — the image has no ls/strings; grep the .so files directly, and the image lives in ROOTLESS storage as docker.io/stilldeadcode/radiance). grep -c receipts: qwen35moe_fp8.so carries the RADIANCE_CALIB tap (2) and gram export (2) plus MTP (4) and NO ngram (0) — i.e. the full calibration→GPTQ path exists for the qwen3.5-MoE family; GPTQ itself lives in rad-convert, not the plugin (0 hits in both .so).

Worked example — Qwen/Qwen3.6-35B-A3B (config: qwen3_5_moe_text 40 layers, full_attention_interval=4 DeltaNet hybrid, 256 experts top-8, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab 248320, multimodal wrapper with vision tower):

• Expert set ~32 B params → w4 ≈ 16 GiB; trunk (attn/DeltaNet/embed/ lm_head ~3 B) at i8 ≈ 3 GiB ⇒ ~20 GiB total — fully VRAM-resident on ONE R9700 (32 GiB card): no TP=2, no host pool, no disk tier, no --ngram-placement, no 47.7 GiB PLE copy, no HC term. The recipe loses three lines (PLE/HC/protected-count scales with 256 experts).
• Convert cost scales with the file: Flash-Next's 335 GiB took 5 h 50 m host-backend ⇒ ~70 GiB BF16 ≈ 1–1.5 h. KLD reference boot is trivial (BF16 ~65 GiB in host pool beside the pool KV needs).
• Unresolved before any real campaign: does qwen35moe decode the Qwen3_5MoeForConditionalGeneration vision wrapper (assume text-only until proven), and does its MTP cast to the fp8/u2 encodings like qwen4exp. The MNBT-2048 disk-tier law is MOOT for a resident model.

Strix Halo — frank-dragoon (halogen) and friends

• .hgn: closed production door. No public quantizer exists (verified absence across peonist-ai repos); the format's lock is the engine's lock. We consume vendor artifacts; the repo-side tooling is compaction/verification only.
• Self-quant work on this card therefore runs through GGUF runtimes: the same imatrix/llama-quantize flow as the 3090 (HIP/Vulkan decode of K/I quants, gufo UD-* precedent, --use-repack off on ROCm), or the custom-type forks (ROCmFP4 STRIX, Kairic IU4) whose PRODUCERS we have never been — the quantizers for types 101/103 ship in those forks' trees; building one is a fork build, not a flag.
• Native-FP4 note: RDNA4/Strix get hardware FP4 matmuls (hipBLASLt MXFP4/DSFP4), which is the engine-level path (radiance/vLLM), not a GGUF decode.

Sources

• REAP: arXiv 2510.13999, github.com/CerebrasResearch/reap, cerebras/GLM-4.5-Air-REAP-82B-A12B-FP8, cerebras/GLM-4.7-REAP-268B-A32B-FP8.
• Heretic: github.com/p-e-w/heretic (AGPLv3).
• Abliteration: arXiv 2406.11717 (Arditi et al.).
• K_P: HauhauCS model cards (e.g. HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive README "What are K_P quants?"), FriendliAI Q8_K_P tensor census, fleet payload tree (BF16/Q4_K_M/Q6_K/Q8_0 + mmproj ladder in HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive).
• EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA; Ampere decode incl. 3090 documented in v1.0 performance write-ups).
• Ampere capability edges: vLLM FP8 docs ("only Hopper and Ada … W8A8; Ampere supported for W8A16 weight-only utilizing Marlin") vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6) TensorRT-LLM Support Matrix (FP8 and INT4-AWQ-with-FP8-acts need SM≥89; NVFP4 Blackwell), SGLang issue #12887 (Ampere FP8-via-Marlin).
• Repo: scripts/convert-swift15-w4dn8.sh scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe scripts/kld-gate-swift15.sh, scripts/build-llama-rdna4.sh scripts/build-kairic-edge-gfx1151.sh; endpoints frank-archon, , ; method and GGUF Compatibility.