docs(LLM): Quantization Production — 3090 as the unlocked card (runtime menu, Ampere ceilings, Playbook B safetensors ladder)

serverAppConfigs-docs/LLM/Quantization Production.md

@@ -18,6 +18,11 @@ survey claims to re-verify before acting.
 | Strix Halo (gfx1151) | halogen-flash-server (peonist, **closed**) | `.hgn` | vendor-defined (w4b era, v2 checkpoint+ngram+vision files) | **NONE public** — no `.hgn` quantizer exists; every `.hgn` on the fleet was made by peonist/Quat3rnion **[repo-verified gap]** |
 | Strix Halo (gfx1151) | llama.cpp / gufo / Kairic / ROCmFPX forks | GGUF | K/I quants (gufo UD-* = imatrix mixed profiles), custom types: `Q4_0_ROCMFP4_STRIX`, `Q8_0_ROCMFPX` (type 103), Kairic IU4 | we built the RUNTIMES (`scripts/build-kairic-edge-gfx1151.sh`, ROCmFPX pin `36e9acd`), never the custom-type quantizers — those files came from agentionai/kingjones **[repo-verified]** |
 
+- **The lock is asymmetric: llama.cpp on the 3090 is a CHOICE.** radiance
+  demands `.rad`, halogen demands `.hgn` (the v2 checkpoint loads in no
+  other engine); sm_86 decodes GGUF, GPTQ/AWQ-Marlin, INT8, weight-only
+  FP8, and EXL3 natively — see the 3090 menu under Per-card playbooks.
+
 Laws that steer everything here:
 - **Production is hardware-agnostic; decode is not.** GPTQ/imatrix/ablation
   surgery runs in torch/CPU anywhere. The card only picks which ENCODED
@@ -101,7 +106,30 @@ Laws that steer everything here:
 
 ## Per-card playbooks
 
-### RTX 3090 — frank-secondary (llama.cpp CUDA)
+### RTX 3090 — the unlocked card (frank-secondary runs llama.cpp BY CHOICE)
+The 3090 is the only fleet device with no format lock: radiance needs
+`.rad`, halogen needs `.hgn`, and the v2 checkpoint loads in NO other
+engine. sm_86 + CUDA is the industry's default target, so the runtime
+menu is open and every self-produced artifact format has a native
+decoder here. This makes the 3090 the natural **quant R&D bench**:
+produce on the host, DECODE AND SCORE on the 3090 against the fleet's
+own KLD/perplexity gates.
+
+Runtime menu on sm_86 (verified against each project's support matrix):
+
+| Runtime | Native quant on Ampere | Notes |
+|---|---|---|
+| llama.cpp CUDA (deployed) | K/I/ML quants, f16 | imatrix + `--tensor-type` K_P ladder |
+| vLLM CUDA | GPTQ INT4 + AWQ via **Marlin** (sm_80/86 native), INT8 W8A8 SmoothQuant, **FP8 as W8A16 weight-only via Marlin** (FP8 W8A8 compute is Ada/Hopper-only) | full OpenAI server, batching |
+| SGLang | GPTQ/AWQ (vLLM kernel lineage); Ampere FP8-via-Marlin tracked upstream | CUDA-first engine |
+| ExLlamaV3 | **EXL3** trellis (QTIP) 2–8 bpw; runs on Ampere incl. the 3090 | the quantizer (`convert.py`) runs on this same card |
+| TensorRT-LLM | INT4 AWQ/GPTQ (FP16 acts), INT8 SmoothQuant; FP8 and INT4-AWQ-with-FP8-acts need SM≥89, NVFP4 is Blackwell | engine-build workflow |
+
+Hardware ceilings that no software unlocks on this card: no FP8 matmul
+hardware (sm_86 < 89), no FP4 (Blackwell). Ampere's fast lanes are
+**INT4 (Marlin/AWQ/GPTQ), INT8 (SmoothQuant), weight-only FP8, f16**.
+
+Playbook A — GGUF ladder (status quo, llama.cpp kept):
 1. Surgery on host/3090: abliterate/prune (Heretic) — torch.
 2. `ggml-imatrix` (CUDA build) over a calibration corpus (our
    `/home/dank/corpora` style set; ≥1M tokens; the fleet's KLD corpus
@@ -110,10 +138,18 @@ Laws that steer everything here:
    overrides for attention/KV/first-layer upgrades; `--use-repack` for
    IQ rungs.
 4. Validate: perplexity + KLD vs BF16 + the K-quant rank checks.
-Decode-side options we do NOT use: EXL3 (turboderp ExLlamaV3, QTIP
-trellis; CUDA production on the 3090 works, decode is exllama-only —
-new runtime, no OpenAI server), Marlin-GPTQ (vLLM path, not a llama.cpp
-runtime).
+
+Playbook B — safetensors ladder (the unlocked option): GPTQModel /
+AutoRound / AWQ run CPU-side, emit GPTQ/AWQ/FP8 checkpoints that vLLM
+or SGLang serves on the 3090 with Marlin kernels — same KLD-gate
+discipline as the radiance pipeline. A REAPed + abliterated Flash-Next
+cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new
+hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the
+quality-per-bit extreme of the same menu.
+Caveats before any swap: the serving contract (OpenAI server,
+`reasoning_content`, vision mmproj, compaction-hub window
+200,704 ×2 slots) must be met by the challenger runtime, and the
+24 GB card runs ONE server — benching follows the stop-and-mask law.
 
 ### R9700 — frank-primary (radiance)
 The full self-quant pipeline already exists end to end: fetch BF16 →
@@ -150,7 +186,13 @@ u2 draft — all recipe-expressible (see the w4dn8 runbook).
   README "What are K_P quants?"), FriendliAI Q8_K_P tensor census,
   fleet payload tree (BF16/Q4_K_M/Q6_K/Q8_0 + mmproj ladder in
   `HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive`).
-- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA).
+- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA; Ampere
+  decode incl. 3090 documented in v1.0 performance write-ups).
+- Ampere capability edges: vLLM FP8 docs ("only Hopper and Ada …
+  W8A8; Ampere supported for W8A16 weight-only utilizing Marlin"),
+  vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6),
+  TensorRT-LLM Support Matrix (FP8 and INT4-AWQ-with-FP8-acts need
+  SM≥89; NVFP4 Blackwell), SGLang issue #12887 (Ampere FP8-via-Marlin).
 - Repo: `scripts/convert-swift15-w4dn8.sh`,
   `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
   `scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,