diff --git a/serverAppConfigs-docs/LLM/Quantization Production.md b/serverAppConfigs-docs/LLM/Quantization Production.md index 6810280..9d301d4 100644 --- a/serverAppConfigs-docs/LLM/Quantization Production.md +++ b/serverAppConfigs-docs/LLM/Quantization Production.md @@ -18,6 +18,11 @@ survey claims to re-verify before acting. | Strix Halo (gfx1151) | halogen-flash-server (peonist, **closed**) | `.hgn` | vendor-defined (w4b era, v2 checkpoint+ngram+vision files) | **NONE public** — no `.hgn` quantizer exists; every `.hgn` on the fleet was made by peonist/Quat3rnion **[repo-verified gap]** | | Strix Halo (gfx1151) | llama.cpp / gufo / Kairic / ROCmFPX forks | GGUF | K/I quants (gufo UD-* = imatrix mixed profiles), custom types: `Q4_0_ROCMFP4_STRIX`, `Q8_0_ROCMFPX` (type 103), Kairic IU4 | we built the RUNTIMES (`scripts/build-kairic-edge-gfx1151.sh`, ROCmFPX pin `36e9acd`), never the custom-type quantizers — those files came from agentionai/kingjones **[repo-verified]** | +- **The lock is asymmetric: llama.cpp on the 3090 is a CHOICE.** radiance + demands `.rad`, halogen demands `.hgn` (the v2 checkpoint loads in no + other engine); sm_86 decodes GGUF, GPTQ/AWQ-Marlin, INT8, weight-only + FP8, and EXL3 natively — see the 3090 menu under Per-card playbooks. + Laws that steer everything here: - **Production is hardware-agnostic; decode is not.** GPTQ/imatrix/ablation surgery runs in torch/CPU anywhere. The card only picks which ENCODED @@ -101,7 +106,30 @@ Laws that steer everything here: ## Per-card playbooks -### RTX 3090 — frank-secondary (llama.cpp CUDA) +### RTX 3090 — the unlocked card (frank-secondary runs llama.cpp BY CHOICE) +The 3090 is the only fleet device with no format lock: radiance needs +`.rad`, halogen needs `.hgn`, and the v2 checkpoint loads in NO other +engine. sm_86 + CUDA is the industry's default target, so the runtime +menu is open and every self-produced artifact format has a native +decoder here. This makes the 3090 the natural **quant R&D bench**: +produce on the host, DECODE AND SCORE on the 3090 against the fleet's +own KLD/perplexity gates. + +Runtime menu on sm_86 (verified against each project's support matrix): + +| Runtime | Native quant on Ampere | Notes | +|---|---|---| +| llama.cpp CUDA (deployed) | K/I/ML quants, f16 | imatrix + `--tensor-type` K_P ladder | +| vLLM CUDA | GPTQ INT4 + AWQ via **Marlin** (sm_80/86 native), INT8 W8A8 SmoothQuant, **FP8 as W8A16 weight-only via Marlin** (FP8 W8A8 compute is Ada/Hopper-only) | full OpenAI server, batching | +| SGLang | GPTQ/AWQ (vLLM kernel lineage); Ampere FP8-via-Marlin tracked upstream | CUDA-first engine | +| ExLlamaV3 | **EXL3** trellis (QTIP) 2–8 bpw; runs on Ampere incl. the 3090 | the quantizer (`convert.py`) runs on this same card | +| TensorRT-LLM | INT4 AWQ/GPTQ (FP16 acts), INT8 SmoothQuant; FP8 and INT4-AWQ-with-FP8-acts need SM≥89, NVFP4 is Blackwell | engine-build workflow | + +Hardware ceilings that no software unlocks on this card: no FP8 matmul +hardware (sm_86 < 89), no FP4 (Blackwell). Ampere's fast lanes are +**INT4 (Marlin/AWQ/GPTQ), INT8 (SmoothQuant), weight-only FP8, f16**. + +Playbook A — GGUF ladder (status quo, llama.cpp kept): 1. Surgery on host/3090: abliterate/prune (Heretic) — torch. 2. `ggml-imatrix` (CUDA build) over a calibration corpus (our `/home/dank/corpora` style set; ≥1M tokens; the fleet's KLD corpus @@ -110,10 +138,18 @@ Laws that steer everything here: overrides for attention/KV/first-layer upgrades; `--use-repack` for IQ rungs. 4. Validate: perplexity + KLD vs BF16 + the K-quant rank checks. -Decode-side options we do NOT use: EXL3 (turboderp ExLlamaV3, QTIP -trellis; CUDA production on the 3090 works, decode is exllama-only — -new runtime, no OpenAI server), Marlin-GPTQ (vLLM path, not a llama.cpp -runtime). + +Playbook B — safetensors ladder (the unlocked option): GPTQModel / +AutoRound / AWQ run CPU-side, emit GPTQ/AWQ/FP8 checkpoints that vLLM +or SGLang serves on the 3090 with Marlin kernels — same KLD-gate +discipline as the radiance pipeline. A REAPed + abliterated Flash-Next +cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new +hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the +quality-per-bit extreme of the same menu. +Caveats before any swap: the serving contract (OpenAI server, +`reasoning_content`, vision mmproj, compaction-hub window +200,704 ×2 slots) must be met by the challenger runtime, and the +24 GB card runs ONE server — benching follows the stop-and-mask law. ### R9700 — frank-primary (radiance) The full self-quant pipeline already exists end to end: fetch BF16 → @@ -150,7 +186,13 @@ u2 draft — all recipe-expressible (see the w4dn8 runbook). README "What are K_P quants?"), FriendliAI Q8_K_P tensor census, fleet payload tree (BF16/Q4_K_M/Q6_K/Q8_0 + mmproj ladder in `HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive`). -- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA). +- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA; Ampere + decode incl. 3090 documented in v1.0 performance write-ups). +- Ampere capability edges: vLLM FP8 docs ("only Hopper and Ada … + W8A8; Ampere supported for W8A16 weight-only utilizing Marlin"), + vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6), + TensorRT-LLM Support Matrix (FP8 and INT4-AWQ-with-FP8-acts need + SM≥89; NVFP4 Blackwell), SGLang issue #12887 (Ampere FP8-via-Marlin). - Repo: `scripts/convert-swift15-w4dn8.sh`, `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,