docs(LLM): 3090 worked example — archon w4dn8 methodology ports to GGUF-native quants; 24 GiB size law for 35B-A3B
This commit is contained in:
@@ -151,6 +151,34 @@ Caveats before any swap: the serving contract (OpenAI server,
|
||||
200,704 ×2 slots) must be met by the challenger runtime, and the
|
||||
24 GB card runs ONE server — benching follows the stop-and-mask law.
|
||||
|
||||
#### Worked example — the frank-archon w4dn8 recipe ported to the 3090
|
||||
(audit 2026-10-08, target `Qwen3.6-35B-A3B`)
|
||||
**The methodology ports; the encoding must be translated.** llama.cpp
|
||||
decodes no GPTQ — the recipe re-expresses on GGUF-native terms:
|
||||
imatrix on our corpora (replaces the radiance grams; runs the 65 GiB
|
||||
BF16 on host CPU), IQ4-class experts, hot-expert protection as
|
||||
`--tensor-type "=f16"` patterns over the derive-the-hot-experts law
|
||||
(statistic: imatrix importances), Q6_K trunk, KLD/perplexity gate at
|
||||
the deployed slot shape. Production stays host-side; the 3090 is only
|
||||
the decode target.
|
||||
KV is cheap on this arch: 10 full layers × 2 KV heads × 256 =
|
||||
**20 KiB/token f16** (3.8 GiB per 200k slot); DeltaNet state is
|
||||
per-slot, tens of MiB. Size verdict (weights ≈ 32.2 B experts + 3 B
|
||||
trunk):
|
||||
- Archon-faithful (w4 experts + bf16 protected + i8 trunk) ≈ 22 GiB →
|
||||
27 GiB with KV+buffers → **over 24; that shape is 2×3090-class.**
|
||||
- Fits 1×200,704 text-only: IQ4_XS experts (17.4) + Q6_K trunk + q8_0
|
||||
KV (1.9; `GGML_CUDA_FA_ALL_QUANTS=ON` build) ≈ 21.5 ✓
|
||||
- Compaction-hub shape (2×200,704 + mmproj): experts at IQ3_M (14.4) ≈
|
||||
24.0 — the exact Swift IQ3_XXS 2-slot edge, MTP precedent exists.
|
||||
- Literal GPTQ-w4 encoding decodes only via vLLM/SGLang Marlin (Playbook
|
||||
B); compressed-tensors expresses the protected set per-module.
|
||||
Shelf check: HauhauCS publishes this model's K_P ladder (IQ2_M 10.86,
|
||||
IQ4_XS 17.44, Q4_K_M 19.71, Q4_K_P 21.82, Q5_K_P 26.1, Q6_K_P 28.5,
|
||||
Q8_K_P 40.6 GiB + mmproj 0.84) — Q4_K_P IS the protected-tensor blend.
|
||||
The self-made delta: calibration on OUR corpora, our hot-expert law,
|
||||
and a KLD gate scored at the serving shape.
|
||||
|
||||
### R9700 — frank-primary (radiance)
|
||||
The full self-quant pipeline already exists end to end: fetch BF16 →
|
||||
record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap,
|
||||
|
||||
Reference in New Issue
Block a user