swift15: campaign decode probe (43.6k greedy, timings-based) + runbook gate results
This commit is contained in:
Frank
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""bench-probe.py — RUNS ON DANKSERVER. The campaign's decode probe: one
|
||||
/completion request, a prompt assembled from calib.jsonl to ~TARGET tokens,
|
||||
temperature 0, max_tokens 512. Reports the engine's own timings: decode rate,
|
||||
prefill rate, draft accept. Run 3x; the median is the arm's number."""
|
||||
import json, sys, urllib.request
|
||||
|
||||
TARGET = 43600 # tokens
|
||||
BYTES_PER_TOK = 3.8
|
||||
URL = "http://localhost:7770/completion"
|
||||
|
||||
def build_prompt():
|
||||
want = int(TARGET * BYTES_PER_TOK)
|
||||
parts = []
|
||||
n = 0
|
||||
with open("/home/dank/corpora/out/calib.jsonl") as f:
|
||||
for line in f:
|
||||
d = json.loads(line)
|
||||
p = d.get("prompt") or d.get("messages", [{}])[0].get("content", "")
|
||||
parts.append(p)
|
||||
n += len(p.encode())
|
||||
if n >= want:
|
||||
break
|
||||
return "".join(parts)
|
||||
|
||||
def probe(salt):
|
||||
prompt = f"[probe {salt}]\n" + build_prompt()
|
||||
body = json.dumps({
|
||||
"prompt": prompt, "temperature": 0.0, "max_tokens": 512,
|
||||
"stream": False,
|
||||
}).encode()
|
||||
req = urllib.request.Request(URL, data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=1800) as r:
|
||||
d = json.load(r)
|
||||
t = d.get("timings", {})
|
||||
return {
|
||||
"prompt_n": t.get("prompt_n"),
|
||||
"prefill_tps": round(t.get("prompt_per_second", 0), 1),
|
||||
"pred_n": t.get("predicted_n"),
|
||||
"decode_tps": round(t.get("predicted_per_second", 0), 2),
|
||||
"draft_n": t.get("draft_n"),
|
||||
"accept": (round(t["draft_n_accepted"] / t["draft_n"], 4)
|
||||
if t.get("draft_n") else None),
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps(probe(sys.argv[1] if len(sys.argv) > 1 else "a")))
|
||||
@@ -61,29 +61,39 @@ policy; 173 GiB, purge when the window closes).
|
||||
and stops the serve at the token threshold. TP=2: whole experts per
|
||||
rank (256/rank); gate_up input is rank-sliced (plugin warning — the
|
||||
recipe only casts the down projection).
|
||||
5. **Protected-expert derivation**:
|
||||
`derive-protected-experts.py /home/dank/corpora/calib` → the 10
|
||||
highest-energy experts (5 per rank half) promoted to bf16. Protection
|
||||
law: energy share ≥ 3 % AND ≥ 3× routing share; even count per rank
|
||||
half; odd half gains the next most energetic.
|
||||
5. **Protected-expert derivation**: `derive-protected-experts.py
|
||||
/home/dank/corpora/calib` → the 24 highest-energy experts (the law
|
||||
is data-driven; Swift drew 24, not the fp8-era 10 — its routing is
|
||||
spikier: `blk.47 e445` alone carries 32 % of its layer's down energy
|
||||
at 70× routing share) promoted to whole-expert bf16.
|
||||
Protection law: energy share ≥ 3 % AND ≥ 3× routing share; even count
|
||||
per rank half; odd half gains the next most energetic.
|
||||
`apply-protected-cast.sh` splices the derived names between the
|
||||
`# PROTECTED-BEGIN/END` markers of the recipe (refuses double-derive).
|
||||
6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, ~2 h, CPU-only)
|
||||
— gates: ≥48 gram files present, bf16 container already deleted
|
||||
(disk), ≥146 GiB free, output ≥126,000 MiB, `rad-info` shows the
|
||||
gptq receipt. No `--reuse`.
|
||||
6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, host-backend
|
||||
CPU-only — no GPU visible to `rad-convert`; GPTQ solve is CPU BLAS) —
|
||||
gates: ≥48 gram files present, bf16 container already deleted
|
||||
(disk), ≥146 GiB free, output ≥108,000 MiB, `rad-info` shows the
|
||||
gptq receipt. No `--reuse`. Measured: 5 h 50 m (6.0–6.5 min/layer on
|
||||
49 GPTQ layers, load-average 8), output **116,416 MiB**.
|
||||
The plugin's `rad_arch_declare` enforces ONE ENCODING PER EXPERT
|
||||
TABLE (`one table of experts takes one encoding, and only a whole
|
||||
expert kept plain bf16 is served beside it`) — a split tier
|
||||
(down=fp8, gate_up=w4) is unservable: the kernel serves a single
|
||||
w4nl64a8h table; both expert tables ride GPTQ w4 (down calibrated,
|
||||
gate_up uncalibrated), protected experts bf16 beside them.
|
||||
|
||||
## The recipe tiers (scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe)
|
||||
|
||||
| Tier | Encoding | Why |
|
||||
|---|---|---|
|
||||
| MoE experts `ffn_down` | **gptq w4n8 (W4Dn8)** | the campaign tier: GPTQ on Hessians from the 10M-token corpus |
|
||||
| MoE `gate_up`, attention, shared expert | w4n8 | matched quantization; KLD-gated |
|
||||
| 10 derived experts | **bf16 protected** | energy-share law (step 5) |
|
||||
| trunk (embeddings, norms, conv) | **int8 (i8)** | trunk precision law |
|
||||
| hyper-cond + MTP head | **fp8** | hc+mtp fp8 law |
|
||||
| lm_head | **int8 (i8)** | i8 lm_head law |
|
||||
| MoE experts, both `gate_up` and `down` (one w4 table/layer) | **GPTQ w4nl64a8h** — u4 codes, fwht128 rotation, fp8 scales/64 + f32 scale plane | the campaign tier: GPTQ on down-input Hessians from the 10M-token corpus (the tap rides the down projection; gate_up shares the table uncalibrated — one-encoding-per-table plugin law) |
|
||||
| 24 derived protected experts (whole) | **bf16** | energy-share law (step 5) |
|
||||
| shared expert + trunk attention/SSM linears | **int8**, scale per 128 cols, searched | trunk precision law |
|
||||
| hyper-cond + MTP head | **fp8_e4m3 row-groups** | hc+mtp fp8 law |
|
||||
| lm_head | **int8** | i8 lm_head law (E4M3 costs 0.6 top-1 alone) |
|
||||
| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
|
||||
| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |
|
||||
|
||||
## The shape (`scripts/frank-primary.env`)
|
||||
|
||||
@@ -92,21 +102,32 @@ Unchanged from the FP8-native era: TP=2 + `wht6`, `--max-num-seqs 3`,
|
||||
collapses decode to 2.6–7.7 t/s), fp8 KV, `expert_tiered` host pool
|
||||
40960 (the load-bearing floor), `--ngram-placement ram`, froggeric
|
||||
v22.5 template override. `MODEL_FILE` points at
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416
|
||||
MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
|
||||
177000 → **110000**.
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8.rad`; the serve size floor
|
||||
`FRANK_PRIMARY_RAD_MIN_MIB` moves 177000 → (real output size, set at
|
||||
cutover from `rad-info`, MEASURED@cutover).
|
||||
|
||||
## Gates (measured at cutover)
|
||||
|
||||
- **KLD gate** (`kld-gate-swift15.sh`, candidate on 1.0.13 vs
|
||||
`kld-ref`): mean KL ≤ 0.03, top-1 ≥ 95 %. Fail → escalate gate_up
|
||||
w4→w5nl (pre-authorized), re-convert, re-gate.
|
||||
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
|
||||
candidate on 1.0.13 vs `kld-ref`: **mean KL 0.0146** (median 0.00126,
|
||||
p90 0.0315, p99 0.194, max 4.12), **top-1 97.84 %**,
|
||||
**ppl ratio 1.0126** (candidate 1.6913 vs reference 1.6702), 50,235
|
||||
positions / 197 docs. Report: `/home/dank/radiance-state/kld-gate.json`
|
||||
(+ `.rows` per position). Gate shape: disk-tier, pool 40960, MNBT 3072
|
||||
(the w4 activation arena scales hard with MNBT: 32768 → 33.3 GiB arena,
|
||||
refused), KV 2048, headroom 512, no speculation. The engine ran rank 0
|
||||
fully on card 0 (25.88 GiB) with rank 1 host-side in this shape —
|
||||
scoring is placement-invariant; scored at 28 positions/s.
|
||||
- **Throughput gate**: decode ≥ 30.8 t/s at the 43.6k greedy probe
|
||||
(control: 1.0.13 + fp8-native drew 30.08 / accept 0.7549). Fail →
|
||||
fall back to the w4 down tier (Q35).
|
||||
- MEASURED@cutover: decode t/s, acceptance, prefill, boot ledger, KV pool,
|
||||
KLD numbers.
|
||||
|
||||
|
||||
## Deploy + verify
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user