73 lines
4.2 KiB
Markdown
73 lines
4.2 KiB
Markdown
# Swift1.5 w4dn8 Campaign — Work Log
|
||
|
||
Campaign branch `swift15-radiance-w4dn8`. Goal: convert
|
||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB) to
|
||
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (GPTQ gate_up w4nl64 + block-fp8 down +
|
||
10 protected bf16 experts + int8 trunk + fixed-scale PLE) and swap it into
|
||
frank-primary on the 1.0.13 pin. Design settled (25-question grill); this log
|
||
carries execution facts only. The cutover runbook
|
||
`frank-primary-radiance-swift15-w4dn8.md` replaces this page at cutover.
|
||
|
||
## 2026-10-06 evening
|
||
|
||
- Purge freed 835.8 GB; Swift1.5 BF16 downloaded 131/131 shards (sha256 ok).
|
||
- Corpora built: calib 10,014,932 tok / 1,294 prompts; KLD 1,800,816 tok /
|
||
197 prompts (`/home/dank/corpora/out/`).
|
||
- Engine verdict: 1.2.0 REJECTED for serving (decode 29.21 vs control 30.08,
|
||
accept 0.7123 vs 0.7549 deterministic; vram-pair translation ledger-exact).
|
||
**Serving pin stays 1.0.13** `sha256:4c3f48…c322e`.
|
||
- bf16 rad-convert done: `/home/dank/models/rad/qwen4exp-swift15-bf16.rad`
|
||
(335.28 GiB, 248,077 vocab tokens baked).
|
||
- **Blocker resolved — engine-version probe law:** the calibration tap lives
|
||
in the architecture plugin `/opt/radiance/share/radiance/architectures/
|
||
qwen4exp_fp8.so`, NOT the radiance binary, and the image has no
|
||
`strings(1)` (greps return 0 for everything — positive controls
|
||
mandatory). 1.0.13 plugin: `RADIANCE_CALIB_DIR` only; 1.2.0 adds
|
||
`RADIANCE_CALIB_TOKENS`. Both `libquant.so` carry the gptq quantizer
|
||
(`calib=/damp/act_order/cd`) and the identical `RADGRAM2` format
|
||
(magic + `gram.r%d.%s.bin` + calib_gram/calib_resolve/calib_stem).
|
||
**Decision: calibrate on 1.2.0, final convert on 1.0.13** — the serving
|
||
pin produces the served `.rad`; only the Gram files cross versions.
|
||
- `run-swift15-bf16.sh` boot-failure trail (all fixed, committed):
|
||
duplicated `radiance` entrypoint arg (rc=2); ratio-derived KV pool SHORT
|
||
217 MiB at 32×16k (explicit `--vram-kv-mib`); explicit 4608 starved the
|
||
disk tier — **weights slab = card − headroom − kv, disk-tier floor
|
||
9.57 GiB** (4.21 layer buffers + 5.36 fixed) → kv 3072; `KLD_MOUNT`
|
||
referenced-never-defined emptied the corpus mount.
|
||
- KLD record launched on 1.0.13 (disk tier, kv 3072, 4×16k, MNBT 32768):
|
||
measured ~116 tok/s at the 32k step (44 docs/58 min).
|
||
- Chain driver `kld-then-calib.sh` (pid 2848206, log
|
||
`/home/dank/logs/swift15-chain.log`): KLD rc=0 → verify kld-ref → boot
|
||
1.2.0 calib (MNBT 65536/seqs 8 per MOE-W4 930 tok/s law, synced commit
|
||
45de2fd) → `calib-push.py` parallel 16 → kill → prod restored by trap.
|
||
- Ready on the branch: `convert-swift15-w4dn8.sh` (1.0.13 convert, gates:
|
||
≥126,000 MiB), `apply-protected-cast.sh` (derive → recipe PROTECTED
|
||
section, refuses double-derive), `kld-gate-swift15.sh` (candidate vs
|
||
reference on 1.0.13, gate mean KL ≤ 0.03 / top-1 ≥ 95 %).
|
||
- `frank-primary` showing `failed` during the campaign is the 143-on-stop
|
||
signature of the foreground podman unit — cosmetic, not a crash.
|
||
|
||
## 2026-10-07 morning
|
||
|
||
- **KLD reference COMPLETE** on 1.0.13: `/home/dank/corpora/kld-ref` 24 GiB,
|
||
`radiance-kld-1`, `rows: 50235`, 197 docs, top-16 log-probs (f16),
|
||
corpus fnv1a64 `a9d62b33c18b3d9c`. Record took ~4 h at the 32k step.
|
||
The wrapper died rc=127 AFTER the engine exited clean: the mid-run scp of
|
||
commit 45de2fd shifted bash's lazy read offset — script-swap law, third
|
||
fleet occurrence. Resume path added: `SKIP_KLD=1` + `rows: 50235` receipt
|
||
gate (cf6c62d). **Never scp a script a running process owns.**
|
||
- Calibration chain3 live on 1.2.0 (big-step shape MNBT 65536/seqs 8): push
|
||
started, requests landing at 7–8k tokens.
|
||
|
||
## Campaign order after the chain
|
||
|
||
1. derive protected experts from Swift's own grams; apply to recipe.
|
||
2. Delete bf16 `.rad` (335 GiB) to `.AGENTTRASH` staging (Freed space gates the convert).
|
||
3. `convert-swift15-w4dn8.sh` (~2 h, CPU-only, prod can stay up).
|
||
4. rad-tune declare TP=2 spec=2; vram-pair boot arms; bench (gate decode ≥ 30.8 t/s).
|
||
5. `kld-gate-swift15.sh` (fail → escalate gate_up w4→w5nl, pre-authorized).
|
||
6. Cutover: docs, tests + e2e, deploy, probes. Throughput fallback: w4 down tier (Q35).
|
||
|
||
Disk ledger at KLD-record time: 104 GiB free → KLD ref −50 → Grams −20 →
|
||
bf16 delete +335 → convert +132 written.
|