docs(campaign): interim work log for the Swift1.5 w4dn8 campaign
Execution facts, blocker trail (plugin-tap probe law, weights-slab formula, KLD_MOUNT), engine verdict, and the post-chain order. Linked from the docs index per the completeness seam; the cutover runbook replaces it.
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# Swift1.5 w4dn8 Campaign — Work Log
|
||||
|
||||
Campaign branch `swift15-radiance-w4dn8`. Goal: convert
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB) to
|
||||
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (GPTQ gate_up w4nl64 + block-fp8 down +
|
||||
10 protected bf16 experts + int8 trunk + fixed-scale PLE) and swap it into
|
||||
frank-primary on the 1.0.13 pin. Design settled (25-question grill); this log
|
||||
carries execution facts only. The cutover runbook
|
||||
`frank-primary-radiance-swift15-w4dn8.md` replaces this page at cutover.
|
||||
|
||||
## 2026-10-06 evening
|
||||
|
||||
- Purge freed 835.8 GB; Swift1.5 BF16 downloaded 131/131 shards (sha256 ok).
|
||||
- Corpora built: calib 10,014,932 tok / 1,294 prompts; KLD 1,800,816 tok /
|
||||
197 prompts (`/home/dank/corpora/out/`).
|
||||
- Engine verdict: 1.2.0 REJECTED for serving (decode 29.21 vs control 30.08,
|
||||
accept 0.7123 vs 0.7549 deterministic; vram-pair translation ledger-exact).
|
||||
**Serving pin stays 1.0.13** `sha256:4c3f48…c322e`.
|
||||
- bf16 rad-convert done: `/home/dank/models/rad/qwen4exp-swift15-bf16.rad`
|
||||
(335.28 GiB, 248,077 vocab tokens baked).
|
||||
- **Blocker resolved — engine-version probe law:** the calibration tap lives
|
||||
in the architecture plugin `/opt/radiance/share/radiance/architectures/
|
||||
qwen4exp_fp8.so`, NOT the radiance binary, and the image has no
|
||||
`strings(1)` (greps return 0 for everything — positive controls
|
||||
mandatory). 1.0.13 plugin: `RADIANCE_CALIB_DIR` only; 1.2.0 adds
|
||||
`RADIANCE_CALIB_TOKENS`. Both `libquant.so` carry the gptq quantizer
|
||||
(`calib=/damp/act_order/cd`) and the identical `RADGRAM2` format
|
||||
(magic + `gram.r%d.%s.bin` + calib_gram/calib_resolve/calib_stem).
|
||||
**Decision: calibrate on 1.2.0, final convert on 1.0.13** — the serving
|
||||
pin produces the served `.rad`; only the Gram files cross versions.
|
||||
- `run-swift15-bf16.sh` boot-failure trail (all fixed, committed):
|
||||
duplicated `radiance` entrypoint arg (rc=2); ratio-derived KV pool SHORT
|
||||
217 MiB at 32×16k (explicit `--vram-kv-mib`); explicit 4608 starved the
|
||||
disk tier — **weights slab = card − headroom − kv, disk-tier floor
|
||||
9.57 GiB** (4.21 layer buffers + 5.36 fixed) → kv 3072; `KLD_MOUNT`
|
||||
referenced-never-defined emptied the corpus mount.
|
||||
- KLD record launched on 1.0.13 (disk tier, kv 3072, 4×16k, MNBT 32768):
|
||||
measured ~116 tok/s at the 32k step (44 docs/58 min).
|
||||
- Chain driver `kld-then-calib.sh` (pid 2848206, log
|
||||
`/home/dank/logs/swift15-chain.log`): KLD rc=0 → verify kld-ref → boot
|
||||
1.2.0 calib (MNBT 65536/seqs 8 per MOE-W4 930 tok/s law, synced commit
|
||||
45de2fd) → `calib-push.py` parallel 16 → kill → prod restored by trap.
|
||||
- Ready on the branch: `convert-swift15-w4dn8.sh` (1.0.13 convert, gates:
|
||||
≥48 grams / bf16 deleted / disk floor / rad-info gptq receipt /
|
||||
≥126,000 MiB), `apply-protected-cast.sh` (derive → recipe PROTECTED
|
||||
section, refuses double-derive), `kld-gate-swift15.sh` (candidate vs
|
||||
reference on 1.0.13, gate mean KL ≤ 0.03 / top-1 ≥ 95 %).
|
||||
- `frank-primary` showing `failed` during the campaign is the 143-on-stop
|
||||
signature of the foreground podman unit — cosmetic, not a crash.
|
||||
|
||||
## Campaign order after the chain
|
||||
|
||||
1. derive protected experts from Swift's own grams; apply to recipe.
|
||||
2. Delete bf16 `.rad` (335 GiB) to `.AGENTTRASH` staging (Freed space gates the convert).
|
||||
3. `convert-swift15-w4dn8.sh` (~2 h, CPU-only, prod can stay up).
|
||||
4. rad-tune declare TP=2 spec=2; vram-pair boot arms; bench (gate decode ≥ 30.8 t/s).
|
||||
5. `kld-gate-swift15.sh` (fail → escalate gate_up w4→w5nl, pre-authorized).
|
||||
6. Cutover: docs, tests + e2e, deploy, probes. Throughput fallback: w4 down tier (Q35).
|
||||
|
||||
Disk ledger at KLD-record time: 104 GiB free → KLD ref −50 → Grams −20 →
|
||||
bf16 delete +335 → convert +132 written.
|
||||
@@ -191,6 +191,9 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
|
||||
FreeToken one-card era (tag `frank-primary-freetoken-pr587-20261005`),
|
||||
then the tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
|
||||
- [[swift15-campaign-worklog]] — ACTIVE campaign log: the Swift1.5 →
|
||||
w4dn8 radiance convert and frank-primary swap in flight (branch
|
||||
`swift15-radiance-w4dn8`); the cutover runbook replaces it at completion.
|
||||
- [[frank-primary-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
|
||||
same day by the radiance cutover above; kept as the rollback runbook):
|
||||
FreeToken built at PR #587 head `77a7201` on the host into
|
||||
|
||||
Reference in New Issue
Block a user