dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
docs/tts-engine-trials
serverAppConfigs/serverAppConfigs-docs/LLM/swift15-campaign-worklog.md
T
Frank 55f87fcdce llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot 
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
2026-10-10 17:03:14 -07:00

4.2 KiB
Raw Permalink Blame History

Swift1.5 w4dn8 Campaign — Work Log

Campaign branch swift15-radiance-w4dn8. Goal: convert ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB) to qwen4exp-swift15-w4dn8-i8-hc8m.rad (GPTQ gate_up w4nl64 + block-fp8 down + 10 protected bf16 experts + int8 trunk + fixed-scale PLE) and swap it into frank-archon on the 1.0.13 pin. Design settled (25-question grill); this log carries execution facts only. The cutover runbook frank-archon-radiance-swift15-w4dn8.md replaces this page at cutover.

2026-10-06 evening

  • Purge freed 835.8 GB; Swift1.5 BF16 downloaded 131/131 shards (sha256 ok).
  • Corpora built: calib 10,014,932 tok / 1,294 prompts; KLD 1,800,816 tok / 197 prompts (/home/dank/corpora/out/).
  • Engine verdict: 1.2.0 REJECTED for serving (decode 29.21 vs control 30.08, accept 0.7123 vs 0.7549 deterministic; vram-pair translation ledger-exact). Serving pin stays 1.0.13 sha256:4c3f48…c322e.
  • bf16 rad-convert done: /home/dank/models/rad/qwen4exp-swift15-bf16.rad (335.28 GiB, 248,077 vocab tokens baked).
  • Blocker resolved — engine-version probe law: the calibration tap lives in the architecture plugin /opt/radiance/share/radiance/architectures/ qwen4exp_fp8.so, NOT the radiance binary, and the image has no strings(1) (greps return 0 for everything — positive controls mandatory). 1.0.13 plugin: RADIANCE_CALIB_DIR only; 1.2.0 adds RADIANCE_CALIB_TOKENS. Both libquant.so carry the gptq quantizer (calib=/damp/act_order/cd) and the identical RADGRAM2 format (magic + gram.r%d.%s.bin + calib_gram/calib_resolve/calib_stem). Decision: calibrate on 1.2.0, final convert on 1.0.13 — the serving pin produces the served .rad; only the Gram files cross versions.
  • run-swift15-bf16.sh boot-failure trail (all fixed, committed): duplicated radiance entrypoint arg (rc=2); ratio-derived KV pool SHORT 217 MiB at 32×16k (explicit --vram-kv-mib); explicit 4608 starved the disk tier — weights slab = card − headroom − kv, disk-tier floor 9.57 GiB (4.21 layer buffers + 5.36 fixed) → kv 3072; KLD_MOUNT referenced-never-defined emptied the corpus mount.
  • KLD record launched on 1.0.13 (disk tier, kv 3072, 4×16k, MNBT 32768): measured ~116 tok/s at the 32k step (44 docs/58 min).
  • Chain driver kld-then-calib.sh (pid 2848206, log /home/dank/logs/swift15-chain.log): KLD rc=0 → verify kld-ref → boot 1.2.0 calib (MNBT 65536/seqs 8 per MOE-W4 930 tok/s law, synced commit 45de2fd) → calib-push.py parallel 16 → kill → prod restored by trap.
  • Ready on the branch: convert-swift15-w4dn8.sh (1.0.13 convert, gates: ≥126,000 MiB), apply-protected-cast.sh (derive → recipe PROTECTED section, refuses double-derive), kld-gate-swift15.sh (candidate vs reference on 1.0.13, gate mean KL ≤ 0.03 / top-1 ≥ 95 %).
  • frank-archon showing failed during the campaign is the 143-on-stop signature of the foreground podman unit — cosmetic, not a crash.

2026-10-07 morning

  • KLD reference COMPLETE on 1.0.13: /home/dank/corpora/kld-ref 24 GiB, radiance-kld-1, rows: 50235, 197 docs, top-16 log-probs (f16), corpus fnv1a64 a9d62b33c18b3d9c. Record took ~4 h at the 32k step. The wrapper died rc=127 AFTER the engine exited clean: the mid-run scp of commit 45de2fd shifted bash's lazy read offset — script-swap law, third fleet occurrence. Resume path added: SKIP_KLD=1 + rows: 50235 receipt gate (cf6c62d). Never scp a script a running process owns.
  • Calibration chain3 live on 1.2.0 (big-step shape MNBT 65536/seqs 8): push started, requests landing at 7–8k tokens.

Campaign order after the chain

  1. derive protected experts from Swift's own grams; apply to recipe.
  2. Delete bf16 .rad (335 GiB) to .AGENTTRASH staging (Freed space gates the convert).
  3. convert-swift15-w4dn8.sh (~2 h, CPU-only, prod can stay up).
  4. rad-tune declare TP=2 spec=2; vram-pair boot arms; bench (gate decode ≥ 30.8 t/s).
  5. kld-gate-swift15.sh (fail → escalate gate_up w4→w5nl, pre-authorized).
  6. Cutover: docs, tests + e2e, deploy, probes. Throughput fallback: w4 down tier (Q35).

Disk ledger at KLD-record time: 104 GiB free → KLD ref −50 → Grams −20 → bf16 delete +335 → convert +132 written.

Reference in New Issue View Git Blame Copy Permalink