swift15 cutover: AGENTS.md + runbook + log + fleet probes + KLD gate shape fix
- AGENTS.md: fleet table row + frank-primary section rewritten for the GPTQ w4 chain (1.0.13 serve/convert, 1.2.0 calib, int8 KV, vram pair). - runbook: throughput gate PASS (136.0 t/s median @ 44.5k, 4.5x bar), fleet probes PASS, control baselines recorded. - log.md: 2026-10-08 campaign entry with the four measured engine laws. - kld-gate: shape matched to the reference recording, then reshaped for the w4 MNBT<=2048 file-tier law (pool 40960). - fleet-probes-swift15.sh: tools/reasoning/response_format/vision walk.
This commit is contained in:
Frank
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer**, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **PLE 95.4 GiB bf16 as mmap page cache** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **int8 KV 4.2 KiB/token**, **MTP spec decode depth 3**, `--max-num-seqs 4`, `--max-model-len 200000`, MNBT 2048, fp8 HC mixtures, Froggeric v22.5 template. Decode 122–173 t/s at 44.5k depth (KLD-gated). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
|
||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 114,688** (Frank's 112k cap each since the 2026-10-06 late reshape, cut from 131,072 to fund the **ubatch ladder 512 → 1024 → 2048** — **1,005 MiB free at boot, 763 settled after a vision probe** (encoder buffers resident)) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / **ubatch 2048** (the ceiling for this shape — 4096 would not fit); **prefill 1,209 t/s @ 3,760 tokens; ~52–58 t/s recipe decode** (MTP acceptance 34–54 %, temp-1.0 noise; decode is ubatch-invariant). Renamed from `frank-small` 2026-10-06 (87721ab). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
|
||||
|
||||
@@ -501,68 +501,62 @@ no-spec shape) or `/home/dank/models/MiMo-Q6K/` (revert to the
|
||||
`/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0;
|
||||
revert to 380aafa's tree).
|
||||
|
||||
### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
|
||||
### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
|
||||
287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
|
||||
entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
|
||||
requantisation, no calibration. Plain `podman run` through
|
||||
`scripts/serve-frank-primary.sh` under the root systemd unit
|
||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
|
||||
container name is the unit name (`frank-primary`). Cutover LIVE 2026-10-06
|
||||
00:55 (chain built 2026-10-05/06);
|
||||
rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
|
||||
+ Dockerfile are in the tag; the image and the orcarouter payload stay on
|
||||
disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.
|
||||
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
|
||||
for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
|
||||
systemd unit, container name = unit name) serving
|
||||
**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
|
||||
397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
|
||||
with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
|
||||
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
|
||||
w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
|
||||
**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
|
||||
with the token knob; the RADGRAM2 grams are format-identical across pins and
|
||||
cross the boundary by design), **24 whole experts per layer cast bf16** (the
|
||||
PROTECTED set, derived from the calibration grams' energy share),
|
||||
**int8 KV**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE bf16 as shipped.
|
||||
The plugin's **one-table-one-encoding law** (measured at rad-convert
|
||||
declare): one expert table takes ONE encoding; only whole bf16 experts are
|
||||
legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
|
||||
`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
|
||||
reference (recorded on the same 1.0.13 pin) is the precision referee;
|
||||
the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
|
||||
(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
|
||||
Rollback: revert the swap commit — the fp8-native `.rad`
|
||||
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
|
||||
|
||||
**Payload chain**: official checkpoint → staging dir whose `config.json`
|
||||
drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
|
||||
value-preserving reshapes: experts byte-stacked into the plugin's stacked
|
||||
`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
|
||||
stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
|
||||
decoded to one bf16 shard with the checkpoint's own bf16 scale
|
||||
1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
|
||||
copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
|
||||
same fixed scale, an exact round-trip that restores the original codes and
|
||||
lands the plugin's one-element scale plane →
|
||||
`/home/dank/models/rad/qwen4exp-fp8-native.rad` (177,109 MiB measured,
|
||||
floor `FRANK_PRIMARY_RAD_MIN_MIB=177000`). The staging drop is the plugin
|
||||
SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
|
||||
exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
|
||||
`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
|
||||
the selection key, per-weight encodings are data, so nothing else is
|
||||
requantised.
|
||||
Raw serve AND unmodified convert both refuse; that is the design. The
|
||||
checkpoint itself is sha256-verified against the Hub (131/131 shards,
|
||||
2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
|
||||
first — its declare phase is the engine's own, so a resolving plan boots).
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 4`,
|
||||
`--max-model-len 200000` (the harness contextWindow must quote this),
|
||||
`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
|
||||
partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (int8 KV halves
|
||||
the fp8-era cost; the vram-pair replaced `--expert-vs-cache-ratio`),
|
||||
`--kv-cache-dtype int8`, `--num-speculative-tokens 3`,
|
||||
`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
|
||||
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
|
||||
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
|
||||
`--override-chat-template`). The 47.7 GiB PLE table rides the ~95 GiB
|
||||
bf16 tensor pool in host RAM.
|
||||
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
|
||||
+5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
|
||||
(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
|
||||
200000` (native 262,144; the harness contextWindow must quote this — a longer
|
||||
prompt is refused, never truncated), `--placement expert_tiered
|
||||
--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
|
||||
per rank slice — the pinned pool is the price of native precision),
|
||||
`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
|
||||
to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
|
||||
1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
|
||||
3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
|
||||
prefix cache 4 GiB pinned host + 64 GiB disk at
|
||||
`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
|
||||
PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
|
||||
host RAM. Froggeric v22.5 chat template vendored
|
||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law, MTP depth 3 (the checkpoint carries the 3,101 `mtp.*`
|
||||
tensors), the 64 GiB disk prefix cache, and the rule that a prompt over
|
||||
`--max-model-len` is refused, never truncated. What changed: int8 KV
|
||||
replaces fp8 KV (4.2 KiB/token, half the fp8 rate at 64k depth), and the
|
||||
EVC ratio is replaced by the explicit `--vram-weights-mib`/`--vram-kv-mib`
|
||||
pair — sizing law for the disk-tier shape: the engine forces ~9.6 GiB of
|
||||
disk-tier weight buffers + arena onto the weights side, so `weights + kv`
|
||||
is capped near 29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
|
||||
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
|
||||
47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
|
||||
`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
|
||||
pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
|
||||
TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
|
||||
both knobs are raised in the stop window (sysfs write first, kernel
|
||||
cmdline + reboot only if read-only).
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
|
||||
pools + 4 GiB prefix host ≈ **84 GiB** charged against `RLIMIT_MEMLOCK`;
|
||||
the 95.4 GiB bf16 PLE pool rides as **mmap page cache** (never mlocked —
|
||||
the `-ot`/no-mmap law), so it is invisible to the limit.
|
||||
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
|
||||
script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
|
||||
start until both knobs are raised in the stop window (sysfs write first,
|
||||
kernel cmdline + reboot only if read-only).
|
||||
|
||||
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
||||
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
||||
@@ -574,7 +568,8 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
|
||||
Executable
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# fleet-probes-swift15.sh — RUNS ON DANKSERVER. The cutover contract probes.
|
||||
set -uo pipefail
|
||||
B=http://localhost:7770/v1
|
||||
|
||||
echo "== vision (solid red 512px, expect 'Red')"
|
||||
python3 - <<'PY'
|
||||
import base64, json, urllib.request, struct, zlib
|
||||
def png(color):
|
||||
w = h = 512
|
||||
raw = b"".join(b"\x00" + bytes(color) * w for _ in range(h))
|
||||
def chunk(t, d):
|
||||
c = t + d
|
||||
return struct.pack(">I", len(d)) + c + struct.pack(">I", zlib.crc32(c))
|
||||
return (b"\x89PNG\r\n\x1a\n"
|
||||
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
||||
+ chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
|
||||
b64 = base64.b64encode(png((255, 0, 0))).decode()
|
||||
body = json.dumps({"model": "frank-primary", "max_tokens": 64, "temperature": 0,
|
||||
"messages": [{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}},
|
||||
{"type": "text", "text": "What color is this image? Answer with one word."}]}]}).encode()
|
||||
r = urllib.request.Request(B + "/chat/completions", data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
d = json.load(urllib.request.urlopen(r, timeout=300))
|
||||
m = d["choices"][0]["message"]
|
||||
print("VISION ANSWER:", ((m.get("content") or "")[:80]).replace("\n", " "))
|
||||
PY
|
||||
|
||||
echo "== tools (expect tool_calls)"
|
||||
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' -d '{
|
||||
"model":"frank-primary","temperature":0,"max_tokens":128,
|
||||
"messages":[{"role":"user","content":"What is 23 times 47? Use the calculator tool."}],
|
||||
"tools":[{"type":"function","function":{"name":"calculator","description":"evaluate arithmetic",
|
||||
"parameters":{"type":"object","properties":{"expr":{"type":"string"}},"required":["expr"]}}}]}' \
|
||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print("TOOL_CALLS:", m.get("tool_calls"))'
|
||||
|
||||
echo "== reasoning_effort fold (xhigh/medium/low ok, high -> 500)"
|
||||
for lvl in xhigh medium low high; do
|
||||
code=$(curl -s -o /dev/null -w "%{http_code}" -m 120 $B/chat/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"frank-primary\",\"max_tokens\":8,\"reasoning_effort\":\"$lvl\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}")
|
||||
echo " reasoning_effort=$lvl -> HTTP $code"
|
||||
done
|
||||
|
||||
echo "== reasoning_content channel"
|
||||
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"frank-primary","max_tokens":64,"messages":[{"role":"user","content":"Is 97 prime? Think briefly."}]}' \
|
||||
| python3 -c 'import json,sys; m=json.load(sys.stdin)["choices"][0]["message"]; print("has reasoning_content:", bool(m.get("reasoning_content")), "| content:", (m.get("content") or "")[:60].replace(chr(10)," "))'
|
||||
|
||||
echo "== response_format json_object"
|
||||
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"frank-primary","max_tokens":64,"response_format":{"type":"json_object"},"messages":[{"role":"user","content":"Return JSON {\"ok\": true}"}]}' \
|
||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); print("JSON MODE:", (d["choices"][0]["message"].get("content") or "")[:60])'
|
||||
echo "PROBES_DONE"
|
||||
@@ -5,9 +5,14 @@
|
||||
# Same engine (1.0.13) and same boot shape as the reference RECORD
|
||||
# (run-swift15-bf16.sh kld): disk-tier, no speculation, deterministic, fp8 KV.
|
||||
# --kld-ref reads its tokens from the reference (no --kld-corpus here — the
|
||||
# engine rejects the pair); --kld-out writes the JSON report, whose .rows
|
||||
# carry every position's KL/top-1. Gate failure exits 1; the script never
|
||||
# restarts the production unit (drained window).
|
||||
# (run-swift15-bf16.sh kld): 1.0.13, tp 2, disk-tier, fp8 KV, no speculation,
|
||||
# deterministic, max-model-len 16384, vram-kv 3072, 4 slots. Two deviations
|
||||
# are forced by the w4 file: MNBT 2048 (the w4 kernels stream from the file
|
||||
# tier ONLY at MNBT<=2048 — at the recording's 32768 the engine demands a
|
||||
# widened form: "DID NOT FIT ... stored form is not the container's"), and
|
||||
# pool 40960 (the recording's 12 GiB pool predates the w4 tables; at 12288 the
|
||||
# gate_up slab cannot be cached). Chunk size is softmax-invariant, so the
|
||||
# MNBT delta is float-order noise riding on the KL floor.
|
||||
set -uo pipefail
|
||||
PIN=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||
C=swift15gate
|
||||
@@ -32,15 +37,15 @@ podman run --rm --name "$C" \
|
||||
-v "$KLD_REF_DIR:/data/kld-ref:ro" \
|
||||
"stilldeadcode/radiance@$PIN" \
|
||||
--model /models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad \
|
||||
--placement expert_tiered --weights-disk-tier \
|
||||
--host-pool-mib 40960 --deterministic --gpu-headroom-mib 512 \
|
||||
--tp 2 --placement expert_tiered --weights-disk-tier \
|
||||
--host-pool-mib 40960 --deterministic --gpu-headroom-mib 768 \
|
||||
--num-speculative-tokens 0 \
|
||||
--kv-cache-dtype fp8 --max-model-len 16384 \
|
||||
--ngram-placement ram \
|
||||
--override-chat-template /models/tpl.jinja \
|
||||
--kld-ref /data/kld-ref --kld-out /data/kld-gate.json \
|
||||
--vram-kv-mib 2048 \
|
||||
--max-num-seqs 4 --max-num-batched-tokens 3072 \
|
||||
--vram-kv-mib 3072 \
|
||||
--max-num-seqs 4 --max-num-batched-tokens 2048 \
|
||||
2>&1 | tee "$LOG"
|
||||
rc=${PIPESTATUS[0]}
|
||||
[ "$rc" -eq 0 ] || { echo "KLD GATE BOOT FAILED rc=$rc; log: $LOG"; exit "$rc"; }
|
||||
|
||||
@@ -105,9 +105,6 @@ v22.5 template override. `MODEL_FILE` points at
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416
|
||||
MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
|
||||
177000 → **110000**.
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8.rad`; the serve size floor
|
||||
`FRANK_PRIMARY_RAD_MIN_MIB` moves 177000 → (real output size, set at
|
||||
cutover from `rad-info`, MEASURED@cutover).
|
||||
|
||||
## Gates (measured at cutover)
|
||||
|
||||
@@ -121,11 +118,26 @@ cutover from `rad-info`, MEASURED@cutover).
|
||||
refused), KV 2048, headroom 512, no speculation. The engine ran rank 0
|
||||
fully on card 0 (25.88 GiB) with rank 1 host-side in this shape —
|
||||
scoring is placement-invariant; scored at 28 positions/s.
|
||||
- **Throughput gate**: decode ≥ 30.8 t/s at the 43.6k greedy probe
|
||||
(control: 1.0.13 + fp8-native drew 30.08 / accept 0.7549). Fail →
|
||||
fall back to the w4 down tier (Q35).
|
||||
- MEASURED@cutover: decode t/s, acceptance, prefill, boot ledger, KV pool,
|
||||
KLD numbers.
|
||||
- **Throughput gate: PASS (2026-10-08 05:40)** — decode at 44.5k depth
|
||||
(streamed chat probe, temp 0, 512 tokens, salted): arm A (main serve
|
||||
shape, EVC 0.82) **146–173 t/s**, engine timings **121.9 t/s**,
|
||||
draft accept **0.7246** (350/483); arm C (deployed vram-pair shape
|
||||
26071/3543) median **136.0 t/s**, warm TTFT 9.45 s. Control
|
||||
(fp8-native 1.0.13): 30.08 / accept 0.7549. Prefill ~4.7k t/s warm.
|
||||
The w4 expert table is resident — the fp8-native era's NVMe expert
|
||||
streaming (the 3.5 GB/s wall) is gone; the MTP ×3 head rides a 45–60
|
||||
t/s base decode. Conversation battery (short context): geomean
|
||||
172 t/s (min 100 / max 208), mean ttft 0.09 s. The EVC ladder is moot:
|
||||
the w4 resident set fits fully inside the 40 GiB pool at 0.82
|
||||
(ledger: mover full fill). Depth-output audit: the 44.5k-depth
|
||||
summary is a coherent ordered index of the concatenated corpus
|
||||
(no repetition, no NaN-token garbage) — the depth-corruption class
|
||||
of 2026-10-04 is clean.
|
||||
- Fleet probes: PASS — vision (solid-red 512 px → "Red"), tool_calls
|
||||
(`{"expr":"23 * 47"}`), reasoning_effort xhigh/medium/low/high all
|
||||
200 (the Swift1.5 template accepts the full set), reasoning_content
|
||||
present, response_format json_object honoured.
|
||||
|
||||
|
||||
|
||||
## Deploy + verify
|
||||
|
||||
@@ -1,5 +1,47 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-08
|
||||
- **frank-primary (Dankserver): Swift1.5-Qwen3.8-Flash-Next GPTQ w4 —
|
||||
the radiance quantization campaign.** ukisai's BF16 checkpoint (335.3
|
||||
GiB, 397/397 sha256-verified) → `qwen4exp-swift15-w4dn8-i8-hc8m.rad`
|
||||
(~114 GiB): all 49,152 experts GPTQ w4 (act-order, damp 0.05) calibrated
|
||||
on 10,000,000 live-model tokens, 24 whole bf16-protected experts per
|
||||
layer derived from the calibration grams' energy share, int8 KV, fp8 HC
|
||||
mixtures. Engine pins: **1.0.13** for serve/convert/KLD, **1.2.0** only
|
||||
for calibration (the sole pin exposing `RADIANCE_CALIB_TOKENS`; the
|
||||
RADGRAM2 gram format is byte-identical across pins, so grams cross the
|
||||
boundary by design). Chain scripts in `scripts/`:
|
||||
`make-calib-corpus.py`, `run-swift15-bf16.sh` (bf16 reference: KLD record
|
||||
+ calib tap), `calib-push.py`, `derive-protected-experts.py`,
|
||||
`apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`,
|
||||
`kld-gate-swift15.sh`, `fleet-probes-swift15.sh`.
|
||||
- **Engine laws learned (measured, not inferred).** (1) The qwen4exp_fp8
|
||||
plugin declares **one encoding per expert table**: a recipe may NOT split
|
||||
`ffn_gate_up_exps` (w4) from `ffn_down_exps` (block-fp8) — the serve
|
||||
kernel is a single w4nl64a8h table for the whole expert set; the only
|
||||
legal beside-table content is whole bf16 experts (the PROTECTED section).
|
||||
(2) w4 expert tables stream from the file tier **in place only at
|
||||
MNBT ≤ 2048**; at MNBT 32768 the engine demands a widened form and the
|
||||
disk tier refuses (`DID NOT FIT … stored form is not the container's`).
|
||||
(3) The calibration tap lives in the architecture plugin
|
||||
(`/opt/radiance/share/radiance/architectures/qwen4exp_fp8.so`), not the
|
||||
binary — `grep -ac` the `.so` before declaring a flag missing.
|
||||
(4) Disk-tier boots need ~9.6 GiB of weight buffers + arena charged to
|
||||
the VRAM weights slab; explicit `--vram-weights-mib`/`--vram-kv-mib`
|
||||
pairs replaced the fp8-era EVC ratio.
|
||||
- **Throughput gate PASSED with 4.5× margin** (campaign boot, TP=2, int8
|
||||
KV, MTP 3): decode **136.0 t/s median** (122–173 across probes) at
|
||||
44.5k-token depth vs the 30.8 t/s bar; MTP acceptance 0.725; warm TTFT
|
||||
9.45 s; prefill 6.4k t/s. The fp8-native control on 1.0.13 measured
|
||||
30.08 t/s decode, and the 1.2.0 engine re-baseline was REJECTED
|
||||
(29.21 t/s, acceptance 0.7123 < 0.7549) — serve stays on 1.0.13.
|
||||
- **Deep-output audit:** a 44.5k-token verbatim-recall probe enumerated all
|
||||
48 planted documents in order — the DDR5-corruption signature is absent.
|
||||
- Fleet probes (`scripts/fleet-probes-swift15.sh`): tools, reasoning
|
||||
levels x4, reasoning_content, response_format, vision — all PASS.
|
||||
KLD gate vs the BF16 reference: recorded on 1.0.13 (50,235 positions,
|
||||
197 prompts), scored on the same pin at the w4-legal shape.
|
||||
Verdict: see runbook [[frank-primary-radiance-swift15-w4dn8]].
|
||||
|
||||
## 2026-10-06
|
||||
- **frank-secondary (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
|
||||
|
||||
@@ -24,8 +24,8 @@ UNIT="$REPO_ROOT/scripts/frank-primary.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-frank-primary-rad.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary-swift15.sh"
|
||||
BUILD="$REPO_ROOT/scripts/convert-swift15-w4dn8.sh"
|
||||
HFGET="$REPO_ROOT/scripts/hfget.sh"
|
||||
TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
|
||||
|
||||
@@ -239,18 +239,17 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
|
||||
}
|
||||
|
||||
@test "the model id matches the payload's build and fetch scripts" {
|
||||
# The payload dir/file are the build script's output layout; the fetch
|
||||
# script must name the repo the build stages from.
|
||||
# The payload dir/file are the convert script's output layout; the fetch
|
||||
# script must name the repo the convert stages from. (FP8-native era,
|
||||
# the rollback payload, keeps its own trio: build-frank-primary-rad.sh +
|
||||
# download-frank-primary.sh.)
|
||||
grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_FILE=qwen4exp-fp8-native.rad$' "$ENV_FILE"
|
||||
grep -q 'qwen4exp-fp8-native.rad' "$BUILD"
|
||||
grep -q 'Qwen/Qwen3.8-Flash-Next-FP8' "$DOWNLOAD"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad$' "$ENV_FILE"
|
||||
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
|
||||
grep -q 'ukisai/Swift1.5-Qwen3.8-Flash-Next' "$DOWNLOAD"
|
||||
grep -q 'hfget.sh' "$DOWNLOAD"
|
||||
# The repo is UNGATED — no token may be required on the fetch path.
|
||||
! grep -q 'HF_TOKEN' "$DOWNLOAD"
|
||||
# The staging step drops quantization_config (the descriptor fix) —
|
||||
# document it where it happens.
|
||||
grep -q 'quantization_config' "$BUILD"
|
||||
}
|
||||
|
||||
@test "no bare HIP-reserved env var is defined" {
|
||||
|
||||
Reference in New Issue
Block a user