dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

merge: swift15-radiance-w4dn8 — frank-primary GPTQ w4 (Swift1.5)

Browse Source 
Conflict resolved: frank-primary row from the branch (w4 chain),
frank-secondary row from main (Qwen3.5-9B swap of 2026-10-07).
This commit is contained in:
Frank
2026-10-08 04:46:11 -07:00
parent fa5a444b64 1aef7418ad
commit 02fa1e22ae
24 changed files with 1779 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+54 -59
View File
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer**, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **PLE 95.4 GiB bf16 as mmap page cache** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **int8 KV 4.2 KiB/token**, **MTP spec decode depth 3**, `--max-num-seqs 4`, `--max-model-len 200000`, MNBT 2048, fp8 HC mixtures, Froggeric v22.5 template. Decode 122–173 t/s at 44.5k depth (KLD-gated). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
@@ -489,68 +489,62 @@ Rollback: revert the swap commit — the Gemma4 payload
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
The **radiance engine** (StillDeadcode, Apache-2.0; image
`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
requantisation, no calibration. Plain `podman run` through
`scripts/serve-frank-primary.sh` under the root systemd unit
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
container name is the unit name (`frank-primary`). Cutover LIVE 2026-10-06
00:55 (chain built 2026-10-05/06);
rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
+ Dockerfile are in the tag; the image and the orcarouter payload stay on
disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
systemd unit, container name = unit name) serving
**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
with the token knob; the RADGRAM2 grams are format-identical across pins and
cross the boundary by design), **24 whole experts per layer cast bf16** (the
PROTECTED set, derived from the calibration grams' energy share),
**int8 KV**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE bf16 as shipped.
The plugin's **one-table-one-encoding law** (measured at rad-convert
declare): one expert table takes ONE encoding; only whole bf16 experts are
legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
reference (recorded on the same 1.0.13 pin) is the precision referee;
the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
Rollback: revert the swap commit — the fp8-native `.rad`
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
**Payload chain**: official checkpoint → staging dir whose `config.json`
drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
value-preserving reshapes: experts byte-stacked into the plugin's stacked
`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
decoded to one bf16 shard with the checkpoint's own bf16 scale
1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
same fixed scale, an exact round-trip that restores the original codes and
lands the plugin's one-element scale plane →
`/home/dank/models/rad/qwen4exp-fp8-native.rad` (177,109 MiB measured,
floor `FRANK_PRIMARY_RAD_MIN_MIB=177000`). The staging drop is the plugin
SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
the selection key, per-weight encodings are data, so nothing else is
requantised.
Raw serve AND unmodified convert both refuse; that is the design. The
checkpoint itself is sha256-verified against the Hub (131/131 shards,
2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
first — its declare phase is the engine's own, so a resolving plan boots).
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 4`,
`--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (int8 KV halves
the fp8-era cost; the vram-pair replaced `--expert-vs-cache-ratio`),
`--kv-cache-dtype int8`, `--num-speculative-tokens 3`,
`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
`--override-chat-template`). The 47.7 GiB PLE table rides the ~95 GiB
bf16 tensor pool in host RAM.
**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
+5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
200000` (native 262,144; the harness contextWindow must quote this — a longer
prompt is refused, never truncated), `--placement expert_tiered
--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
per rank slice — the pinned pool is the price of native precision),
`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
prefix cache 4 GiB pinned host + 64 GiB disk at
`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
host RAM. Froggeric v22.5 chat template vendored
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law, MTP depth 3 (the checkpoint carries the 3,101 `mtp.*`
tensors), the 64 GiB disk prefix cache, and the rule that a prompt over
`--max-model-len` is refused, never truncated. What changed: int8 KV
replaces fp8 KV (4.2 KiB/token, half the fp8 rate at 64k depth), and the
EVC ratio is replaced by the explicit `--vram-weights-mib`/`--vram-kv-mib`
pair — sizing law for the disk-tier shape: the engine forces ~9.6 GiB of
disk-tier weight buffers + arena onto the weights side, so `weights + kv`
is capped near 29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
both knobs are raised in the stop window (sysfs write first, kernel
cmdline + reboot only if read-only).
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
pools + 4 GiB prefix host ≈ **84 GiB** charged against `RLIMIT_MEMLOCK`;
the 95.4 GiB bf16 PLE pool rides as **mmap page cache** (never mlocked —
the `-ot`/no-mmap law), so it is invisible to the limit.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
start until both knobs are raised in the stop window (sysfs write first,
kernel cmdline + reboot only if read-only).
**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -562,7 +556,8 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.