diff --git a/AGENTS.md b/AGENTS.md index 95e71c8..d812ab9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer**, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **PLE 95.4 GiB bf16 as mmap page cache** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **int8 KV 4.2 KiB/token**, **MTP spec decode depth 3**, `--max-num-seqs 4`, `--max-model-len 200000`, MNBT 2048, fp8 HC mixtures, Froggeric v22.5 template. Decode 122–173 t/s at 44.5k depth (KLD-gated). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 | +| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 | | Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770| | Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 | @@ -504,7 +504,8 @@ w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine with the token knob; the RADGRAM2 grams are format-identical across pins and cross the boundary by design), **24 whole experts per layer cast bf16** (the PROTECTED set, derived from the calibration grams' energy share), -**int8 KV**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE bf16 as shipped. +**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram +table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16). The plugin's **one-table-one-encoding law** (measured at rad-convert declare): one expert table takes ONE encoding; only whole bf16 experts are legal beside it — no w4/fp8 split of the down/gate_up pair (24-value @@ -515,32 +516,37 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar Rollback: revert the swap commit — the fp8-native `.rad` (`qwen4exp-fp8-native.rad`) and every prior payload remain on disk. -**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 4`, +**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`, `--max-model-len 200000` (the harness contextWindow must quote this), `--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM -partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (int8 KV halves -the fp8-era cost; the vram-pair replaced `--expert-vs-cache-ratio`), -`--kv-cache-dtype int8`, `--num-speculative-tokens 3`, +partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair +replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the +deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native +env), `--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`, -`--override-chat-template`). The 47.7 GiB PLE table rides the ~95 GiB -bf16 tensor pool in host RAM. +`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8 +tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free. Shape notes carried from the fp8-native era and unchanged: wht6 gains, -MNDT 2048 law, MTP depth 3 (the checkpoint carries the 3,101 `mtp.*` -tensors), the 64 GiB disk prefix cache, and the rule that a prompt over -`--max-model-len` is refused, never truncated. What changed: int8 KV -replaces fp8 KV (4.2 KiB/token, half the fp8 rate at 64k depth), and the -EVC ratio is replaced by the explicit `--vram-weights-mib`/`--vram-kv-mib` -pair — sizing law for the disk-tier shape: the engine forces ~9.6 GiB of -disk-tier weight buffers + arena onto the weights side, so `weights + kv` -is capped near 29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`. +MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk +prefix cache, and the rule that a prompt over `--max-model-len` is +refused, never truncated. What changed: the fp8-native era's NVMe expert +streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the +40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s +on the deployed shape (512-token streamed probe). The campaign also +measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth, +accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover — +the KLD gate scored at the fp8-KV shape, so the gate and the boot agree. +Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight +buffers + arena onto the weights side, so `weights + kv` is capped near +29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`. **The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert -pools + 4 GiB prefix host ≈ **84 GiB** charged against `RLIMIT_MEMLOCK`; -the 95.4 GiB bf16 PLE pool rides as **mmap page cache** (never mlocked — -the `-ot`/no-mmap law), so it is invisible to the limit. +pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM +the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against +`RLIMIT_MEMLOCK` — the TTM floor is set just above it. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until both knobs are raised in the stop window (sysfs write first,