frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index
The parallel session's uncommitted swap, committed at Frank's direction (commit whatever's in the working tree): - llama.cpp master on the RTX 3090 now serves ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE 131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to rollback target. - env/serve/download + seam twins updated together (deployer payload gates rode the previous commit's deploy-remote.sh). - docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135 seam green). - comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/ workflows/ (12 KiB).
This commit is contained in:
Frank
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (~161 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, **2 × 200,000** (first boot; native 262,144), froggeric v22.5 template, vision + tools + `reasoning_content` auto, `/metrics` scraped. Cutover 2026-10-05 (rollback `frank-primary-freetoken-pr587-20261005`, FreeToken one-card image + payload on disk) | 7770 |
|
||||
| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; **MTP head IN the file** — `--spec-type draft-mtp`, depth 3) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **1 × 131,072** (Frank's 128k cap; a second 131,072 slot misses the 2-3 GiB floor) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024. Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -264,7 +264,7 @@ merely wedged kernel usually still answers ARP.
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-primary` (root unit, FreeToken on one R9700):
|
||||
- Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
|
||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
||||
2. Bench.
|
||||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
|
||||
@@ -299,8 +299,9 @@ a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
|
||||
image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
|
||||
ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
|
||||
`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision
|
||||
tower and streams its block weights from host RAM two at a time by default),
|
||||
`frank-primary` takes them natively (radiance runs the checkpoint's
|
||||
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
||||
are the engine core; `--mm-max-patches` available),
|
||||
and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
@@ -378,132 +379,145 @@ rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
**llama.cpp master** (CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
|
||||
**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
|
||||
sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
|
||||
projector is a hardlink of the Q5KM copy — the dir is self-contained), on
|
||||
port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
|
||||
reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
|
||||
one reason:
|
||||
**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
|
||||
together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
|
||||
from three slots to two made room for **fully unquantized V**
|
||||
(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
|
||||
2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
|
||||
file is the entire vision path (and ~1.4 GiB of the card).
|
||||
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
|
||||
(`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving `ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF`
|
||||
at **IQ3_S -mtp** (12,120,016,896 B, sha256 `9aecf1cd…`) **plus
|
||||
`mmproj-Qwen3.8-27B-F16.gguf`** (927,607,488 B, sha256 `cbb841a9…`), both
|
||||
`/home/dank/models/Swift15-IQ3S/`, on port 7771. Cutover 2026-10-05
|
||||
(evening) on Frank's explicit contract: **128k context cap, KV at FP8,
|
||||
vision on, as many 128k contexts as fit** — the answer is ONE, by
|
||||
arithmetic. The same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K
|
||||
shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa)
|
||||
but never deployed (blocked at the docs gate; the docs debt is what this
|
||||
section settles).
|
||||
|
||||
**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
|
||||
DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
|
||||
the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
|
||||
drives that term.
|
||||
**Architecture: qwen3_5 hybrid, bigger** — block_count 65 (64 real layers +
|
||||
the blk.64 MTP head), `full_attention_interval 4` ⇒ **16 full-attention +
|
||||
48 Gated DeltaNet**, native 262,144 window, dense. KV exists only in the 16
|
||||
attention layers (4 KV heads × 256 dim): 64 KiB/token at f16, **34 KiB/token
|
||||
at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at
|
||||
48 linear layers × 48 value heads). Header facts were read from the Hub
|
||||
file by HTTP range + struct parse before anything was downloaded.
|
||||
|
||||
**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
|
||||
conversations of 200,704 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
|
||||
**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
|
||||
2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
|
||||
runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
|
||||
+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
|
||||
which sits below either ubatch, did not move at all. The old "ubatch 2048
|
||||
failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
|
||||
generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
|
||||
free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
|
||||
byte-identically the same hour — the doubling costs **616 MiB** and lands at
|
||||
the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
|
||||
if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
|
||||
compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
|
||||
quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
|
||||
~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
|
||||
q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
|
||||
CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
|
||||
there). Three native windows need 18.4 GiB of KV alone and were never on the
|
||||
table.
|
||||
**The shape (2026-10-05):** `--ctx-size 131072 --parallel 1` = ONE
|
||||
conversation at Frank's 128k cap (Swift's own benchmark protocol ran at
|
||||
131,072), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's FP8
|
||||
class — no true fp8 KV dtype exists; 8.5 effective bits; the order
|
||||
SUPERSEDES the same-day f16/f16 rule written for the Ornith 9B fit), all
|
||||
layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024 (lower
|
||||
ubatch first, Frank's standing law; 2048 is the first A/B if boot lands
|
||||
high). Non-KV ≈ 15.5 GiB (weights 11,561 + projector ~1,350 + buffers
|
||||
~1.6k + CUDA ~500 + DeltaNet ~150/slot + MTP draft ctx ~0.3-0.5k); one q8_0
|
||||
slot = 4,352 MiB ⇒ free ≈ 4.5 GiB; a SECOND slot wants +4,352 ⇒ ~0.2 GiB —
|
||||
far under the 2-3 GiB floor. The pool is capped by the 128k law, not the
|
||||
card; leftover headroom is honest slack.
|
||||
|
||||
Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
|
||||
and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
|
||||
that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
|
||||
decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
|
||||
bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
|
||||
`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
|
||||
`image_url` content part (two traps: the legacy top-level `images` field is
|
||||
ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
|
||||
the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
|
||||
is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
|
||||
`contextWindow: 200704`.
|
||||
**Speculative decoding is the file's own MTP head**: the `-mtp` GGUF
|
||||
carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`), so
|
||||
`--spec-type draft-mtp --spec-draft-n-max 3` loads it against the main
|
||||
model's weights — no `--model-draft`, no second draft KV pool. Flags move
|
||||
as a set (a no-spec baseline strips BOTH). Acceptance reads from the
|
||||
response `timings` (`draft_n`/`draft_n_accepted`) on real traffic, never a
|
||||
synthetic repeat.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
|
||||
(the boot-pair table that prices pool changes lives in the retired
|
||||
[[frank-small-llama-mimo-q5km]]).
|
||||
Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
|
||||
buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
|
||||
measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
|
||||
pairs with Q5 only; SGLang rollback in
|
||||
[[frank-small-sglang-mimo-v26]] (venv on disk at
|
||||
`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
|
||||
**Vision without a Swift projector**: no Swift repo ships one (ukisai's
|
||||
README is explicit), so the projector is unsloth's mmproj built from the
|
||||
SHARED BASE `Qwen/Qwen3.8-27B` — legal because Swift-1.5's vision stack was
|
||||
hash-proven byte-identical to the base (333/333 tensor names, 6/6 sampled
|
||||
tensors: merger fc1/fc2, blocks 0/13/26) on 2026-10-05. The probe traps
|
||||
carry over: ≥256 px probe images or the model answers "video"; only the
|
||||
OpenAI `image_url` content part reaches the model.
|
||||
|
||||
### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
|
||||
The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** of stock quant
|
||||
types (IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …)
|
||||
— stock llama.cpp loads it; the tensor-type histogram is the law (see Key
|
||||
Lessons). Publisher KLD vs Swift BF16: 0.0513 development / 0.024–0.052
|
||||
held-out; the MTP packaging is NOT separately evaluated by the publisher.
|
||||
|
||||
The **FreeToken engine at PR #587 head `77a7201`** (branch
|
||||
`rocm714_torch214`, built on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own
|
||||
docs/install_amd.md stack: digest-pinned
|
||||
`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base =
|
||||
torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`,
|
||||
curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so`
|
||||
dev-symlink the pip-ROCm wheel lacks) serving
|
||||
**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards,
|
||||
standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no
|
||||
`input_scale`, loaded directly from HF safetensors, no FTW conversion) on
|
||||
**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run`
|
||||
through `scripts/serve-frank-primary.sh` under the root systemd unit
|
||||
Runbook:
|
||||
`serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md`.
|
||||
Rollback: `/home/dank/models/MiMo-Q6K/` (the LIVE payload until the cutover;
|
||||
revert to the 2×200,704 f16 shape — [[frank-small-llama-mimo-q6k-2x200704]])
|
||||
or `/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0;
|
||||
revert to 380aafa's tree).
|
||||
|
||||
### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
|
||||
287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
|
||||
entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
|
||||
requantisation, no calibration. Plain `podman run` through
|
||||
`scripts/serve-frank-primary.sh` under the root systemd unit
|
||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
|
||||
container name is the unit name (`frank-primary`; the vLLM-era
|
||||
`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05;
|
||||
rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio
|
||||
and its 362-line env lore live in that tag; the tcclaviger image stays in
|
||||
dank's rootless storage).
|
||||
container name is the unit name (`frank-primary`). Cutover 2026-10-05;
|
||||
rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
|
||||
+ Dockerfile are in the tag; the image and the orcarouter payload stay on
|
||||
disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.
|
||||
|
||||
**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a
|
||||
GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given
|
||||
the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM
|
||||
(→ root unit + memlock law), **KV pool `--num-tokens 299968`**
|
||||
(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be
|
||||
a multiple of 64** — the auto attention backend is `qsa_sparse`, which
|
||||
overrides page size to 64; 300,000 was refused at first boot), 2 decode
|
||||
slots, `--max-output-tokens 36864` (the harness maxTokens; requests that
|
||||
omit one get this default). **Parsers are AUTO**: thinking lands in
|
||||
`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml
|
||||
flips `reasoningContentField` to `reasoning_content` on the harness side.
|
||||
Vision works (models.yml keeps `input: [text, image]`); the vision tower's
|
||||
blocks stream from host RAM two at a time by default.
|
||||
**Payload chain**: official checkpoint → staging dir whose `config.json`
|
||||
drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
|
||||
value-preserving reshapes: experts byte-stacked into the plugin's stacked
|
||||
`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
|
||||
stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
|
||||
decoded to one bf16 shard with the checkpoint's own bf16 scale
|
||||
1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
|
||||
copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
|
||||
same fixed scale, an exact round-trip that restores the original codes and
|
||||
lands the plugin's one-element scale plane →
|
||||
`/home/dank/models/rad/qwen4exp-fp8-native.rad` (~161 GiB, floor
|
||||
`FRANK_PRIMARY_RAD_MIN_MIB=158000`). The staging drop is the plugin
|
||||
SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
|
||||
exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
|
||||
`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
|
||||
the selection key, per-weight encodings are data, so nothing else is
|
||||
requantised.
|
||||
Raw serve AND unmodified convert both refuse; that is the design. The
|
||||
checkpoint itself is sha256-verified against the Hub (131/131 shards,
|
||||
2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
|
||||
first — its declare phase is the engine's own, so a resolving plan boots).
|
||||
|
||||
**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health
|
||||
status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent
|
||||
measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU,
|
||||
and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT
|
||||
(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed
|
||||
2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw
|
||||
Prometheus llm-fleet scrape of :7770 reads down until repointed.
|
||||
`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a
|
||||
restart); `ft ctl stats` reports occupancy.
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
|
||||
+5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
|
||||
(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
|
||||
200000` (native 262,144; the harness contextWindow must quote this — a longer
|
||||
prompt is refused, never truncated), `--placement expert_tiered
|
||||
--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
|
||||
per rank slice — the pinned pool is the price of native precision),
|
||||
`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
|
||||
to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
|
||||
1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
|
||||
3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
|
||||
prefix cache 4 GiB pinned host + 64 GiB disk at
|
||||
`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
|
||||
PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
|
||||
host RAM. Froggeric v22.5 chat template vendored
|
||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).
|
||||
|
||||
**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with
|
||||
`{"status":"loading"}`** while the engine loads — the FastAPI front end
|
||||
binds before the scheduler is up, so a bare `curl` false-greens (measured:
|
||||
"healthy after 6 s" mid-page-in). The deployer loops until the body says
|
||||
`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate
|
||||
is `podman image exists` (a local build carries no RepoDigests — the old
|
||||
digest grep refused exactly this shape; pinning lives in the Dockerfile).
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
|
||||
47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
|
||||
`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
|
||||
pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
|
||||
TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
|
||||
both knobs are raised in the stop window (sysfs write first, kernel
|
||||
cmdline + reboot only if read-only).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
||||
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
||||
200 serving / 503 stopping); sampling params temperature…preserve_thinking
|
||||
accepted, **unknown params 400** (models.yml's extraBody set is fully
|
||||
inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
|
||||
`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
|
||||
chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
|
||||
@@ -950,8 +964,8 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
|
||||
| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` |
|
||||
| frank-small (llama.cpp CUDA, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1×131,072 q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md` |
|
||||
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -961,6 +975,9 @@ Retired (rollback targets and history):
|
||||
|---------|------|--------|---------|
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-small-llama-mimo-q6k-2x200704.md` |
|
||||
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-small`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-small-llama-swift15-iq3s-mtp-131072.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
|
||||
Reference in New Issue
Block a user