frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index
The parallel session's uncommitted swap, committed at Frank's direction (commit whatever's in the working tree): - llama.cpp master on the RTX 3090 now serves ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE 131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to rollback target. - env/serve/download + seam twins updated together (deployer payload gates rode the previous commit's deploy-remote.sh). - docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135 seam green). - comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/ workflows/ (12 KiB).
This commit is contained in:
Frank
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (~161 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, **2 × 200,000** (first boot; native 262,144), froggeric v22.5 template, vision + tools + `reasoning_content` auto, `/metrics` scraped. Cutover 2026-10-05 (rollback `frank-primary-freetoken-pr587-20261005`, FreeToken one-card image + payload on disk) | 7770 |
|
||||
| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; **MTP head IN the file** — `--spec-type draft-mtp`, depth 3) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **1 × 131,072** (Frank's 128k cap; a second 131,072 slot misses the 2-3 GiB floor) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024. Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -264,7 +264,7 @@ merely wedged kernel usually still answers ARP.
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-primary` (root unit, FreeToken on one R9700):
|
||||
- Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
|
||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
||||
2. Bench.
|
||||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
|
||||
@@ -299,8 +299,9 @@ a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
|
||||
image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
|
||||
ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
|
||||
`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision
|
||||
tower and streams its block weights from host RAM two at a time by default),
|
||||
`frank-primary` takes them natively (radiance runs the checkpoint's
|
||||
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
||||
are the engine core; `--mm-max-patches` available),
|
||||
and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
@@ -378,132 +379,145 @@ rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
**llama.cpp master** (CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
|
||||
**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
|
||||
sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
|
||||
projector is a hardlink of the Q5KM copy — the dir is self-contained), on
|
||||
port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
|
||||
reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
|
||||
one reason:
|
||||
**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
|
||||
together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
|
||||
from three slots to two made room for **fully unquantized V**
|
||||
(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
|
||||
2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
|
||||
file is the entire vision path (and ~1.4 GiB of the card).
|
||||
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
|
||||
(`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving `ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF`
|
||||
at **IQ3_S -mtp** (12,120,016,896 B, sha256 `9aecf1cd…`) **plus
|
||||
`mmproj-Qwen3.8-27B-F16.gguf`** (927,607,488 B, sha256 `cbb841a9…`), both
|
||||
`/home/dank/models/Swift15-IQ3S/`, on port 7771. Cutover 2026-10-05
|
||||
(evening) on Frank's explicit contract: **128k context cap, KV at FP8,
|
||||
vision on, as many 128k contexts as fit** — the answer is ONE, by
|
||||
arithmetic. The same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K
|
||||
shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa)
|
||||
but never deployed (blocked at the docs gate; the docs debt is what this
|
||||
section settles).
|
||||
|
||||
**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
|
||||
DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
|
||||
the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
|
||||
drives that term.
|
||||
**Architecture: qwen3_5 hybrid, bigger** — block_count 65 (64 real layers +
|
||||
the blk.64 MTP head), `full_attention_interval 4` ⇒ **16 full-attention +
|
||||
48 Gated DeltaNet**, native 262,144 window, dense. KV exists only in the 16
|
||||
attention layers (4 KV heads × 256 dim): 64 KiB/token at f16, **34 KiB/token
|
||||
at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at
|
||||
48 linear layers × 48 value heads). Header facts were read from the Hub
|
||||
file by HTTP range + struct parse before anything was downloaded.
|
||||
|
||||
**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
|
||||
conversations of 200,704 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
|
||||
**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
|
||||
2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
|
||||
runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
|
||||
+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
|
||||
which sits below either ubatch, did not move at all. The old "ubatch 2048
|
||||
failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
|
||||
generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
|
||||
free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
|
||||
byte-identically the same hour — the doubling costs **616 MiB** and lands at
|
||||
the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
|
||||
if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
|
||||
compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
|
||||
quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
|
||||
~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
|
||||
q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
|
||||
CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
|
||||
there). Three native windows need 18.4 GiB of KV alone and were never on the
|
||||
table.
|
||||
**The shape (2026-10-05):** `--ctx-size 131072 --parallel 1` = ONE
|
||||
conversation at Frank's 128k cap (Swift's own benchmark protocol ran at
|
||||
131,072), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's FP8
|
||||
class — no true fp8 KV dtype exists; 8.5 effective bits; the order
|
||||
SUPERSEDES the same-day f16/f16 rule written for the Ornith 9B fit), all
|
||||
layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024 (lower
|
||||
ubatch first, Frank's standing law; 2048 is the first A/B if boot lands
|
||||
high). Non-KV ≈ 15.5 GiB (weights 11,561 + projector ~1,350 + buffers
|
||||
~1.6k + CUDA ~500 + DeltaNet ~150/slot + MTP draft ctx ~0.3-0.5k); one q8_0
|
||||
slot = 4,352 MiB ⇒ free ≈ 4.5 GiB; a SECOND slot wants +4,352 ⇒ ~0.2 GiB —
|
||||
far under the 2-3 GiB floor. The pool is capped by the 128k law, not the
|
||||
card; leftover headroom is honest slack.
|
||||
|
||||
Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
|
||||
and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
|
||||
that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
|
||||
decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
|
||||
bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
|
||||
`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
|
||||
`image_url` content part (two traps: the legacy top-level `images` field is
|
||||
ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
|
||||
the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
|
||||
is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
|
||||
`contextWindow: 200704`.
|
||||
**Speculative decoding is the file's own MTP head**: the `-mtp` GGUF
|
||||
carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`), so
|
||||
`--spec-type draft-mtp --spec-draft-n-max 3` loads it against the main
|
||||
model's weights — no `--model-draft`, no second draft KV pool. Flags move
|
||||
as a set (a no-spec baseline strips BOTH). Acceptance reads from the
|
||||
response `timings` (`draft_n`/`draft_n_accepted`) on real traffic, never a
|
||||
synthetic repeat.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
|
||||
(the boot-pair table that prices pool changes lives in the retired
|
||||
[[frank-small-llama-mimo-q5km]]).
|
||||
Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
|
||||
buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
|
||||
measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
|
||||
pairs with Q5 only; SGLang rollback in
|
||||
[[frank-small-sglang-mimo-v26]] (venv on disk at
|
||||
`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
|
||||
**Vision without a Swift projector**: no Swift repo ships one (ukisai's
|
||||
README is explicit), so the projector is unsloth's mmproj built from the
|
||||
SHARED BASE `Qwen/Qwen3.8-27B` — legal because Swift-1.5's vision stack was
|
||||
hash-proven byte-identical to the base (333/333 tensor names, 6/6 sampled
|
||||
tensors: merger fc1/fc2, blocks 0/13/26) on 2026-10-05. The probe traps
|
||||
carry over: ≥256 px probe images or the model answers "video"; only the
|
||||
OpenAI `image_url` content part reaches the model.
|
||||
|
||||
### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
|
||||
The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** of stock quant
|
||||
types (IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …)
|
||||
— stock llama.cpp loads it; the tensor-type histogram is the law (see Key
|
||||
Lessons). Publisher KLD vs Swift BF16: 0.0513 development / 0.024–0.052
|
||||
held-out; the MTP packaging is NOT separately evaluated by the publisher.
|
||||
|
||||
The **FreeToken engine at PR #587 head `77a7201`** (branch
|
||||
`rocm714_torch214`, built on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own
|
||||
docs/install_amd.md stack: digest-pinned
|
||||
`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base =
|
||||
torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`,
|
||||
curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so`
|
||||
dev-symlink the pip-ROCm wheel lacks) serving
|
||||
**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards,
|
||||
standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no
|
||||
`input_scale`, loaded directly from HF safetensors, no FTW conversion) on
|
||||
**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run`
|
||||
through `scripts/serve-frank-primary.sh` under the root systemd unit
|
||||
Runbook:
|
||||
`serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md`.
|
||||
Rollback: `/home/dank/models/MiMo-Q6K/` (the LIVE payload until the cutover;
|
||||
revert to the 2×200,704 f16 shape — [[frank-small-llama-mimo-q6k-2x200704]])
|
||||
or `/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0;
|
||||
revert to 380aafa's tree).
|
||||
|
||||
### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
|
||||
287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
|
||||
entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
|
||||
requantisation, no calibration. Plain `podman run` through
|
||||
`scripts/serve-frank-primary.sh` under the root systemd unit
|
||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
|
||||
container name is the unit name (`frank-primary`; the vLLM-era
|
||||
`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05;
|
||||
rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio
|
||||
and its 362-line env lore live in that tag; the tcclaviger image stays in
|
||||
dank's rootless storage).
|
||||
container name is the unit name (`frank-primary`). Cutover 2026-10-05;
|
||||
rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
|
||||
+ Dockerfile are in the tag; the image and the orcarouter payload stay on
|
||||
disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.
|
||||
|
||||
**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a
|
||||
GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given
|
||||
the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM
|
||||
(→ root unit + memlock law), **KV pool `--num-tokens 299968`**
|
||||
(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be
|
||||
a multiple of 64** — the auto attention backend is `qsa_sparse`, which
|
||||
overrides page size to 64; 300,000 was refused at first boot), 2 decode
|
||||
slots, `--max-output-tokens 36864` (the harness maxTokens; requests that
|
||||
omit one get this default). **Parsers are AUTO**: thinking lands in
|
||||
`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml
|
||||
flips `reasoningContentField` to `reasoning_content` on the harness side.
|
||||
Vision works (models.yml keeps `input: [text, image]`); the vision tower's
|
||||
blocks stream from host RAM two at a time by default.
|
||||
**Payload chain**: official checkpoint → staging dir whose `config.json`
|
||||
drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
|
||||
value-preserving reshapes: experts byte-stacked into the plugin's stacked
|
||||
`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
|
||||
stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
|
||||
decoded to one bf16 shard with the checkpoint's own bf16 scale
|
||||
1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
|
||||
copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
|
||||
same fixed scale, an exact round-trip that restores the original codes and
|
||||
lands the plugin's one-element scale plane →
|
||||
`/home/dank/models/rad/qwen4exp-fp8-native.rad` (~161 GiB, floor
|
||||
`FRANK_PRIMARY_RAD_MIN_MIB=158000`). The staging drop is the plugin
|
||||
SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
|
||||
exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
|
||||
`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
|
||||
the selection key, per-weight encodings are data, so nothing else is
|
||||
requantised.
|
||||
Raw serve AND unmodified convert both refuse; that is the design. The
|
||||
checkpoint itself is sha256-verified against the Hub (131/131 shards,
|
||||
2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
|
||||
first — its declare phase is the engine's own, so a resolving plan boots).
|
||||
|
||||
**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health
|
||||
status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent
|
||||
measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU,
|
||||
and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT
|
||||
(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed
|
||||
2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw
|
||||
Prometheus llm-fleet scrape of :7770 reads down until repointed.
|
||||
`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a
|
||||
restart); `ft ctl stats` reports occupancy.
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
|
||||
+5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
|
||||
(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
|
||||
200000` (native 262,144; the harness contextWindow must quote this — a longer
|
||||
prompt is refused, never truncated), `--placement expert_tiered
|
||||
--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
|
||||
per rank slice — the pinned pool is the price of native precision),
|
||||
`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
|
||||
to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
|
||||
1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
|
||||
3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
|
||||
prefix cache 4 GiB pinned host + 64 GiB disk at
|
||||
`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
|
||||
PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
|
||||
host RAM. Froggeric v22.5 chat template vendored
|
||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).
|
||||
|
||||
**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with
|
||||
`{"status":"loading"}`** while the engine loads — the FastAPI front end
|
||||
binds before the scheduler is up, so a bare `curl` false-greens (measured:
|
||||
"healthy after 6 s" mid-page-in). The deployer loops until the body says
|
||||
`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate
|
||||
is `podman image exists` (a local build carries no RepoDigests — the old
|
||||
digest grep refused exactly this shape; pinning lives in the Dockerfile).
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
|
||||
47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
|
||||
`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
|
||||
pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
|
||||
TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
|
||||
both knobs are raised in the stop window (sysfs write first, kernel
|
||||
cmdline + reboot only if read-only).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
||||
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
||||
200 serving / 503 stopping); sampling params temperature…preserve_thinking
|
||||
accepted, **unknown params 400** (models.yml's extraBody set is fully
|
||||
inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
|
||||
`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
|
||||
chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
|
||||
@@ -950,8 +964,8 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
|
||||
| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` |
|
||||
| frank-small (llama.cpp CUDA, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1×131,072 q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md` |
|
||||
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -961,6 +975,9 @@ Retired (rollback targets and history):
|
||||
|---------|------|--------|---------|
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-small-llama-mimo-q6k-2x200704.md` |
|
||||
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-small`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-small-llama-swift15-iq3s-mtp-131072.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"3": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"_meta": { "title": "Positive prompt" },
|
||||
"inputs": {
|
||||
"text": "A detailed illustration of a 1930s Soviet industrial design bureau at night: a well-measured engineer at a drafting table, a warm desk lamp, rolled blueprints, gears and calipers on the desk, deep red and cream palette, gouache painterly style, intricate mechanical detail, soft shadows",
|
||||
"clip": [ "6", 0 ]
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"_meta": { "title": "Negative prompt" },
|
||||
"inputs": {
|
||||
"text": "lowres, blurry, deformed, extra limbs, watermark, signature, jpeg artifacts",
|
||||
"clip": [ "6", 0 ]
|
||||
}
|
||||
},
|
||||
"5": {
|
||||
"class_type": "UNETLoader",
|
||||
"_meta": { "title": "Qwen Image 2.1 (int8)" },
|
||||
"inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
|
||||
},
|
||||
"6": {
|
||||
"class_type": "CLIPLoader",
|
||||
"_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
|
||||
"inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
|
||||
},
|
||||
"7": {
|
||||
"class_type": "VAELoader",
|
||||
"_meta": { "title": "VAE" },
|
||||
"inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
|
||||
},
|
||||
"8": {
|
||||
"class_type": "EmptySD3LatentImage",
|
||||
"_meta": { "title": "Empty Latent 1024x1024" },
|
||||
"inputs": { "width": 1024, "height": 1024, "batch_size": 1 }
|
||||
},
|
||||
"9": {
|
||||
"class_type": "KSampler",
|
||||
"_meta": { "title": "KSampler" },
|
||||
"inputs": {
|
||||
"model": [ "5", 0 ],
|
||||
"seed": 1930,
|
||||
"steps": 20,
|
||||
"cfg": 1.0,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "simple",
|
||||
"positive": [ "3", 0 ],
|
||||
"negative": [ "4", 0 ],
|
||||
"latent_image": [ "8", 0 ],
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "VAEDecode",
|
||||
"_meta": { "title": "VAE Decode" },
|
||||
"inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
|
||||
},
|
||||
"11": {
|
||||
"class_type": "SaveImage",
|
||||
"_meta": { "title": "Save Image (art)" },
|
||||
"inputs": { "filename_prefix": "qwen21-art", "images": [ "10", 0 ] }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"3": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"_meta": { "title": "Positive prompt (legible text)" },
|
||||
"inputs": {
|
||||
"text": "Clean minimalist movie-poster design, dark charcoal background with a subtle vintage grain. Centered large bold white sans-serif text reading \"SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS\", perfectly legible, crisp letterforms, even letter spacing. Below it, a smaller line of yellow text reading \"DANKCLAW DANKSERVER EST 2026\", equally legible. Flat graphic design, high contrast, generous margins, no other words, no watermark",
|
||||
"clip": [ "6", 0 ]
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"_meta": { "title": "Negative prompt" },
|
||||
"inputs": {
|
||||
"text": "blurry, illegible, misspelled, warped letters, extra text, watermark, signature, lowres, jpeg artifacts",
|
||||
"clip": [ "6", 0 ]
|
||||
}
|
||||
},
|
||||
"5": {
|
||||
"class_type": "UNETLoader",
|
||||
"_meta": { "title": "Qwen Image 2.1 (int8)" },
|
||||
"inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
|
||||
},
|
||||
"6": {
|
||||
"class_type": "CLIPLoader",
|
||||
"_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
|
||||
"inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
|
||||
},
|
||||
"7": {
|
||||
"class_type": "VAELoader",
|
||||
"_meta": { "title": "VAE" },
|
||||
"inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
|
||||
},
|
||||
"8": {
|
||||
"class_type": "EmptySD3LatentImage",
|
||||
"_meta": { "title": "Empty Latent 832x1216 poster" },
|
||||
"inputs": { "width": 832, "height": 1216, "batch_size": 1 }
|
||||
},
|
||||
"9": {
|
||||
"class_type": "KSampler",
|
||||
"_meta": { "title": "KSampler" },
|
||||
"inputs": {
|
||||
"model": [ "5", 0 ],
|
||||
"seed": 2026,
|
||||
"steps": 25,
|
||||
"cfg": 1.0,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "simple",
|
||||
"positive": [ "3", 0 ],
|
||||
"negative": [ "4", 0 ],
|
||||
"latent_image": [ "8", 0 ],
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "VAEDecode",
|
||||
"_meta": { "title": "VAE Decode" },
|
||||
"inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
|
||||
},
|
||||
"11": {
|
||||
"class_type": "SaveImage",
|
||||
"_meta": { "title": "Save Image (text poster)" },
|
||||
"inputs": { "filename_prefix": "qwen21-text", "images": [ "10", 0 ] }
|
||||
}
|
||||
}
|
||||
@@ -1,32 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-small payload for llama.cpp on the RTX 3090:
|
||||
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 plus its vision projector
|
||||
# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
|
||||
# without this file the endpoint is text-only by construction). Public repo —
|
||||
# no auth header, and never a token on the command line (it is world-readable
|
||||
# in /proc/<pid>/cmdline).
|
||||
# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **with the MTP head in the
|
||||
# file** (the -mtp variant), plus a vision projector. No Swift repo ships a
|
||||
# projector — ukisai's release is language-model-only ("A Swift 27B vision
|
||||
# projector has not been verified for this release") — so this script fetches
|
||||
# unsloth's mmproj-F16.gguf built from the shared base Qwen/Qwen3.8-27B.
|
||||
# That substitution is proven, not assumed: on 2026-10-05 Swift-1.5's vision
|
||||
# stack was hash-compared against the base over HTTP range reads — 333/333
|
||||
# visual tensor names and 6/6 sampled tensors byte-identical (merger fc1/fc2,
|
||||
# blocks 0/13/26) — so the base-model projector IS the model's own projector.
|
||||
# Public repos — no auth header, and never a token on the command line (it is
|
||||
# world-readable in /proc/<pid>/cmdline).
|
||||
#
|
||||
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
|
||||
# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
|
||||
# on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a success,
|
||||
# then byte count and the Hub LFS sha256. Idempotent: complete files are
|
||||
# skipped.
|
||||
# skipped. The projector is stored under a self-documenting dest name; the
|
||||
# URL name and sha stay pinned in the table below.
|
||||
#
|
||||
# 2026-10-05 (Frank): the rung is Ornith-1.5-9B Q8_0. The payload moved off
|
||||
# MiMo-V2.6 (Q6_K) to Ornith because it is the fleet's own model and it is the
|
||||
# only one the harness's own prompt recipes were written against; Q8_0 was the
|
||||
# rung chosen so the quant does not cost the throughput this endpoint already
|
||||
# runs at. The Q5_K_M file this script served from 2026-10-02 stays on disk at
|
||||
# /home/dank/models/MiMo-Q5KM/ (6876124704 B, LFS oid
|
||||
# 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f) as the
|
||||
# rollback target; re-fetch it from the same repo with the same recipe.
|
||||
# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5-27B GSQ-RCO
|
||||
# IQ3_S-mtp — a 27B on the 24 GiB card: ONE 131,072-token slot, q8_0 K + q8_0
|
||||
# V, the in-file MTP head drafting 3. This supersedes the Ornith-1.5-9B Q8_0
|
||||
# swap (commits 77decb5..380aafa, same day), which was committed but never
|
||||
# deployed — the live service was still MiMo-V2.6 Q6_K when Swift cut over.
|
||||
# Rollback payloads stay on disk: /home/dank/models/Ornith-Q8/ and
|
||||
# /home/dank/models/MiMo-Q6K/ (the payload that was live until this swap).
|
||||
# Their re-fetch recipes are in git history (predecessors of this script).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="ornith-ai/Ornith-1.5-9B-GGUF"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
@@ -35,19 +39,20 @@ for arg in "$@"; do
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/Ornith-Q8}"
|
||||
DEST="${DEST:-/home/dank/models/Swift15-IQ3S}"
|
||||
|
||||
# name bytes sha256 (the sha256 is the LFS oid in the repo's pointer file)
|
||||
# repo | hub-name | dest-name | bytes | sha256 (the Hub LFS oid)
|
||||
FILES=(
|
||||
"Ornith-1.5-9B-Q8_0.gguf 9786060384 22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010"
|
||||
"mmproj-Ornith-1.5-9B-BF16.gguf 921704672 626f9f90627402a6bf4a999111d0fbd69b5fcca7aa8ba089d69e5f10e8858e1d"
|
||||
"ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF|Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf|Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf|12120016896|9aecf1cd41b2cb2f32a74e0d889e33855ebef43b26f43b43feb5720239e677e5"
|
||||
"unsloth/Qwen3.8-27B-GGUF|mmproj-F16.gguf|mmproj-Qwen3.8-27B-F16.gguf|927607488|cbb841a9ee0636b2ec172f5bb8df2ea8dfeb01e90fe7c6126581d662a0b4e43e"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
IFS='|' read -r repo hubname destname size sha <<<"$entry"
|
||||
url="https://huggingface.co/${repo}/resolve/main/${hubname}"
|
||||
f="$DEST/$destname"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
@@ -55,29 +60,29 @@ for entry in "${FILES[@]}"; do
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
echo "OK $destname"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
echo "MISSING $destname"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
echo "FETCHING $destname (from $repo)"
|
||||
tmp="$f.part"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$url"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
echo "SHA FAIL $destname"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
echo "VERIFIED $destname"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
|
||||
+83
-116
@@ -1,79 +1,64 @@
|
||||
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
||||
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
|
||||
# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **-mtp** on the RTX 3090,
|
||||
# :7771.
|
||||
#
|
||||
# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
|
||||
# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
|
||||
# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
|
||||
# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
|
||||
# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
|
||||
# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
|
||||
# the file — capability only; no spec decode ships, same law as before).
|
||||
# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5 Qwen3.8-27B
|
||||
# GSQ-RCO IQ3_S WITH the MTP head in the file, each conversation capped at
|
||||
# 131,072 tokens, KV at FP8 class, vision on. Header-verified facts that
|
||||
# re-price the shape (read from the Hub file 2026-10-05, HTTP range + struct
|
||||
# parse): qwen35 hybrid like MiMo/Ornith but bigger — block_count 65 (64 real
|
||||
# layers + the blk.64 MTP head), full_attention_interval 4 => SIXTEEN
|
||||
# full-attention layers (MiMo had 8, Ornith 9), 24 Q heads / 4 KV heads,
|
||||
# k/v length 256, native window 262144, dense, nextn_predict_layers=1 with
|
||||
# blk.64.nextn.{enorm,hnorm,shared_head_norm} (the -mtp file's head tensors
|
||||
# are Q6_K/F32 per the publisher RECIPE, "MTP_OK" verified).
|
||||
#
|
||||
# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
|
||||
# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
|
||||
# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
|
||||
# for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
|
||||
# full-attn layer AND one fewer KV head).
|
||||
# Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
|
||||
# 4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
|
||||
# the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
|
||||
# the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
|
||||
# on the live card is the accepted method for llama.cpp sizing here: boot,
|
||||
# read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
|
||||
# each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
|
||||
# buffers allocate at graph capture during BOOT, so a failure is loud and
|
||||
# fast, not a surprise at the first long prompt.
|
||||
# THE KV CONTRACT IS NOW q8_0 K + q8_0 V — Frank's order for THIS swap
|
||||
# ("k/v at FP8", 2026-10-05), superseding the same-day f16/f16 rule, which
|
||||
# was written for the Ornith 9B fit ("no further quantization anywhere on
|
||||
# this endpoint"). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit
|
||||
# class (1.0625 B/elem => 8.5 effective bits) and the only FP8-class lever
|
||||
# this runtime moves. Arithmetic that makes the order load-bearing:
|
||||
# KV rate = 16 attn layers x 4 heads x 256 x 1.0625 B, K and V again
|
||||
# => 34 KiB/token (f16 would be 64 KiB/token — halogen's 27B card quotes
|
||||
# the same f16 number; at f16 the 131,072 slot alone would cost 8.0 GiB).
|
||||
#
|
||||
# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
|
||||
# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
|
||||
# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
|
||||
# of weights.
|
||||
# Why ONE slot at 131,072 is the maximum (Frank's cap per conversation):
|
||||
# card 24,576 MiB. Weights 11,561 MiB (12,120,016,896 B, the IQ3_S-mtp
|
||||
# file; the GSQ "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram
|
||||
# IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ...
|
||||
# all stock types, stock llama.cpp loads it). Projector ~1,350 MiB
|
||||
# (927,607,488 B F16 file at the measured ~1.45x card law). Compute
|
||||
# buffers at ubatch 1024 ~1.6 GiB. CUDA ctx ~500 MiB. DeltaNet state
|
||||
# ~150 MiB/slot (48 linear layers x 48 value heads x 128x128 x 4 B).
|
||||
# MTP draft context ~0.3-0.5 GiB (one nextn layer over the slot window;
|
||||
# the boot ledger is the truth). Non-KV ~= 15.5 GiB. One q8_0 slot =
|
||||
# 4,352 MiB => free ~= 4.5 GiB. A SECOND slot wants +4,352 MiB => free
|
||||
# ~0.2 GiB, far below the 2-3 GiB floor. "As many 128k contexts as fit"
|
||||
# = 1. MEASURE nvidia-smi free at boot (quote the free field): under
|
||||
# ~1.5 GiB or a boot failure, step the pool down in 12,288-token
|
||||
# increments (~410 MiB each at this rate). Compute buffers allocate at
|
||||
# graph capture during BOOT, so a failure is loud and fast.
|
||||
#
|
||||
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
|
||||
# tokens per conversation, above the 200,700 the harness compacts at — and the
|
||||
# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
|
||||
# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
|
||||
# up is exactly the VRAM that pays for unquantized V. No further quantization
|
||||
# anywhere on this endpoint.
|
||||
#
|
||||
# Three measurements on this card, this runtime, price the terms:
|
||||
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
||||
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
||||
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
||||
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
||||
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
||||
# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
|
||||
# shape) => non-KV = 8,748 MiB
|
||||
# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
|
||||
# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
|
||||
# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
|
||||
# shape hit, bought with precision instead of a slot. The 1024 ubatch
|
||||
# re-prices the compute buffers upward, so MEASURE after any change: if the
|
||||
# boot lands under 2 GiB free, step the pool down in 12,288-token
|
||||
# (384 MiB at this rate) increments.
|
||||
#
|
||||
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
||||
# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
|
||||
# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
|
||||
# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
|
||||
# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
|
||||
# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
|
||||
# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
|
||||
# precision buys; q8_0 on V would buy 262,144 instead.
|
||||
#
|
||||
# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
|
||||
# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
|
||||
# and the third slot is gone. The flags stay explicit so the choice is visible
|
||||
# in the argv either way.
|
||||
# Payload history (for the rollback trail): the Ornith-1.5-9B Q8_0 swap of
|
||||
# earlier on 2026-10-05 (commits 77decb5..380aafa) was committed but never
|
||||
# deployed — the live service was still MiMo-V2.6 Q6_K (2 x 200,704, f16
|
||||
# KV) when Swift cut over. Rollback payloads on disk: Ornith-Q8/ and
|
||||
# MiMo-Q6K/ (one MODEL_FILE + shape edit back; see the runbook).
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||
MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
|
||||
MODEL_FILE=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
|
||||
|
||||
# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
|
||||
# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
|
||||
# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
|
||||
# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
|
||||
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
||||
MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
|
||||
# Vision: no Swift repo ships a projector (ukisai's release is
|
||||
# language-model-only by their own README), so the projector comes from the
|
||||
# SHARED BASE — unsloth's mmproj-F16.gguf built from Qwen/Qwen3.8-27B,
|
||||
# stored as mmproj-Qwen3.8-27B-F16.gguf (927,607,488 B, LFS sha256
|
||||
# cbb841a9..., fetched by scripts/download-frank-small.sh). The substitution
|
||||
# is proven, not assumed: Swift-1.5's vision stack (tower + merger) was
|
||||
# hash-compared against the base on 2026-10-05 — 333/333 visual tensor names
|
||||
# and 6/6 sampled tensors byte-identical — so the base projector IS the
|
||||
# model's own projector (Swift's RL/OPD post-training touched text only).
|
||||
# Empty string = text-only on purpose.
|
||||
MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf
|
||||
|
||||
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
||||
# harness models.yml id and the homepage probe do not move.
|
||||
@@ -81,56 +66,38 @@ ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
|
||||
# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
|
||||
# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
|
||||
# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
|
||||
# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
|
||||
# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
|
||||
# pool step fits BOTH the 2-3 GiB floor and the harness compaction
|
||||
# invariant (slot >= 191,148). One native slot fits with room; the window
|
||||
# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
|
||||
# request queues behind the released slot (subagent concurrency 2 -> 1).
|
||||
CTX_SIZE=262144
|
||||
# ONE conversation at Frank's 128k cap: --ctx-size is the POOL llama.cpp
|
||||
# divides across slots; 131072 / 1 = 131,072 per conversation. The cap is a
|
||||
# product decision (Swift's own benchmark protocol ran at 131,072), and a
|
||||
# second 131,072 slot does not fit the 2-3 GiB floor (arithmetic above).
|
||||
CTX_SIZE=131072
|
||||
PARALLEL=1
|
||||
|
||||
# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
|
||||
# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
|
||||
# Both are tunables because the pair is a measurement, not a law — and the
|
||||
# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
|
||||
# curve peaks at 512 and every step up costs GiB of compute buffer for no
|
||||
# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
|
||||
# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
|
||||
# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
|
||||
# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
|
||||
#
|
||||
# The logical batch is free — two independent sweeps agree that at a fixed
|
||||
# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
|
||||
# because the compute buffers are sized by the ubatch alone. Doubling it costs
|
||||
# nothing and buys nothing; it is here because it is the value the pair was
|
||||
# measured at.
|
||||
#
|
||||
# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
|
||||
# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
|
||||
# smaller compute tensors, so the delta should be well under that, but the
|
||||
# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
|
||||
# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
|
||||
# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
|
||||
# claw back, before any pool move). Compute buffers are allocated at graph
|
||||
# capture during BOOT, so a failure is loud and fast, not a surprise at the
|
||||
# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
|
||||
# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
|
||||
# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
|
||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
|
||||
# on this CUDA 3090 the curve climbs monotonically to 2048, but Frank's
|
||||
# standing law for a resize is LOWER UBATCH FIRST (the measured 616 MiB rung
|
||||
# on the 9B; the 27B's buffers are larger, so the claw-back is worth more).
|
||||
# The pair is a measurement, not a law: if boot lands well above the floor,
|
||||
# ubatch 2048 is the first thing to A/B for prefill (loud failure at graph
|
||||
# capture if it does not fit).
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
|
||||
# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
|
||||
# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
|
||||
# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
|
||||
# guess-and-check on the live card is the accepted method: boot, read
|
||||
# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
|
||||
# in the serve script: it is the fast path for f16 KV.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
|
||||
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
|
||||
# the argv either way, and --flash-attn stays pinned on in the serve script:
|
||||
# it is required for quantized V (the build carries
|
||||
# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
|
||||
# not a silent f16 fallback).
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
# Speculative decoding: the MTP head is IN the -mtp file (blk.64.nextn.*,
|
||||
# nextn_predict_layers=1), so draft-mtp needs no second model and no second
|
||||
# KV pool — the head shares the main model's weights. Depth 3 matches the
|
||||
# fleet's radiance MTP depth; measure draft acceptance from the response
|
||||
# timings object and retune here if the head earns more.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
|
||||
NGL=99
|
||||
|
||||
@@ -1,20 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve ornith-ai/Ornith-1.5-9B-GGUF (Q8_0) as frank-small on
|
||||
# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
# Serve ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF (IQ3_S **-mtp**) as
|
||||
# frank-small on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA
|
||||
# sm_86).
|
||||
#
|
||||
# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
|
||||
# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
|
||||
# runtime on this host that moves K and V independently — the asymmetric
|
||||
# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
|
||||
# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
|
||||
# unquantized too. The per-tensor flags stay explicit either way.
|
||||
# 2026-10-05 evening (Frank): a 27B on the 24 GiB card — ONE 131,072-token
|
||||
# conversation (Frank's cap), KV at FP8 class (q8_0 both halves), the MTP
|
||||
# head that lives IN the -mtp file drafting 3, vision on. The q8_0 KV
|
||||
# contract supersedes the same-day f16/f16 rule, which was written for the
|
||||
# Ornith 9B fit; the per-tensor flags stay explicit either way.
|
||||
#
|
||||
# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
|
||||
# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
|
||||
# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
|
||||
# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
|
||||
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
|
||||
# text-only opt-out.
|
||||
# Architecture: qwen35 hybrid — 64 real layers, 16 full-attention + 48
|
||||
# Gated DeltaNet (full_attention_interval 4), native 262144 window, plus
|
||||
# the blk.64 MTP head (nextn_predict_layers=1). KV lives only in the 16
|
||||
# attention layers (34 KiB/token at q8_0); the DeltaNet state is per-slot
|
||||
# fixed size. Vision comes from the projector file (the GGUF carries no
|
||||
# vision tensors; unsloth's mmproj built from the shared Qwen3.8-27B base —
|
||||
# proven byte-identical to Swift's own tower, see download-frank-small.sh),
|
||||
# so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the
|
||||
# documented text-only opt-out.
|
||||
#
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends its
|
||||
# recipe from models.yml); greedy is the harness default.
|
||||
@@ -33,31 +36,37 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
|
||||
# 200,704 per slot — two independent conversations, each one wide enough to
|
||||
# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
|
||||
# no longer fits twice on a 24 GiB card once the vision tower is loaded;
|
||||
# frank-small.env carries the three measurements that set this number).
|
||||
: "${CTX_SIZE:=401408}"
|
||||
: "${PARALLEL:=2}"
|
||||
# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
|
||||
# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
|
||||
# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
|
||||
# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
|
||||
# and the logical batch is VRAM- and throughput-neutral. frank-small.env
|
||||
# carries the sweep tables and the VRAM risk note.
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 131072 / 1 gives
|
||||
# 131,072 per conversation — Frank's 128k cap. A second 131,072 slot does
|
||||
# not fit the 2-3 GiB floor on the 24 GiB card with the 27B payload
|
||||
# (frank-small.env carries the arithmetic); "as many 128k contexts as fit"
|
||||
# = 1.
|
||||
: "${CTX_SIZE:=131072}"
|
||||
: "${PARALLEL:=1}"
|
||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves),
|
||||
# and Frank's standing law for a resize is LOWER UBATCH FIRST; the shipped
|
||||
# pair is 4096/1024. If boot lands well above the 2-3 GiB floor, ubatch
|
||||
# 2048 is the first A/B for prefill (a miss fails loud at graph capture).
|
||||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch.
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits;
|
||||
# there is no true fp8 KV dtype). Explicit either way so the contract is
|
||||
# visible in the argv.
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
# Speculative decoding: draft-mtp loads the MTP head that is IN the -mtp
|
||||
# GGUF; the flags move as a set.
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${NGL:=99}"
|
||||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||||
: "${MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
|
||||
: "${MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf}"
|
||||
|
||||
# Fail loudly rather than half-loading: a missing file is a download that
|
||||
# has not finished, and Restart=always would turn it into a crash loop that
|
||||
@@ -71,14 +80,21 @@ if [[ -n "$MMPROJ_FILE" ]]; then
|
||||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||||
fi
|
||||
|
||||
# The MTP head ships IN the -mtp GGUF (blk.64.nextn.*,
|
||||
# nextn_predict_layers=1): draft-mtp loads it against the main model's own
|
||||
# weights — no --model-draft and no second KV pool. Speculative flags move
|
||||
# as a set: a no-spec baseline must strip BOTH of these, not just one.
|
||||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
|
||||
# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
|
||||
# ever set back to q8_0 this flag must not be lost, or the build falls back
|
||||
# silently and the KV contract in frank-small.env becomes a lie.
|
||||
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
|
||||
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
|
||||
# the q8_0 KV contract in frank-small.env is a real path — without this
|
||||
# flag or that build option the engine silently falls back and the
|
||||
# contract becomes a lie.
|
||||
ARGS=(
|
||||
--model "$MODEL_FILE"
|
||||
--alias "$ALIAS"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
|
||||
description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
|
||||
description: "The RETIRED frank-small runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -14,13 +14,15 @@ timestamp: 2026-10-03
|
||||
|
||||
# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
|
||||
> this card: the freed third slot paid for **exact KV values**
|
||||
> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
|
||||
> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
|
||||
> — the same file this endpoint served on 2026-10-01, come back as an
|
||||
> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
|
||||
> only the weights moved up.
|
||||
> **RETIRED 2026-10-05 (evening).** Superseded by
|
||||
> [[frank-small-llama-swift15-iq3s-mtp-131072]] — the Swift-1.5-27B GSQ-RCO
|
||||
> IQ3_S-mtp swap. Two facts for the trail: (1) the Ornith-1.5-9B Q8_0 swap
|
||||
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
|
||||
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
|
||||
> contract Frank set here (f16 K + f16 V, "no further quantization on this
|
||||
> endpoint") was superseded the same evening by the Swift swap's explicit
|
||||
> q8_0/q8_0 order. This page stays as the rollback and the measured
|
||||
> boot-pair record. Payload on disk: `/home/dank/models/MiMo-Q6K/`.
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-10-05
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening).** A 27B replaces the 9B on the same
|
||||
> 24 GiB card, bought by Frank's explicit contract for this swap: context
|
||||
> capped at 128k per conversation, KV at FP8 class, vision on, "as many of
|
||||
> those contexts as you can" — the answer turned out to be **one**, by
|
||||
> arithmetic, not by choice (see *Sizing*). This supersedes BOTH earlier
|
||||
> 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape
|
||||
> ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the Ornith-1.5-9B
|
||||
> Q8_0 swap that was committed (77decb5..380aafa) but never deployed.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) |
|
||||
| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out) |
|
||||
|
||||
## Why a base-model projector is legal here
|
||||
|
||||
No Swift repo ships a projector — ukisai's GSQ-RCO release is
|
||||
language-model-only by their own README ("A Swift 27B vision projector has
|
||||
not been verified for this release"). The substitution is proven, not
|
||||
assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its
|
||||
vision stack was hash-compared against the base on 2026-10-05 over HTTP
|
||||
range reads of the two BF16 repos — **333/333 visual tensor names match and
|
||||
6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`,
|
||||
blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the
|
||||
base projector IS the model's own projector. The mmproj header matches
|
||||
Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16,
|
||||
merge 2, ffn 4304, 16 heads, out 5120).
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 131072 --parallel 1` — ONE conversation at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072). `CTX_SIZE`
|
||||
stays divisible by `PARALLEL`; the seam enforces it.
|
||||
|
||||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||||
⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The
|
||||
order supersedes the same-day f16/f16 rule, which was written for the
|
||||
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
|
||||
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
|
||||
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
|
||||
pinned: quantized V requires it, and the build's
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a
|
||||
silent f16 fallback.
|
||||
|
||||
**One slot is the maximum, by arithmetic.** Card 24,576 MiB; weights
|
||||
11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card
|
||||
law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet
|
||||
state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP
|
||||
draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒
|
||||
free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under
|
||||
the 2–3 GiB floor. The pool is capped by the 128k law, not by the card —
|
||||
leftover headroom is honest slack, not an undersized pool.
|
||||
|
||||
## MTP (speculative decoding)
|
||||
|
||||
The MTP head ships **in the file** — the `-mtp` GGUF carries
|
||||
`blk.64.nextn.{enorm,hnorm,shared_head_norm}` + the head layer's tensors at
|
||||
Q6_K/F32 (publisher RECIPE: "MTP_OK verified"),
|
||||
`qwen35.nextn_predict_layers=1`. `--spec-type draft-mtp` loads it against
|
||||
the main model's own weights — no `--model-draft`, no second draft KV pool
|
||||
(an MTP head can NEVER load as a separate draft model on this stack; two
|
||||
context-backed drafters cannot share a process). Depth 3
|
||||
(`--spec-draft-n-max 3`) matches the fleet's radiance MTP depth. The flags
|
||||
move as a set: a no-spec baseline must strip BOTH. Acceptance reads from
|
||||
the response `timings` object (`draft_n`, `draft_n_accepted`) — measure on
|
||||
real traffic, never on a synthetic repeat (greedy-text acceptance lies).
|
||||
|
||||
## Vision contract
|
||||
|
||||
`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
|
||||
Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
|
||||
probe undersupplies image tokens and the model invents ("video"); the
|
||||
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
|
||||
content part reaches the model. Same traps the MiMo eras documented.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking
|
||||
tokens COUNT against `max_tokens`.
|
||||
- The embedded chat template gates `reasoning_effort` to
|
||||
`{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
|
||||
`models.yml` `compat.reasoningEffortMap` folds the harness levels onto
|
||||
that set (mandatory, same as every Qwen3.8-template endpoint).
|
||||
- Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0
|
||||
/ top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly
|
||||
this recipe at ctx 131,072).
|
||||
- Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking
|
||||
tokens vs base at equal-or-better accuracy. Token budgets go further;
|
||||
the 32,768 `maxTokens` is unchanged from the MiMo era and still clears
|
||||
the compaction invariant (91,750 + 32,768 ≤ 131,072).
|
||||
- The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 /
|
||||
IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant
|
||||
types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513
|
||||
(development), held-out 0.024–0.052 per domain; the MTP packaging is NOT
|
||||
separately evaluated by the publisher.
|
||||
|
||||
## Rollback
|
||||
|
||||
Two payloads on disk, both fetchable by the same script's git-history
|
||||
recipes:
|
||||
|
||||
1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||||
this cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||||
2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||||
262,144 slot, f16/f16 KV).
|
||||
|
||||
## Trail
|
||||
|
||||
- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
|
||||
committed and pushed but its deploy never completed (blocked at the
|
||||
whole-tree docs gate); the live service was still MiMo Q6_K when Swift
|
||||
cut over. Both Ornith payloads were verified on disk before Swift.
|
||||
- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
|
||||
read from the Hub file by HTTP range + struct parse — no download
|
||||
needed — before the payload was fetched.
|
||||
@@ -1,25 +1,24 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
|
||||
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
|
||||
image `frank-primary-freetoken:pr587-77a7201` built on the host from
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
|
||||
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
|
||||
driving `podman run` directly, serving orcarouter
|
||||
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
|
||||
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
|
||||
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
|
||||
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
|
||||
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
|
||||
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw
|
||||
runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||||
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
|
||||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||||
and frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
|
||||
**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B
|
||||
GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072
|
||||
conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via
|
||||
the base-model projector, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -33,12 +32,12 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[FreeToken PR587 ROCm, local image
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8
|
||||
1x R9700, experts in pinned host RAM + GPU LRU 10%,
|
||||
KV pool 299,968 tokens, 2 slots, parsers AUTO,
|
||||
TP=2 untested on ROCm; second R9700 idle]
|
||||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
2 x 200000, froggeric v22.5]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -50,8 +49,8 @@ frank-large, port 7770]
|
||||
1 x 200700 ctx, d2 KV, text-only]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision]
|
||||
H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF
|
||||
1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -81,22 +80,52 @@ frank-small, port 7771]
|
||||
No server-side sampler — the thinking recipe rides the wire in models.yml.
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
|
||||
slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
|
||||
f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
|
||||
measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048
|
||||
since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card
|
||||
left free. Probe-size trap (>=256 px images or the model says "video")
|
||||
documented. Rollback: one MODEL_FILE line back to Q5_K_M
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
|
||||
(evening)** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B, the
|
||||
checkpoint's own MTP head IN the file — `--spec-type draft-mtp`, depth 3)
|
||||
on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a
|
||||
second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8
|
||||
class, 34 KiB/token vs 64 at f16) and vision via unsloth's
|
||||
mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base
|
||||
vision stack was hash-proven byte-identical to Swift's (6/6 tensors).
|
||||
The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types.
|
||||
Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and
|
||||
the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
never deployed
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||||
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
|
||||
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
|
||||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||||
probe-size trap (>=256 px images or the model says "video") live inside
|
||||
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
|
||||
(branch rocm714_torch214) built on the host into the local image
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision — converted no-recipe to
|
||||
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
|
||||
in a staging dir (the plugin descriptor-selection law: nothing claims
|
||||
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
|
||||
per-weight encodings are data, not the descriptor) — on BOTH R9700s
|
||||
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
|
||||
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
|
||||
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
|
||||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||||
`/metrics` scraped. Rollback: git tag
|
||||
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||||
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
|
||||
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||||
rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
@@ -105,8 +134,9 @@ frank-small, port 7771]
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
|
||||
frank-primary-vllm-shapeG-20261003
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||||
radiance swap for TP=2; the image + payload stay on disk until the
|
||||
rollback window closes
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
|
||||
@@ -181,11 +181,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
### Deployment and Topology
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-primary-freetoken-pr587]] — the ACTIVE frank-primary runbook
|
||||
(2026-10-05): FreeToken built at PR #587 head `77a7201` on the host into
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — the ACTIVE frank-primary
|
||||
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
|
||||
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
|
||||
after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts
|
||||
in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV
|
||||
7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
|
||||
FreeToken one-card era (tag `frank-primary-freetoken-pr587-20261005`),
|
||||
then the tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
|
||||
- [[frank-primary-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
|
||||
same day by the radiance cutover above; kept as the rollback runbook):
|
||||
FreeToken built at PR #587 head `77a7201` on the host into
|
||||
`frank-primary-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
|
||||
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO; rollback is the
|
||||
tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
|
||||
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
|
||||
@@ -359,14 +368,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook
|
||||
since 2026-10-03: llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090,
|
||||
port 7771 — TWO conversations at 200,704 tokens each with fully
|
||||
unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung
|
||||
costs a measured −10 % vs Q5_K_M — weight-bandwidth), batch 4096 / ubatch
|
||||
2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and
|
||||
the ≥256 px image-probe trap (tiny images answer "video")
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small
|
||||
runbook since 2026-10-05 (evening): llama.cpp master (CUDA sm_86) serving
|
||||
ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on
|
||||
the RTX 3090 — ONE 131,072-token conversation at Frank's 128k cap, q8_0
|
||||
K + q8_0 V, vision through the base-model projector; the arithmetic that
|
||||
makes one slot the maximum
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
superseded by swift15): llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
|
||||
200,704 tokens each with fully unquantized f16 KV and mmproj vision;
|
||||
94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M —
|
||||
weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
|
||||
2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
|
||||
images answer "video")
|
||||
- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target, payload on disk; owns the measured boot-pair table): same runtime
|
||||
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
|
||||
|
||||
+51
-49
@@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-small llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint; since 2026-10-05 it serves
|
||||
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 via a CUDA (sm_86) build of
|
||||
# llama.cpp master, chosen because it is the only runtime here that moves K and
|
||||
# V independently. Since 2026-10-03 it runs TWO unquantized contexts of
|
||||
# 200,704 tokens each and carries a vision projector, and the card must keep
|
||||
# frank-small is the RTX 3090 endpoint; since 2026-10-05 evening it serves
|
||||
# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S-mtp (27B, MTP head in the
|
||||
# file) via a CUDA (sm_86) build of llama.cpp master: ONE 131,072-token
|
||||
# conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), the
|
||||
# in-file MTP head drafting 3, and a vision projector. The card must keep
|
||||
# 2-3 GiB free. This seam runs the serve script against a stub engine and
|
||||
# asserts on the argv it actually execs, so what is pinned is the
|
||||
# command the operator gets.
|
||||
@@ -16,9 +16,9 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
|
||||
TARGET_STUB="Ornith-1.5-9B-Q8_0.gguf"
|
||||
TARGET_STUB="Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf"
|
||||
# Same basename as the shipped projector: the test env repoints it into $STAGE.
|
||||
MM_STUB="mmproj-Ornith-1.5-9B-BF16.gguf"
|
||||
MM_STUB="mmproj-Qwen3.8-27B-F16.gguf"
|
||||
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
@@ -81,9 +81,9 @@ argv_value() {
|
||||
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the shipped env points at the llama.cpp runtime and the Ornith-1.5-9B Q8_0 GGUF" {
|
||||
@test "the shipped env points at the llama.cpp runtime and the Swift-1.5-27B IQ3_S-mtp GGUF" {
|
||||
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
|
||||
grep -qE "^MODEL_FILE=/home/dank/models/Ornith-Q8/$TARGET_STUB$" "$ENV_FILE"
|
||||
grep -qE "^MODEL_FILE=/home/dank/models/Swift15-IQ3S/$TARGET_STUB$" "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the deployer gates on exactly the payload paths the env file ships" {
|
||||
@@ -100,7 +100,7 @@ argv_value() {
|
||||
done
|
||||
}
|
||||
|
||||
@test "the shipped env serves the fleet shape: one native 262144-token context" {
|
||||
@test "the shipped env serves the fleet shape: one conversation at the 128k cap" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --alias)" = "frank-small" ]
|
||||
@@ -110,54 +110,50 @@ argv_value() {
|
||||
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
||||
# llama.cpp divides --ctx-size across slots and a mamba hybrid cannot
|
||||
# share one cache, so the observable contract is the per-slot window.
|
||||
# 2026-10-05 (Frank): NO KV quantization forces ONE slot at the NATIVE
|
||||
# window — Ornith's 9 full-attn layers price f16 KV at 36 KiB/token, and
|
||||
# two 200,704 slots cannot fit the 2-3 GiB floor on 24 GiB with Q8
|
||||
# weights. The window per conversation ROSE 200,704 -> 262,144.
|
||||
# 2026-10-05 evening (Frank): the 27B Swift-1.5 swap caps each
|
||||
# conversation at 131,072, and ONE slot is the maximum — a second
|
||||
# 131,072 slot does not fit the 2-3 GiB floor (KV budget test below).
|
||||
[ "$(argv_value --parallel)" -eq 1 ]
|
||||
ctx="$(argv_value --ctx-size)"
|
||||
par="$(argv_value --parallel)"
|
||||
[ $((ctx % par)) -eq 0 ]
|
||||
[ $((ctx / par)) -eq 262144 ]
|
||||
[ $((ctx / par)) -eq 131072 ]
|
||||
}
|
||||
|
||||
@test "the KV contract is unquantized both halves, flash-attn on" {
|
||||
@test "the KV contract is q8_0 both halves (FP8 class), flash-attn on" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --cache-type-k)" = "f16" ]
|
||||
# Frank, 2026-10-05: NO KV quantization on this endpoint. The Q8_0 rung
|
||||
# funds itself by UBATCH first (2048 -> 1024), then pool steps — a q8_0-V
|
||||
# proposal was written and overruled. The flag pair is what makes this
|
||||
# Frank, 2026-10-05 evening: "k/v at FP8" for the Swift-1.5 swap —
|
||||
# llama.cpp's FP8 class is q8_0 (1.0625 B/elem => 8.5 effective bits;
|
||||
# there is no true fp8 KV dtype). This SUPERSEDES the same-day f16/f16
|
||||
# rule written for the Ornith 9B fit. The flag pair is what makes this
|
||||
# observable — a build without the FA path silently falls back, so both
|
||||
# must appear in the argv the supervisor runs.
|
||||
[ "$(argv_value --cache-type-v)" = "f16" ]
|
||||
[ "$(argv_value --cache-type-k)" = "q8_0" ]
|
||||
[ "$(argv_value --cache-type-v)" = "q8_0" ]
|
||||
[ "$(argv_value --flash-attn)" = "on" ]
|
||||
}
|
||||
|
||||
@test "the KV budget leaves 2-3 GiB of the 3090 free" {
|
||||
# Rate, from the Ornith GGUF header (read from the Hub file, 2026-10-05):
|
||||
# NINE full-attention layers (block_count 33, full_attention_interval 4)
|
||||
# x 4 kv heads x 256 dim x 2 B = 18 KiB/token for K and the same for V
|
||||
# => 36 KiB/token (budgeted 37 with slack; MiMo measured 24.5 — one
|
||||
# fewer layer AND one fewer KV head). The non-KV half: Q8_0 weights
|
||||
# 9,334 MiB (9,786,060,384 B) + vision tower ~1.3 GiB (mmproj 879 MiB at
|
||||
# the measured ~1.5x cost law) + DeltaNet state + compute buffers +
|
||||
# CUDA graphs => 11 GiB budgeted. Two asserts, both about the output
|
||||
# the operator sees: the pool must still fit with 2 GiB spare, and it
|
||||
# must be large enough that 5 GiB spare is NOT reachable (an undersized
|
||||
# pool is as wrong as an oversized one).
|
||||
@test "the KV budget fits the 2-3 GiB floor and forbids a second 128k slot" {
|
||||
# Rate, from the Swift-1.5 IQ3_S-mtp GGUF header (read from the Hub
|
||||
# file, 2026-10-05): SIXTEEN full-attention layers (block_count 65 = 64
|
||||
# real layers + the blk.64 MTP head, full_attention_interval 4) x 4 kv
|
||||
# heads x 256 dim x 1.0625 B = 17 KiB/token for K and the same for V
|
||||
# => 34 KiB/token at q8_0 (budgeted 37 with slack; f16 would be 64).
|
||||
# The non-KV half: IQ3_S-mtp weights 11,561 MiB (12,120,016,896 B) +
|
||||
# vision tower ~1.35 GiB (927,607,488 B F16 mmproj at the measured
|
||||
# ~1.45x cost law) + DeltaNet states + compute buffers at ubatch 1024 +
|
||||
# CUDA graphs + the MTP draft context => 17 GiB budgeted. Two asserts,
|
||||
# both about the output the operator sees: the pool must still fit with
|
||||
# 2 GiB spare, and a SECOND 131,072 slot must not fit — Frank's "as
|
||||
# many 128k contexts as fit" has the answer 1 on this card, and the
|
||||
# pool itself is capped by the 128k law, not by the card.
|
||||
# Arithmetic, not a boot log: change CTX_SIZE and this bites before the
|
||||
# deploy does. At Ornith's rate the two-200704-slot pool (14.1 GiB KV)
|
||||
# FAILS the first assert — which is exactly why the shipped shape is
|
||||
# ONE native slot.
|
||||
# deploy does.
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
kv_gib=$((ctx * 37 / 1048576))
|
||||
[ $((kv_gib + 11 + 2)) -le 24 ]
|
||||
[ $((kv_gib + 11 + 5)) -gt 24 ]
|
||||
# And the ceiling that sets the per-slot window: two NATIVE windows cannot
|
||||
# be paid for in full precision on this card.
|
||||
native=$((2 * 262144))
|
||||
[ $((native * 37 / 1048576 + 11)) -gt 24 ]
|
||||
[ $((kv_gib + 17 + 2)) -le 24 ]
|
||||
[ $((2 * kv_gib + 17)) -gt 24 ]
|
||||
}
|
||||
|
||||
@test "the compute-buffer pair follows the env file and stays self-consistent" {
|
||||
@@ -187,16 +183,22 @@ argv_value() {
|
||||
argv_value --ubatch-size | grep -qE '^[0-9]+$'
|
||||
}
|
||||
|
||||
@test "no speculative draft ships by default" {
|
||||
# The qwen35 GGUF has no MTP tensors, and the cross-model DFlash draft
|
||||
# is a measured decision the harness makes later, not a default.
|
||||
! grep -q -- '--model-draft' "$SERVE"
|
||||
! grep -q -- '--spec-type' "$SERVE"
|
||||
! grep -q '^DRAFT' "$ENV_FILE"
|
||||
@test "the in-file MTP head ships as draft-mtp at depth 3" {
|
||||
# The -mtp GGUF carries the head (blk.64.nextn.*, nextn_predict_layers
|
||||
# =1): draft-mtp loads it against the main model's own weights — no
|
||||
# --model-draft and no second draft KV pool. Speculative flags move as
|
||||
# a set; both must appear in the argv the supervisor runs, and the env
|
||||
# must carry the same contract the argv executes.
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --spec-type)" = "draft-mtp" ]
|
||||
[ "$(argv_value --spec-draft-n-max)" = "3" ]
|
||||
grep -qE '^SPEC_TYPE=draft-mtp$' "$ENV_FILE"
|
||||
grep -qE '^SPEC_DRAFT_N_MAX=3$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
|
||||
# The Q6_K GGUF carries no `v.*` tensors, so the projector argument IS
|
||||
# The Swift GGUF carries no vision tensors, so the projector argument IS
|
||||
# the vision capability: no flag, no eyes.
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
Reference in New Issue
Block a user