dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record the two-slot no-spec frank-small shape (measured 1,397 MiB free, 48.0 t/s)

Browse Source 
Runbook reshaped: 2 x 131072 no-spec identity, slot-2 arithmetic (drafter ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), ngram-mod skipped on the recurrent-rollback law, MTP references kept (44.8-52.1% acceptance, 58.9-66.9 t/s), rollback = one revert of 812d9eb. AGENTS fleet row + section, LLM/index.md intro + entry, root index entry all flipped to the 2026-10-06 shape. 360/360 suite green; e2e gate 33/33 green at deploy.
This commit is contained in:
Frank
2026-10-06 00:15:55 -07:00
parent 812d9ebff5
commit 5e166543fa
4 changed files with 134 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+25 -28
View File
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (~161 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, **2 × 200,000** (first boot; native 262,144), froggeric v22.5 template, vision + tools + `reasoning_content` auto, `/metrics` scraped. Cutover 2026-10-05 (rollback `frank-primary-freetoken-pr587-20261005`, FreeToken one-card image + payload on disk) | 7770 |
| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; **MTP head IN the file** — `--spec-type draft-mtp`, depth 3) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **1 × 131,072** (Frank's 128k cap; a second 131,072 slot misses the 2-3 GiB floor) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024. Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 |
| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; the -mtp file's MTP head present but NOT drafted — **spec OFF** since 2026-10-06, ngram skipped on the recurrent-rollback law) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tight fit he ordered 2026-10-06 — **1,397 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **48.0 t/s** no-spec decode (MTP reference 58.9-66.9). Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -402,34 +402,31 @@ at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at
48 linear layers × 48 value heads). Header facts were read from the Hub
file by HTTP range + struct parse before anything was downloaded.
**The shape (2026-10-05):** `--ctx-size 131072 --parallel 1` = ONE
conversation at Frank's 128k cap (Swift's own benchmark protocol ran at
131,072), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's FP8
class — no true fp8 KV dtype exists; 8.5 effective bits; the order
SUPERSEDES the same-day f16/f16 rule written for the Ornith 9B fit), all
layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024 (lower
ubatch first, Frank's standing law; 2048 is the first A/B if more headroom
is ever wanted). Measured at boot: **19,404 used / 4,721 MiB free**
(`nvidia-smi` free field — the 4.5 GiB prediction held; f16 KV would have
left ~700 MiB, so the free number is also the q8_0 proof). The pool is
capped by the 128k law, not the card; a SECOND slot wants +4,352 MiB and
would land ~0.2 GiB free — far under the 2-3 GiB floor.
**The shape (2026-10-06, Frank's tight-fit order):** `--ctx-size 262144
--parallel 2` = **TWO 131,072-token conversations** (Frank's 128k cap each;
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` pinned, **q8_0 K +
q8_0 V** (llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective
bits; supersedes the same-day f16/f16 rule written for the Ornith 9B fit),
all layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 512.
**Speculative decoding is OFF** (same order): the -mtp file's head stays but
is not drafted with; ngram-mod skipped on purpose — on a qwen35 hybrid
ngram disables recurrent-state rollback, so a rejected draft corrupts the
DeltaNet state.
**Measured decode (2026-10-05 probes, draft_n counts from the live
`timings`)**: temp 0.8 → **66.1 tok/s**, acceptance 182/349 (52.1 %); temp 0
→ 66.9 tok/s, 166/318; **deployed recipe temp 1.0/top_p 0.95/top_k 20 →
58.9 tok/s, 159/355 (44.8 %)** — the higher temperature costs both acceptance
and rate. Vision: a 512×512 solid-red PNG answered `Red` through an
`image_url` part. `reasoning_effort` probe: `xhigh` 200, `high` → 500
Jinja raise (the models.yml fold stays mandatory).
**Speculative decoding is the file's own MTP head**: the `-mtp` GGUF
carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`), so
`--spec-type draft-mtp --spec-draft-n-max 3` loads it against the main
model's weights — no `--model-draft`, no second draft KV pool. Flags move
as a set (a no-spec baseline strips BOTH). Acceptance reads from the
response `timings` (`draft_n`/`draft_n_accepted`) on real traffic, never a
synthetic repeat.
**Measured at the two-slot boot (2026-10-06): 22,728 used / 1,397 MiB
free** (`nvidia-smi` free field) — under the old 2-3 GiB floor BY ORDER ("a
very tight fit"); a 512×512 solid-red image still answers `Red` afterwards
(1,363 MiB free after the probe), so the fit carries the vision transients.
The slot-2 pricing came from the measured 1×131,072 MTP boot (2026-10-05:
19,404 used / 4,721 free — also the q8_0 proof, f16 would have left ~700):
the second q8_0 slot costs 4,352 MiB KV + ~150 MiB DeltaNet, funded by
dropping the MTP draft ctx (~0.4 GiB) and ubatch 1024→512 (~0.3 GiB,
inferred from the measured 616 MiB/doubling rung on the 9B). **Decode
without the drafter: 48.0 tok/s (temp 0) / 47.8 (recipe temp 1.0)** — the
MTP references stay in frank-small.env for the day the head returns (44.8 %
acceptance at the recipe temp ⇒ 58.9 tok/s; 52.1 % at temp 0.8 ⇒ 66.1).
`reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 Jinja raise
(the models.yml fold stays mandatory).
**Vision without a Swift projector**: no Swift repo ships one (ukisai's
README is explicit), so the projector is unsloth's mmproj built from the