frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index
The parallel session's uncommitted swap, committed at Frank's direction (commit whatever's in the working tree): - llama.cpp master on the RTX 3090 now serves ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE 131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to rollback target. - env/serve/download + seam twins updated together (deployer payload gates rode the previous commit's deploy-remote.sh). - docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135 seam green). - comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/ workflows/ (12 KiB).
This commit is contained in:
Frank
@@ -1,25 +1,24 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
|
||||
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
|
||||
image `frank-primary-freetoken:pr587-77a7201` built on the host from
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
|
||||
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
|
||||
driving `podman run` directly, serving orcarouter
|
||||
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
|
||||
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
|
||||
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
|
||||
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
|
||||
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
|
||||
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw
|
||||
runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||||
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
|
||||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||||
and frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
|
||||
**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B
|
||||
GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072
|
||||
conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via
|
||||
the base-model projector, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -33,12 +32,12 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[FreeToken PR587 ROCm, local image
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8
|
||||
1x R9700, experts in pinned host RAM + GPU LRU 10%,
|
||||
KV pool 299,968 tokens, 2 slots, parsers AUTO,
|
||||
TP=2 untested on ROCm; second R9700 idle]
|
||||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
2 x 200000, froggeric v22.5]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -50,8 +49,8 @@ frank-large, port 7770]
|
||||
1 x 200700 ctx, d2 KV, text-only]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision]
|
||||
H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF
|
||||
1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -81,22 +80,52 @@ frank-small, port 7771]
|
||||
No server-side sampler — the thinking recipe rides the wire in models.yml.
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
|
||||
slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
|
||||
f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
|
||||
measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048
|
||||
since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card
|
||||
left free. Probe-size trap (>=256 px images or the model says "video")
|
||||
documented. Rollback: one MODEL_FILE line back to Q5_K_M
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
|
||||
(evening)** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B, the
|
||||
checkpoint's own MTP head IN the file — `--spec-type draft-mtp`, depth 3)
|
||||
on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a
|
||||
second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8
|
||||
class, 34 KiB/token vs 64 at f16) and vision via unsloth's
|
||||
mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base
|
||||
vision stack was hash-proven byte-identical to Swift's (6/6 tensors).
|
||||
The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types.
|
||||
Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and
|
||||
the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
never deployed
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||||
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
|
||||
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
|
||||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||||
probe-size trap (>=256 px images or the model says "video") live inside
|
||||
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
|
||||
(branch rocm714_torch214) built on the host into the local image
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision — converted no-recipe to
|
||||
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
|
||||
in a staging dir (the plugin descriptor-selection law: nothing claims
|
||||
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
|
||||
per-weight encodings are data, not the descriptor) — on BOTH R9700s
|
||||
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
|
||||
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
|
||||
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
|
||||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||||
`/metrics` scraped. Rollback: git tag
|
||||
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||||
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
|
||||
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||||
rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
@@ -105,8 +134,9 @@ frank-small, port 7771]
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
|
||||
frank-primary-vllm-shapeG-20261003
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||||
radiance swap for TP=2; the image + payload stay on disk until the
|
||||
rollback window closes
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
|
||||
Reference in New Issue
Block a user