frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index
The parallel session's uncommitted swap, committed at Frank's direction (commit whatever's in the working tree): - llama.cpp master on the RTX 3090 now serves ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE 131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to rollback target. - env/serve/download + seam twins updated together (deployer payload gates rode the previous commit's deploy-remote.sh). - docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135 seam green). - comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/ workflows/ (12 KiB).
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
|
||||
description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
|
||||
description: "The RETIRED frank-small runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -14,13 +14,15 @@ timestamp: 2026-10-03
|
||||
|
||||
# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
|
||||
> this card: the freed third slot paid for **exact KV values**
|
||||
> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
|
||||
> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
|
||||
> — the same file this endpoint served on 2026-10-01, come back as an
|
||||
> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
|
||||
> only the weights moved up.
|
||||
> **RETIRED 2026-10-05 (evening).** Superseded by
|
||||
> [[frank-small-llama-swift15-iq3s-mtp-131072]] — the Swift-1.5-27B GSQ-RCO
|
||||
> IQ3_S-mtp swap. Two facts for the trail: (1) the Ornith-1.5-9B Q8_0 swap
|
||||
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
|
||||
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
|
||||
> contract Frank set here (f16 K + f16 V, "no further quantization on this
|
||||
> endpoint") was superseded the same evening by the Swift swap's explicit
|
||||
> q8_0/q8_0 order. This page stays as the rollback and the measured
|
||||
> boot-pair record. Payload on disk: `/home/dank/models/MiMo-Q6K/`.
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-10-05
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening).** A 27B replaces the 9B on the same
|
||||
> 24 GiB card, bought by Frank's explicit contract for this swap: context
|
||||
> capped at 128k per conversation, KV at FP8 class, vision on, "as many of
|
||||
> those contexts as you can" — the answer turned out to be **one**, by
|
||||
> arithmetic, not by choice (see *Sizing*). This supersedes BOTH earlier
|
||||
> 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape
|
||||
> ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the Ornith-1.5-9B
|
||||
> Q8_0 swap that was committed (77decb5..380aafa) but never deployed.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) |
|
||||
| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out) |
|
||||
|
||||
## Why a base-model projector is legal here
|
||||
|
||||
No Swift repo ships a projector — ukisai's GSQ-RCO release is
|
||||
language-model-only by their own README ("A Swift 27B vision projector has
|
||||
not been verified for this release"). The substitution is proven, not
|
||||
assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its
|
||||
vision stack was hash-compared against the base on 2026-10-05 over HTTP
|
||||
range reads of the two BF16 repos — **333/333 visual tensor names match and
|
||||
6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`,
|
||||
blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the
|
||||
base projector IS the model's own projector. The mmproj header matches
|
||||
Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16,
|
||||
merge 2, ffn 4304, 16 heads, out 5120).
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 131072 --parallel 1` — ONE conversation at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072). `CTX_SIZE`
|
||||
stays divisible by `PARALLEL`; the seam enforces it.
|
||||
|
||||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||||
⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The
|
||||
order supersedes the same-day f16/f16 rule, which was written for the
|
||||
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
|
||||
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
|
||||
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
|
||||
pinned: quantized V requires it, and the build's
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a
|
||||
silent f16 fallback.
|
||||
|
||||
**One slot is the maximum, by arithmetic.** Card 24,576 MiB; weights
|
||||
11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card
|
||||
law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet
|
||||
state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP
|
||||
draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒
|
||||
free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under
|
||||
the 2–3 GiB floor. The pool is capped by the 128k law, not by the card —
|
||||
leftover headroom is honest slack, not an undersized pool.
|
||||
|
||||
## MTP (speculative decoding)
|
||||
|
||||
The MTP head ships **in the file** — the `-mtp` GGUF carries
|
||||
`blk.64.nextn.{enorm,hnorm,shared_head_norm}` + the head layer's tensors at
|
||||
Q6_K/F32 (publisher RECIPE: "MTP_OK verified"),
|
||||
`qwen35.nextn_predict_layers=1`. `--spec-type draft-mtp` loads it against
|
||||
the main model's own weights — no `--model-draft`, no second draft KV pool
|
||||
(an MTP head can NEVER load as a separate draft model on this stack; two
|
||||
context-backed drafters cannot share a process). Depth 3
|
||||
(`--spec-draft-n-max 3`) matches the fleet's radiance MTP depth. The flags
|
||||
move as a set: a no-spec baseline must strip BOTH. Acceptance reads from
|
||||
the response `timings` object (`draft_n`, `draft_n_accepted`) — measure on
|
||||
real traffic, never on a synthetic repeat (greedy-text acceptance lies).
|
||||
|
||||
## Vision contract
|
||||
|
||||
`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
|
||||
Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
|
||||
probe undersupplies image tokens and the model invents ("video"); the
|
||||
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
|
||||
content part reaches the model. Same traps the MiMo eras documented.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking
|
||||
tokens COUNT against `max_tokens`.
|
||||
- The embedded chat template gates `reasoning_effort` to
|
||||
`{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
|
||||
`models.yml` `compat.reasoningEffortMap` folds the harness levels onto
|
||||
that set (mandatory, same as every Qwen3.8-template endpoint).
|
||||
- Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0
|
||||
/ top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly
|
||||
this recipe at ctx 131,072).
|
||||
- Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking
|
||||
tokens vs base at equal-or-better accuracy. Token budgets go further;
|
||||
the 32,768 `maxTokens` is unchanged from the MiMo era and still clears
|
||||
the compaction invariant (91,750 + 32,768 ≤ 131,072).
|
||||
- The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 /
|
||||
IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant
|
||||
types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513
|
||||
(development), held-out 0.024–0.052 per domain; the MTP packaging is NOT
|
||||
separately evaluated by the publisher.
|
||||
|
||||
## Rollback
|
||||
|
||||
Two payloads on disk, both fetchable by the same script's git-history
|
||||
recipes:
|
||||
|
||||
1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||||
this cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||||
2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||||
262,144 slot, f16/f16 KV).
|
||||
|
||||
## Trail
|
||||
|
||||
- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
|
||||
committed and pushed but its deploy never completed (blocked at the
|
||||
whole-tree docs gate); the live service was still MiMo Q6_K when Swift
|
||||
cut over. Both Ornith payloads were verified on disk before Swift.
|
||||
- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
|
||||
read from the Hub file by HTTP range + struct parse — no download
|
||||
needed — before the payload was fetched.
|
||||
@@ -1,25 +1,24 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-05):
|
||||
**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
|
||||
image `frank-primary-freetoken:pr587-77a7201` built on the host from
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
|
||||
`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
|
||||
driving `podman run` directly, serving orcarouter
|
||||
/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
|
||||
pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
|
||||
`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
|
||||
decode slots, parsers AUTO (thinking → `reasoning_content` natively;
|
||||
measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
|
||||
runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw
|
||||
runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||||
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
|
||||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||||
and frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
|
||||
**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B
|
||||
GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072
|
||||
conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via
|
||||
the base-model projector, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -33,12 +32,12 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[FreeToken PR587 ROCm, local image
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8
|
||||
1x R9700, experts in pinned host RAM + GPU LRU 10%,
|
||||
KV pool 299,968 tokens, 2 slots, parsers AUTO,
|
||||
TP=2 untested on ROCm; second R9700 idle]
|
||||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
2 x 200000, froggeric v22.5]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -50,8 +49,8 @@ frank-large, port 7770]
|
||||
1 x 200700 ctx, d2 KV, text-only]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision]
|
||||
H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF
|
||||
1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -81,22 +80,52 @@ frank-small, port 7771]
|
||||
No server-side sampler — the thinking recipe rides the wire in models.yml.
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
|
||||
slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
|
||||
f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
|
||||
measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048
|
||||
since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card
|
||||
left free. Probe-size trap (>=256 px images or the model says "video")
|
||||
documented. Rollback: one MODEL_FILE line back to Q5_K_M
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
|
||||
(evening)** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B, the
|
||||
checkpoint's own MTP head IN the file — `--spec-type draft-mtp`, depth 3)
|
||||
on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a
|
||||
second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8
|
||||
class, 34 KiB/token vs 64 at f16) and vision via unsloth's
|
||||
mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base
|
||||
vision stack was hash-proven byte-identical to Swift's (6/6 tensors).
|
||||
The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types.
|
||||
Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and
|
||||
the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
never deployed
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||||
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
|
||||
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
|
||||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||||
probe-size trap (>=256 px images or the model says "video") live inside
|
||||
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
|
||||
(branch rocm714_torch214) built on the host into the local image
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision — converted no-recipe to
|
||||
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
|
||||
in a staging dir (the plugin descriptor-selection law: nothing claims
|
||||
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
|
||||
per-weight encodings are data, not the descriptor) — on BOTH R9700s
|
||||
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
|
||||
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
|
||||
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
|
||||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||||
`/metrics` scraped. Rollback: git tag
|
||||
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||||
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
|
||||
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||||
rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
@@ -105,8 +134,9 @@ frank-small, port 7771]
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
|
||||
frank-primary-vllm-shapeG-20261003
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||||
radiance swap for TP=2; the image + payload stay on disk until the
|
||||
rollback window closes
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
|
||||
@@ -181,11 +181,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
### Deployment and Topology
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-primary-freetoken-pr587]] — the ACTIVE frank-primary runbook
|
||||
(2026-10-05): FreeToken built at PR #587 head `77a7201` on the host into
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — the ACTIVE frank-primary
|
||||
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
|
||||
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
|
||||
after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts
|
||||
in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV
|
||||
7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
|
||||
FreeToken one-card era (tag `frank-primary-freetoken-pr587-20261005`),
|
||||
then the tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
|
||||
- [[frank-primary-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
|
||||
same day by the radiance cutover above; kept as the rollback runbook):
|
||||
FreeToken built at PR #587 head `77a7201` on the host into
|
||||
`frank-primary-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
|
||||
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO; rollback is the
|
||||
tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
|
||||
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
|
||||
@@ -359,14 +368,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook
|
||||
since 2026-10-03: llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090,
|
||||
port 7771 — TWO conversations at 200,704 tokens each with fully
|
||||
unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung
|
||||
costs a measured −10 % vs Q5_K_M — weight-bandwidth), batch 4096 / ubatch
|
||||
2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and
|
||||
the ≥256 px image-probe trap (tiny images answer "video")
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small
|
||||
runbook since 2026-10-05 (evening): llama.cpp master (CUDA sm_86) serving
|
||||
ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on
|
||||
the RTX 3090 — ONE 131,072-token conversation at Frank's 128k cap, q8_0
|
||||
K + q8_0 V, vision through the base-model projector; the arithmetic that
|
||||
makes one slot the maximum
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
superseded by swift15): llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
|
||||
200,704 tokens each with fully unquantized f16 KV and mmproj vision;
|
||||
94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M —
|
||||
weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
|
||||
2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
|
||||
images answer "video")
|
||||
- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target, payload on disk; owns the measured boot-pair table): same runtime
|
||||
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
|
||||
|
||||
Reference in New Issue
Block a user