Compare commits
41
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Comrade Chief Designer
|
30904bac1b | ||
 Frank
|
f8f845d86b | ||
|
Frank
|
934285c2b2 | ||
|
Frank
|
7fe881de01 | ||
|
Frank
|
a02850203d | ||
|
Frank
|
8f129180d8 | ||
|
Frank
|
72bd4c967e | ||
|
Frank
|
125d04967f | ||
|
Frank
|
d82c5c03b7 | ||
|
Frank
|
4bb10cd20e | ||
|
Frank
|
a7c68db9d0 | ||
|
Frank
|
8551bb006b | ||
|
Frank
|
e452a1afbf | ||
|
Frank
|
01ba41cab4 | ||
|
Frank
|
cde91c1a00 | ||
|
Frank
|
6513672f31 | ||
|
Frank
|
5df6ef3166 | ||
|
Frank
|
3c339eeba2 | ||
|
Frank
|
780c307354 | ||
|
Frank
|
de383fe53e | ||
|
Frank
|
1127a52078 | ||
|
Frank
|
dab483c79f | ||
|
Frank
|
b3313340fa | ||
|
Frank
|
e703b0c15c | ||
|
Frank
|
d3d180fd3f | ||
|
Frank
|
db54fa3af1 | ||
|
Frank
|
8029119b19 | ||
|
Frank
|
35e24db7e6 | ||
|
Frank
|
389b605baf | ||
|
Frank
|
3f1860aa5f | ||
|
Frank
|
e3c3fad0f1 | ||
|
Frank
|
a1b650aabc | ||
|
Frank
|
6bbcaf3999 | ||
|
Frank
|
148a5f2575 | ||
|
Frank
|
667ec2987c | ||
|
Frank
|
0c5e054fb8 | ||
|
Frank
|
7da504a651 | ||
|
Frank
|
bd311bf591 | ||
|
Frank
|
7cab15ac39 | ||
|
Frank
|
e8524d4046 | ||
|
Frank
|
c1248d4a5d |
No files matched your search
@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp
|
||||
fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **2 seqs × 200704 ctx** at KV_BYTES 3,150,000,000/rank (fp8 measures 7,701 B/token), pool deliberately small so the expert arena keeps what it reclaims (~7,500 slots vs 6,248 at the 6.1e9 shape) — util 0.98, and the client contract wired: `--reasoning-parser …
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -209,14 +209,14 @@ archive.
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||
endpoints, `gufo bench` for frank-large. Rolling your own bench
|
||||
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
|
||||
requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
**`gufo bench` cannot rank speculative-decode settings, and will invert the
|
||||
ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus
|
||||
(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
|
||||
Measured 2026-09-28 on frank-large at depth 32768: its corpus
|
||||
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
|
||||
draft widths, i.e. the same highly-predictable text every run, and 93 % of
|
||||
every proposal accepted. It reported 29–37 t/s decode where real traffic
|
||||
@@ -241,7 +241,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
||||
merely wedged kernel usually still answers ARP.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (root unit, gufo on the iGPU):
|
||||
- Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
@@ -277,151 +277,147 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-26 both `frank-small` and `frank-primary` accept images
|
||||
(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
|
||||
(this build never autoloads projectors); `frank-large` remains text-only
|
||||
because its stock llama.cpp deployment loads no
|
||||
vision projector (the mmproj is not loaded). Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||
`reasoning_content`, vLLM emits `reasoning` — academic since the
|
||||
2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
|
||||
of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
|
||||
`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
|
||||
`frank-primary` through a BF16 mmproj loaded explicitly (this build never
|
||||
autoloads projectors), and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
**180224**, the per-slot window, not the model's native 262144: the harness
|
||||
must compact before llama.cpp answers a longer prompt with a 400. Note also
|
||||
that
|
||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||
models.yml sets `reasoningContentField` per endpoint to match.
|
||||
There is **no audio**
|
||||
input anywhere in the fleet except frank-small: that capability left with the
|
||||
FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
|
||||
projector's audio encoder (frank-small transcribes WAV verbatim even though
|
||||
`audio` is omitted from its `input` list — OMP's schema rejects it).
|
||||
input anywhere in the fleet: that capability left with the FLM NPU
|
||||
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(the MiMo tower is vision-only).
|
||||
|
||||
### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
|
||||
|
||||
**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
|
||||
vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
|
||||
Radeon 8060S, `gfx1151`) — as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), serving
|
||||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
|
||||
(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
|
||||
the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
|
||||
One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
|
||||
DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
|
||||
**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
|
||||
extracted to `reasoning_content`, and **no vision projector**.
|
||||
The **kingjones30/ROCmFPX llama.cpp fork** at commit
|
||||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
|
||||
`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
|
||||
as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||||
`Restart=always`) running a bare `llama-server` (no container) on port 7770
|
||||
from the Strix Halo iGPU. Boot to healthy: **16 s**.
|
||||
|
||||
**Two swaps landed on 2026-09-28**, both same day: the engine (from the
|
||||
unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
|
||||
and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
|
||||
disk and the payload-only rollback). The two targets are structurally
|
||||
interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
|
||||
in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
|
||||
head at `blk.64` — verified by parsing and diffing both GGUF headers before
|
||||
either was loaded. **The DFlash2 drafter did not move**, so it has its own
|
||||
`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
|
||||
filename, not the target's.
|
||||
**The runtime is not interchangeable — it is chosen by the FILE.**
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung,
|
||||
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`,
|
||||
ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103,
|
||||
`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader:
|
||||
stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The
|
||||
file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant
|
||||
keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file),
|
||||
which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality
|
||||
AND the better drafter base. Vision comes from the bundled
|
||||
`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B).
|
||||
|
||||
**The runtime is built from source on the host** —
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
|
||||
`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships
|
||||
no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty.
|
||||
Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo
|
||||
links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must
|
||||
stay installed. Two host workarounds the build script encodes:
|
||||
**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
|
||||
`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
|
||||
checks the commit out by SHA, so a fork-side force-push cannot silently
|
||||
re-break the endpoint.
|
||||
|
||||
- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated
|
||||
rocWMMA header set into `/usr/include` that shadows AMD's complete one
|
||||
and breaks the build with `'internal/accessors.hpp' file not found`. A
|
||||
`-I` cannot fix it — in HIP mode the host pass searches the system
|
||||
directories first. The package has no reverse dependencies.
|
||||
- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships
|
||||
`/usr/include/c++/16`, which the ROCm clang prefers; its `<atomic_wait.h>`
|
||||
needs a `std::__detail::__notify_impl` symbol version the installed
|
||||
`libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails.
|
||||
**Spec decode is the performance feature, and it is measured.** The draft is
|
||||
the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`,
|
||||
`nextn_predict_layers = 1`), pinned to the same card:
|
||||
`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
|
||||
Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens,
|
||||
`cache_prompt: false`, production service stopped for each arm:
|
||||
**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance
|
||||
length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**;
|
||||
prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's
|
||||
0.95 (that number is the card's own text at a shallow prompt); the gain is
|
||||
real and n-max 4 is where it was measured, never swept. This replaces the
|
||||
gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more
|
||||
speed at once.
|
||||
|
||||
`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`.
|
||||
It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
|
||||
`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which
|
||||
matters because ComfyUI holds queues on the same device; four HIP processes
|
||||
is the device ceiling.
|
||||
**Shape and budget:** ONE session at the native 262,144 window, unquantized
|
||||
f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
|
||||
aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
|
||||
--cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model +
|
||||
draft + projector resident. Prompt cache verified: a repeat pass over the
|
||||
same 6k prompt replays in 0.5 s.
|
||||
|
||||
**The client contract changed, and the chat template is the sharp edge.**
|
||||
**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
|
||||
587 lines of hand-written C++ that appends strings, with no
|
||||
`--chat-template-file`, no env override, and no template body to point at.
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
|
||||
frank-large: that is a port, not a config line. Given the choice on
|
||||
2026-09-28 the decision was to ship on gufo's stock renderer and accept the
|
||||
loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
|
||||
control, the `json|xml` `tool_call_format` switch, the tool-argument
|
||||
truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
|
||||
does keep the effort instruction strings and `preserve_thinking`, which are
|
||||
what the harness depends on. **This is the thing to watch**: the endpoint
|
||||
now runs an *abliterated* model without the prompt that tells it to stop
|
||||
waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
|
||||
as an ordinary system message from `models.yml` — not a template change.
|
||||
`reasoning_effort` accepts the full OpenAI set (all six levels verified
|
||||
200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
|
||||
it is kept deliberately, so that removing it does not silently move every
|
||||
`high` role from `xhigh` to `high` effort inside a swap. And there is no
|
||||
KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
|
||||
`--cache-disk` is opt-in and off.
|
||||
**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
|
||||
(without it `chat_template_kwargs` are silently ignored), tools are native
|
||||
(`get_weather` → `{"city":"Seattle"}`), thinking lands in
|
||||
**`reasoning_content`**, `reasoning_effort` accepts ONLY
|
||||
`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and
|
||||
`max` to `xhigh`), and thinking is enabled server-side with the card's
|
||||
recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus
|
||||
`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request
|
||||
whose whole budget is consumed by reasoning returns `content: null` (bench
|
||||
clients must fall back to `reasoning_content` or they crash *after*
|
||||
measuring), and the fork logs
|
||||
`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
|
||||
resets — self-recovering, not a fault.
|
||||
|
||||
**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
|
||||
(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
|
||||
faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at
|
||||
65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the
|
||||
crossover is ~32k of cached context, and frank-large runs one 262,144-token
|
||||
conversation, so long agent sessions sit on the losing side of that line.
|
||||
The Q4_K_XL target shows no such regression; revisiting the quantization is
|
||||
a separate decision from the runtime. Peak memory is 50.58 GiB at the full
|
||||
262,144 context against the iGPU's 114 GiB GTT aperture.
|
||||
Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS
|
||||
sha256 verified, `--verify-only`, no token on the command line).
|
||||
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
|
||||
GGUF → drafter and exits with a pointer message if any is missing, then
|
||||
`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
|
||||
…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
|
||||
The deployer's `deploy_llm_large` gates on the **runtime binary** as well
|
||||
as both payload paths, and returns before it stops anything.
|
||||
|
||||
Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
|
||||
count + sha256 against the Hub LFS oid, no auth header — the repo is public
|
||||
and a token on the curl command line is world-readable) and
|
||||
`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
|
||||
Both take `--verify-only`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
|
||||
Rollback, two ways: **payload only** — re-fetch with
|
||||
`scripts/download-frank-large.sh` and repoint `MODEL_FILE` at
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`; that
|
||||
29.3 GiB target was **reclaimed on 2026-09-28** (staged to
|
||||
`~/.AGENTTRASH/20260928-retired-models/`, restorable from there), so it is
|
||||
no longer a one-minute flip. **Engine** —
|
||||
[[frank-large-unsloth-qwen38-27b-q8xl]], the b11030 prebuilt at
|
||||
`/home/dank/GitPackages/unsloth-llama-b11030/`; the only way to get
|
||||
Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
|
||||
`Qwen3.8-27B-UD-Q8_K_XL/`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`.
|
||||
Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1
|
||||
(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B
|
||||
payload and the DFlash2 drafter all remain on the host; restore the tag's
|
||||
unit/env/serve trio and redeploy.
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
Mainline llama.cpp master `37b53fd` (CUDA build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
|
||||
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
|
||||
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
|
||||
(four FULL 262,144-token slots — the measured card ceiling), batch
|
||||
**8192/1024**, greedy default. Deployed flags are in
|
||||
`scripts/frank-small.env`; the serve script refuses to start if the
|
||||
weights, the projector, the sidecar or the binary are missing.
|
||||
**llama.cpp master** (CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
|
||||
**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
|
||||
sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
|
||||
port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
|
||||
runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
|
||||
**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
|
||||
Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
|
||||
the projector file is the entire vision path (and ~1.4 GiB of the card).
|
||||
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; an image request is answered from the projector;
|
||||
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
|
||||
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
|
||||
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
|
||||
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
100 is WORSE at long ctx) is tabled in the runbook.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
|
||||
DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
|
||||
the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
|
||||
drives that term.
|
||||
|
||||
**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
|
||||
180,224 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
|
||||
only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
|
||||
layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
|
||||
free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
|
||||
solving the pool against two measured boots: non-KV cost is 7,390 MiB without
|
||||
a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
|
||||
vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
|
||||
windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
|
||||
table. Full f16 V would add ~4 GiB more and bust the card — that is the
|
||||
justification for the asymmetric contract.
|
||||
|
||||
Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
|
||||
`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
|
||||
**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
|
||||
2 × 262,144 shape, because a shorter slot means a shorter attention scan;
|
||||
three concurrent requests answered in parallel and a fourth queued behind a
|
||||
released slot; native `tool_calls`; thinking in `reasoning_content`; a
|
||||
solid-red PNG answers "Red." through an `image_url` content part (the legacy
|
||||
top-level `images` field is ignored by `/v1/chat/completions` — the model then
|
||||
invents a color). A prompt over the slot is a **400
|
||||
`exceed_context_size_error`**, which is why `models.yml` sets
|
||||
`contextWindow: 180224`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
|
||||
Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
|
||||
(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
|
||||
staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
|
||||
|
||||
@@ -630,6 +626,26 @@ linked runbooks.
|
||||
non-greedy requests and reasoning budgets. Re-test chat before production.
|
||||
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
|
||||
stock binaries fail the draft load with a tensor-shape mismatch.
|
||||
- **A GGUF's custom tensor type picks the runtime — read the histogram before
|
||||
promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
|
||||
ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
|
||||
llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
|
||||
the only door. Counting tensor types out of the GGUF KV section takes one
|
||||
python read and settles a runtime argument in minutes. The build identity
|
||||
matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
|
||||
`ggml_compute_forward_set_rows` on this same file, so
|
||||
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
|
||||
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
|
||||
`--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
|
||||
bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
|
||||
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
|
||||
`--spec-type`, or it measures a crash instead of the absence of drafting.
|
||||
- **An orphaned scratch server taxes the whole host.** A bench child left
|
||||
alive by a client-side exception kept ~26 GiB of weights resident on a
|
||||
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
|
||||
Record the child pid and kill it in a `finally`, then prove `free` came back
|
||||
before believing a number. Kill by pid — never `pkill -f llama-server`,
|
||||
which also hits frank-small.
|
||||
|
||||
**llama.cpp serving**
|
||||
|
||||
@@ -702,6 +718,35 @@ linked runbooks.
|
||||
host-RAM hash table) and its window decides which traffic it can see: too
|
||||
short and it proposes on ordinary prose fragments, which costs verification
|
||||
without acceptance.
|
||||
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
|
||||
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
|
||||
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
|
||||
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
|
||||
available context size (180224 tokens)`) even when the pool has room. So
|
||||
frank-small's 2 → 3 reslice moved each conversation from the model's native
|
||||
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
|
||||
the harness would discover the ceiling as a 400 on a long conversation. Side
|
||||
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
|
||||
100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
|
||||
Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
|
||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
||||
costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
|
||||
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
||||
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
|
||||
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
|
||||
that bills ~1.4 GiB, because the encoder's activation and compute buffers
|
||||
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
|
||||
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
|
||||
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
||||
`PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
|
||||
starts from measurement.
|
||||
- **`images` and `image_url` are NOT interchangeable: only the content part
|
||||
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
|
||||
model still invents an answer if the client used the legacy top-level
|
||||
`images` field on `/v1/chat/completions` — frank-small answered "Brown." for
|
||||
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
|
||||
failure is silent because a hallucinated color is a plausible answer. Probe
|
||||
vision with a solid color and expect an exact word.
|
||||
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
|
||||
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
|
||||
then served frank-large: every tool-using agent on that endpoint was broken
|
||||
@@ -802,8 +847,8 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -812,6 +857,10 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
|
||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
|
||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
|
||||
@@ -826,6 +875,7 @@ Retired (rollback targets and history):
|
||||
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bash
|
||||
# autoresearch.sh — canonical benchmark entrypoint: speed of frank-primary.
|
||||
#
|
||||
# Runs on the operator box and drives one bench window on DANKSERVER:
|
||||
# 1. fingerprint the candidate trio (env + serve script + unit)
|
||||
# 2. memo fast-path: an identical fingerprint that already produced a FULLY
|
||||
# successful window (bench + verified production restore) returns the
|
||||
# cached metrics without touching the fleet
|
||||
# 3. stage the candidate trio plus the measure client into
|
||||
# /home/dank/.autoresearch-frank-primary/ (ephemeral; the repo checkout
|
||||
# on the host stays the production source of truth)
|
||||
# 4. run the window (scripts/autoresearch/frank-primary-window.sh):
|
||||
# stop + mask the production unit, boot the staged config AS THE SERVICE
|
||||
# (same serve script, same container, same port 7770), wait for health,
|
||||
# run the in-container usage-counter measure, then restore the production
|
||||
# unit and verify its health — AGENTS.md Benchmarking Policy
|
||||
# 5. print METRIC lines (parse by run_experiment)
|
||||
#
|
||||
# Primary metric: decode_tps (HIGHER is better) — median across the three
|
||||
# prompt depths (115/1056/4096) of per-depth medians of
|
||||
# completion_tokens / (wall - ttft), 3 runs each after a warm pass,
|
||||
# temperature 0, the campaign's deterministic word-seed workload.
|
||||
#
|
||||
# Exit 0 with METRIC lines on success; non-zero on any failure (the window
|
||||
# script's EXIT trap restores production on every path).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
|
||||
STAGE="/home/dank/.autoresearch-frank-primary"
|
||||
|
||||
ENV_FILE="$SCRIPT_DIR/scripts/frank-primary.env"
|
||||
SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
|
||||
UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
|
||||
MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
|
||||
WINDOW="$SCRIPT_DIR/scripts/autoresearch/frank-primary-window.sh"
|
||||
|
||||
for f in "$ENV_FILE" "$SERVE" "$UNIT" "$MEASURE" "$WINDOW"; do
|
||||
[ -f "$f" ] || { echo "[autoresearch] missing harness file: $f" >&2; exit 1; }
|
||||
done
|
||||
command -v python3 >/dev/null 2>&1 || { echo "[autoresearch] python3 required on operator box" >&2; exit 1; }
|
||||
|
||||
# Fingerprint of everything that can change the engine's argv or environment.
|
||||
HASH="$(cat "$ENV_FILE" "$SERVE" "$UNIT" | sha256sum | cut -d' ' -f1)"
|
||||
echo "[autoresearch] candidate fingerprint $HASH" >&2
|
||||
|
||||
ssh_run() { ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST" "$@"; }
|
||||
|
||||
emit_metrics() { # $1 = run.json
|
||||
python3 -c '
|
||||
import json, sys
|
||||
run = json.loads(sys.argv[1])
|
||||
m = run["measure"]
|
||||
print("METRIC decode_tps=%s" % m["decode_tps"])
|
||||
for k in ("decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
|
||||
print("METRIC %s=%s" % (k, m[k]))
|
||||
print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
|
||||
print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
|
||||
print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
|
||||
print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
|
||||
' "$1"
|
||||
}
|
||||
|
||||
# --- memo fast-path -------------------------------------------------------
|
||||
cached_hash="$(ssh_run "cat '$STAGE/last/hash' 2>/dev/null" 2>/dev/null || true)"
|
||||
if [ -n "$cached_hash" ] && [ "$cached_hash" = "$HASH" ]; then
|
||||
cached_run="$(ssh_run "cat '$STAGE/last/run.json' 2>/dev/null" || true)"
|
||||
if [ -n "$cached_run" ]; then
|
||||
echo "[autoresearch] memo hit — cached metrics for $HASH, fleet untouched" >&2
|
||||
emit_metrics "$cached_run"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- stage the candidate ----------------------------------------------------
|
||||
echo "[autoresearch] staging candidate on $HOST:$STAGE" >&2
|
||||
ssh_run "mkdir -p '$STAGE'"
|
||||
scp -q "$ENV_FILE" "$HOST:$STAGE/frank-primary.env"
|
||||
scp -q "$SERVE" "$HOST:$STAGE/serve.sh"
|
||||
scp -q "$MEASURE" "$HOST:$STAGE/measure.py"
|
||||
scp -q "$WINDOW" "$HOST:$STAGE/window.sh"
|
||||
ssh_run "printf '%s\n' '$HASH' > '$STAGE/hash'"
|
||||
|
||||
# --- the window -------------------------------------------------------------
|
||||
# Budgets: bench boot up to 20 min (warm boots measured ~7 min), measure up
|
||||
# to 30 min, restore boot up to 20 min — call run_experiment with
|
||||
# timeout_seconds >= 3600.
|
||||
echo "[autoresearch] opening bench window (stop+mask -> boot -> measure -> restore)" >&2
|
||||
set +e
|
||||
ssh_run "bash '$STAGE/window.sh'"
|
||||
rc=$?
|
||||
set -e
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "[autoresearch] window failed (rc=$rc); host-side evidence:" >&2
|
||||
ssh_run "tail -60 '$STAGE/fail.log' 2>/dev/null || tail -40 '$STAGE/serve.out' 2>/dev/null || tail -40 '$STAGE/measure.out' 2>/dev/null || true" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
|
||||
echo "[autoresearch] could not read $STAGE/run.json back" >&2; exit 1; }
|
||||
echo "[autoresearch] window result:" >&2
|
||||
printf '%s\n' "$run_json" >&2
|
||||
emit_metrics "$run_json"
|
||||
exit 0
|
||||
Executable
+76
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
|
||||
# speculative-decode arm (vLLM --speculative-config method=dflash). Public
|
||||
# repo — no auth header on the command line.
|
||||
#
|
||||
# The drafter is a 6-layer qwen3 draft whose dflash_config pins
|
||||
# target_layer_ids [1,5,9,13,17,21,25,29] and block_size 16; its hidden_size
|
||||
# 4096 / vocab_size 248320 match the MiMo-V2.6-Distill-Qwen-9B target exactly
|
||||
# (both preserve the Qwen3.5-9B geometry), which is what lets vLLM load it as
|
||||
# the draft for that target.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="z-lab/Qwen3.5-9B-DFlash"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/Qwen3.5-9B-DFlash}"
|
||||
|
||||
# name bytes sha256(- = non-LFS, size-only check)
|
||||
FILES=(
|
||||
"model.safetensors 2583816465 0a42274b32554f48de1faa0d42824e9c2ceda649c30ae0a731cddf410dd698c7"
|
||||
"config.json 1222 -"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
tmp="$f.part"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
Executable
+90
@@ -0,0 +1,90 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
|
||||
# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
|
||||
# command line (a Bearer token in argv is world-readable via /proc).
|
||||
#
|
||||
# Per-file FRESH downloads, no resume: the house rule (see
|
||||
# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the
|
||||
# Hub LFS oid is what makes an access error fail instead of "succeeding".
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
|
||||
#
|
||||
# The verified set below is exactly what the 2026-10-01 bench window served:
|
||||
# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
|
||||
set -euo pipefail
|
||||
|
||||
REPO="XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16}"
|
||||
|
||||
# name bytes sha256(- = non-LFS, size-only check)
|
||||
FILES=(
|
||||
"model-00001-of-00004.safetensors 5276436216 aab052180118aee34abc3029b54eaa49096aac606b97d703866b420dceb703c3"
|
||||
"model-00002-of-00004.safetensors 5033171280 7a0486565f06d25ac4628e9dba470dc3f604353471d240d5a0bf7128f64df396"
|
||||
"model-00003-of-00004.safetensors 5234499504 6c73207563d1879bfd6c143a028cc70be68edff56280458c71a43df4240300f4"
|
||||
"model-00004-of-00004.safetensors 3275613848 1379a7cf8c0b8555a39ab65a47e830e0eb45e776b46045734da3afd73c09eea2"
|
||||
"model.safetensors.index.json 69217 -"
|
||||
"config.json 2784 -"
|
||||
"generation_config.json 126 -"
|
||||
"chat_template.jinja 3916 -"
|
||||
"tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
|
||||
"tokenizer_config.json 1165 -"
|
||||
"vocab.json 6722759 -"
|
||||
"merges.txt 3353259 -"
|
||||
"preprocessor_config.json 443 -"
|
||||
"processor_config.json 1191 -"
|
||||
"video_preprocessor_config.json 385 -"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
tmp="$f.part"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
Executable
+85
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
|
||||
# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
|
||||
# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
|
||||
# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00
|
||||
# acceptance). Public repo — no auth header on the command line.
|
||||
#
|
||||
# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="Qwen/Qwen3.5-9B"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/Qwen3.5-9B}"
|
||||
|
||||
# name bytes sha256(- = non-LFS, size-only check)
|
||||
FILES=(
|
||||
"model.safetensors-00001-of-00004.safetensors 5276436216 db6f444b43d318c92f360a13a25561a6a65b10c0631b8ed305a426dbaa6c380e"
|
||||
"model.safetensors-00002-of-00004.safetensors 5335161512 31c7d7e2dd5d207840b31cc59083c8f4c4718959149e0358c0364052bb9a0330"
|
||||
"model.safetensors-00003-of-00004.safetensors 5368717440 7ec36ba3a4176a44c3c0876ad80c56a2f70c84bf008d82e9501df642f17dadec"
|
||||
"model.safetensors-00004-of-00004.safetensors 3325995712 b62b0c4cd7e44edee103ee8f4fe225f246d5e768e07bfd5f25b63a8aa1fdd0c6"
|
||||
"model.safetensors.index.json 79657 -"
|
||||
"config.json 3126 -"
|
||||
"chat_template.jinja 7756 -"
|
||||
"tokenizer.json 12807982 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
|
||||
"tokenizer_config.json 16710 -"
|
||||
"vocab.json 6722759 -"
|
||||
"merges.txt 3353259 -"
|
||||
"preprocessor_config.json 390 -"
|
||||
"video_preprocessor_config.json 385 -"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
tmp="$f.part"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
|
||||
# (SGLang, RTX 3090). Public repo — no auth header on the command line.
|
||||
#
|
||||
# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
|
||||
# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
|
||||
# the house rule.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
|
||||
#
|
||||
# LFS shards are verified against the Hub LFS sha256; non-LFS text files
|
||||
# against their declared byte size. Idempotent: complete files are skipped.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
|
||||
|
||||
# name bytes sha256(- = non-LFS, size-only check)
|
||||
FILES=(
|
||||
"model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
|
||||
"model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
|
||||
"model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
|
||||
"model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
|
||||
"model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
|
||||
"model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
|
||||
"model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
|
||||
"model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
|
||||
"tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
|
||||
"model.safetensors.index.json 106278 -"
|
||||
"config.json 4608 -"
|
||||
"generation_config.json 161 -"
|
||||
"quantization_config.json 1654 -"
|
||||
"chat_template.jinja 3916 -"
|
||||
"tokenizer_config.json 1168 -"
|
||||
"preprocessor_config.json 443 -"
|
||||
"processor_config.json 1191 -"
|
||||
"README.md 8448 -"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
tmp="$f.part"
|
||||
# no -C resume; -f + --fail-with-body so an HTTP error is not a success
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
Executable
+141
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""autoresearch-bench.py — deterministic greedy battery for frank-large.
|
||||
|
||||
Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
|
||||
Fixed embedded corpus, temperature 0, streaming so decode speed is measured
|
||||
between real token arrivals (not request totals). Median of REPS samples per
|
||||
metric is the reported value; medians are far more stable than means under
|
||||
GPU thermal drift.
|
||||
|
||||
Metrics emitted (stdout, one per line):
|
||||
METRIC decode_tokens_per_sec=<float> primary, higher better
|
||||
METRIC prefill_tokens_per_sec=<float> secondary
|
||||
METRIC ttft_ms=<float> secondary, lower better
|
||||
Exit non-zero if the endpoint is unhealthy or the served alias is wrong.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
# Fixed corpus: deterministic, ~28k tokens after tokenization (same scale as
|
||||
# the 2026-10-01 bench window). Built by repeating a fixed paragraph block a
|
||||
# fixed number of times - NO random, NO clock, NO network.
|
||||
_BLOCK = (
|
||||
"The rendering pipeline walks the scene graph once per frame, resolving "
|
||||
"material bindings into descriptor sets before any draw call is recorded. "
|
||||
"Because the allocator recycles command buffers only after their fence "
|
||||
"signals, a submission that outlives its frame must pin the pool entry; "
|
||||
"otherwise the next frame overwrites recorded commands in flight. The "
|
||||
"validation layer flags exactly this class of lifetime bug, but only for "
|
||||
"buffers it can see - GPU-side timestamps and queue-family ownership "
|
||||
"transfers hide in the driver's own bookkeeping, which is why release "
|
||||
"builds must lean on explicit barriers and why a frame-time regression "
|
||||
"often shows up as corruption one frame later than its cause.\n"
|
||||
)
|
||||
LONG_CTX = _BLOCK * 220 # ~28k-token prefill, matching the deployed workload
|
||||
_LONG = LONG_CTX + "\nNow, in one focused paragraph (80-150 words), explain " \
|
||||
"which lifetime bugs this pipeline can still suffer and what explicit " \
|
||||
"mitigations the text implies."
|
||||
_SHORT = "Design a rate limiter for a homelab reverse proxy that must not " \
|
||||
"lock out the LAN. Think it through, then give the algorithm."
|
||||
|
||||
REPS = 3
|
||||
MAX_TOKENS = 256
|
||||
REQ_TIMEOUT = 900
|
||||
|
||||
|
||||
def stream_completion(url: str, model: str, prompt: str):
|
||||
"""Greedy streaming completion.
|
||||
|
||||
Returns (ttft_s, decode_tps, prefill_tps, prompt_tokens, n_tokens).
|
||||
ttft is RELATIVE to request send: time.perf_counter()'s epoch is host
|
||||
boot, and returning the raw stamp read as an 11-day first token.
|
||||
"""
|
||||
body = json.dumps({
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": MAX_TOKENS,
|
||||
"temperature": 0,
|
||||
"stream": True,
|
||||
"stream_options": {"include_usage": True},
|
||||
}).encode()
|
||||
req = urllib.request.Request(
|
||||
url.rstrip("/") + "/v1/chat/completions", data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
t0 = time.perf_counter()
|
||||
first = None
|
||||
stamps = []
|
||||
usage = {}
|
||||
with urllib.request.urlopen(req, timeout=REQ_TIMEOUT) as resp:
|
||||
for raw in resp:
|
||||
line = raw.decode(errors="replace").strip()
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
payload = line[5:].strip()
|
||||
if payload == "[DONE]":
|
||||
break
|
||||
chunk = json.loads(payload)
|
||||
if chunk.get("usage"):
|
||||
usage = chunk["usage"]
|
||||
for ch in chunk.get("choices", []):
|
||||
d = ch.get("delta") or {}
|
||||
# reasoning AND content deltas both arrive token by token;
|
||||
# count either so greedy text length drives the rate.
|
||||
# (llama.cpp names the thinking field reasoning_content.)
|
||||
if d.get("content") or d.get("reasoning") or d.get("reasoning_content"):
|
||||
now = time.perf_counter()
|
||||
if first is None:
|
||||
first = now
|
||||
stamps.append(now)
|
||||
if first is None or len(stamps) < 2:
|
||||
# A zero-argmax degenerate answer would poison the median; raise.
|
||||
raise RuntimeError(f"stream produced {len(stamps)} tokens")
|
||||
ttft_s = first - t0
|
||||
decode_s = stamps[-1] - stamps[0]
|
||||
decode_tps = (len(stamps) - 1) / decode_s if decode_s > 0 else 0.0
|
||||
prompt_tokens = usage.get("prompt_tokens") or 0
|
||||
prefill_tps = prompt_tokens / ttft_s if prompt_tokens else 0.0
|
||||
return ttft_s, decode_tps, prefill_tps, prompt_tokens, len(stamps)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--url", default="http://127.0.0.1:7770")
|
||||
ap.add_argument("--model", default="frank-large")
|
||||
args = ap.parse_args()
|
||||
|
||||
with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
|
||||
timeout=10) as r:
|
||||
ids = [m["id"] for m in json.load(r)["data"]]
|
||||
if args.model not in ids:
|
||||
print(f"FATAL: served ids {ids}, expected {args.model}",
|
||||
file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
# Warmup (compile/graph warm + allocator settle), NOT measured.
|
||||
stream_completion(args.url, args.model, _SHORT)
|
||||
|
||||
decodes, prefills, ttfts = [], [], []
|
||||
for rep in range(REPS):
|
||||
for name, prompt in (("long", _LONG), ("short", _SHORT)):
|
||||
ttft, dt, pt, ptk, ntok = stream_completion(
|
||||
args.url, args.model, prompt)
|
||||
decodes.append(dt)
|
||||
ttfts.append(ttft * 1000.0)
|
||||
if name == "long":
|
||||
prefills.append(pt)
|
||||
print(f" rep{rep} {name:5s} ttft={ttft*1000:8.1f}ms "
|
||||
f"prefill={pt:8.1f} t/s decode={dt:6.2f} t/s "
|
||||
f"({ntok} tok, prompt {ptk})", file=sys.stderr)
|
||||
|
||||
med = lambda xs: statistics.median(xs)
|
||||
print(f"METRIC decode_tokens_per_sec={med(decodes):.2f}")
|
||||
print(f"METRIC prefill_tokens_per_sec={med(prefills):.1f}")
|
||||
print(f"METRIC ttft_ms={med(ttfts):.1f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env python3
|
||||
"""frank-primary decode measurement — runs INSIDE the engine container.
|
||||
|
||||
The method is the fleet's established one (the 2026-09-29 / 2026-10-01
|
||||
tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
|
||||
under speculative decoding because rejected drafts change the accounting, so
|
||||
decode is measured from the server's own usage counters:
|
||||
|
||||
decode_tok_s = completion_tokens / (wall - ttft)
|
||||
|
||||
with ttft = wall time to the FIRST content-bearing chunk (thinking arrives in
|
||||
`reasoning`/`delta.reasoning` on this stack and is excluded from the decode
|
||||
window — and is deterministic per prompt at temperature 0, so the exclusion
|
||||
cancels in any A/B on the same workload).
|
||||
|
||||
Workload (fixed, deterministic, offline):
|
||||
* prompts built from a large fixed-seed shuffled word list so tokens rarely
|
||||
repeat (no n-gram/prefix predictability of the text itself), sized with
|
||||
the server's own /v1/messages/count_tokens counter
|
||||
* depths 115 / 1056 / 4096 target prompt tokens
|
||||
* one 64-token warm pass per depth (warms the expert arena), then
|
||||
max_tokens=256 at temperature 0, 3 runs, median
|
||||
Everything is greedy with a fixed seed — the same workload every run, no
|
||||
network, no time-of-day dependence.
|
||||
|
||||
Env:
|
||||
AR_BASE endpoint (default http://127.0.0.1:7770)
|
||||
AR_MODEL served name (default frank-primary)
|
||||
AR_PROMPT_CEILING hard prompt ceiling in tokens (default 100000)
|
||||
|
||||
Output: progress on stderr, ONE json object on stdout (the last line).
|
||||
Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
|
||||
MODEL = os.environ.get("AR_MODEL", "frank-primary")
|
||||
PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
|
||||
DEPTHS = (115, 1056, 4096)
|
||||
WARM_TOKENS = 64
|
||||
MAX_TOKENS = 256
|
||||
RUNS = 3
|
||||
|
||||
# 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
|
||||
# and seed, so prompts are byte-identical to the historical measurements.
|
||||
_VOCAB = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima "
|
||||
"mike november oscar papa quebec romeo sierra tango uniform victor whiskey "
|
||||
"xray yankee zulu cardinal ordinal topology kernel scheduler resident "
|
||||
"unified aperture expert arena slab offload prefetch gemm triton quant "
|
||||
"tensor shard checkpoint manifest embed router cache page table window "
|
||||
"gradient sparse dense moe attention head dim block stream decode prefill "
|
||||
"bandwidth latency throughput residency eviction tier swap pin memlock").split()
|
||||
_R = random.Random(1234)
|
||||
_ORDER = list(range(len(_VOCAB)))
|
||||
_R.shuffle(_ORDER)
|
||||
|
||||
|
||||
def log(msg):
|
||||
print("[measure] %s" % msg, file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def post_json(path, body, timeout):
|
||||
req = urllib.request.Request(
|
||||
BASE + path, data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def count_tokens(text):
|
||||
"""Token count without a prefill, from the server's own counter."""
|
||||
body = {"model": MODEL,
|
||||
"messages": [{"role": "user", "content": text}]}
|
||||
r = post_json("/v1/messages/count_tokens", body, timeout=300)
|
||||
n = r.get("input_tokens") or r.get("count") or r.get("token_count")
|
||||
if not n:
|
||||
raise RuntimeError("count_tokens returned no count: %r" % (r,))
|
||||
return int(n)
|
||||
|
||||
|
||||
def words(n):
|
||||
return " ".join(_VOCAB[_ORDER[i % len(_VOCAB)]] for i in range(n))
|
||||
|
||||
|
||||
def calibrate():
|
||||
"""Measure words->tokens from the server's own counter."""
|
||||
return 2000.0 / count_tokens(words(2000))
|
||||
|
||||
|
||||
def prompt_for(target_tokens, wpt):
|
||||
n = max(1, int(target_tokens * wpt))
|
||||
text = words(n)
|
||||
pt = count_tokens(text)
|
||||
while pt > PROMPT_CEILING:
|
||||
n = max(1, int(n * (PROMPT_CEILING / pt) * 0.98))
|
||||
text = words(n)
|
||||
pt = count_tokens(text)
|
||||
return text, pt
|
||||
|
||||
|
||||
def measured(prompt, max_tokens):
|
||||
"""Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
|
||||
body = {"model": MODEL,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens, "temperature": 0,
|
||||
"stream": True, "stream_options": {"include_usage": True}}
|
||||
req = urllib.request.Request(
|
||||
BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
t0 = time.time()
|
||||
ttft = None
|
||||
usage = {}
|
||||
with urllib.request.urlopen(req, timeout=1800) as r:
|
||||
for raw in r:
|
||||
line = raw.decode("utf-8", "replace").strip()
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
s = line[5:].strip()
|
||||
if s == "[DONE]":
|
||||
break
|
||||
try:
|
||||
ev = json.loads(s)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if ev.get("usage"):
|
||||
usage = ev["usage"]
|
||||
for ch in ev.get("choices", []):
|
||||
if (ch.get("delta") or {}).get("content"):
|
||||
if ttft is None:
|
||||
ttft = time.time() - t0
|
||||
wall = time.time() - t0
|
||||
ct = int(usage.get("completion_tokens") or 0)
|
||||
if ct <= 0:
|
||||
# No usage on the stream: the endpoint is unhealthy or the model
|
||||
# returned nothing (e.g. HTTP 200 with an error payload).
|
||||
raise RuntimeError("no completion_tokens in usage: %r" % (usage,))
|
||||
dec_window = max(wall - (ttft or 0.0), 1e-9)
|
||||
return (ttft or wall), (ct / dec_window), ct
|
||||
|
||||
|
||||
def main():
|
||||
log("base=%s model=%s ceiling=%d" % (BASE, MODEL, PROMPT_CEILING))
|
||||
try:
|
||||
wpt = calibrate()
|
||||
except Exception as e:
|
||||
log("endpoint unhealthy during calibration: %s" % e)
|
||||
return 2
|
||||
log("words/token: %.3f" % wpt)
|
||||
|
||||
result = {"words_per_token": round(wpt, 5)}
|
||||
for tgt in DEPTHS:
|
||||
try:
|
||||
p, pt = prompt_for(tgt, wpt)
|
||||
measured(p, WARM_TOKENS) # warm the arena
|
||||
except Exception as e:
|
||||
log("warm pass failed at depth %d: %s" % (tgt, e))
|
||||
return 2
|
||||
rates, tts, cts = [], [], []
|
||||
for i in range(RUNS):
|
||||
try:
|
||||
tt, r, ct = measured(p, MAX_TOKENS)
|
||||
except Exception as e:
|
||||
log("run %d/%d failed at depth %d: %s" % (i + 1, RUNS, tgt, e))
|
||||
continue
|
||||
rates.append(r)
|
||||
tts.append(tt)
|
||||
cts.append(ct)
|
||||
if len(rates) < 2:
|
||||
log("only %d usable run(s) at depth %d — refusing to median" % (len(rates), tgt))
|
||||
return 2
|
||||
result["decode_tps_%d" % tgt] = round(statistics.median(rates), 2)
|
||||
result["ttft_ms_%d" % tgt] = round(1000.0 * statistics.median(tts), 1)
|
||||
result["completion_tokens_%d" % tgt] = int(statistics.median(cts))
|
||||
result["prompt_tokens_%d" % tgt] = pt
|
||||
log("depth %5d tok decode %.2f t/s ttft %.3f s (runs=%d)"
|
||||
% (pt, result["decode_tps_%d" % tgt], result["ttft_ms_%d" % tgt] / 1000.0, len(rates)))
|
||||
|
||||
meds = [result["decode_tps_%d" % t] for t in DEPTHS]
|
||||
result["decode_tps"] = round(statistics.median(meds), 2)
|
||||
print(json.dumps(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+200
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env bash
|
||||
# frank-primary-window.sh — one full bench window for frank-primary, run ON
|
||||
# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
|
||||
#
|
||||
# Discipline (AGENTS.md Benchmarking Policy): the production unit is
|
||||
# STOPPED + MASKED first, the candidate config boots AS THE SERVICE itself
|
||||
# (same serve script, same container name, same port 7770 — no scratch
|
||||
# ports, no co-tenant GPU), the in-container measure runs, and the window
|
||||
# ALWAYS ends by restoring the production unit and verifying its health.
|
||||
# The EXIT trap makes the restore unconditional: a boot failure, a measure
|
||||
# failure, or a dead ssh session all end with production back up.
|
||||
#
|
||||
# Staging layout (created by the orchestrator, ephemeral — never a config
|
||||
# source; the repo checkout is):
|
||||
# /home/dank/.autoresearch-frank-primary/
|
||||
# serve.sh staged candidate serve script
|
||||
# frank-primary.env staged candidate env (the candidate knobs)
|
||||
# measure.py the in-container measure client
|
||||
# hash candidate fingerprint
|
||||
# run.json result (metrics + boot + kv + fingerprint)
|
||||
# serve.out/measure.out/fail.log
|
||||
# last/ memo: run.json + hash of the last FULLY
|
||||
# successful window (bench AND verified restore)
|
||||
#
|
||||
# This script logs to stderr and exits 0 only when the measure produced a
|
||||
# usable result (the restore happens in the trap regardless of the bench
|
||||
# outcome; a failed restore forces exit 1).
|
||||
set -uo pipefail
|
||||
|
||||
STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
|
||||
UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
|
||||
CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
|
||||
HEALTH_URL="http://127.0.0.1:7770/health"
|
||||
MODELS_URL="http://127.0.0.1:7770/v1/models"
|
||||
BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
|
||||
RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
|
||||
MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
|
||||
SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
|
||||
SERVED="${SERVED:-frank-primary}"
|
||||
|
||||
RESTORED=0
|
||||
|
||||
log() { printf '[window %s] %s\n' "$(date '+%H:%M:%S')" "$*" >&2; }
|
||||
|
||||
container_state() { podman inspect -f '{{.State.Status}}' "$CTR" 2>/dev/null || echo absent; }
|
||||
|
||||
wait_port_free() {
|
||||
local i
|
||||
for i in $(seq 1 90); do
|
||||
[ "$(container_state)" != "running" ] && return 0
|
||||
sleep 2
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_health() { # $1 = timeout seconds
|
||||
local t0 code
|
||||
t0=$(date +%s)
|
||||
while :; do
|
||||
code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$HEALTH_URL" 2>/dev/null || echo 000)"
|
||||
[ "$code" = "200" ] && return 0
|
||||
[ "$(( $(date +%s) - t0 ))" -ge "$1" ] && return 1
|
||||
sleep 2
|
||||
done
|
||||
}
|
||||
|
||||
capture_bench_failure() {
|
||||
log "capturing bench-boot failure evidence"
|
||||
{
|
||||
echo "=== podman logs (tail 400) ==="
|
||||
podman logs --tail 400 "$CTR" 2>&1 || true
|
||||
echo "=== staged serve.out (tail 60) ==="
|
||||
tail -60 "$STAGE/serve.out" 2>/dev/null || true
|
||||
echo "=== engine log (tail 60) ==="
|
||||
tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
|
||||
} >"$STAGE/fail.log" 2>&1 || true
|
||||
grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
|
||||
"$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
|
||||
}
|
||||
|
||||
restore() {
|
||||
[ "$RESTORED" = 1 ] && return 0
|
||||
RESTORED=1
|
||||
log "restoring production unit $UNIT"
|
||||
if [ -f "$STAGE/serve.pid" ]; then
|
||||
kill "$(cat "$STAGE/serve.pid" 2>/dev/null)" 2>/dev/null || true
|
||||
fi
|
||||
podman stop -t 120 "$CTR" >/dev/null 2>&1 || true
|
||||
wait_port_free || log "WARNING: bench container still draining after 180s"
|
||||
sudo systemctl unmask "$UNIT" 2>/dev/null || true
|
||||
if ! sudo systemctl start "$UNIT"; then
|
||||
log "ERROR: systemctl start $UNIT failed — operator intervention required"
|
||||
return 1
|
||||
fi
|
||||
if wait_health "$RESTORE_TIMEOUT"; then
|
||||
if curl -s --max-time 10 "$MODELS_URL" 2>/dev/null | grep -q "$SERVED"; then
|
||||
log "production restored and verified ($SERVED answering on :7770)"
|
||||
# Memo promotion lives HERE: a cached result is only ever the
|
||||
# product of a fully successful window (bench AND verified
|
||||
# restore), never of a bare bench success.
|
||||
cp "$STAGE/run.json" "$STAGE/last/" 2>/dev/null || true
|
||||
cp "$STAGE/hash" "$STAGE/last/" 2>/dev/null || true
|
||||
else
|
||||
log "ERROR: health 200 but /v1/models does not list $SERVED"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
log "ERROR: production not healthy within ${RESTORE_TIMEOUT}s — unit left to systemd (Restart=always)"
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
|
||||
|
||||
main() {
|
||||
local f
|
||||
for f in serve.sh frank-primary.env measure.py hash; do
|
||||
[ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
|
||||
done
|
||||
mkdir -p "$STAGE/last"
|
||||
rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out"
|
||||
|
||||
local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
|
||||
t_start="$(date +%s)"
|
||||
|
||||
log "stopping and masking production unit $UNIT"
|
||||
sudo systemctl stop "$UNIT" 2>/dev/null || true
|
||||
# Explicit stop is not auto-restarted even with Restart=always; the mask
|
||||
# is belt-and-braces against a stray start during the window.
|
||||
sudo systemctl mask "$UNIT" 2>/dev/null || \
|
||||
log "WARNING: mask failed — relying on explicit-stop semantics"
|
||||
wait_port_free || log "note: production container still draining"
|
||||
|
||||
t0="$(date +%s)"
|
||||
log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
|
||||
FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
|
||||
setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
|
||||
echo "$!" >"$STAGE/serve.pid"
|
||||
|
||||
for i in $(seq 1 45); do
|
||||
[ "$(container_state)" = "running" ] && break
|
||||
sleep 2
|
||||
done
|
||||
if [ "$(container_state)" != "running" ]; then
|
||||
log "bench container never reached running state"
|
||||
tail -40 "$STAGE/serve.out" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if wait_health "$BOOT_TIMEOUT"; then
|
||||
boot_seconds=$(( $(date +%s) - t0 ))
|
||||
log "bench healthy after ${boot_seconds}s"
|
||||
else
|
||||
if [ "$(container_state)" = "running" ]; then
|
||||
log "bench boot timed out after ${BOOT_TIMEOUT}s (still loading)"
|
||||
else
|
||||
log "bench container died during boot"
|
||||
capture_bench_failure
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log "running measure in container (timeout ${MEASURE_TIMEOUT}s)"
|
||||
if ! podman cp "$STAGE/measure.py" "$CTR":/tmp/ar-measure.py; then
|
||||
log "podman cp of measure client failed"
|
||||
exit 1
|
||||
fi
|
||||
if ! timeout "$MEASURE_TIMEOUT" podman exec "$CTR" python3 /tmp/ar-measure.py \
|
||||
>"$STAGE/measure.out" 2>&1; then
|
||||
log "measure failed or timed out"
|
||||
tail -40 "$STAGE/measure.out" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Capacity facts for the run (context-size experiments change these).
|
||||
kvline="$(podman logs "$CTR" 2>&1 | grep -E 'GPU KV cache size' | tail -1 || true)"
|
||||
kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
|
||||
kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
|
||||
|
||||
measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
|
||||
if [ -z "$measure_json" ]; then
|
||||
log "measure produced no JSON result line"
|
||||
tail -40 "$STAGE/measure.out" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The measure line is already JSON, so the run object can be assembled
|
||||
# without any host-side interpreter.
|
||||
printf '{"hash":"%s","boot_seconds":%s,"kv_tokens":%s,"kv_concurrency":"%s","total_seconds":%s,"measure":%s}\n' \
|
||||
"$(tr -d '[:space:]' <"$STAGE/hash")" \
|
||||
"$boot_seconds" "${kv_tokens:-0}" "${kv_conc:-unknown}" \
|
||||
"$(( $(date +%s) - t_start ))" "$measure_json" >"$STAGE/run.json"
|
||||
|
||||
log "bench window complete in $(( $(date +%s) - t_start ))s"
|
||||
# Memo only after a FULL success; the restore below still has to verify.
|
||||
exit 0
|
||||
}
|
||||
|
||||
main "$@"
|
||||
exit "$?"
|
||||
Executable
+89
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
|
||||
#
|
||||
# bash scripts/build-rocmfpx.sh # clone/fetch + build the pin
|
||||
# bash scripts/build-rocmfpx.sh --verify # only check the installed tree
|
||||
#
|
||||
# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
|
||||
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
|
||||
# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
|
||||
# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
|
||||
# stock llama.cpp, gufo and every other loader on the host reject the file.
|
||||
# This is the same tree that served the Qwen3.8-Flash-Next ROCmFP4 STRIX
|
||||
# endpoint 2026-08-30 → 09-11 (see the kingjones runbook), rebuilt here so a
|
||||
# fresh host can be brought up from the repo alone.
|
||||
#
|
||||
# Why this commit and not a newer one: 36e9acd (2026-08-28) is what was
|
||||
# qualified against this payload on 2026-10-02 — it loads ftype 115, survives
|
||||
# warmup, and drafts with the MTP head at acceptance 0.65. The five legacy
|
||||
# builds of 2026-08-25 crashed this same file in ggml_compute_forward_set_rows
|
||||
# (see kingjones-q8agent-rocmfpx.md), so the pin is load-bearing: do not bump
|
||||
# it without re-running that crash test.
|
||||
#
|
||||
# Host notes:
|
||||
# * -DLLAMA_BUILD_WEBUI=OFF is REQUIRED on this box. The tree provisions its
|
||||
# embedded UI assets via npm or the HF llama-ui bucket; npm is absent here
|
||||
# and the bucket download half-failed, aborting the link at 98%. The API
|
||||
# server is unaffected.
|
||||
# * The build links against the host ROCm (/opt/rocm); no bundled toolchain
|
||||
# like the gfx1201 R9700 build needed.
|
||||
set -euo pipefail
|
||||
|
||||
ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
|
||||
# The commit frank-large was qualified with (measured 2026-10-02).
|
||||
ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
|
||||
SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
|
||||
BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
|
||||
GPU_TARGET="${ROCMFPX_GPU_TARGET:-gfx1151}"
|
||||
JOBS="${JOBS:-$(nproc)}"
|
||||
|
||||
# check_pin — refuse a source tree that is not the qualified commit. A moved
|
||||
# fork means different kernels under the same path, and on this model line a
|
||||
# wrong build is a segfault at first request, not a clean error.
|
||||
check_pin() {
|
||||
local commit
|
||||
commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
|
||||
if [[ -n "$commit" && "$commit" != "$ROCMFPX_COMMIT" ]]; then
|
||||
echo "ROCMFPX_COMMIT mismatch: $SRC_DIR is at $commit, pinned $ROCMFPX_COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--verify" ]]; then
|
||||
check_pin
|
||||
[[ -x "$BUILD_DIR/bin/llama-server" ]] || {
|
||||
echo "no ROCmFPX llama-server at $BUILD_DIR/bin/llama-server" >&2
|
||||
exit 1
|
||||
}
|
||||
# The format the endpoint exists for: the fork must carry the ROCmFPx
|
||||
# tensor type, or the payload cannot load at all.
|
||||
"$BUILD_DIR/bin/llama-server" --help 2>&1 | grep -q -- '--spec-type' || {
|
||||
echo "$BUILD_DIR/bin/llama-server has no --spec-type: MTP drafting is impossible" >&2
|
||||
exit 1
|
||||
}
|
||||
"$BUILD_DIR/bin/llama-server" --version | head -1
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "== source: $ROCMFPX_REPO @ $ROCMFPX_COMMIT =="
|
||||
if [[ ! -d "$SRC_DIR/.git" ]]; then
|
||||
mkdir -p "$(dirname "$SRC_DIR")"
|
||||
git clone "$ROCMFPX_REPO" "$SRC_DIR"
|
||||
fi
|
||||
git -C "$SRC_DIR" fetch --quiet origin "$ROCMFPX_COMMIT" || true
|
||||
git -C "$SRC_DIR" checkout -q --detach "$ROCMFPX_COMMIT"
|
||||
check_pin
|
||||
echo " $(git -C "$SRC_DIR" rev-parse HEAD)"
|
||||
|
||||
echo "== configure + build (release, HIP $GPU_TARGET) =="
|
||||
cmake -S "$SRC_DIR" -B "$BUILD_DIR" \
|
||||
-DGGML_HIP=ON \
|
||||
-DGPU_TARGETS="$GPU_TARGET" \
|
||||
-DGGML_NATIVE=ON \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DLLAMA_BUILD_WEBUI=OFF
|
||||
cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
|
||||
|
||||
echo "== verify =="
|
||||
"$BUILD_DIR/bin/llama-server" --version | head -1
|
||||
echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
|
||||
+72
-44
@@ -1077,27 +1077,29 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
|
||||
# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
|
||||
# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
|
||||
# ROOT unit: the runtime is a native binary linked against the host ROCm,
|
||||
# so there is no container image to install and no generated unit to dodge.
|
||||
# relink_unit points /etc/systemd/system/frank-large.service at the repo
|
||||
# file and boot persistence is that unit's [Install] section.
|
||||
# frank-large is the ROCmFPX fork of llama.cpp (kingjones30/ROCmFPX, pinned by
|
||||
# scripts/build-rocmfpx.sh), serving
|
||||
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF:Q8_0_ROCMFPX_AGENT with the
|
||||
# repo's own Q4_0 MTP draft head and its BF16 vision projector, on the Strix
|
||||
# Halo iGPU. It is a ROOT unit: the runtime is a native binary linked against
|
||||
# the host ROCm, so there is no container image to install and no generated
|
||||
# unit to dodge. relink_unit points /etc/systemd/system/frank-large.service at
|
||||
# the repo file and boot persistence is that unit's [Install] section.
|
||||
#
|
||||
# The runtime gate matters as much as the payload gate: gufo is built from
|
||||
# source on this host (there is no prebuilt release), so a machine whose
|
||||
# The runtime gate matters as much as the payload gate: the fork is built from
|
||||
# source on this host, and the payload is ftype 115 (tensor type 103,
|
||||
# GGML_TYPE_Q8_0_ROCMFPX) which only this fork can read — so a machine whose
|
||||
# /home/dank/GitPackages tree was never built has a unit that can never
|
||||
# become healthy, and a green deploy would leave the endpoint dark.
|
||||
#
|
||||
# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
|
||||
# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
|
||||
# same as every cutover since 2026-09-16: nothing starts until 7770 is
|
||||
# released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
# It replaces gufo v0.1.1 + unsloth UD-Q4_K_XL (the rollback target, still
|
||||
# built by scripts/build-gufo.sh with its payload on disk), which had replaced
|
||||
# the stock llama.cpp prebuilt. The port gate is the same as every cutover
|
||||
# since 2026-09-16: nothing starts until 7770 is released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
LLM_LARGE_MMPROJ="${LLM_LARGE_MMPROJ:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
|
||||
retire_halogen_large_quadlet() {
|
||||
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
|
||||
@@ -1132,17 +1134,19 @@ wait_port_free() {
|
||||
deploy_llm_large() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
# endpoint dark with nothing in the deploy log to say why. The gates
|
||||
# mirror serve-frank-large.sh's own predicates exactly: runtime
|
||||
# executable, target GGUF, MTP head, mmproj.
|
||||
if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
|
||||
echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
echo "ERROR: ROCmFPX runtime absent at $LLM_LARGE_RUNTIME" >&2
|
||||
echo "frank-large: build it: bash scripts/build-rocmfpx.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
local missing=0 f
|
||||
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
|
||||
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT" "$LLM_LARGE_MMPROJ"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "ERROR: payload absent at $f" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large-kingjones.sh" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
@@ -1152,18 +1156,14 @@ deploy_llm_large() {
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
# Then sweep a stray manual run -- that is how a bench harness leaves the
|
||||
# port busy. Match the DRAFTER path, which every frank-large command
|
||||
# line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
|
||||
# payload swap: the target moved to a different repository while the
|
||||
# drafter stayed put, so keying on the target filename would have quietly
|
||||
# stopped matching. It is also the right key in principle — it names this
|
||||
# endpoint's speculative drafter, not a binary. Match that, not a binary
|
||||
# name: frank-small on this host serves a different GGUF over llama-server
|
||||
# with no DFlash2 drafter and must not be killed. The pattern never
|
||||
# appears in the deploying shell's own command line, so -f is safe here
|
||||
# (it is not in `bash deploy-remote.sh`).
|
||||
# port busy. Match the TARGET GGUF name, which every frank-large command
|
||||
# line carries via --model and which no other process on this host uses:
|
||||
# frank-small also runs llama-server here (with the MiMo GGUF) and must
|
||||
# not be killed, so keying on the binary name would take the 3090 endpoint
|
||||
# down with it. The pattern never appears in the deploying shell's own
|
||||
# command line, so -f is safe here (it is not in `bash deploy-remote.sh`).
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT' 2>/dev/null || true
|
||||
# Nothing may start until the predecessor has released the port. On
|
||||
# 2026-09-16 the cutover failed exactly here: the old server logged
|
||||
# "Deactivated successfully", the next engine tried to bind 7770 a second
|
||||
@@ -1177,28 +1177,56 @@ deploy_llm_large() {
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
}
|
||||
|
||||
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
|
||||
# ROOT unit: the runtime is a native CUDA binary, so there is no container
|
||||
# image to install and no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
|
||||
# is that unit's [Install] section.
|
||||
# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on
|
||||
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
|
||||
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
|
||||
# no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-small.service at the repo file and boot
|
||||
# persistence is that unit's [Install] section.
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
|
||||
LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
# The projector IS the vision capability here: the Q5_K_M GGUF ships no `v.*`
|
||||
# tensors, so a deploy that starts without this file silently ships a
|
||||
# text-only endpoint. Checked unconditionally, like frank-large's.
|
||||
LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
|
||||
|
||||
deploy_llm_small() {
|
||||
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
|
||||
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
|
||||
echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
|
||||
echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2
|
||||
echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
|
||||
echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
|
||||
echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
|
||||
return 1
|
||||
fi
|
||||
for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "ERROR: frank-small payload file absent at $f" >&2
|
||||
echo "frank-small: run scripts/download-frank-small.sh first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
relink_unit frank-small
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-small.service
|
||||
run_privileged systemctl restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1
|
||||
# False-green hardening (frank-large's lesson): the port answering
|
||||
# proves nothing about WHO answers — the unit must be active with a
|
||||
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
|
||||
local main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-small: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
|
||||
# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
|
||||
# command line (it is world-readable in /proc/<pid>/cmdline).
|
||||
#
|
||||
# The payload is the repo's highest-quality target plus its MTP draft head and
|
||||
# vision projector:
|
||||
# * Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — ftype 115, 187 tensors at plain
|
||||
# Q8_0 and 311 at GGML_TYPE_Q8_0_ROCMFPX (103). The AGENT routing keeps the
|
||||
# output-head tensors at full Q8_0 (187 vs 2 in the non-AGENT build), which
|
||||
# is the variant to take when serving with the MTP head.
|
||||
# * mtp-Qwen3.8-27B-Q4_0.gguf — the draft head. Q4_0, NOT Q8_0: the head's
|
||||
# per-step cost is its projection over a 248K vocabulary, so the cheaper
|
||||
# head is faster end to end at equal acceptance (author: +6.0% at n-max 1,
|
||||
# +10.2% at n-max 2).
|
||||
# * mmproj-Qwen3.8-27B-BF16.gguf — the vision tower (verified 4/4 on spatial
|
||||
# ground truth with -fa on).
|
||||
#
|
||||
# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
|
||||
# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
|
||||
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
|
||||
#
|
||||
# The repo's own SHA256SUMS covers only the ROCmFP4 files, so the hashes below
|
||||
# come from the Hub LFS oids (api/models/<repo>/tree/main).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
|
||||
|
||||
# name bytes sha256
|
||||
FILES=(
|
||||
"Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 28214425952 f3ef2bdbf56e244fff14b72114dc2aa300b84a4cb4eaf54f2e5e4da1cfcae10b"
|
||||
"mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e"
|
||||
"mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
tmp="$f.part"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
Executable
+76
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the Q4_K_XL frank-large payload on Dankclaw:
|
||||
# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf
|
||||
#
|
||||
# bash scripts/download-frank-large-q4.sh
|
||||
# bash scripts/download-frank-large-q4.sh --verify-only
|
||||
#
|
||||
# Integrity: the file is checked against its declared byte count and sha256
|
||||
# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
|
||||
# the repo is public and ungated (gated: false), and a token on the curl
|
||||
# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
|
||||
# a token, put it in a mode-600 file and pass it with `curl -K`.
|
||||
#
|
||||
# Layout after a successful run (MODEL_DIR):
|
||||
# Qwen3.8-27B-UD-Q4_K_XL.gguf 17559178144 (16.35 GiB)
|
||||
#
|
||||
# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
|
||||
# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
|
||||
# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
|
||||
# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
|
||||
# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
|
||||
#
|
||||
# The speculative drafter is UNCHANGED and still fetched by its own script
|
||||
# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
|
||||
# appear here at all — the frank-large seam greps THIS file for it to prove
|
||||
# the drafter was not folded into the payload fetch.
|
||||
#
|
||||
# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
|
||||
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
|
||||
# spec decode here is the block-diffusion drafter, not MTP.
|
||||
set -euo pipefail
|
||||
|
||||
MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
|
||||
REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
|
||||
SIZE=17559178144
|
||||
SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
|
||||
URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
|
||||
|
||||
mode="download"
|
||||
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
||||
|
||||
if [[ "$mode" == "download" ]]; then
|
||||
mkdir -p "$MODEL_DIR"
|
||||
if [[ -s "$MODEL_DIR/$REL" ]]; then
|
||||
echo "already present: $MODEL_DIR/$REL"
|
||||
else
|
||||
echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
|
||||
# -f so an HTTP error is a failure, not a 153-byte "artifact".
|
||||
# -C - so an interrupted run resumes instead of restarting.
|
||||
curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "verifying $MODEL_DIR against declared size + sha256"
|
||||
python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
|
||||
import hashlib, os, sys
|
||||
model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
|
||||
dest = os.path.join(model_dir, rel)
|
||||
if not os.path.isfile(dest):
|
||||
print(f" MISSING {rel}")
|
||||
sys.exit(1)
|
||||
actual = os.path.getsize(dest)
|
||||
if actual != size:
|
||||
print(f" SIZE MISMATCH {rel}: {actual} != {size}")
|
||||
sys.exit(1)
|
||||
h = hashlib.sha256()
|
||||
with open(dest, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1 << 24), b""):
|
||||
h.update(chunk)
|
||||
if h.hexdigest() != sha:
|
||||
print(f" SHA256 MISMATCH {rel}: {h.hexdigest()}")
|
||||
sys.exit(1)
|
||||
print(f" OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
|
||||
PY
|
||||
|
||||
echo "Q4_K_XL payload complete: $MODEL_DIR"
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-small payload for llama.cpp on the RTX 3090:
|
||||
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M plus its vision projector
|
||||
# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
|
||||
# without this file the endpoint is text-only by construction). Public repo —
|
||||
# no auth header, and never a token on the command line (it is world-readable
|
||||
# in /proc/<pid>/cmdline).
|
||||
#
|
||||
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
|
||||
# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
|
||||
# then byte count and the Hub LFS sha256. Idempotent: complete files are
|
||||
# skipped.
|
||||
#
|
||||
# The Q6_K file this endpoint served until 2026-10-01 night stays on disk as
|
||||
# the rollback target (7793710624 B, LFS oid
|
||||
# ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d) — re-fetch
|
||||
# it from the same repo with the same recipe if it is ever needed again.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF"
|
||||
BASE="https://huggingface.co/${REPO}/resolve/main"
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/MiMo-Q5KM}"
|
||||
|
||||
# name bytes sha256
|
||||
FILES=(
|
||||
"MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf 6876124704 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f"
|
||||
"mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf 918166048 ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
read -r name size sha <<<"$entry"
|
||||
f="$DEST/$name"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
if (( have )); then
|
||||
echo "OK $name"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $name"
|
||||
tmp="$f.part"
|
||||
curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
|
||||
mv "$tmp" "$f"
|
||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $name"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $name"
|
||||
done
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
+78
-38
@@ -1,45 +1,85 @@
|
||||
# frank-large — gufo (gufo-org/gufo) serving
|
||||
# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU.
|
||||
# frank-large — ROCmFPX llama.cpp serving
|
||||
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
|
||||
# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
|
||||
#
|
||||
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
||||
# list is owned by the serve script; this file only sets fleet values.
|
||||
# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
|
||||
# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
|
||||
# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
|
||||
# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
|
||||
# tells you to take with MTP, because it keeps the output-head tensors at full
|
||||
# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
|
||||
# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
|
||||
#
|
||||
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
||||
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
|
||||
# exception because the serve script itself consumes it for --host.
|
||||
# The shape: one session at the model's full native window (262,144), which is
|
||||
# the whole point — the Q8_0 head costs quality the fleet wanted back, and
|
||||
# the draft head pays for the bandwidth the bigger file spends.
|
||||
#
|
||||
# Budget (from the GGUF header, not from a boot log): attention KV lives only
|
||||
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
|
||||
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
|
||||
# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
|
||||
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
|
||||
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
|
||||
# here in a way it is not on frank-small's 24 GiB card.
|
||||
|
||||
# The target payload (scripts/download-frank-large-swift.sh): the serve
|
||||
# script refuses to start without the main GGUF and the DFlash2 drafter.
|
||||
# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
|
||||
# are structurally interchangeable for gufo (same qwen35 architecture, the
|
||||
# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
|
||||
# layout, the same MTP head), so only this path changed.
|
||||
MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
|
||||
MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||
# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
|
||||
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
|
||||
# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
|
||||
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
||||
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
|
||||
# projection over a 248K vocabulary, so the cheaper head is faster end to end
|
||||
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
|
||||
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
# Vision tower, separate from the LLM file. Empty string = text-only.
|
||||
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
|
||||
|
||||
# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
|
||||
# the previous target, so it carries its own directory. It has a sliding
|
||||
# window of 2048, which is what keeps its KV small (~40 MiB).
|
||||
DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
|
||||
DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
|
||||
# The gufo runtime, built from source by scripts/build-gufo.sh.
|
||||
RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
|
||||
|
||||
# Fleet endpoint: the same port and alias every frank-large has served, so
|
||||
# no client, probe or models.yml entry changes address.
|
||||
# Fleet endpoint: same port and alias as every frank-large before it, so the
|
||||
# harness models.yml id and the homepage probe do not move.
|
||||
ALIAS=frank-large
|
||||
HOST=0.0.0.0
|
||||
PORT=7770
|
||||
# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
|
||||
# avoids a bare TEMP.
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# One session, full 262144 context (the model's native window).
|
||||
CONTEXT_SIZE=262144
|
||||
SESSIONS=1
|
||||
# One full native window. llama.cpp divides --ctx-size across --parallel
|
||||
# slots, so 262144/1 IS the model's trained window; the card validated 65,536
|
||||
# and the fleet has been running this family at 262,144 since the gufo era.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
|
||||
# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
|
||||
# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
|
||||
# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy
|
||||
# shortens the block when acceptance drops) and DRAFT_POLICY is the block
|
||||
# length controller.
|
||||
DRAFT_TOKENS=7
|
||||
DRAFT_POLICY=adaptive
|
||||
# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
|
||||
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
|
||||
# speed), and --fit must stay off: --fit's auto placement does not know that
|
||||
# 311 tensors of type 103 belong on the iGPU.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
NGL=999
|
||||
FIT=off
|
||||
FLASH_ATTN=on
|
||||
# Unquantized KV — the quality-first choice, affordable at one slot.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
|
||||
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
|
||||
# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
|
||||
# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
|
||||
# The draft runs entirely on the iGPU, so cross-device traffic stays off the
|
||||
# bandwidth-limited link.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_NGL=99
|
||||
SPEC_DRAFT_DEVICE=ROCm0
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.0
|
||||
|
||||
# The card's thinking sampling profile, which is also what the harness sends
|
||||
# in extraBody, so the endpoint behaves the same to a client that omits it.
|
||||
TEMPERATURE=1.0
|
||||
TOP_P=0.95
|
||||
TOP_K=20
|
||||
MIN_P=0.0
|
||||
PRESENCE_PENALTY=0.0
|
||||
|
||||
NUM_THREADS=16
|
||||
THREAD_BATCH=16
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
Description=ROCmFPX llama.cpp (gfx1151) kingjones777 Qwen3.8-27B Q8_0_ROCMFPX_AGENT, MTP spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -13,7 +13,9 @@ Group=dank
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
||||
Restart=on-failure
|
||||
# always, not on-failure: an engine that drains and exits 0 after a worker
|
||||
# fault would otherwise leave the endpoint down until someone notices.
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
|
||||
+97
-29
@@ -1,37 +1,44 @@
|
||||
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
||||
#
|
||||
# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
|
||||
# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
|
||||
# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
|
||||
# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
|
||||
# digest-pinned image the 2026-09-29 tuning campaign measured — serving
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
|
||||
# the MTP4 drafter on the two R9700s.
|
||||
# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
|
||||
# campaign's verdict, Frank's call): same digest-pinned engine and payload as
|
||||
# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
|
||||
# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
|
||||
# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
|
||||
# restorable from the tag.
|
||||
#
|
||||
# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
|
||||
# * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
|
||||
# the engine measured "GPU KV cache size: 729,545 tokens; Maximum
|
||||
# concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
|
||||
# per rank. KV_BYTES is PER RANK — sizing it to the global requirement
|
||||
# OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
|
||||
# free; 7.2 GiB boots).
|
||||
# * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
|
||||
# — the stock flag quantizes them together, which is exactly what was
|
||||
# requested here). Accepted by the attention path; qualified with a
|
||||
# completion probe.
|
||||
# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
|
||||
# * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
|
||||
# shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
|
||||
# engine measured "KV cache size: 732,359 tokens; Maximum concurrency
|
||||
# for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
|
||||
# at graph capture (76 MiB short / 0 bytes free); 237568 passed at
|
||||
# 718,606 tokens and the +4096 bonus step passed and won.
|
||||
# * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
|
||||
# the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
|
||||
# the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
|
||||
# 11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
|
||||
# * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
|
||||
# 7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
|
||||
# the gather win) — traded deliberately for clean cache + 262k windows.
|
||||
# * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
|
||||
# R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
|
||||
# and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
|
||||
# era). Do not drop it.
|
||||
# * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
|
||||
# vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
|
||||
# and arena. 1024 keeps the workspace floor so the arena stays max.
|
||||
# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
|
||||
# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
|
||||
# roughly half the gather traffic was waste; prefetch off is the tuned
|
||||
# incumbent and the +40-51% decode came from deleting the flag. Never
|
||||
# re-add --expert-prefetch without a fresh A/B.
|
||||
# roughly half the gather traffic was waste. Never re-add
|
||||
# --expert-prefetch without a fresh A/B.
|
||||
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
|
||||
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
|
||||
# numerics (repetition loops, early halts, invalid measurements).
|
||||
# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
|
||||
# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
|
||||
# numerics.
|
||||
#
|
||||
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
||||
# prefixed FRANK_PRIMARY_ so none can leak.
|
||||
#
|
||||
|
||||
# --- runtime ----------------------------------------------------------------
|
||||
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
||||
@@ -52,10 +59,71 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape ---------------------------------------------------
|
||||
FRANK_PRIMARY_MAXLEN=200000
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
FRANK_PRIMARY_KV_BYTES=7730934784
|
||||
# --- qualified engine shape (2026-10-02, Frank's own correction) ---
|
||||
# 2026-10-01 left a shape that did not stay up: bf16 KV at 2 x 262144 with
|
||||
# --expert-precise-memory crash-looped at graph capture (the arena is
|
||||
# DEVICE-resident and the shape had 0.13 GiB to spare), and the plain bf16
|
||||
# 2 x 262144 boot booted but left 0 bytes free at runtime. This is the
|
||||
# correction: FP8 KV, TWO windows, each 200704 tokens, and the pool cut so
|
||||
# the expert arena gets the room back.
|
||||
#
|
||||
# LADDER (2026-10-01 night, subtract-16k method): 262144 and 245760 OOM at
|
||||
# graph capture; 237568 passed at 718,606 tokens; 241664 x 3 passed at
|
||||
# 732,359 tokens / 3.03x but is the shape this cutover retired. 200704 was
|
||||
# chosen to sit below the 241664 rung while leaving a full 2.95 GiB/rank of
|
||||
# pool to cut loose for the arena — the two contexts are a capacity
|
||||
# contract, not a throughput one (2-way concurrency summed BELOW parity at
|
||||
# 4064 ctx), so the slot buys the VRAM headroom long generations need.
|
||||
#
|
||||
# KV_BYTES is PER RANK. fp8 costs 7,701 B/token (measured 6,138 attention
|
||||
# + 1,563 mamba/side, dtype-independent). 2 x 200704 = 401,408 tokens
|
||||
# -> 3.091e9 B floor, pinned at 3.15e9 (~405,000 tokens, ~44 MB margin —
|
||||
# the same margin convention the 6.1e9 shape used).
|
||||
#
|
||||
# THE ARENA IS WHAT REMAINS: the fork's expert planner sizes it AFTER the
|
||||
# KV override, so every byte cut here is arena. The 6.1e9 shape ran an
|
||||
# arena of 6,248 slots; this one reclaims ~2.95 GiB/rank, roughly 1,260
|
||||
# more slots at 2.344 MiB each. That is the point of the change — the
|
||||
# gather read path is the measured decode bound, and a bigger arena means
|
||||
# more experts resident instead of streamed.
|
||||
#
|
||||
# MNBT 1024 (down from 2048): 1024-4096 are interchangeable on decode AND
|
||||
# identical on arena (the sweep measured the same slot count at every
|
||||
# rung <= 4096), so this is a pure workspace-floor choice, not a decode
|
||||
# trade. The floor matters because the arena budget is what is left.
|
||||
#
|
||||
# PRECISE MEMORY stays DROPPED: it crashed the 2 x 262144 bf16 boot
|
||||
# (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45
|
||||
# = 31.73 of 31.86 GiB, ~0.13 GiB free, NRestarts=3). The flag survives in
|
||||
# the serve script as an opt-in; re-enable it only with a pool/context
|
||||
# reduction that actually pays for the extra overhead.
|
||||
#
|
||||
# TUNABLEOP stays OFF (silently skipped under offload unless
|
||||
# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics).
|
||||
#
|
||||
# SEQS 2 x MAXLEN 200704 at FP8 KV, pool cut for the arena. No
|
||||
# speculative decode, no prefetch.
|
||||
FRANK_PRIMARY_MAXLEN=200704
|
||||
FRANK_PRIMARY_SEQS=2
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_SPEC=4
|
||||
FRANK_PRIMARY_MNBT=1024
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
FRANK_PRIMARY_GATHER_LANES=64
|
||||
FRANK_PRIMARY_UTIL=0.98
|
||||
|
||||
# --- client contract: reasoning split, tools, preserve_thinking ----------------
|
||||
# Without --reasoning-parser the engine leaves the thinking text INLINE in
|
||||
# `content` with a dangling `</think>` and a null `reasoning` field (measured
|
||||
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
|
||||
# malformed thinking block clients choke on. The qwen3 parser splits it
|
||||
# properly. The GLM rollback (d91379d) had dropped all three of these; the
|
||||
# working values are the ones ecaad36 shipped for this model.
|
||||
FRANK_PRIMARY_REASONING_PARSER=qwen3
|
||||
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
|
||||
# tool_calls work here (without it, tool_choice auto returns 400).
|
||||
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
|
||||
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
|
||||
# turns; enable_thinking makes thinking the server-side default. Verified the
|
||||
# payload's tokenizer_config template consumes both keys.
|
||||
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
|
||||
@@ -8,8 +8,10 @@
|
||||
#
|
||||
# Same port, same served alias; the harness models.yml id is unchanged.
|
||||
# The serve script drives `podman run` directly (the image entrypoint execs
|
||||
# `vllm serve`) and then waits on the container, so a crashed engine is a
|
||||
# nonzero ExecStart and Restart=on-failure is real.
|
||||
# `vllm serve`) and then waits on the container. NOTE: that wait can return
|
||||
# 0 even when the engine died mid-request (vLLM's API server drains and
|
||||
# exits clean on a worker fault), which is why the restart policy is
|
||||
# `always` — see the [Service] note.
|
||||
#
|
||||
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
|
||||
# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
|
||||
@@ -23,6 +25,9 @@
|
||||
Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
||||
# endpoint that systemd refuses to retry is worse than a retrying one).
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
User=dank
|
||||
@@ -39,8 +44,23 @@ ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
|
||||
# Boot cost: FP8 payload page-in plus torch compile/graph capture —
|
||||
# ~15 min cold (measured 2026-09-29), faster warm.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
|
||||
# torch.OutOfMemoryError kills the engine while vLLM's API server still
|
||||
# exits 0, so ExecStart returned clean, systemd logged "Deactivated
|
||||
# successfully", and on-failure NEVER fired — the endpoint stayed down until
|
||||
# a human noticed. `always` restarts on any container death; an explicit
|
||||
# `systemctl stop` still wins (systemd does not restart a unit the operator
|
||||
# stopped), so bench windows and deploys are unaffected.
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
|
||||
# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
|
||||
# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
|
||||
# Must exceed the podman grace with margin for the engine's own shutdown.
|
||||
TimeoutStopSec=180
|
||||
# The engine pins tens of GiB of host RAM (expert host slabs); it is the
|
||||
# last process on this host that should be chosen by the kernel OOM killer.
|
||||
OOMScoreAdjust=-500
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
+49
-107
@@ -1,118 +1,60 @@
|
||||
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
||||
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
||||
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
||||
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
||||
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
|
||||
#
|
||||
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
||||
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
||||
# CUDA (see the CUDA lesson in AGENTS.md).
|
||||
# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
|
||||
# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
|
||||
# window shrinks — see the budget below.
|
||||
#
|
||||
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
||||
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
||||
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
||||
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
||||
# only a 1024-position window, which is why the KV pool is a fraction of
|
||||
# the full-ctx x all-layers number.
|
||||
# Two measurements on this card, this runtime, price the terms:
|
||||
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
||||
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
||||
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
||||
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
||||
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
||||
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
|
||||
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
|
||||
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
|
||||
# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
|
||||
# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
|
||||
#
|
||||
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
||||
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
||||
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
||||
# context loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
||||
# is the only legal shape that carries both.
|
||||
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
||||
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
|
||||
# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
|
||||
# is one 293 MiB step — that is the ladder to walk if this moves again. The
|
||||
# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
|
||||
# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
||||
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
||||
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
||||
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
||||
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
||||
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
||||
# which would shrink each slot below the full 262144 window once
|
||||
# parallel > 1.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
||||
# params; it carries an audio encoder too). The fleet's image path lives
|
||||
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
||||
# WORKS: input_audio content parts transcribe verbatim (verified
|
||||
# 2026-09-17 with a 16k mono WAV).
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
# temperature gets its own, and the harness applies per-model overrides
|
||||
# from models.yml.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
||||
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
|
||||
# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
|
||||
# the tensor the asymmetric-KV contract exists to protect.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
|
||||
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
||||
# FA_ALL_QUANTS=ON.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
|
||||
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
||||
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
||||
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
|
||||
|
||||
# Model package. The MTP sidecar file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
||||
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
||||
# what clients ask for -- the harness entry in models.yml names it.
|
||||
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
||||
# harness models.yml id and the homepage probe do not move.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
||||
# pool; --parallel divides it). See the SIZING note above for the fit
|
||||
# arithmetic; a fifth slot does not fit.
|
||||
CTX_SIZE=1048576
|
||||
PARALLEL=4
|
||||
CACHE_TYPE_K=q8_0
|
||||
# Three independent conversations. llama.cpp divides --ctx-size across
|
||||
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
||||
# cannot use unified KV, so each slot gets its own cache and its own window.
|
||||
# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
|
||||
# third slot: a longer single prompt needs compaction instead of one shot.
|
||||
# Consequence worth knowing: frank-primary's compaction trigger is 183,501
|
||||
# tokens, which no longer fits a frank-small slot — see the runbook's
|
||||
# "three slots vs compaction" note.
|
||||
CTX_SIZE=540672
|
||||
PARALLEL=3
|
||||
|
||||
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
||||
# q8_0 on V is only served by the flash-attention path; the serve script
|
||||
# pins --flash-attn on and must never lose it.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=q8_0
|
||||
# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
|
||||
# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
|
||||
# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
|
||||
# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
|
||||
# model, so the >12.8 GiB CACHE_RAM floor does not apply.
|
||||
CACHE_RAM=2048
|
||||
|
||||
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
|
||||
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
|
||||
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
|
||||
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
|
||||
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
|
||||
# tried. Idle VRAM is 23,300 MiB at this geometry.
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
||||
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention (required for q8 KV).
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
||||
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
||||
# explicit here (both were the implicit build defaults: 3 and 50).
|
||||
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
|
||||
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
|
||||
# costs more attention re-read at long context than the extra accepted
|
||||
# token earns. Keep 3/50.
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
POLL_LEVEL=50
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
NGL=99
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
|
||||
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -8,8 +8,7 @@ Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||
# though the weights themselves are copied into VRAM. Parity with the halogen
|
||||
# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
|
||||
# though the weights themselves are copied into VRAM.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
||||
|
||||
+163
-105
@@ -1,121 +1,179 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on
|
||||
# Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the
|
||||
# inference engine built specifically for this hardware.
|
||||
# Serve kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF as frank-large on
|
||||
# Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of llama.cpp.
|
||||
#
|
||||
# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
|
||||
# against the host ROCm by scripts/build-gufo.sh. It replaced the stock
|
||||
# llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client
|
||||
# contract did not move.
|
||||
# 2026-10-02 (Frank): swapped from gufo + Q4_K_XL to the highest-quality file
|
||||
# in the repo, one session at the native 262,144 window.
|
||||
#
|
||||
# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's
|
||||
# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally
|
||||
# interchangeable for this engine — same `qwen35` architecture, the same 866
|
||||
# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the
|
||||
# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune
|
||||
# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which
|
||||
# is not an OSI licence.
|
||||
# Why this runtime and not the previous one: the payload is ftype 115 — 311
|
||||
# tensors of GGML_TYPE_Q8_0_ROCMFPX (103), a tensor type that exists only in
|
||||
# the ROCmFPX fork. Stock llama.cpp, SGLang and gufo all refuse the file, and
|
||||
# gufo additionally gates every "Qwen3.8" artifact on a template-hash
|
||||
# allowlist. The qualified build is scripts/build-rocmfpx.sh's pin: it loads
|
||||
# ftype 115, survives warmup (the 2026-08-25 builds of the older fork line
|
||||
# segfaulted in ggml_compute_forward_set_rows on this exact file), and drafts
|
||||
# with the model's own MTP head.
|
||||
#
|
||||
# Shape: one session, full 262144 context (the model's native window). The
|
||||
# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
||||
# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
|
||||
# context measured 66.5 GiB of the 116.7 GiB device.
|
||||
# Why this file: Q8_0_ROCMFPX_AGENT is the top of the repo's quality ladder
|
||||
# (26.28 GiB, 8.39 bpw) and the AGENT routing keeps 187 tensors — the
|
||||
# output-head set the draft scores against — at plain Q8_0 instead of the 2
|
||||
# the non-AGENT build keeps, which is the variant the card names for MTP
|
||||
# serving. Drafting uses the repo's Q4_0 MTP head: the head's cost is its
|
||||
# projection over the 248K vocabulary, so the 4-bit head is faster end to end
|
||||
# at equal-or-better acceptance than its Q8_0 sibling.
|
||||
#
|
||||
# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
|
||||
# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
|
||||
# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
|
||||
# (small draft KV). It is UNCHANGED by the model swap and still lives beside
|
||||
# the previous target, which is why DRAFTER_DIR is its own tunable.
|
||||
#
|
||||
# Sampling: greedy (--temperature 0) server-side, as every frank-large has
|
||||
# served. The harness overrides it per request via extraBody (1.0 / 0.95 /
|
||||
# top-k 20).
|
||||
#
|
||||
# Template: gufo compiles the official Qwen Jinja into the binary and takes
|
||||
# NO --chat-template-file — and it is not a Jinja engine at all
|
||||
# (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the
|
||||
# fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law
|
||||
# was given up when this endpoint moved to gufo on 2026-09-28, by decision:
|
||||
# the terse system-prompt injection, the in-band <|think_*|> control, the
|
||||
# json|xml tool_call_format switch and the tool-argument truncation are all
|
||||
# absent. gufo's own renderer keeps the effort instruction strings and
|
||||
# preserve_thinking, which are the parts the harness depends on. Accepted
|
||||
# 2026-09-28; recovering the terse prompt from the client side is the
|
||||
# cheapest partial restore.
|
||||
#
|
||||
# Text-only: gufo auto-discovers a mmproj beside the model and this payload
|
||||
# has none (the base repo ships one separately, unused), so the endpoint
|
||||
# stays text-only.
|
||||
#
|
||||
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
|
||||
# it; anything not set there falls through to the defaults below.
|
||||
# The client contract lives here, not in frank-large.env:
|
||||
# * --jinja — without it llama.cpp silently IGNORES the harness's
|
||||
# chat_template_kwargs (enable_thinking / preserve_thinking), so the
|
||||
# endpoint would look like it had no thinking control at all. The GGUF
|
||||
# carries the official Qwen template, so no --chat-template-file.
|
||||
# * --reasoning-format deepseek — thinking is emitted as reasoning_content,
|
||||
# the field the harness reads for this endpoint.
|
||||
# * the template accepts reasoning_effort xhigh|medium|low only, which is
|
||||
# why models.yml folds the harness's levels onto that set.
|
||||
# * --alias frank-large — that is the model id the harness resolves.
|
||||
# * --ctx-checkpoints / --cache-ram — prompt-cache geometry for the 16
|
||||
# sliding-window-style layers; harmless where the hybrid state forces a
|
||||
# full re-process, and required for the checkpoint feature to work at all.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
||||
# Fleet values live in frank-large.env. Anything already in the environment
|
||||
# wins over the file, so an operator can override one value on a single run:
|
||||
# the caller's values are captured, the file is sourced, then they are put
|
||||
# back. (A plain `source` would let the file clobber the environment.)
|
||||
TUNABLES=(LLAMA_BIN MODEL_FILE DRAFT_FILE MMPROJ_FILE ALIAS BIND_ADDR
|
||||
LISTEN_PORT CTX_SIZE PARALLEL BATCH_SIZE UBATCH_SIZE NGL FIT
|
||||
FLASH_ATTN CACHE_TYPE_K CACHE_TYPE_V SPEC_TYPE SPEC_DRAFT_NGL
|
||||
SPEC_DRAFT_DEVICE SPEC_DRAFT_N_MAX SPEC_DRAFT_N_MIN SPEC_DRAFT_P_MIN
|
||||
TEMPERATURE TOP_P TOP_K MIN_P PRESENCE_PENALTY NUM_THREADS
|
||||
THREAD_BATCH ROCM_PATH HIP_VISIBLE HSA_GFX_VERSION)
|
||||
_OVERRIDES=()
|
||||
for _var in "${TUNABLES[@]}"; do
|
||||
[[ -n "${!_var+set}" ]] && _OVERRIDES+=("$_var=${!_var}")
|
||||
done
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$(dirname "$0")/frank-large.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) ------------------
|
||||
: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}"
|
||||
: "${ALIAS:=frank-large}"
|
||||
: "${HOST:=0.0.0.0}"
|
||||
: "${PORT:=7770}"
|
||||
: "${CONTEXT_SIZE:=262144}"
|
||||
: "${SESSIONS:=1}"
|
||||
: "${DRAFT_TOKENS:=7}"
|
||||
: "${DRAFT_POLICY:=adaptive}"
|
||||
|
||||
GUFO_BIN="$RUNTIME_DIR/bin/gufo"
|
||||
|
||||
# The target GGUF and the drafter are named explicitly because they do not
|
||||
# share a directory any more: the target is a different repository's artifact
|
||||
# and keeps its own long filename, while the DFlash2 drafter is unchanged and
|
||||
# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
|
||||
# the deployed payload, so the env file only has to state them when it moves.
|
||||
: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
|
||||
: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
||||
: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
|
||||
if [[ ! -x "$GUFO_BIN" ]]; then
|
||||
echo "runtime not found at $GUFO_BIN" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2
|
||||
exit 1
|
||||
fi
|
||||
for _kv in "${_OVERRIDES[@]+${_OVERRIDES[@]}}"; do
|
||||
export "${_kv?}"
|
||||
done
|
||||
unset _var _kv
|
||||
|
||||
# --- env -------------------------------------------------------------------
|
||||
# gufo links against the host ROCm through its RPATH (/opt/rocm ->
|
||||
# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
|
||||
# already gfx1151 — `gufo diagnose` reports it natively — so no
|
||||
# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
|
||||
# llama.cpp runtime and can only misreport the arch here.
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
: "${LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
|
||||
: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
: "${ALIAS=frank-large}"
|
||||
: "${BIND_ADDR=0.0.0.0}"
|
||||
: "${LISTEN_PORT=7770}"
|
||||
: "${CTX_SIZE=262144}"
|
||||
: "${PARALLEL=1}"
|
||||
: "${BATCH_SIZE=2048}"
|
||||
: "${UBATCH_SIZE=512}"
|
||||
: "${NGL=999}"
|
||||
: "${FIT=off}"
|
||||
: "${FLASH_ATTN=on}"
|
||||
: "${CACHE_TYPE_K=f16}"
|
||||
: "${CACHE_TYPE_V=f16}"
|
||||
: "${SPEC_TYPE=draft-mtp}"
|
||||
: "${SPEC_DRAFT_NGL=99}"
|
||||
: "${SPEC_DRAFT_DEVICE=ROCm0}"
|
||||
: "${SPEC_DRAFT_N_MAX=4}"
|
||||
: "${SPEC_DRAFT_N_MIN=0}"
|
||||
: "${SPEC_DRAFT_P_MIN=0.0}"
|
||||
: "${TEMPERATURE=1.0}"
|
||||
: "${TOP_P=0.95}"
|
||||
: "${TOP_K=20}"
|
||||
: "${MIN_P=0.0}"
|
||||
: "${PRESENCE_PENALTY=0.0}"
|
||||
: "${NUM_THREADS=16}"
|
||||
: "${THREAD_BATCH=16}"
|
||||
: "${ROCM_PATH=/opt/rocm}"
|
||||
: "${HIP_VISIBLE=0}"
|
||||
: "${HSA_GFX_VERSION=11.5.1}"
|
||||
|
||||
# --- the serve command ---------------------------------------------------
|
||||
# Server options precede the subcommand, model options follow it.
|
||||
exec "$GUFO_BIN" serve \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
--sessions "$SESSIONS" \
|
||||
llm \
|
||||
[[ -n "$LLAMA_BIN" ]] || { echo "frank-large: LLAMA_BIN is empty" >&2; exit 1; }
|
||||
[[ -x "$LLAMA_BIN" ]] || {
|
||||
echo "frank-large: llama-server '$LLAMA_BIN' is not executable;" >&2
|
||||
echo " build it with scripts/build-rocmfpx.sh" >&2
|
||||
exit 1
|
||||
}
|
||||
# Refuse anything but the real payload. A path that is not a readable file
|
||||
# makes llama-server hang rather than fail clean.
|
||||
[[ -r "$MODEL_FILE" ]] || {
|
||||
echo "frank-large: model file '$MODEL_FILE' is not readable;" >&2
|
||||
echo " fetch it with scripts/download-frank-large-kingjones.sh" >&2
|
||||
exit 1
|
||||
}
|
||||
# The MTP head is the performance feature: without it the Q8_0 weights decode
|
||||
# at memory-bandwidth speed and the endpoint loses the ~1.4x the head buys.
|
||||
[[ -r "$DRAFT_FILE" ]] || {
|
||||
echo "frank-large: MTP draft head '$DRAFT_FILE' is not readable;" >&2
|
||||
echo " fetch it with scripts/download-frank-large-kingjones.sh" >&2
|
||||
exit 1
|
||||
}
|
||||
# The GGUF carries no vision tower; the projector file is the whole tower, so
|
||||
# a missing file means a vision request would 500 rather than degrade.
|
||||
# MMPROJ_FILE="" is the documented opt-out for a text-only endpoint.
|
||||
MM_ARGS=()
|
||||
if [[ -n "${MMPROJ_FILE:-}" ]]; then
|
||||
[[ -r "$MMPROJ_FILE" ]] || {
|
||||
echo "frank-large: mmproj '$MMPROJ_FILE' is not readable; set MMPROJ_FILE='' for text-only" >&2
|
||||
exit 1
|
||||
}
|
||||
MM_ARGS=(--mmproj "$MMPROJ_FILE")
|
||||
fi
|
||||
|
||||
# HIP/ROCm host environment. HIP reads bare env vars (TEMP, HOST,
|
||||
# GPU_MAX_HEAP_SIZE), which is why the tunables above are named TEMPERATURE
|
||||
# and BIND_ADDR. HSA_OVERRIDE_GFX_VERSION is what lets a runtime built for a
|
||||
# neighbouring gfx target run on this iGPU; the qualified pin targets gfx1151
|
||||
# natively and the override is a no-op for it.
|
||||
export ROCM_PATH
|
||||
export HIP_VISIBLE_DEVICES="$HIP_VISIBLE"
|
||||
export HSA_OVERRIDE_GFX_VERSION="$HSA_GFX_VERSION"
|
||||
export LD_LIBRARY_PATH="$(dirname "$LLAMA_BIN"):${ROCM_PATH}/lib:${ROCM_PATH}/lib/rocm_sysdeps/lib:${ROCM_PATH}/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
||||
|
||||
exec /usr/bin/env "$LLAMA_BIN" \
|
||||
--model "$MODEL_FILE" \
|
||||
--served-model-name "$ALIAS" \
|
||||
--context "$CONTEXT_SIZE" \
|
||||
--speculative dflash2 \
|
||||
--dflash-model "$DRAFTER_FILE" \
|
||||
--draft-tokens "$DRAFT_TOKENS" \
|
||||
--draft-policy "$DRAFT_POLICY" \
|
||||
--temperature 0 \
|
||||
--think auto \
|
||||
--preserve-thinking on
|
||||
"${MM_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--jinja \
|
||||
--device ROCm0 \
|
||||
--n-gpu-layers "$NGL" \
|
||||
--parallel "$PARALLEL" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--fit "$FIT" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
--direct-io \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--ctx-checkpoints 16 \
|
||||
--cache-ram 8192 \
|
||||
--cache-idle-slots \
|
||||
--temp "$TEMPERATURE" \
|
||||
--top-p "$TOP_P" \
|
||||
--top-k "$TOP_K" \
|
||||
--min-p "$MIN_P" \
|
||||
--presence-penalty "$PRESENCE_PENALTY" \
|
||||
--reasoning on \
|
||||
--reasoning-format deepseek \
|
||||
--reasoning-budget -1 \
|
||||
--spec-type "$SPEC_TYPE" \
|
||||
--model-draft "$DRAFT_FILE" \
|
||||
--spec-draft-ngl "$SPEC_DRAFT_NGL" \
|
||||
--spec-draft-device "$SPEC_DRAFT_DEVICE" \
|
||||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
|
||||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
|
||||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
|
||||
--metrics
|
||||
@@ -11,12 +11,16 @@
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
#
|
||||
# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
|
||||
# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
|
||||
# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
|
||||
# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
|
||||
# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
|
||||
# precision — the tuned incumbent deletes it), TunableOp off.
|
||||
# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
|
||||
# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
|
||||
# fp8; the bf16 unquantized shape is retired to git tag
|
||||
# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
|
||||
# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
|
||||
# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
|
||||
# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
|
||||
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
|
||||
# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
|
||||
# the pool deliberately small so the expert arena keeps what it reclaims.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
@@ -45,7 +49,10 @@ set +a
|
||||
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
||||
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
||||
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
|
||||
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
|
||||
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
|
||||
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
|
||||
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
||||
# classic `podman images | grep -q` pattern races — grep exits on its first
|
||||
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
||||
@@ -100,17 +107,24 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
|
||||
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
|
||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||
"$FRANK_PRIMARY_IMAGE" /models \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||
--tensor-parallel-size 2 \
|
||||
${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
|
||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
||||
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
--enable-prefix-caching \
|
||||
--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
|
||||
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
||||
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
||||
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
||||
--trust-remote-code \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
|
||||
+57
-116
@@ -1,39 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
|
||||
# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
|
||||
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
|
||||
# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
#
|
||||
# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
|
||||
# weights plus the BF16 vision projector and q8_0 KV inside the card's
|
||||
# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
|
||||
# belongs to the halogen engine that serves frank-large.
|
||||
# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
|
||||
# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
|
||||
# runtime on this host that can serve the MiMo qwen35 hybrid with
|
||||
# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v
|
||||
# q8_0 (quantized values), which is Frank's explicit KV contract for this
|
||||
# endpoint. SGLang's --kv-cache-dtype only moves K and V together.
|
||||
#
|
||||
# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
|
||||
# passed via --model-draft with --spec-type draft-mtp: the draft context
|
||||
# loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so this chain is the
|
||||
# only shape that carries both.
|
||||
# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
|
||||
# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
|
||||
# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
|
||||
# separate projector (the Q5_K_M GGUF carries no `v.*` tensors at all), so
|
||||
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
|
||||
# text-only opt-out.
|
||||
#
|
||||
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
|
||||
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
|
||||
# maximises the TOTAL pool instead, which would divide a smaller window
|
||||
# across slots. The 4-slot layout is measured to fit the card (see the
|
||||
# SIZING note in frank-small.env); more slots would fail the KV allocation.
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends its
|
||||
# recipe from models.yml); greedy is the harness default.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector, which also carries
|
||||
# an audio encoder — input_audio content parts transcribe verbatim
|
||||
# (verified 2026-09-17). The fleet's image path lives here, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
# models.yml. --jinja is pinned even though this build defaults it on: the
|
||||
# model's own template supplies reasoning_effort, preserve_thinking and the
|
||||
# tool-call grammar, and a future build that flips that default must not
|
||||
# silently take tools away.
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -46,101 +32,56 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}"
|
||||
: "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
# NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO
|
||||
# payload, which frank-small served on 2026-09-16/17 and which no longer
|
||||
# exists on disk. frank-small.env has always overridden it, so the drift was
|
||||
# invisible — but a missing or unreadable env file would have left this
|
||||
# script pointing at a payload that is not there, turning a config problem
|
||||
# into a "missing asset" that names the wrong model.
|
||||
: "${DRAFT_FILE:=}"
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
: "${CTX_SIZE:=}"
|
||||
: "${PARALLEL:=1}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
|
||||
# 180224 per slot — three independent conversations (the native 262144 no
|
||||
# longer fits three times on a 24 GiB card once the vision tower is loaded;
|
||||
# frank-small.env carries the two measurements that set this number).
|
||||
: "${CTX_SIZE:=540672}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
|
||||
# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
|
||||
# busy-wait level (build default 50).
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${POLL_LEVEL:=50}"
|
||||
: "${NUM_THREADS:=8}"
|
||||
: "${THREAD_BATCH:=8}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
: "${NGL:=99}"
|
||||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||||
: "${MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
# is a download that has not finished, and the unit's restart policy would
|
||||
# otherwise turn that into a crash loop that reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
|
||||
# Fail loudly rather than half-loading: a missing file is a download that
|
||||
# has not finished, and Restart=always would turn it into a crash loop that
|
||||
# reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||||
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
||||
done
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
[[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
|
||||
EXTRA_ARGS=()
|
||||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
|
||||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||||
fi
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
fi
|
||||
|
||||
DRAFT_ARGS=()
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
|
||||
fi
|
||||
|
||||
# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
|
||||
# whatever the weights, projector, draft context and compute buffers leave
|
||||
# on the card; the engine prints the chosen value at startup.
|
||||
if [[ -n "${CTX_SIZE:-}" ]]; then
|
||||
CTX_ARGS=(--ctx-size "$CTX_SIZE")
|
||||
else
|
||||
CTX_ARGS=(--fit on)
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
"${DRAFT_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
"${CTX_ARGS[@]}" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--poll "$POLL_LEVEL" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--cache-ram "$CACHE_RAM" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET"
|
||||
# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
|
||||
# on the FA path); an FA-less build silently falls back and the KV contract
|
||||
# in frank-small.env becomes a lie, so pin the flag pair here.
|
||||
ARGS=(
|
||||
--model "$MODEL_FILE"
|
||||
--alias "$ALIAS"
|
||||
--host "$BIND_ADDR"
|
||||
--port "$LISTEN_PORT"
|
||||
--ctx-size "$CTX_SIZE"
|
||||
--parallel "$PARALLEL"
|
||||
--n-gpu-layers "$NGL"
|
||||
--flash-attn on
|
||||
--cache-type-k "$CACHE_TYPE_K"
|
||||
--cache-type-v "$CACHE_TYPE_V"
|
||||
--jinja
|
||||
--reasoning auto
|
||||
--reasoning-format deepseek
|
||||
)
|
||||
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|
||||
@@ -244,13 +244,155 @@ scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
|
||||
are `podman cp`'d into the container after every recreate (the serve script
|
||||
re-copies them; a recreate wipes container-local files).
|
||||
|
||||
## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
|
||||
|
||||
Two same-day cutovers moved this page's findings into production:
|
||||
|
||||
1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
|
||||
window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
|
||||
`frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
|
||||
production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
|
||||
measured 37.4-40.6 tok/s at ~54% acceptance.
|
||||
2. **The overnight redo campaign re-measured EVERY lever at matched
|
||||
geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
|
||||
and concluded the day: the incumbent cutover to **bf16 KV, no
|
||||
speculative decode** (Frank's call).
|
||||
|
||||
### What the redo campaign measured (all at shallow depth, 166/1087/4064)
|
||||
|
||||
| lever | verdict |
|
||||
|---|---|
|
||||
| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
|
||||
| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
|
||||
| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
|
||||
| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
|
||||
| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
|
||||
| UVA | 2048 POISON (-39%); 8192 ~parity |
|
||||
| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
|
||||
| six slots | every SEQS-6 KV sizing refused at engine init |
|
||||
|
||||
Depth caveat: every number above is at a near-empty KV. Deep-context
|
||||
behavior (the gather win growing with depth, bf16's decode cost at depth)
|
||||
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
|
||||
### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=200704 x SEQS 2 (401,408 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
|
||||
`--expert-precise-memory` — which crash-looped at graph capture: the arena is
|
||||
DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
|
||||
5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
|
||||
graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
|
||||
NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
|
||||
stay up: it left 0 bytes free at runtime and a real client request (995-token
|
||||
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
|
||||
... 0 bytes free`.
|
||||
|
||||
The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
|
||||
(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
|
||||
200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
|
||||
above the floor — the same convention the 6.1e9 shape used. The arena is what
|
||||
remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
|
||||
6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
|
||||
contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
|
||||
ctx), so the slot buys the VRAM headroom long generations need.
|
||||
|
||||
Boot-probed 2026-10-02 (transient systemd units at the production uid +
|
||||
memlock, full-log capture; the rootless bench probe had masked the production
|
||||
memory environment). The night searched two concurrency contracts:
|
||||
|
||||
| rung | result |
|
||||
|---|---|
|
||||
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
|
||||
| 3 x 245760, bf16 | **OOM identically** |
|
||||
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
|
||||
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
|
||||
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
|
||||
| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
|
||||
7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
|
||||
decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
|
||||
graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
|
||||
needs 12.47 GB even at fp8.
|
||||
|
||||
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
|
||||
reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
|
||||
reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
|
||||
which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
|
||||
read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
|
||||
only available while the container lives).
|
||||
|
||||
Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
|
||||
|
||||
## Parallelism options on this box (2026-10-01 survey)
|
||||
|
||||
Surveyed from the running image's own CLI surface. Verdicts are specific to
|
||||
dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
|
||||
with only 2 KV heads.
|
||||
|
||||
| option | flag | verdict on this box |
|
||||
|---|---|---|
|
||||
| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
|
||||
| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
|
||||
| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
|
||||
| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
|
||||
| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
|
||||
| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
|
||||
|
||||
### DCP is structurally unavailable here — do not re-test on 2 GPUs
|
||||
`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
|
||||
|
||||
```
|
||||
Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
|
||||
greater than the model's total number of KV heads (2). Increase
|
||||
`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
|
||||
```
|
||||
|
||||
DCP shards the KV along the *sequence* dimension after KV heads have been spread
|
||||
across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
|
||||
heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
|
||||
fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
|
||||
`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
|
||||
(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
|
||||
each rank would hold half the sequence's KV — but it is unreachable without two
|
||||
more cards.
|
||||
|
||||
### Measuring decode: the harness lives INSIDE the container
|
||||
`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
|
||||
run in the engine's namespace and the files must be copied in first:
|
||||
|
||||
```
|
||||
podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
|
||||
podman cp /home/dank/bench_iter.py frank-primary-qwen:/
|
||||
podman exec frank-primary-qwen python3 /mtp_measure.py
|
||||
```
|
||||
|
||||
Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
|
||||
No such file or directory` — measured 2026-10-01, which silently cost a whole
|
||||
arm's numbers. A window script must do the copy, not assume the path exists.
|
||||
|
||||
### The deployed shape, measured again (2026-10-01 A/B window)
|
||||
A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
|
||||
2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
|
||||
concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
|
||||
deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
|
||||
reproducible across boots.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
|
||||
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
|
||||
V4.1 scale)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
|
||||
(production is the R9V fork; this page's container is the ad hoc detour)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
|
||||
fork era; production has been the tcclaviger fork since)
|
||||
- [[x4-pair-experiment]] — the same residency law measured on a narrower
|
||||
link: streaming over PCIe loses, computing in RAM does not care
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
@@ -16,9 +16,20 @@ tags:
|
||||
timestamp: 2026-09-28
|
||||
---
|
||||
|
||||
# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
|
||||
# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
|
||||
|
||||
> **ACTIVE since 2026-09-28.** Replaces
|
||||
> **RETIRED 2026-10-02 — the rollback target.** Superseded by
|
||||
> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
|
||||
> MTP4, 17.0 t/s measured). This era was itself a return after one day of
|
||||
> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
|
||||
> ≥6-bit vLLM quant path dead on gfx1151). The shape
|
||||
> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
|
||||
> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
|
||||
> reclaimed), and the session grid is **one session at the full native
|
||||
> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
|
||||
> Swift-1.5 Q8 history below is retained for the engine/template law,
|
||||
> which still governs.
|
||||
> **Originally ACTIVE 2026-09-28.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP"
|
||||
description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rocmfpx
|
||||
- gfx1151
|
||||
- gguf
|
||||
- spec-decode
|
||||
- mtp
|
||||
timestamp: 2026-10-02
|
||||
---
|
||||
|
||||
# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151)
|
||||
|
||||
> **ACTIVE since 2026-10-02.** Cut over from
|
||||
> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from
|
||||
> Frank: *highest quality that runs*, and *one full context*. This endpoint
|
||||
> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and
|
||||
> measures **17.0 t/s** decode instead of 15, so quality was not bought with
|
||||
> speed.
|
||||
>
|
||||
> The capability that forced this whole shape is a GGUF tensor type:
|
||||
> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
|
||||
> host, so the runtime was decided by the file, not by preference.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
|
||||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
|
||||
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
|
||||
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
|
||||
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
|
||||
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
|
||||
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
|
||||
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
|
||||
| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
|
||||
| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
|
||||
|
||||
## Why this file, and why only this runtime
|
||||
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
|
||||
headers of the two candidates:
|
||||
|
||||
| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
|
||||
|---|---|---|---|---|
|
||||
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
|
||||
| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
|
||||
|
||||
Two consequences, both measured off the header rather than guessed:
|
||||
|
||||
1. **Quality.** The AGENT routing variant keeps the output-head (and other
|
||||
routing-sensitive) tensors at plain Q8_0 instead of folding them into the
|
||||
ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
|
||||
highest-precision tensors in the file. It is also the card's MTP-tuned
|
||||
variant, so it is the *faster* drafter base at the same time.
|
||||
2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
|
||||
and gufo both fail at load (`unknown tensor type`). Verified on the host
|
||||
before deploying — the bake-off was not a horse race, it was a lock-in
|
||||
check.
|
||||
|
||||
**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
|
||||
builds read this file and then died in `ggml_compute_forward_set_rows` — the
|
||||
exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
|
||||
first attempted on Dankclaw and routed to Dankserver as
|
||||
[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
|
||||
the commit out **by SHA** and refuses to build anything else; the pin is
|
||||
load-bearing, not tidiness.
|
||||
|
||||
## Shape and budget
|
||||
|
||||
One session, the model's full native window, unquantized KV:
|
||||
|
||||
| Dial | Value | Why |
|
||||
|---|---|---|
|
||||
| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
|
||||
| `--parallel` | 1 | one conversation, all memory to it |
|
||||
| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
|
||||
| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
|
||||
| `--flash-attn on` | on | required for the long-window attention path to fit |
|
||||
| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
|
||||
| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
|
||||
| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
|
||||
| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
|
||||
| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
|
||||
|
||||
Measured steady state (production, one model copy resident):
|
||||
`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
|
||||
0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
|
||||
in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
|
||||
endpoint bring-up in the fleet, because there is no engine compile step.
|
||||
|
||||
## Speculative decode — measured, not assumed
|
||||
|
||||
```
|
||||
--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl 99
|
||||
```
|
||||
|
||||
MTP in llama.cpp is a property of the file, not of the flags: the head is a
|
||||
trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
|
||||
`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
|
||||
there is no second 26 GiB copy in memory. See the fleet law in
|
||||
`AGENTS.md → llama.cpp serving`.
|
||||
|
||||
**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
|
||||
`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
|
||||
benchmarking law), baseline arm stripped of `--model-draft` **and** every
|
||||
`--spec-*` flag:
|
||||
|
||||
| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
|
||||
|---|---|---|---|
|
||||
| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
|
||||
| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
|
||||
|
||||
**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
|
||||
re-litigates this:
|
||||
|
||||
- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
|
||||
its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
|
||||
card's number as a file property, not a throughput promise.
|
||||
- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
|
||||
gufo-era law applies in spirit — never rank spec settings on a
|
||||
self-predictable corpus.
|
||||
- End-to-end on the chat path is slower than the `/completion` number because
|
||||
thinking tokens are included: a 2,253-token chat prompt generating 512
|
||||
tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
|
||||
unbounded reasoning budget, not a regression in decoding.
|
||||
|
||||
## Client contract (live-verified 2026-10-02)
|
||||
|
||||
| Behavior | Observation |
|
||||
|---|---|
|
||||
| Model id | `/v1/models` → `frank-large` |
|
||||
| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
|
||||
| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
|
||||
| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
|
||||
| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
|
||||
| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
|
||||
| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
|
||||
|
||||
Three traps, all hit for real:
|
||||
|
||||
1. **`content: null` when reasoning eats the budget.** With
|
||||
`--reasoning-parser`-style separation, a request whose `max_tokens` is
|
||||
consumed entirely by thinking returns `message.content = None`. A bench
|
||||
client that indexes `content` crashes *after* measuring and loses the
|
||||
numbers — fall back to `reasoning_content`, or size the budget (200 was
|
||||
enough for the vision probe).
|
||||
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
|
||||
appears in the log under draft resets. Self-recovering; not an outage signal.
|
||||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
|
||||
`llama-server`. The sweep key here is the payload name:
|
||||
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
|
||||
|
||||
## Operations
|
||||
|
||||
```bash
|
||||
# deploy (planner selects this endpoint only via its own unit/env/serve trio)
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
|
||||
# status / logs / cause of a crash loop
|
||||
systemctl status frank-large
|
||||
journalctl -u frank-large -n 200 --no-pager
|
||||
|
||||
# rebuild the pinned fork (only when deliberately moving the pin)
|
||||
scripts/build-rocmfpx.sh
|
||||
|
||||
# re-verify the payload without re-downloading
|
||||
scripts/download-frank-large-kingjones.sh --verify-only
|
||||
```
|
||||
|
||||
`serve-frank-large.sh` refuses to start — before touching the port — unless the
|
||||
fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
|
||||
projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
|
||||
broken payload never stops the running unit. Caller environment **overrides**
|
||||
`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
|
||||
as the error it reads as.
|
||||
|
||||
## Rollback
|
||||
|
||||
git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
|
||||
(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the
|
||||
DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
|
||||
and redeploy — nothing here deletes anything.
|
||||
|
||||
The rollback payloads stay exactly where the tag's env file expects them:
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2
|
||||
drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`
|
||||
(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
|
||||
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
|
||||
recoverable until the trash is purged, and never part of a rollback plan.
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- **`unknown tensor type` / ftype 115 refused** — something moved off the
|
||||
ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
|
||||
and the fork commit; this file cannot run on stock llama.cpp or gufo.
|
||||
- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
|
||||
older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
|
||||
History: [[kingjones-q8agent-rocmfpx]].
|
||||
- **Slow decode with no log error** — check whether the draft is engaged
|
||||
(`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
|
||||
did not load), and check GTT pressure with
|
||||
`cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
|
||||
of this model (an orphaned scratch server) drives the whole host into
|
||||
eviction and halves the rate silently.
|
||||
- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
|
||||
without `--jinja`.
|
||||
- **HTTP 500 naming the template on the first request of a role** — an
|
||||
unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.
|
||||
@@ -0,0 +1,186 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
|
||||
description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- dankclaw
|
||||
- vllm
|
||||
- strix-halo
|
||||
- mimo
|
||||
- mtp
|
||||
- spec-decode
|
||||
- bf16
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||||
|
||||
> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
|
||||
> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
|
||||
> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
|
||||
> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
|
||||
> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
|
||||
> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
|
||||
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
|
||||
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
|
||||
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
|
||||
`podman run` driven by `scripts/serve-frank-large.sh` under a root unit
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
|
||||
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
|
||||
(17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
|
||||
W8A16 AutoRound conversion on the 3090, here unquantized.
|
||||
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
|
||||
are loaded into the draft pass).
|
||||
|
||||
## Why this shape (the roof math)
|
||||
|
||||
Decode speed for a dense model on Strix Halo is bandwidth-bound:
|
||||
BF16 weights x one pass per token against the LPDDR5x aperture. Measured
|
||||
2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
|
||||
a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
|
||||
because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
|
||||
wall is generating more than one token per weight pass: **speculative decode
|
||||
is the performance feature of this endpoint**, not a garnish.
|
||||
|
||||
The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
|
||||
attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
|
||||
the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
|
||||
sequence. `--max-model-len 262144` (the native window) with
|
||||
`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
|
||||
four-session contract the gufo shape ran, with the pool shared by the
|
||||
scheduler instead of carved per session.
|
||||
|
||||
## Spec method: cross-model MTP — measured, not assumed
|
||||
|
||||
| arm | prefill | decode | acceptance |
|
||||
|---|---|---|---|
|
||||
| BF16 base | 17,876 t/s | 13.1 t/s | — |
|
||||
| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
|
||||
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
|
||||
|
||||
One session, temperature 0, 28,647-token prefill / 512-token decode, same
|
||||
client for all three arms (2026-10-01 bench window, frank-large stopped per
|
||||
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
|
||||
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
|
||||
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
|
||||
`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
|
||||
fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
|
||||
the compute queues) — this endpoint's thinking-heavy traffic spends its wall
|
||||
clock in decode, which is the regime where that trade wins.
|
||||
|
||||
**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
|
||||
Qwen3.5-9B that preserves the hidden-state geometry — identical
|
||||
4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
|
||||
per-position acceptance is the direct measurement that the distillation
|
||||
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
|
||||
(verified in both the BF16 base and the INT8 conversion) — which is exactly
|
||||
why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||||
reads the head from the TARGET checkpoint, vLLM reads the draft from a
|
||||
separate path. That difference is the whole trick:
|
||||
|
||||
```
|
||||
--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
|
||||
```
|
||||
|
||||
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
|
||||
[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
|
||||
`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
|
||||
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
|
||||
|
||||
## The numerics trap (greedy text is NOT preserved by spec decode)
|
||||
|
||||
Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
|
||||
`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
|
||||
reduction order, argmax near-ties diverge, and the divergence cascades.
|
||||
vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
|
||||
The gufo-era "greedy == autoregressive text" law does not carry over; do not
|
||||
chase text drift between spec and no-spec arms as a regression.
|
||||
|
||||
## Payload
|
||||
|
||||
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
|
||||
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
|
||||
byte-count + LFS sha256-verified by
|
||||
`old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
|
||||
the curl command line).
|
||||
- MTP source: `old/download-frank-large-qwen35.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
|
||||
`model.safetensors.index.json`; the shards carry the odd HF asset names
|
||||
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
|
||||
the engine boots from exactly this tree).
|
||||
- DFlash drafter: `old/download-frank-large-dflash.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B-DFlash/`.
|
||||
- **Payload status 2026-10-02:** all three trees moved to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
|
||||
the other retired frank payloads) and the fetch trio moved to `old/` — the
|
||||
endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
|
||||
- The serve script gates the image by REPO DIGEST (rootless podman resolves
|
||||
tags from the network; the pin exists to boot exactly these bytes
|
||||
offline), then config + index + exact shard count, then the spec draft
|
||||
(either weight layout: single `model.safetensors` or a sharded index),
|
||||
then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
|
||||
Dankserver 2026-09-29 lesson).
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
|
||||
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
|
||||
`models.yml` compat block for this provider sets
|
||||
`reasoningContentField: "reasoning"`. Miss that and thinking silently
|
||||
vanishes from transcripts.
|
||||
- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
|
||||
- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
|
||||
serve script passes `--language-model-only` — the endpoint stays
|
||||
text-only by decision; dropping the flag brings vision back.
|
||||
- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
|
||||
(temp 0.7, top_p 0.8) via `compat.extraBody`, and
|
||||
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
|
||||
`chat_template_kwargs` form is what vLLM reads; a top-level
|
||||
`preserve_thinking` is ignored there.
|
||||
- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
|
||||
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
|
||||
the env file with `set -a`.
|
||||
|
||||
## Ops
|
||||
|
||||
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
|
||||
- The serve script sweeps stale container identities before starting:
|
||||
`frank-large-mimo` itself and the bench-window container `vllm-mimo`.
|
||||
- Container death → `podman wait` returns non-zero from ExecStart →
|
||||
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
|
||||
frank-primary lesson).
|
||||
- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
|
||||
relinked unit; the deploy restores).
|
||||
- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
|
||||
boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
|
||||
262,144 tokens per request: 3.13x` — four sessions share the pool
|
||||
(the scheduler, not a carve). Live battery green: alias `frank-large`,
|
||||
reasoning split into `reasoning`, tool_calls with correct args, and
|
||||
`/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
|
||||
matching the bench window.
|
||||
|
||||
## Rollback
|
||||
|
||||
Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
|
||||
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
|
||||
payload + DFlash2 drafter, all still on the host. Restoring the tag's
|
||||
unit/env/serve trio and redeploying returns the 27B endpoint.
|
||||
|
||||
# Citations
|
||||
|
||||
- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
|
||||
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
|
||||
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
|
||||
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
|
||||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||||
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
|
||||
[[frank-small-sglang-mimo-v26]]
|
||||
@@ -0,0 +1,171 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
- mimo
|
||||
- gguf
|
||||
- kv-cache
|
||||
timestamp: 2026-10-02
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
|
||||
|
||||
> **ACTIVE since 2026-10-02.** One quant step down from
|
||||
> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
|
||||
> contract, then **resized twice the same day** on Frank's orders: first to
|
||||
> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
|
||||
> card free. Frank's arithmetic request — "as much as can fit while keeping
|
||||
> overhead open space to 2-3gb on the 3090" — is answered by the two
|
||||
> measurements in *Shape and the KV contract* below; the answer is
|
||||
> **3 × 180,224**, not 3 × 262,144.
|
||||
|
||||
> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
|
||||
> of per-conversation memory versus the two-slot shape: 180,224 tokens each
|
||||
> instead of the model's native 262,144. See *What three slots cost*.
|
||||
|
||||
## Identity
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
|
||||
each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
|
||||
n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
|
||||
`PARALLEL`; the bats seam enforces it.
|
||||
|
||||
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
|
||||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||||
|
||||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
|
||||
Without the flag the engine falls back silently and the contract becomes a
|
||||
lie. The serve script pins the trio; the bats seam pins the script.
|
||||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||||
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
|
||||
|
||||
**The pool size came from two boots, not from a formula.** `free`-style
|
||||
arithmetic on this stack is unreliable because the compute buffers, CUDA
|
||||
graphs and now the vision tower all bill separately, so the fit was measured
|
||||
at two shapes and solved between them:
|
||||
|
||||
|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
|
||||
|---|---|---|---|---|
|
||||
|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
|
||||
|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
|
||||
|
||||
The two rows price the projector and its image buffers at **~1.4 GiB** — not
|
||||
the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
|
||||
activation and compute buffers ride along. Solving the second row's non-KV cost
|
||||
against Frank's 2–3 GiB band: the card is 24,576 MiB, so
|
||||
`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
|
||||
down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
|
||||
**2,441 MiB free** (measured, 21,684 MiB used).
|
||||
|
||||
**Why 3 × 262,144 was never possible.** Three native windows are 786,432
|
||||
tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
|
||||
27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
|
||||
6 GiB.
|
||||
|
||||
## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
|
||||
|
||||
|Check|Result|
|
||||
|---|---|
|
||||
|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
|
||||
|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
|
||||
|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
|
||||
|Prefill|**3,814 tok/s** (same request)|
|
||||
|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
|
||||
|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
|
||||
|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
|
||||
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
|
||||
|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
|
||||
|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
|
||||
|
||||
Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
|
||||
day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
|
||||
scan at equal prompt depth, and one less slot's DeltaNet state resident.
|
||||
|
||||
## What three slots cost
|
||||
|
||||
The per-slot window is the whole price. Two consequences the fleet depends on:
|
||||
|
||||
- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
|
||||
sets `contextWindow: 180224` for this endpoint — the harness compacts before
|
||||
it reaches the ceiling instead of discovering it.
|
||||
- **frank-primary's compaction prompt no longer fits with room to answer.**
|
||||
Its trigger is 183,501 tokens — already above this slot — and its summary
|
||||
budget is 32,768 output tokens. On the two-slot shape (262,144) that left
|
||||
32.8k of output room, which is why that number was chosen; on this shape a
|
||||
single compaction request must be truncated or split. Nothing in the fleet
|
||||
routes compaction here automatically, but any runbook that claims frank-small
|
||||
"fits a compaction request in ONE request" is describing the 262,144 shape.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
|
||||
the field `models.yml` reads for this endpoint.
|
||||
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
|
||||
`enable_thinking` false, so `models.yml` ships
|
||||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
|
||||
`extraBody`. A raw client that omits it gets a direct answer — measured, and
|
||||
not a fault. Trivial questions produce no thinking even with the flag on.
|
||||
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
|
||||
- **Vision is on**, through the projector file (`--mmproj`). Use the
|
||||
OpenAI-standard `image_url` content part — that path answers correctly
|
||||
("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
|
||||
read by `/v1/chat/completions`: the same request sent that way answered
|
||||
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
|
||||
but refuses a request without `messages` in this build, so chat is the only
|
||||
vision path.
|
||||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
|
||||
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
|
||||
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
|
||||
refusal, never a silent fallback to blindness.
|
||||
|
||||
## No spec decode
|
||||
|
||||
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
|
||||
ghost), and the cross-model DFlash draft
|
||||
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
|
||||
optimization candidate, not a default — acceptance has to be measured on this
|
||||
traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|
||||
## Rollback
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|
||||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
|
||||
naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
- Requests queue behind each other: three concurrent long generations are the
|
||||
slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
|
||||
for the three, 3.0 s for the fourth).
|
||||
- `400 … exceeds the available context size (180224 tokens)`: the conversation
|
||||
outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
|
||||
and the per-slot window is the binding limit. Shorten the prompt, compact, or
|
||||
roll back to the 262,144 shape above.
|
||||
- Vision answers a color it never saw: the client sent `images` instead of an
|
||||
`image_url` content part (see *Client contract*).
|
||||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||||
of VRAM into unified memory halves the rate silently.
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||||
description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
- mimo
|
||||
- gguf
|
||||
- kv-cache
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
|
||||
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
|
||||
> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||||
> target.
|
||||
>
|
||||
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
|
||||
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
|
||||
> exposes per-tensor. That contract is what the current deployment kept.
|
||||
|
||||
## Identity
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
|
||||
so each session gets `n_ctx_slot = 262144`, the model's full native window
|
||||
(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
|
||||
|
||||
Frank's KV contract: **K unquantized (f16), V `q8_0`** —
|
||||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||||
|
||||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
|
||||
path. Without the flag the engine falls back silently and the contract
|
||||
becomes a lie. The serve script pins the trio; the bats seam pins the
|
||||
script.
|
||||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||||
(4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
|
||||
≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
|
||||
**20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
|
||||
order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
|
||||
K to fix it.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format
|
||||
deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
|
||||
`reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
|
||||
- **Thinking is off unless the request asks**: the MiMo GGUF template
|
||||
defaults `enable_thinking` false; `models.yml` now ships
|
||||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
|
||||
in `extraBody`. A raw client that omits it gets a no-thinking answer
|
||||
(measured — this is not a fault).
|
||||
- Tools: `--jinja` + server-side tool parsing, verified
|
||||
`tool_calls` round-trip 2026-10-01 23:0x.
|
||||
- **Text-only now.** The GGUF carries no vision projector; the
|
||||
in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
|
||||
`input: [text]`.
|
||||
|
||||
## No spec decode (yet)
|
||||
|
||||
The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
|
||||
cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
|
||||
Q8_0) is an optimization-loop candidate, not a default — `--spec-type
|
||||
draft-dflash` needs acceptance measured on this traffic first. The bats
|
||||
seam fails if a draft flag ships unannounced.
|
||||
|
||||
## Build
|
||||
|
||||
No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
|
||||
compiler (nvcc 12.4 rejects gcc ≥ 14):
|
||||
|
||||
```
|
||||
cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
|
||||
-DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
|
||||
cmake --build build-cuda -j 16 --target llama-server
|
||||
```
|
||||
|
||||
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
|
||||
AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
|
||||
stay on disk as the rollback: repoint the unit's serve script and env from
|
||||
git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
|
||||
use the parent of the 2026-10-01 night swap commit).
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
|
||||
exits 1 naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
|
||||
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
- Requests hang at first token while `/health` is 200: both slots busy —
|
||||
two concurrent long generations are the ceiling by contract.
|
||||
- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
|
||||
VRAM into unified memory halves the rate silently.
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- sglang
|
||||
- rtx3090
|
||||
- qwen
|
||||
- mtp
|
||||
- vision
|
||||
- int8
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
|
||||
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
|
||||
> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
|
||||
> payload was staged to
|
||||
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
|
||||
> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
|
||||
> is purged.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
|
||||
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
|
||||
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
|
||||
systemd unit `scripts/frank-small.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
|
||||
tunables from `scripts/frank-small.env`.
|
||||
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
|
||||
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
|
||||
|
||||
## Shape
|
||||
|
||||
- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
|
||||
(4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
|
||||
DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
|
||||
tower (27 layers) in the safetensors — no separate mmproj file.
|
||||
- **Context:** two 262,144-token slots at the model's native window
|
||||
(`--context-length 262144`, `--max-running-requests 2`) against a
|
||||
**pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
|
||||
reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
|
||||
~82k tokens of pool slack, and a compaction request (frank-primary's
|
||||
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
|
||||
one request. A single request may grow to the whole pool — a
|
||||
170,019-token prompt was verified at the 4-slot shape — the grid is
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
|
||||
180,224, which 183,501 tokens already exceeds — see *What three slots
|
||||
cost* in [[frank-small-llama-mimo-q5km]].
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
ssm state pool is a first-class consumer. The engine autofills the
|
||||
mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
|
||||
(fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
|
||||
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
|
||||
MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
|
||||
leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
|
||||
CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
|
||||
dies in capture, or later in the first long request once Triton's
|
||||
deferred kernel loads eat the remainder). Hence
|
||||
`--disable-prefill-cuda-graph`; decode graphs are untouched.
|
||||
- **Quantization:** `--quantization auto-round` — the checkpoint's
|
||||
`quantization_config.json` declares `quant_method: "auto-round"` and
|
||||
SGLang loads it natively (vision tower, `embed_tokens` and
|
||||
`linear_attn.in_proj` stay fp16).
|
||||
|
||||
## Spec method
|
||||
|
||||
**None — and this is a checkpoint fact, not a tuning choice.** The
|
||||
`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
|
||||
safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
|
||||
`model.safetensors.index.json` for both this INT8 conversion and the base
|
||||
`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
|
||||
`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
|
||||
boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
|
||||
decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
|
||||
paying the 4-token verify tax for no token. The serve script passes no
|
||||
`--speculative-*` flag, and the seam gates that.
|
||||
|
||||
## CUDA 13 toolchain (the sharp edge)
|
||||
|
||||
The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
|
||||
flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
|
||||
venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
|
||||
`--compress-mode`) and its headers clash with the 13.0 device passes.
|
||||
|
||||
Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
|
||||
|
||||
| Package | Supplies |
|
||||
|---|---|
|
||||
| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
|
||||
| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
|
||||
| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
|
||||
| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
|
||||
| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
|
||||
| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
|
||||
|
||||
Three local modifications the tree carries:
|
||||
|
||||
1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
|
||||
`-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
|
||||
2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
|
||||
`../../../include` / `../../../lib64` (the deb ships them as real
|
||||
near-empty dirs; the stub pass must see the 13.0 headers, not
|
||||
`/usr/include` 12.4).
|
||||
3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
|
||||
includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
|
||||
`noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
|
||||
the unpatched 13.0 declarations are incompatible in the C++ host stub).
|
||||
|
||||
JIT caches: `~/.cache/sglang/jit/sm86/` and
|
||||
`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
|
||||
change lands.
|
||||
|
||||
## Payload
|
||||
|
||||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||||
all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
|
||||
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
|
||||
cut over to llama.cpp + GGUF on 2026-10-01 night, and
|
||||
`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
|
||||
serve script refuses to start with a missing shard.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-small` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
|
||||
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
|
||||
as llama.cpp; models.yml sets `reasoningContentField:
|
||||
"reasoning_content"`). `--reasoning-parser qwen3`.
|
||||
- **Tools:** `--tool-call-parser qwen3_coder`.
|
||||
- **Vision:** supported (in-checkpoint tower).
|
||||
- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
|
||||
`--top-k` — sampling rides on the request; the harness sends the Qwen
|
||||
recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
|
||||
- **Reasoning effort:** the full OpenAI set is accepted (the chat template
|
||||
has no effort gate).
|
||||
|
||||
## Measured
|
||||
|
||||
- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
|
||||
compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
|
||||
above is what makes that compile succeed).
|
||||
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
|
||||
- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
|
||||
50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
|
||||
incl. 8 decode tokens, ~2,056 t/s).
|
||||
- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
|
||||
(216k tokens of live KV at once). 2-slot shape: two concurrent
|
||||
50,000-token requests both answered (28.8 s wall).
|
||||
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
|
||||
well past frank-primary's 183,501-token compaction trigger (70% of its
|
||||
262,144 window) — prefills and completes in ONE request.
|
||||
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
|
||||
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
|
||||
Gemma4-12B QAT payload, was staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
|
||||
2026-10-02; its unit/env/serve trio remains in `old/`.
|
||||
|
||||
# Citations
|
||||
|
||||
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
- SGLang: https://github.com/sgl-project/sglang
|
||||
- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "halogen-flash-server on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
|
||||
title: halogen-flash-server on Dankclaw (frank-large)
|
||||
description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
|
||||
tags:
|
||||
- llm
|
||||
- halogen
|
||||
@@ -13,3 +13,466 @@ tags:
|
||||
- rocm
|
||||
timestamp: 2026-09-13
|
||||
---
|
||||
|
||||
# halogen-flash-server on Dankclaw (frank-large)
|
||||
|
||||
> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
|
||||
> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
|
||||
> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
|
||||
> [[frank-large-ik-uncensored]], the current rollback), and came back as the
|
||||
> shape the fleet asked for: two 200k-token conversations rather than the
|
||||
> native-context pair. Deployed as a rootless Quadlet container,
|
||||
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
|
||||
> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
|
||||
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
|
||||
Upstream's own comparison table measures it at **roughly 4x the prefill
|
||||
throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
|
||||
ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
|
||||
prompt with real context. That, not decode, is the reason for the swap.
|
||||
|
||||
## Shape
|
||||
|
||||
One container runs both halves of the stack:
|
||||
|
||||
| half | port | reachable |
|
||||
|---|---|---|
|
||||
| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
|
||||
| OpenAI front-end | 8731 | published as host **7770** |
|
||||
|
||||
The `all` mode is upstream's default and the shape a single-container host
|
||||
wants: the front-end talks to the engine over loopback inside the container, so
|
||||
there is one process tree, one log, and one published port.
|
||||
|
||||
`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
|
||||
`/health`, and in every response body. Requests are **never rejected for naming
|
||||
a different model** — the server serves whatever asks. Keeping the fleet alias
|
||||
is therefore free, and it means Hermes, the harness config, the OpenWebUI
|
||||
discovery, the `services.json` probe and the docs needed no rename.
|
||||
|
||||
## Weights
|
||||
|
||||
`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
|
||||
|
||||
| file | size | role |
|
||||
|---|---|---|
|
||||
| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
|
||||
| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
|
||||
| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
|
||||
| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
|
||||
|
||||
`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
|
||||
only** — the entrypoint acts only when the checkpoint is absent, so a restart
|
||||
never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
|
||||
mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
|
||||
`/home/dank/models/halogen`.
|
||||
|
||||
**Refreshing the weights.** Delete the mount's contents (or the whole
|
||||
directory, then recreate it **empty**) and start the unit: the entrypoint sees
|
||||
the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
|
||||
traps. First, the bind mount's source directory must *exist* — podman refuses
|
||||
the container with `Error: statfs /home/dank/models/halogen: no such file or
|
||||
directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
|
||||
directory and the fetch starts on the next start. Second, "refresh" only means
|
||||
"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
|
||||
(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
|
||||
2,383,306,048 → **2,478,095,488**) while the checkpoint returned
|
||||
byte-identical in size (124,068,083,904) — so an update can live entirely in
|
||||
the sidecars, and equal file sizes are not proof of an unchanged repo.
|
||||
|
||||
`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
|
||||
refresh) comes along with the repo but is unused: it trades the `o_proj`
|
||||
calibration for about 2 % of decode, and the quality sidecar is the default for
|
||||
a reason.
|
||||
|
||||
## Deployed flags
|
||||
|
||||
Everything else is configured by environment variable — there is no config
|
||||
file, and the flags are read once at startup. These are the departures from
|
||||
upstream's defaults:
|
||||
|
||||
| flag | value | why |
|
||||
|---|---|---|
|
||||
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
|
||||
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
|
||||
| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
|
||||
| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
|
||||
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
|
||||
| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
|
||||
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
|
||||
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
|
||||
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
|
||||
|
||||
Everything else is upstream's default, deliberately: prompt cache mode `2`
|
||||
(two entries per conversation — end-of-system-prompt and end-of-history — which
|
||||
already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
|
||||
stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
|
||||
temperature 0, and it stands down whenever a second conversation is
|
||||
generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
|
||||
gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
|
||||
|
||||
## Memory model
|
||||
|
||||
Three numbers explain the whole box:
|
||||
|
||||
- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
|
||||
GPU in place, never copied. The kernel counts the locked pages as reclaimable
|
||||
file cache, so `free`, `MemAvailable` and every dashboard built on them
|
||||
**overstate free memory by the size of the model**. The server prints the
|
||||
corrected figure at startup; believe that line, not Grafana.
|
||||
- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
|
||||
~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
|
||||
reads the two together in the engine's own words: `400384 positions, ~27.2
|
||||
GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
|
||||
~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
|
||||
vision tower. The engine's device-side budget stops at roughly 47 GiB on a
|
||||
128 GB machine *with the weights pinned* — and lower on a machine carrying
|
||||
more besides this server, which Dankclaw is.
|
||||
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
|
||||
table is read through. Too little and long prompts crawl with the disk busy
|
||||
and the process in uninterruptible sleep — that is a cache problem, not a
|
||||
memory problem, and the fix is a *smaller* pool, not a bigger one.
|
||||
|
||||
`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
|
||||
the pool itself if the configured one will not fit, printing what it chose.
|
||||
Read those lines on every restart; they are the ground truth for sizing.
|
||||
|
||||
**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
|
||||
startup account, including three deliberate experiments:
|
||||
|
||||
| pool | max_tok | reserve | KV | arena | need | budget | verdict |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
|
||||
| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
|
||||
| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
|
||||
| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
|
||||
| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
|
||||
| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
|
||||
| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
|
||||
| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
|
||||
|
||||
### The sizing rule
|
||||
|
||||
One budget covers the pool, the arena **and** the vision tower together:
|
||||
|
||||
```
|
||||
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
|
||||
= 121.2 - 67.7 - reserve
|
||||
reserve 24 -> 29.5 GiB
|
||||
reserve 15 -> 38.5 GiB
|
||||
|
||||
need = KV (28 KiB/position) + arena + vision (1.97 measured)
|
||||
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
||||
```
|
||||
|
||||
The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
|
||||
121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
|
||||
Use the unrounded value. Two of the three terms are fixed hardware and
|
||||
checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
|
||||
buys the *fit margin*, not the host's memory. It is a term in this inequality,
|
||||
not an allocation, and the section at the end of this page shows the live test
|
||||
that proves it.
|
||||
|
||||
Trust the engine's printed requirement over this model wherever they disagree:
|
||||
a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
|
||||
of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
|
||||
enough to plan with, and the authoritative number to cut the other way.
|
||||
|
||||
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
|
||||
point to 0.1 GiB — including upstream's three published "on the device" figures
|
||||
(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
|
||||
**pool + arena totals at the default `max_tok`**, not pool alone. That is why
|
||||
their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
|
||||
49.4. Reading those figures as pool-only is the trap that makes a doubling look
|
||||
affordable when it is not.
|
||||
|
||||
At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
|
||||
13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
|
||||
ceiling, not a conservative setting.** The corner check agrees: at 32768 only
|
||||
7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
|
||||
262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
|
||||
= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
|
||||
fits unmodified.
|
||||
|
||||
### What the reserve actually costs
|
||||
|
||||
The device budget is not the whole story. Memory position by configuration
|
||||
(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
|
||||
|
||||
| config | held | host left | contiguous 2 MiB blocks |
|
||||
|---|---|---|---|
|
||||
| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
|
||||
| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
|
||||
| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
|
||||
| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
|
||||
| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
|
||||
|
||||
At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
|
||||
engine volunteered the stall note. It did not print it at the 401,408 trim, and
|
||||
it prints it again after the 2026-09-16 retune. Three data points, and they do
|
||||
not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
|
||||
not, 475 did. So treat the note as "the machine is tight" and read the
|
||||
`host memory left` line rather than waiting to be told:
|
||||
|
||||
```
|
||||
halogen: NOTE this server now holds most of the machine, and 310 contiguous
|
||||
2 MiB blocks remain. It runs best on a host of its own. If other large
|
||||
processes share this machine they will compete for what is left, and when it
|
||||
runs out BOTH they and this server can stall for minutes at a time at full
|
||||
CPU with no disk activity and no output, which is not a crash and needs no
|
||||
restart.
|
||||
```
|
||||
|
||||
That is the failure mode to watch, and it is the least diagnosable of the
|
||||
three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
|
||||
looks like a hang, for the neighbours as much as for frank-large. `free` is
|
||||
useless for spotting it — `MemAvailable` still reads ~74 GiB because the
|
||||
page cache counts as reclaimable. Watch the `host memory left for everything
|
||||
else` line and the contiguous-block count at boot instead.
|
||||
|
||||
28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
|
||||
so it is survivable.
|
||||
|
||||
### The reserve cannot buy air — ceiling 16
|
||||
|
||||
The last row above is the experiment that proves it. Raising the reserve from
|
||||
15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
|
||||
GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
|
||||
allocation; the actual footprint is `weights + KV + arena + vision`, none of
|
||||
which the reserve touches. So the knob has exactly two regimes:
|
||||
|
||||
| reserve | budget | vs required 36.5 | effect |
|
||||
|---|---|---|---|
|
||||
| 15 | 38.5 | +2.0 | deployed |
|
||||
| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
|
||||
| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
|
||||
| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
|
||||
| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
|
||||
|
||||
The 36.5 requirement is the engine's own, read from a clamped boot:
|
||||
`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
|
||||
thing a bigger reserve can do is take the doubling away.
|
||||
|
||||
**To actually free host memory, change the pool or the arena**, and the reserve
|
||||
then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
|
||||
7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
|
||||
are one-line edits that the fit check will confirm on the next boot. The
|
||||
2026-09-13 trim to 401,408 was this remedy applied at half strength —
|
||||
`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
|
||||
with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
|
||||
modelled 3.4 because the engine holds the freed KV as working slack.
|
||||
|
||||
### The `WARNING` line only appears when the fit check had to move
|
||||
|
||||
This one cost me a wrong conclusion, so it is worth stating plainly. When the
|
||||
configured pool already fits, the engine reserves it **and prints nothing** —
|
||||
the 09-11 baseline boot goes straight from the pre-flight line to
|
||||
`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
|
||||
its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
|
||||
after a clamp**.
|
||||
|
||||
So the absence of a warning in old logs is not evidence of headroom. Both
|
||||
`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
|
||||
baseline has lived in that state since it was deployed without incident,
|
||||
because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
|
||||
at that point). The engine's budget is a conservative floor, not a hard wall.
|
||||
|
||||
### Two failure modes, and only one is dangerous
|
||||
|
||||
Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
|
||||
outgrows the budget, and the server cannot allocate at all:
|
||||
|
||||
```
|
||||
kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
|
||||
kv pool: WARNING even 262144 positions (the context, the smallest pool that can
|
||||
serve one request) needs ~44.5 GiB and host RAM cannot spare it.
|
||||
dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
|
||||
```
|
||||
|
||||
This is the quiet one: the unit stays `active`, the container reports
|
||||
`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
|
||||
`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
|
||||
large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
|
||||
slower".
|
||||
|
||||
Under-running the ceiling is far more benign and far easier to miss. At 524288
|
||||
with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
|
||||
`deploy.sh` still reported healthy in 12 seconds:
|
||||
|
||||
```
|
||||
kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
|
||||
RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
|
||||
```
|
||||
|
||||
A green health check does not mean the pool you asked for is the pool you got —
|
||||
read the `kv pool:` lines after every restart.
|
||||
|
||||
The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
|
||||
the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
|
||||
`/proc/meminfo` figure at all.
|
||||
|
||||
**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
|
||||
decode numbers were measured with, and it means this container holds most of
|
||||
the host. If a co-tenant then takes the remainder, both it and the engine can
|
||||
stall for minutes at a time at full CPU with no disk activity and no output.
|
||||
That is documented behaviour, not a crash, and it needs no restart.
|
||||
`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
|
||||
decode speed — a last resort, not a tuning option.
|
||||
|
||||
## Client contract
|
||||
|
||||
The differences from llama.cpp that bite are all in the request, not the
|
||||
deployment:
|
||||
|
||||
- **No server-side sampler defaults.** `temperature` absent or `0` is
|
||||
**greedy** decode; above 0 the request samples on the same drafter. Supported
|
||||
fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
|
||||
`presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
|
||||
with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
|
||||
value outside its defined range (values are refused, never clamped).
|
||||
The fleet runs greedy for clients that stay silent — deterministic, and
|
||||
byte-identical with the drafter.
|
||||
- **The token budget covers thinking.** The default is **8192**, and a budget
|
||||
that runs out mid-thought returns `finish_reason: "length"` with an **empty
|
||||
`content`** and the partial reasoning in `reasoning_content` — which most
|
||||
clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
|
||||
request is a 400 rather than a silent truncation. `max_tokens`,
|
||||
`max_completion_tokens` and `max_output_tokens` all work and must agree if
|
||||
more than one is sent.
|
||||
- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
|
||||
The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
|
||||
only compacts at its own threshold — with OMP at 70 % of a 200,192 window
|
||||
(140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
|
||||
the window while sitting below the compaction trigger. That collided live on
|
||||
2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
|
||||
harness-side: keep the declared `maxTokens` at or under
|
||||
`window − threshold` (57,344 chosen) so no pre-compaction request can miss
|
||||
the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
|
||||
breaks the two-resident-conversations property.
|
||||
- **`reasoning_effort`** is accepted at the top level and inside
|
||||
`chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
|
||||
model's default is `xhigh`. `minimal` aliases the template's `low` and still
|
||||
reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
|
||||
honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
|
||||
- `/v1/models`, `/health` and completions all report the served id, and
|
||||
`reasoning_content` carries the reasoning trace.
|
||||
|
||||
**Pool admission is the one arithmetic trap.** A request reserves its prompt
|
||||
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
|
||||
the request waits in arrival order, and after the queue timeout (3600 s) it
|
||||
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
|
||||
the pool can therefore never be admitted at all. The 2026-09-16 shape is a
|
||||
400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
|
||||
conversations at the ceiling fit the pool exactly, so a pair that is both
|
||||
near-full and both generating is pool-bound — the second waits for the first to
|
||||
release its region. The pool binds **how many** conversations stay resident,
|
||||
not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
|
||||
tables above.
|
||||
|
||||
## Vision
|
||||
|
||||
**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
|
||||
the tower is a 1.97 GiB charge against the same device budget as the pool and
|
||||
the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
|
||||
which loads the BF16 projector alongside its own weights. This server refuses
|
||||
image content parts and says why, rather than ignoring them:
|
||||
|
||||
```json
|
||||
{"error": {"message": "this server does not accept images: the engine was
|
||||
started without a vision tower (HALOGEN_VISION_TOWER)", "type":
|
||||
"invalid_request_error"}}
|
||||
```
|
||||
|
||||
The llama.cpp side handles images the same way this engine used to: a `data:`
|
||||
URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
|
||||
refused deliberately (fetching one would make the server issue outbound
|
||||
requests to wherever a client pointed it). Audio and video have no path
|
||||
anywhere in the fleet — that left with [[frank-npu]].
|
||||
|
||||
## Deploy and operate
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
```
|
||||
|
||||
The handler retires the ik-era **root** unit first (stop, disable, move
|
||||
`/etc/systemd/system/frank-large.service` into
|
||||
`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
|
||||
`disable` may delete the file outright, which the handler now reports), stops
|
||||
the Quadlet's own unit if it is running, then pulls
|
||||
`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
|
||||
`localhost/` alias the unit references, and **waits for port 7770 to be free**
|
||||
before installing and starting anything.
|
||||
|
||||
That last gate is not paranoia. On the first cutover attempt the ik server
|
||||
logged "Deactivated successfully", and pasta — rootless podman's userspace
|
||||
network stack, which binds the published port at container CREATE — still
|
||||
reported `Address already in use` a second later, exiting the unit with code
|
||||
126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
|
||||
starts a doomed container has not deployed anything, so `wait_port_free` polls
|
||||
every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
|
||||
the socket table if it never clears.
|
||||
|
||||
Boot persistence comes from `[Install] WantedBy=default.target` in the unit
|
||||
file plus linger on the deploy user — generated units **cannot** be
|
||||
`systemctl --user enable`d on this host.
|
||||
|
||||
```bash
|
||||
systemctl --user status frank-large.service # the generated unit
|
||||
podman logs frank-large | head -40 # version banner + memory ledger
|
||||
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
|
||||
curl -s http://127.0.0.1:7770/v1/models
|
||||
```
|
||||
|
||||
`/health` asks the **engine** for a PONG before answering, so a 200 means
|
||||
"ready to serve", not merely "process alive". During the first-start download
|
||||
it stays unhealthy by design.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| symptom | cause |
|
||||
|---|---|
|
||||
| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
|
||||
| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
|
||||
| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
|
||||
| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
|
||||
| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
|
||||
| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
|
||||
|
||||
`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
|
||||
already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
|
||||
deliberately **not** set: it logs every allocation over 64 MB, which is
|
||||
steady-state noise on a shared host — raise it only for a live OOM report.
|
||||
The lines worth grepping first:
|
||||
|
||||
```bash
|
||||
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
```
|
||||
|
||||
Retrieval: `podman logs frank-large` covers the **current** instance only —
|
||||
the Quadlet runs `--replace --rm`, so everything before the last recreate is
|
||||
gone from the container log. `journalctl --user -u frank-large.service` is
|
||||
the durable record. Both are rootless: `sudo podman logs frank-large` looks
|
||||
in root's storage and answers "no such container".
|
||||
|
||||
## Rollback
|
||||
|
||||
The predecessor is intact: the ik payload stays at
|
||||
`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
|
||||
ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
|
||||
serve script and the Qwen-Sharp template are archived in the repo under
|
||||
`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
|
||||
is a `git revert` of this change set plus a deploy — **but the Quadlet must be
|
||||
removed from `~/.config/containers/systemd/` first**, or the restarted root
|
||||
unit and the still-installed container both claim port 7770. The port gate in
|
||||
the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
|
||||
|
||||
The `amd_iommu=off` kernel argument that came with this change is separate and
|
||||
reverts on its own — see [[dankclaw-iommu-translation-off]].
|
||||
|
||||
## Citations
|
||||
|
||||
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
|
||||
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
|
||||
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
|
||||
@@ -1,17 +1,21 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-30):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
|
||||
Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
|
||||
of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
|
||||
bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
|
||||
Dankclaw runs two llama.cpp endpoints —
|
||||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||||
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
|
||||
(the model's full native window)**, fp8 KV (5 boot but leave no runtime
|
||||
headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
|
||||
file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
|
||||
session at 262,144, f16 KV, vision via the bundled projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -27,22 +31,22 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[GLM-5.3-Flash W4A16 int4
|
||||
TP=2, expert offload 73.64 GiB/rank pinned,
|
||||
14.51 GiB LRU arena, 131072 ctx, vision, tools]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 4 x 262144 ctx,
|
||||
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[stock llama.cpp (unsloth b11030, gfx1151)
|
||||
E[ROCmFPX llama.cpp gfx1151, pinned fork build
|
||||
frank-large, port 7770]
|
||||
F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
|
||||
262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
|
||||
1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[Gemma4-12B QAT Q4_K_M
|
||||
4 x 262144 ctx, MTP + vision + audio]
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -61,7 +65,23 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
|
||||
(root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
|
||||
the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
|
||||
(26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
|
||||
tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
|
||||
tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
|
||||
runtime is not interchangeable, the pinned-build set_rows history, the
|
||||
measured envelope, rollback to gufo
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
|
||||
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||||
@@ -158,14 +178,24 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
|
||||
Strix Halo inference engine, built from source on Dankclaw, serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
|
||||
Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
|
||||
262,144-token conversation, greedy, text-only, the two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode trade
|
||||
against llama.cpp, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
|
||||
target)** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||||
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode
|
||||
trade against llama.cpp
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||||
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
|
||||
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
|
||||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -199,12 +229,27 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
|
||||
target)**: llama.cpp (master, CUDA sm_86 build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||||
`/home/dank/models/MiMo-Q6K/`
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv still on disk, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
|
||||
vLLM detour (2026-09-19) was reverted the same day; see below.
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
|
||||
@@ -213,14 +213,27 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1,
|
||||
the Strix Halo inference engine, built from source on Dankclaw and
|
||||
serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the
|
||||
z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token
|
||||
conversation, adaptive block policy, greedy, text-only, the two
|
||||
from-source build workarounds, what changed in the client contract, the
|
||||
measured prefill/decode trade against llama.cpp
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook
|
||||
since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/
|
||||
ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
|
||||
Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115,
|
||||
311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the
|
||||
host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16
|
||||
projector, one session at the native 262,144 window
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback
|
||||
target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1
|
||||
built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
|
||||
drafter on the gfx1151 iGPU — the two from-source build workarounds, the
|
||||
template-hash allowlist, the measured prefill/decode trade
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
|
||||
replaced by the gufo rollback, then by
|
||||
[[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
|
||||
the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -269,10 +282,26 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
|
||||
2026-09-17: mainline llama.cpp serving
|
||||
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
|
||||
2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
|
||||
3090, port 7771 — THREE conversations at 180,224 tokens each, the
|
||||
asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
|
||||
and the two VRAM measurements that set the pool size; ~100 tok/s decode
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
|
||||
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
|
||||
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
|
||||
2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
|
||||
SGLang in a self-contained python3.12 venv serving
|
||||
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
|
||||
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
|
||||
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
|
||||
2 x 262,144 slots, bf16 mamba states, eager prefill
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
|
||||
@@ -662,3 +662,334 @@
|
||||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails)
|
||||
|
||||
bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
|
||||
--expert-precise-memory:
|
||||
|
||||
- with precise: NRestarts 1->3, each attempt ~6.7 min then
|
||||
`Deactivated successfully` (API server drains after an EngineCore fatal
|
||||
error) — cause not captured, the serve script rm'd the container.
|
||||
- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
|
||||
climbing, container `Exited (1)`, and the journal finally showed the real
|
||||
error:
|
||||
`RuntimeError: Worker failed with error 'Constraints violated
|
||||
(L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
|
||||
as dynamic but your code specialized it to be a constant (3)'`
|
||||
— a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
|
||||
`~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
|
||||
fresh container filesystem, so a stale compile cache is ruled out: the
|
||||
conflict is produced by this configuration's own compile path.
|
||||
|
||||
Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
|
||||
KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
|
||||
(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
|
||||
`--enforce-eager` is the bypass that would sidestep the constraint path.
|
||||
|
||||
Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
|
||||
Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
|
||||
|
||||
## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
|
||||
|
||||
**Reverted.** The two changes Frank asked for were applied together and the
|
||||
result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
|
||||
~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
|
||||
and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
|
||||
new `Restart=always` turned a silent death into a visible loop).
|
||||
|
||||
The budget arithmetic explains it without needing the error text, because the
|
||||
arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
|
||||
that is exactly what `rocm-smi` measured):
|
||||
|
||||
| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
|
||||
|---|---|---|
|
||||
| non-expert weights | 6.28 | 6.28 |
|
||||
| pinned pool | 6.80 | 5.68 |
|
||||
| CUDA graphs | 0.55 | 0.55 |
|
||||
| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
|
||||
| non-torch | ~1.45 | ~1.45 |
|
||||
| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
|
||||
| free at boot | **~0.13 GiB** | 3.59 |
|
||||
|
||||
`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
|
||||
the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
|
||||
put graph capture ~0.13 GiB from the ceiling, where every transient allocation
|
||||
dies. So the flag is dropped and bf16 2 x 262144 ships with the default
|
||||
overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
|
||||
|
||||
**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
|
||||
default — it is the budget for the phase that allocates last. Reclaim it only
|
||||
by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
|
||||
|
||||
## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
|
||||
|
||||
Frank's final shape for the night. Two changes at once, both measured in
|
||||
principle before the boot:
|
||||
|
||||
- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
|
||||
bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
|
||||
11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
|
||||
1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
|
||||
524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
|
||||
3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
|
||||
- **`--expert-precise-memory`.** The registry's default overhead for this
|
||||
architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
|
||||
nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
|
||||
The flag selects the fitted figure, which moves only the plan's `room` term
|
||||
— arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
|
||||
cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
|
||||
1024 nbt above 4096, so MNBT 2048 pays nothing.
|
||||
|
||||
Budget note worth keeping: the arena is `(budget - non_expert_weights -
|
||||
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
|
||||
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
|
||||
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
|
||||
planned arena; it comes out of free VRAM instead.
|
||||
|
||||
## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions
|
||||
|
||||
Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
|
||||
UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
|
||||
sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
|
||||
uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2
|
||||
drafter and its fetcher are unchanged.
|
||||
|
||||
Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
|
||||
KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
|
||||
to `kFp16`), so "unquantized" is inherent.
|
||||
|
||||
KV arithmetic (why this fits where one 262144 session used to sit):
|
||||
f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
|
||||
`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
|
||||
2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
|
||||
per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
|
||||
655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
|
||||
weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
|
||||
~62 GiB against the 114 GiB GTT.
|
||||
|
||||
## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
|
||||
|
||||
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
|
||||
deploy gate twice and then passed cleanly on a manual re-run:
|
||||
- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
|
||||
— Plex redirects its web UI to the hosted app; the same failure appeared on
|
||||
the previous deploy too.
|
||||
- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/`
|
||||
— the homepage answered `http=200` in 40 ms from the dev box at the same
|
||||
moment, so this is a browser-level flake.
|
||||
- run 3 (manual): **33 passed**, exit 0.
|
||||
|
||||
So a red gate on this spec alone is NOT proof the fleet is broken, and the
|
||||
deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
|
||||
`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one
|
||||
stable underlying issue and it lives in Plex's web-client/image config, not in
|
||||
any repo service.
|
||||
|
||||
## 2026-10-01 — frank-primary bf16 no-spec cutover
|
||||
|
||||
- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
|
||||
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
|
||||
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
|
||||
FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
|
||||
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
|
||||
request: 3.03x".
|
||||
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
|
||||
(`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
|
||||
allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
|
||||
(KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
|
||||
PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
|
||||
model's native 262,144 does not fit with a 3-seq contract. Every rung was
|
||||
booted under a transient systemd unit at uid dank with
|
||||
`LimitMEMLOCK=infinity` (the rootless bench probe masked the production
|
||||
memory environment) with full engine-log capture.
|
||||
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
|
||||
`--gpu-memory-utilization` shrinks the engine's own budget rather than
|
||||
buying slack; Frank forbade it and the winning ladder ran at 0.98.
|
||||
- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
|
||||
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
|
||||
container-env passthrough added; shape header rewritten.
|
||||
- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
|
||||
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
|
||||
qualified-shape and flags tests flipped to the new env keys.
|
||||
- Overnight redo campaign (prefetch OFF everywhere) tabled in
|
||||
`/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
|
||||
with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
|
||||
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
|
||||
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
|
||||
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
|
||||
- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
|
||||
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
|
||||
updated out of the GLM era.
|
||||
- **Client contract fixed (thinking + tools).** The live endpoint returned a
|
||||
null `reasoning` field and dumped the thinking text INLINE in `content`
|
||||
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
|
||||
— the malformed thinking block clients choke on — because the deployed argv
|
||||
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
|
||||
three contract flags and the qwen rollback inherited the gap. Restored from
|
||||
`ecaad36`'s values, confirmed against this fork's registries:
|
||||
`--reasoning-parser qwen3`, `--enable-auto-tool-choice
|
||||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||||
"NO tool dialect" test is flipped to a PRESENT test.
|
||||
- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
|
||||
fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
|
||||
free) — six full-length contexts do not fit at any KV dtype — so the
|
||||
contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
|
||||
rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
|
||||
5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
|
||||
three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
|
||||
same night) is the documented max at 3 contexts.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
|
||||
with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
|
||||
<= 262,144) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
|
||||
passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
|
||||
at runtime; a real client request (995-token prompt, `max_tokens=261149`,
|
||||
sampling params straight from models.yml) killed Worker_TP1 with
|
||||
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
|
||||
the engine shut down. Boot is not proof of runtime survivability — after a
|
||||
knife-edge shape change, send a large-`max_tokens` request before calling it
|
||||
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
|
||||
GiB per rank for activations/graph capture. Frank's call 2026-10-01.
|
||||
- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
|
||||
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
|
||||
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
|
||||
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
|
||||
down until a human noticed. An explicit `systemctl stop` still wins, so
|
||||
bench windows and deploys are unaffected.
|
||||
- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
|
||||
MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
|
||||
contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
|
||||
and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
|
||||
the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
|
||||
decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
|
||||
arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
|
||||
- **DCP investigated and closed** (see the parallelism survey in the runbook):
|
||||
`--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
|
||||
TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
|
||||
knob on this box.
|
||||
- Production bring-up: the shape boots and serves on :7770 — engine log
|
||||
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
|
||||
contract probes (reasoning split, tool_calls). Rollback target for the
|
||||
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
|
||||
|
||||
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
|
||||
|
||||
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
|
||||
frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
|
||||
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
|
||||
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
|
||||
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
|
||||
vLLM reads the draft from a SEPARATE path — so cross-model MTP from
|
||||
`Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
|
||||
geometry) works. Frank's call: ship that as frank-large.
|
||||
- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
|
||||
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
|
||||
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
|
||||
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
|
||||
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
|
||||
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
|
||||
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
|
||||
text (bf16 batched verification reorders reductions; near-tie argmaxes
|
||||
cascade). vLLM spec decode is distribution-preserving, not
|
||||
bitwise-greedy-preserving.
|
||||
- **Client-contract delta wired:** vLLM emits `reasoning`, not
|
||||
`reasoning_content` - `models.yml` frank-large block now sets
|
||||
`reasoningContentField: "reasoning"`, and the extraBody moved to the
|
||||
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
|
||||
(a top-level `preserve_thinking` is ignored by vLLM).
|
||||
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
|
||||
`model.safetensors` (the dflash layout) and would have rejected the
|
||||
deployed MTP arm (4 shards behind an index); fixed to accept both
|
||||
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
|
||||
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
|
||||
0.30 (verified in-image); the drafter stays on disk as the alternative.
|
||||
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
|
||||
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
|
||||
rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
|
||||
Q4_K_XL payload and DFlash2 drafter all remain on the host.
|
||||
- First deploy attempt #2 crash-looped the container 51 times: the serve
|
||||
script handed vLLM the HOST model path while the tree is mounted at
|
||||
`/models` inside the container — transformers rejected it as a malformed
|
||||
repo id. Fixed by converting to the container path (the spec draft was
|
||||
already converted; the target was not), pinned by bats. Also fixed the
|
||||
deployer's own draft gate (same single-file bug as the serve script's).
|
||||
- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
|
||||
`scripts/download-frank-large-swift.sh`) and the orphaned flux-2-klein
|
||||
tree under `models/checkpoints` (23 G; ComfyUI mounts
|
||||
`comfyui-data/models`, not this) purged via
|
||||
`~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
|
||||
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
|
||||
drafter, not a remnant).
|
||||
|
||||
## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
|
||||
|
||||
Frank's order was the highest quality of
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
|
||||
runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
|
||||
carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
|
||||
llama.cpp, gufo and every other loader on the fleet cannot read — only the
|
||||
kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
|
||||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
|
||||
`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
|
||||
bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
|
||||
(n-max 4, `--spec-draft-device ROCm0`).
|
||||
|
||||
- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
|
||||
(+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
|
||||
mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
|
||||
routing variant was chosen over the card's plain Q8_0 because it keeps 187
|
||||
output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
|
||||
0.953 vs 0.911.
|
||||
- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
|
||||
- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
|
||||
fetchers moved to `old/`. Runbook
|
||||
[[frank-large-rocmfpx-q8agent-mtp]].
|
||||
|
||||
## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
|
||||
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
|
||||
with two contexts ("whatever context size fits the 3090"), then **three
|
||||
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
|
||||
the 3090", then **"frank-small should have vision"**.
|
||||
|
||||
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
|
||||
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
|
||||
**21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
|
||||
- **The pool came from two boots, not a formula.** 2 x 262,144 with no
|
||||
projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
|
||||
with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
|
||||
difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
|
||||
encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
|
||||
booted healthy but left only 1,153 MiB free, which is how the band got
|
||||
violated and why the seam now prices non-KV at the measured 9 GiB and
|
||||
REJECTS that shape.
|
||||
- **Three contexts cost window, and that is the tradeoff Frank was told:**
|
||||
180,224 tokens per conversation, not the native 262,144. A longer prompt is
|
||||
a hard `400 exceed_context_size_error`, so `models.yml` sets
|
||||
`contextWindow: 180224`. Consequence: frank-primary's 183,501-token
|
||||
compaction request no longer fits here with summary room; the two-slot
|
||||
rollback is the way back.
|
||||
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
|
||||
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
|
||||
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
|
||||
queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
|
||||
`reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
|
||||
- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
|
||||
top-level `images` field — that request answered "Brown." for the same red
|
||||
image, i.e. it generated from text alone. Only the `image_url` content part
|
||||
reaches the model. `/completion` accepts `images` but this build still
|
||||
requires `messages`.
|
||||
- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
|
||||
mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
|
||||
empty) in the env file is the documented text-only opt-out and returns ~1.4
|
||||
GiB to KV.
|
||||
- Reclaim staged, **not purged, awaiting approval**:
|
||||
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
|
||||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||||
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
|
||||
@@ -303,8 +303,8 @@
|
||||
"section": "AI",
|
||||
"details": {
|
||||
"role": "Fast utility model",
|
||||
"spec": "Gemma4-12B QAT (Q4_K_M, 7.4 GiB) — 1M-token pool across four conversations; fastest at images.",
|
||||
"inputs": ["text", "image", "audio"],
|
||||
"spec": "MiMo-V2.6-Distill-Qwen-9B (INT8 W8A16, 12 GiB) on SGLang — 606k-token pool across four 151k-token conversations; 84 tok/s decode; sees images.",
|
||||
"inputs": ["text", "image"],
|
||||
"use": "Agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence."
|
||||
},
|
||||
"url": "",
|
||||
|
||||
Executable
+150
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: autoresearch measure client behavior.
|
||||
#
|
||||
# The measure client is the whole yardstick of the frank-primary speed
|
||||
# session, so its METRIC arithmetic gets a behavior test: against a
|
||||
# deterministic mock endpoint that emits a KNOWN number of completion
|
||||
# tokens at a KNOWN per-token delay, the reported decode rate must land
|
||||
# near the nominal rate, the usage accounting must come back exact, and an
|
||||
# unhealthy endpoint must fail loudly (no half-measured numbers).
|
||||
#
|
||||
# The mock implements only what the client calls: POST
|
||||
# /v1/messages/count_tokens (word count as the token count, which makes the
|
||||
# client's words/token calibration come out to exactly 1.0) and a streaming
|
||||
# POST /v1/chat/completions (one content chunk per completion token, then a
|
||||
# usage chunk, then [DONE]).
|
||||
|
||||
load helpers
|
||||
|
||||
MEASURE="$REPO_ROOT/scripts/autoresearch/frank-primary-measure.py"
|
||||
MOCK="$BATS_TEST_TMPDIR/mock.py"
|
||||
PORT=""
|
||||
MOCK_PID=""
|
||||
|
||||
setup() {
|
||||
# Deterministic mock; argv: <port> <delay-seconds>; env MOCK_FAIL=1
|
||||
# makes the chat endpoint answer 500.
|
||||
cat >"$MOCK" <<'PYEOF'
|
||||
import json, os, sys, time
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
PORT = int(sys.argv[1])
|
||||
DELAY = float(sys.argv[2])
|
||||
FAIL = os.environ.get("MOCK_FAIL", "0") == "1"
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def _send(self, obj, code=200):
|
||||
b = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(b)))
|
||||
self.end_headers()
|
||||
self.wfile.write(b)
|
||||
|
||||
def do_POST(self):
|
||||
n = int(self.headers.get("Content-Length", "0"))
|
||||
body = json.loads(self.rfile.read(n) or b"{}")
|
||||
if self.path == "/v1/messages/count_tokens":
|
||||
text = body["messages"][0]["content"]
|
||||
self._send({"input_tokens": max(1, len(text.split()))})
|
||||
elif self.path == "/v1/chat/completions":
|
||||
if FAIL:
|
||||
self._send({"error": "engine down"}, 500)
|
||||
return
|
||||
max_tokens = int(body.get("max_tokens", 64))
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/event-stream")
|
||||
self.end_headers()
|
||||
for i in range(max_tokens):
|
||||
ev = {"choices": [{"delta": {"content": "tok%d " % i}}]}
|
||||
self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
|
||||
self.wfile.flush()
|
||||
time.sleep(DELAY)
|
||||
prompt_words = max(1, len(body["messages"][0]["content"].split()))
|
||||
ev = {"choices": [{"delta": {}}],
|
||||
"usage": {"prompt_tokens": prompt_words,
|
||||
"completion_tokens": max_tokens}}
|
||||
self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
|
||||
self.wfile.write(b"data: [DONE]\n\n")
|
||||
self.wfile.flush()
|
||||
else:
|
||||
self._send({"error": "unknown path"}, 404)
|
||||
|
||||
|
||||
HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
|
||||
PYEOF
|
||||
}
|
||||
free_port() {
|
||||
python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()'
|
||||
}
|
||||
|
||||
start_mock() { # $1 = ok|fail
|
||||
PORT="$(free_port)"
|
||||
if [ "$1" = "fail" ]; then
|
||||
MOCK_FAIL=1 python3 "$MOCK" "$PORT" 0.005 &
|
||||
else
|
||||
python3 "$MOCK" "$PORT" 0.005 &
|
||||
fi
|
||||
MOCK_PID=$!
|
||||
local i
|
||||
for i in $(seq 1 100); do
|
||||
if python3 -c "import socket; socket.create_connection(('127.0.0.1', $PORT), 0.2).close()" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 0.1
|
||||
done
|
||||
echo "mock did not start" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
stop_mock() {
|
||||
if [ -n "$MOCK_PID" ]; then
|
||||
kill "$MOCK_PID" 2>/dev/null || true
|
||||
wait "$MOCK_PID" 2>/dev/null || true
|
||||
MOCK_PID=""
|
||||
fi
|
||||
}
|
||||
|
||||
@test "measure exists and compiles" {
|
||||
[ -f "$MEASURE" ]
|
||||
python3 -m py_compile "$MEASURE"
|
||||
}
|
||||
|
||||
@test "measure computes decode rate and exact usage from a controlled endpoint" {
|
||||
start_mock ok || return 1
|
||||
run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
|
||||
stop_mock
|
||||
[ "$status" -eq 0 ] || { echo "$output"; return 1; }
|
||||
# run() merges stderr in; the contract is: exactly ONE json result line
|
||||
# (the window script parses the same way), progress on stderr.
|
||||
printf '%s\n' "$output" | python3 -c '
|
||||
import json, sys
|
||||
lines = [l for l in sys.stdin.read().splitlines() if l.startswith("{")]
|
||||
assert len(lines) == 1, "expected exactly one JSON line, got %d" % len(lines)
|
||||
d = json.loads(lines[0])
|
||||
# nominal 200 t/s (0.005 s/chunk) minus small per-chunk overhead
|
||||
for k in ("decode_tps", "decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
|
||||
v = d[k]
|
||||
assert 60.0 <= v <= 320.0, "%s=%s outside [60, 320]" % (k, v)
|
||||
# the mock emits exactly max_tokens completion tokens; the accounting must be exact
|
||||
assert d["completion_tokens_4096"] == 256, d["completion_tokens_4096"]
|
||||
assert d["completion_tokens_115"] == 256
|
||||
# prompt sizing: calibration is exactly 1.0 words/token in the mock, so the
|
||||
# measured prompt lengths equal the targets
|
||||
assert d["prompt_tokens_115"] == 115
|
||||
assert d["prompt_tokens_4096"] == 4096
|
||||
print("measure claims verified")
|
||||
'
|
||||
}
|
||||
|
||||
@test "measure fails loudly on an unhealthy endpoint instead of half-measuring" {
|
||||
start_mock fail || return 1
|
||||
run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
|
||||
stop_mock
|
||||
[ "$status" -ne 0 ]
|
||||
! printf '%s\n' "$output" | grep -qE '^\{'
|
||||
}
|
||||
+242
-296
@@ -1,334 +1,280 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-large root unit on dankclaw.
|
||||
# Seam: frank-large serves kingjones777's Qwen3.8-27B Q8_0_ROCMFPX_AGENT (a
|
||||
# ftype-115 ROCmFPX file) with the repo's MTP draft head on the Strix Halo
|
||||
# iGPU, via the ROCmFPX fork of llama.cpp — one session at the native
|
||||
# 262,144 window.
|
||||
#
|
||||
# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
|
||||
# Strix Halo inference engine built from the pinned v0.1.1 tag, serving
|
||||
# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) UD-Q8_K_XL with the z-lab
|
||||
# DFlash2 Q4_K_M drafter. Two changes landed on this endpoint that day: the
|
||||
# engine (from the stock llama.cpp prebuilt) and the payload (from
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL). Port, alias and client contract did
|
||||
# not move.
|
||||
# This seam exists because the format is the constraint: GGML_TYPE_Q8_0_ROCMFPX
|
||||
# (tensor type 103) is understood by the ROCmFPX fork and by nothing else on
|
||||
# this host, so the runtime, the payload and the deployer's gate have to agree
|
||||
# on paths that no other endpoint uses. On 2026-10-01 a serve-script gate and
|
||||
# its deployer twin disagreed during a cutover and the endpoint crash-looped;
|
||||
# both sides are pinned here.
|
||||
#
|
||||
# Most of this seam runs the serve script against a stub engine and asserts
|
||||
# on the argv it actually execs, so what is pinned is the command the
|
||||
# operator gets rather than the text of the script: a tunable renamed,
|
||||
# dropped, or silently defaulted away fails here the way it would fail a
|
||||
# deployment.
|
||||
load helpers
|
||||
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
|
||||
DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-gufo.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# The deployed target's filename, kept here so a rename in the env file and a
|
||||
# rename in this stub cannot drift apart unnoticed.
|
||||
TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
|
||||
# Everything is tested from what an operator or a supervisor can observe — the
|
||||
# command line, the exit code, the deployer's error text — never from anything
|
||||
# inside llama.cpp.
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
RUNTIME_DIR="$STAGE/gufo"
|
||||
MODEL_DIR="$STAGE/models"
|
||||
DRAFTER_DIR="$STAGE/draft"
|
||||
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" "$DRAFTER_DIR"
|
||||
# A stub engine: the serve script execs it, so $@ is exactly what gufo
|
||||
# would be invoked with.
|
||||
cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$@"
|
||||
STUB
|
||||
chmod +x "$RUNTIME_DIR/bin/gufo"
|
||||
# The target and the drafter live in SEPARATE directories since the
|
||||
# 2026-09-28 payload swap: the target came from a different repository,
|
||||
# the DFlash2 drafter did not move. A stub that colocated them would
|
||||
# never notice DRAFTER_DIR going stale.
|
||||
: > "$MODEL_DIR/$TARGET_STUB"
|
||||
: > "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
STUB_ENV="$STAGE/stub.env"
|
||||
cat > "$STUB_ENV" <<EOF
|
||||
RUNTIME_DIR=$RUNTIME_DIR
|
||||
MODEL_DIR=$MODEL_DIR
|
||||
DRAFTER_DIR=$DRAFTER_DIR
|
||||
EOF
|
||||
load helpers
|
||||
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
DEPLOYER="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-rocmfpx.sh"
|
||||
RUNTIME_STUB="/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server"
|
||||
MODEL_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf"
|
||||
DRAFT_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
|
||||
MMPROJ_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf"
|
||||
|
||||
TEST_TMP="$(mktemp -d)"
|
||||
SRC="$TEST_TMP/src"
|
||||
mkdir -p "$SRC"
|
||||
git -C "$SRC" init -q
|
||||
git -C "$SRC" -c user.email=t -c user.name=t commit -q --allow-empty -m x
|
||||
}
|
||||
|
||||
teardown() {
|
||||
if [ -n "${STAGE:-}" ]; then
|
||||
rm -rf "$STAGE"
|
||||
fi
|
||||
rm -rf "$TEST_TMP"
|
||||
}
|
||||
|
||||
# serve_argv [env-file] — run the serve script and capture the engine's argv,
|
||||
# one token per line, in $output. No argument exercises the built-in defaults
|
||||
# with the stub's paths.
|
||||
serve_argv() {
|
||||
local envfile="${1:-$STUB_ENV}"
|
||||
if [ -n "$envfile" ]; then
|
||||
FRANK_LARGE_ENV_FILE="$envfile" run bash "$SERVE"
|
||||
else
|
||||
run env FRANK_LARGE_ENV_FILE="$STAGE/absent.env" bash "$SERVE"
|
||||
fi
|
||||
# make_stub — an executable that prints its whole argv on one line, so
|
||||
# "what command line would the supervisor run" is observable output.
|
||||
make_stub() {
|
||||
cat > "$TEST_TMP/bin" <<'STUB'
|
||||
#!/bin/sh
|
||||
printf '%s\n' "$*"
|
||||
STUB
|
||||
chmod +x "$TEST_TMP/bin"
|
||||
}
|
||||
|
||||
# argv_has <token> — exact token match, so --port never satisfies a test
|
||||
# looking for a different flag.
|
||||
argv_has() {
|
||||
grep -qxF -- "$1" <<<"$output"
|
||||
@test "the env file defines the deployment values" {
|
||||
grep -qE '^LLAMA_BIN=/' "$ENV_FILE"
|
||||
grep -qE '^MODEL_FILE=/' "$ENV_FILE"
|
||||
grep -qE '^DRAFT_FILE=/' "$ENV_FILE"
|
||||
grep -qE '^ALIAS=[a-z0-9-]+$' "$ENV_FILE"
|
||||
grep -qE '^LISTEN_PORT=[0-9]+$' "$ENV_FILE"
|
||||
grep -qE '^CTX_SIZE=[0-9]+$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
# argv_value <flag> — the token following <flag>, empty if the flag is absent.
|
||||
argv_value() {
|
||||
local i
|
||||
i="$(grep -nx -- "$1" <<<"$output" | cut -d: -f1)"
|
||||
[ -n "$i" ] || return 0
|
||||
sed -n "$((i + 1))p" <<<"$output"
|
||||
}
|
||||
|
||||
@test "the frank-large unit, env, serve and build files exist where the deployer installs them from" {
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$SERVE" ]
|
||||
[ -x "$BUILD" ]
|
||||
}
|
||||
|
||||
@test "frank-large is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
# The engine pins buffers; the memlock budget must be unbounded or the
|
||||
# load fails on the RLIMIT.
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
# The unit runs the repo's own serve script with the repo's env.
|
||||
grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT"
|
||||
grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the fleet endpoint and alias survive the cutover" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# frank-large at :7770, whatever engine answers it.
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
}
|
||||
|
||||
@test "the built-in defaults are one session of the full 262144 window with DFlash2" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# 262144 is the model's native window (the operator's call, 2026-09-21).
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
# One session is the single stream every prior frank-large served.
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
# A draft ceiling of 0 would silently serve the autoregressive rate.
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
}
|
||||
|
||||
@test "the shipped env file, not just the defaults, serves the deployed shape" {
|
||||
# The defaults are a floor; systemd sources the env file. Only the
|
||||
# on-host paths are redirected at the stub, so every fleet value in the
|
||||
# shipped file is the one under test. MODEL_FILE and DRAFTER_FILE are
|
||||
# absolute in the shipped file, so they need redirecting too — a sed that
|
||||
# only rewrote MODEL_DIR would leave them pointing at /home/dank and the
|
||||
# test would pass for the wrong reason (or fail on a box that has them).
|
||||
sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
|
||||
-e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
|
||||
-e "s#^MODEL_FILE=.*#MODEL_FILE=$MODEL_DIR/$TARGET_STUB#" \
|
||||
-e "s#^DRAFTER_DIR=.*#DRAFTER_DIR=$DRAFTER_DIR#" \
|
||||
-e "s#^DRAFTER_FILE=.*#DRAFTER_FILE=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf#" \
|
||||
"$ENV_FILE" > "$STAGE/shipped.env"
|
||||
serve_argv "$STAGE/shipped.env"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
[ "$(argv_value --draft-policy)" = "adaptive" ]
|
||||
# The shipped file must name the target the fleet actually serves, not a
|
||||
# path that merely happens to exist.
|
||||
[ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
|
||||
[ "$(argv_value --dflash-model)" = "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ]
|
||||
}
|
||||
|
||||
@test "the engine is pointed at the payload that is actually on disk" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ -f "$(argv_value --model)" ]
|
||||
# A drafter path that does not exist would load the target and decode at
|
||||
# the autoregressive rate, which reads as "the new runtime is slow".
|
||||
[ -f "$(argv_value --dflash-model)" ]
|
||||
[ "$(argv_value --model)" != "$(argv_value --dflash-model)" ]
|
||||
}
|
||||
|
||||
@test "frank-large stays text-only" {
|
||||
# The payload has no mmproj, so an --mmproj would point at nothing. The
|
||||
# fleet's image path is frank-small; models.yml declares frank-large
|
||||
# text-only.
|
||||
serve_argv
|
||||
! argv_has --mmproj
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
}
|
||||
|
||||
@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
|
||||
# Greedy is what this endpoint has always served; a silent switch to
|
||||
# sampled output would be a model-behaviour change disguised as a
|
||||
# runtime swap. Prior thinking blocks must survive across turns or a
|
||||
# multi-turn agent loses its own reasoning between tool calls.
|
||||
serve_argv
|
||||
[ "$(argv_value --temperature)" = "0" ]
|
||||
[ "$(argv_value --preserve-thinking)" = "on" ]
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing runtime with the build pointer" {
|
||||
rm -f "$RUNTIME_DIR/bin/gufo"
|
||||
serve_argv
|
||||
@test "the serve script fails when the runtime is missing" {
|
||||
# Only the model, the head and the projector exist; the fork was never
|
||||
# built. llama.cpp is not reached, so this fails fast and says what to
|
||||
# run instead of failing at bind time.
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/nope/llama-server" \
|
||||
"MODEL_FILE=$TEST_TMP/model.gguf" "DRAFT_FILE=$TEST_TMP/draft.gguf" \
|
||||
"MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"runtime not found"* ]]
|
||||
[[ "$output" == *"build-gufo.sh"* ]]
|
||||
# A restart loop would be the alternative; nothing may be exec'd.
|
||||
! argv_has serve
|
||||
[[ "$output" == *"build it with scripts/build-rocmfpx.sh"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing drafter with a fetch pointer" {
|
||||
# The drafter lives in its own directory now, so the target can be present
|
||||
# and this still has to be caught.
|
||||
rm -f "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
serve_argv
|
||||
@test "the serve script refuses to start without the model file" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/absent.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"Qwen3.8-27B-DFlash2-Q4_K_M.gguf"* ]]
|
||||
[[ "$output" == *"download-frank-large"* ]]
|
||||
! argv_has serve
|
||||
[[ "$output" == *"is not readable"* ]]
|
||||
[[ "$output" == *"download-frank-large-kingjones.sh"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing target and names the file" {
|
||||
rm -f "$MODEL_DIR/$TARGET_STUB"
|
||||
serve_argv
|
||||
@test "the serve script refuses to start without the MTP draft head" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/absent.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"$TARGET_STUB"* ]]
|
||||
[[ "$output" == *"MTP draft head"* ]]
|
||||
}
|
||||
|
||||
@test "both payload files are fetched resumably and verified against declared sizes + sha256" {
|
||||
# The target and the drafter come from DIFFERENT repositories, fetched by
|
||||
# two scripts. Anchor the repo checks to the URL ASSIGNMENT, not to the
|
||||
# file: the same repo string also appears in each script's header comment,
|
||||
# so a plain grep stayed green after the URL was repointed at another
|
||||
# repository -- the script would then download something the serve script
|
||||
# never asked for.
|
||||
grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
|
||||
grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
|
||||
for f in "$FETCH" "$DRAFTER_FETCH"; do
|
||||
# -f so an HTTP error is a failure rather than a small "artifact";
|
||||
# -C - so an interrupted 29 GiB run resumes.
|
||||
grep -q -- '-fL -C -' "$f"
|
||||
# The size and the sha256 are compared after the transfer, not trusted.
|
||||
grep -q 'SHA256 MISMATCH' "$f"
|
||||
grep -q 'SIZE MISMATCH' "$f"
|
||||
@test "the serve script refuses a missing mmproj unless vision is opted out" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/absent.gguf" "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"set MMPROJ_FILE='' for text-only"* ]]
|
||||
# The documented opt-out must actually start, without a --mmproj argument.
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=" "$SERVE"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" != *"--mmproj"* ]]
|
||||
}
|
||||
|
||||
@test "the serve script fails when LLAMA_BIN is empty" {
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env LLAMA_BIN= "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"LLAMA_BIN is empty"* ]]
|
||||
}
|
||||
|
||||
@test "the shipped env points at the ROCmFPX runtime and the AGENT payload" {
|
||||
grep -qE "^LLAMA_BIN=$RUNTIME_STUB$" "$ENV_FILE"
|
||||
grep -qE "^MODEL_FILE=$MODEL_STUB$" "$ENV_FILE"
|
||||
grep -qE "^DRAFT_FILE=$DRAFT_STUB$" "$ENV_FILE"
|
||||
grep -qE "^MMPROJ_FILE=$MMPROJ_STUB$" "$ENV_FILE"
|
||||
# HIP reads bare HOST/TEMP/GPU_MAX_HEAP_SIZE out of the environment that
|
||||
# `set -a` creates; a tunable named HOST leaks into the runtime.
|
||||
! grep -qE '^(HOST|TEMP|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the argv contract keeps the fleet's port and alias" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" \
|
||||
ALIAS=frank-large BIND_ADDR=127.0.0.1 LISTEN_PORT=7770 "$SERVE"
|
||||
[ "$status" -eq 0 ]
|
||||
# The caller's values must survive the env file, or a single-run operator
|
||||
# override silently does nothing.
|
||||
[[ "$output" == *"--host 127.0.0.1"* ]]
|
||||
[[ "$output" == *"--port 7770"* ]]
|
||||
[[ "$output" == *"--alias frank-large"* ]]
|
||||
# Vision is wired: the projector is the whole vision tower.
|
||||
[[ "$output" == *"--mmproj $TEST_TMP/mmproj.gguf"* ]]
|
||||
}
|
||||
|
||||
@test "the shipped argv is one full native window on one session" {
|
||||
grep -qE '^CTX_SIZE=262144$' "$ENV_FILE"
|
||||
grep -qE '^PARALLEL=1$' "$ENV_FILE"
|
||||
# 262144 IS the model's trained window (general.context_length in the
|
||||
# GGUF), so 1 slot gets the whole thing — no truncation, no yarn scaling.
|
||||
#
|
||||
# Budget, from the GGUF header: 26.28 GiB of weights + 1.57 GiB draft head
|
||||
# + 0.87 GiB projector + 16 full-attention layers x 4 kv heads x 256 dim x
|
||||
# 2 (K+V) x 2 B = 64 KiB/token of f16 KV (~16 GiB at 262,144) + ~0.14 GiB
|
||||
# of DeltaNet state + a couple GiB of compute buffers, against the iGPU's
|
||||
# ~114 GiB GTT aperture. Change CTX_SIZE and this arithmetic, not a boot
|
||||
# log, is what tells you it no longer fits.
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
kv_gib=$((ctx * 65536 / 1073741824))
|
||||
total_gib=$((27 + 2 + 1 + kv_gib + 3))
|
||||
[ "$total_gib" -le 114 ]
|
||||
}
|
||||
|
||||
@test "the MTP draft contract matches the model card's measured optimum" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -eq 0 ]
|
||||
# The head is the model's own nextn layer, loaded as a separate draft
|
||||
# model because this checkpoint ships it standalone (no blk.<n>.nextn.*
|
||||
# tensors in the target file).
|
||||
[[ "$output" == *"--spec-type draft-mtp"* ]]
|
||||
[[ "$output" == *"--model-draft $TEST_TMP/draft.gguf"* ]]
|
||||
# n-max 4 is the measured optimum; n-max 8 measured WORSE (acceptance
|
||||
# fell to 0.757 and the extra verification cost outran the savings), so
|
||||
# the flag is a measurement, not a wish.
|
||||
[[ "$output" == *"--spec-draft-n-max 4"* ]]
|
||||
[[ "$output" == *"--spec-draft-n-min 0"* ]]
|
||||
[[ "$output" == *"--spec-draft-p-min 0.0"* ]]
|
||||
# The draft runs on the iGPU beside the target: no device-to-device hop
|
||||
# on a link that measures single-digit GB/s.
|
||||
[[ "$output" == *"--spec-draft-device ROCm0"* ]]
|
||||
[[ "$output" == *"--spec-draft-ngl 99"* ]]
|
||||
}
|
||||
|
||||
@test "KV and placement stay unquantized with flash attention on" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"--cache-type-k f16"* ]]
|
||||
[[ "$output" == *"--cache-type-v f16"* ]]
|
||||
# ROCmFPx quantized KV has no kernel on this branch, and the endpoint has
|
||||
# the aperture to keep KV unquantized anyway.
|
||||
[[ "$output" == *"--flash-attn on"* ]]
|
||||
# --fit's placement heuristics do not know that 311 tensors of type 103
|
||||
# belong on the iGPU, so the whole stack is placed explicitly instead.
|
||||
[[ "$output" == *"--fit off"* ]]
|
||||
[[ "$output" == *"--n-gpu-layers 999"* ]]
|
||||
}
|
||||
|
||||
@test "the client contract wires thinking and template kwargs" {
|
||||
make_stub
|
||||
touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
|
||||
run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
|
||||
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
|
||||
[ "$status" -eq 0 ]
|
||||
# Without --jinja, llama.cpp silently ignores the harness's
|
||||
# chat_template_kwargs, so thinking control disappears from the API.
|
||||
[[ "$output" == *"--jinja"* ]]
|
||||
# The harness reads `reasoning_content` for this endpoint, which is what
|
||||
# the deepseek emission format produces (vLLM's `reasoning` is not it).
|
||||
[[ "$output" == *"--reasoning-format deepseek"* ]]
|
||||
# The template needs an unbounded thinking budget.
|
||||
[[ "$output" == *"--reasoning-budget -1"* ]]
|
||||
}
|
||||
|
||||
@test "the deployer gates exactly the paths the serve script opens" {
|
||||
# A gate that checks a different file than the engine opens is the
|
||||
# 2026-10-01 crash-loop: deploy passed, the unit restart-looped. Read the
|
||||
# engine's own paths out of the env file and require the deployer's
|
||||
# defaults to name the same ones.
|
||||
for pair in "LLAMA_BIN:LLM_LARGE_RUNTIME" \
|
||||
"MODEL_FILE:LLM_LARGE_MODEL" \
|
||||
"DRAFT_FILE:LLM_LARGE_DRAFT" \
|
||||
"MMPROJ_FILE:LLM_LARGE_MMPROJ"; do
|
||||
env_key="${pair%%:*}"
|
||||
dep_key="${pair##*:}"
|
||||
want="$(sed -nE "s|^${env_key}=(.*)\$|\1|p" "$ENV_FILE")"
|
||||
[ -n "$want" ]
|
||||
got="$(grep -oE "${dep_key}:-[^}]+" "$DEPLOYER" | head -1 | sed "s|^${dep_key}:-||")"
|
||||
[ "$got" = "$want" ]
|
||||
done
|
||||
}
|
||||
|
||||
@test "the serve script and the fetch scripts agree on the payload filenames" {
|
||||
# The serve script's MODEL_FILE and the fetch script's REL are two
|
||||
# independent spellings of one filename, and a repo is not required to
|
||||
# keep them in step: a re-upload under a new name, or an env edited
|
||||
# without the fetch, leaves the endpoint unable to start with a
|
||||
# "missing asset" that no test would have predicted. Bind them here.
|
||||
local served fetched drafter_served drafter_fetched
|
||||
served="$(basename "$(grep '^MODEL_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
|
||||
fetched="$(sed -n 's/^REL="\(.*\)"$/\1/p' "$FETCH")"
|
||||
[ -n "$served" ]
|
||||
[ "$served" = "$fetched" ]
|
||||
drafter_served="$(basename "$(grep '^DRAFTER_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
|
||||
drafter_fetched="$(sed -n 's/^ *"\(Qwen3\.8-27B-DFlash2[^ ]*\.gguf\) .*/\1/p' "$DRAFTER_FETCH")"
|
||||
[ -n "$drafter_served" ]
|
||||
[ "$drafter_served" = "$drafter_fetched" ]
|
||||
@test "the deployer points a missing runtime at the fork build script" {
|
||||
grep -qE 'build it: bash scripts/build-rocmfpx\.sh' "$DEPLOYER"
|
||||
grep -qE 'fetch it: bash scripts/download-frank-large-kingjones\.sh' "$DEPLOYER"
|
||||
}
|
||||
|
||||
@test "the drafter is still fetched by the script that carries its Hub repo" {
|
||||
# Guards a future "simplification" that folds the drafter into the Swift
|
||||
# script and silently orphans the artifact the serve script still points
|
||||
# at via DRAFTER_DIR.
|
||||
grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
|
||||
! grep -q 'DFlash2' "$FETCH"
|
||||
@test "the stray-process sweep cannot take frank-small down with frank-large" {
|
||||
# frank-small serves a different GGUF over llama-server on the SAME host.
|
||||
# A sweep keyed on the binary name (`pkill -f llama-server`) would kill
|
||||
# the 3090 endpoint too, so it is keyed on frank-large's own payload file.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'" "$DEPLOYER"
|
||||
! grep -qE "pkill -f 'llama-server'" "$DEPLOYER"
|
||||
}
|
||||
|
||||
@test "the build script refuses a source tree that is not the pinned commit" {
|
||||
# gufo ships no prebuilt Linux binaries, so the runtime is built here.
|
||||
# A tag that moves under us would silently install different kernels
|
||||
# under the same RUNTIME_DIR, so the pin is enforced, not documented.
|
||||
local src="$STAGE/not-the-pinned-commit"
|
||||
mkdir -p "$src"
|
||||
git -C "$src" init -q
|
||||
git -C "$src" -c user.email=t@t -c user.name=t commit -q --allow-empty -m drift
|
||||
run env GUFO_SRC_DIR="$src" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
|
||||
# The pin is load-bearing: the older builds of this fork line segfault in
|
||||
# ggml_compute_forward_set_rows on this exact payload. A runtime that
|
||||
# differs from what was qualified must be heard about at deploy time.
|
||||
wrong="$(git -C "$SRC" rev-parse HEAD)"
|
||||
run env ROCMFPX_SRC_DIR="$SRC" ROCMFPX_COMMIT="ffff$wrong" \
|
||||
"$BUILD" --verify
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"GUFO_COMMIT mismatch"* ]]
|
||||
[[ "$output" == *"ROCMFPX_COMMIT"*mismatch* ]]
|
||||
}
|
||||
|
||||
@test "the build script reports a runtime that was never built" {
|
||||
# --verify on a host with no install must say so rather than exit 0 on
|
||||
# an empty check.
|
||||
run env GUFO_SRC_DIR="$STAGE/no-source-tree" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"no gufo at"* ]]
|
||||
@test "the build script pins a real commit and the gfx1151 target" {
|
||||
grep -qE 'ROCMFPX_COMMIT=.*[0-9a-f]{40}' "$BUILD"
|
||||
grep -qE 'GPU_TARGET=.*gfx1151' "$BUILD"
|
||||
grep -qE 'DGPU_TARGETS=' "$BUILD"
|
||||
# The webui asset fetch half-failed on this box and aborted the link at
|
||||
# 98%; the build must not depend on it.
|
||||
grep -qE 'DLLAMA_BUILD_WEBUI=OFF' "$BUILD"
|
||||
}
|
||||
|
||||
@test "the deploy handler gates on the runtime, the payload and the port" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_RUNTIME' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
|
||||
@test "the deployed unit keeps the fleet's restart and boot-persistence shape" {
|
||||
grep -qE '^Restart=(always|on-failure)$' "$UNIT"
|
||||
grep -qE '^WantedBy=' "$UNIT"
|
||||
grep -qE '^ExecStart=.*/serve-frank-large\.sh$' "$UNIT"
|
||||
grep -qE '^User=dank$' "$UNIT"
|
||||
# The payload is 26 GiB of mmap'd weights plus pinned staging buffers; an
|
||||
# RLIMIT smaller than the pinned set fails the load in a way that looks
|
||||
# like an engine crash.
|
||||
grep -qE '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the deploy handler will not stop a serving endpoint it cannot replace" {
|
||||
# The runtime gate must be live (a real conditional on an executable
|
||||
# path, not a branch that never fires) and it must return BEFORE the
|
||||
# stop, or a host whose gufo was never built takes frank-large down on
|
||||
# its way to reporting the failure.
|
||||
local large gate bail stop
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
[[ "$large" == *'build-gufo.sh'* ]]
|
||||
gate="$(grep -n 'if \[\[ ! -x "\$LLM_LARGE_RUNTIME" \]\]' <<<"$large" | head -1 | cut -d: -f1)"
|
||||
stop="$(grep -n 'systemctl stop frank-large' <<<"$large" | head -1 | cut -d: -f1)"
|
||||
[ -n "$gate" ]
|
||||
[ -n "$stop" ]
|
||||
bail="$(awk -v g="$gate" 'NR > g && /return 1/ { print NR; exit }' <<<"$large")"
|
||||
[ -n "$bail" ]
|
||||
[ "$bail" -lt "$stop" ]
|
||||
}
|
||||
|
||||
@test "the stray-run sweep keys on the drafter and cannot take frank-small down" {
|
||||
# The sweep used to key on the TARGET filename, which the 2026-09-28
|
||||
# payload swap moved to a different repository. It would have stopped
|
||||
# matching the live process, and a stray run would then hold 7770 past the
|
||||
# port gate and fail the deploy. The drafter is the part that did NOT
|
||||
# move, and every frank-large command line carries it via --dflash-model.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M'" "$REMOTE_DEPLOY"
|
||||
# frank-small serves a different GGUF on the 3090 with no DFlash2 drafter,
|
||||
# so a binary-name sweep is the only thing that would endanger it.
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
! grep -q 'pkill -x gufo' <<<"$large"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the halogen quadlet without deleting it" {
|
||||
grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY"
|
||||
grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
|
||||
# Nothing live ships for this endpoint as a container any more. The
|
||||
# archived copy is host state in an ignored .AGENTTRASH — deliberately
|
||||
# untracked, and present only in a primary checkout — so search the
|
||||
# tracked tree, not the filesystem.
|
||||
[ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ]
|
||||
! git -C "$REPO_ROOT" grep -q 'Image=.*halogen' -- dankclaw
|
||||
@test "the unit points at no retired frank-large runtime path" {
|
||||
# The host keeps every retired payload and build on disk, so a stale path
|
||||
# in a unit still resolves — to a runtime nobody has measured in weeks.
|
||||
! grep -qE 'GitPackages/(ROCmFPX-kairic|llama.cpp-ik_llama|llama-builds|ciru-orca)' "$UNIT"
|
||||
! grep -q 'halogen' "$UNIT"
|
||||
}
|
||||
+147
-56
@@ -2,25 +2,25 @@
|
||||
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
|
||||
#
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
|
||||
# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which
|
||||
# reversed the failed GLM/DFlash2 experiment): the serve script drives
|
||||
# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
|
||||
# engine flag qualified by boot probe and carried in the repo env file. The
|
||||
# seam pins what the flags alone do not say: the KV qualification law
|
||||
# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB
|
||||
# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV
|
||||
# qualification (e4m3 for BOTH K and V — the stock flag quantizes them
|
||||
# together), the automatic-offload law (the fork plans the tiered-expert
|
||||
# offload itself; no flag exists to set), the self-healing law (the run
|
||||
# returns before the engine is up, so the serve script must wait on the
|
||||
# container for Restart=on-failure to mean anything), the memlock law that
|
||||
# makes the pinned host rows legal, the deployer's image/payload gates with
|
||||
# the false-green MainPID check, and the measured laws: expert prefetch
|
||||
# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved
|
||||
# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance
|
||||
# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading
|
||||
# carried the prefetch tax), and NO tool dialect (the GLM era added
|
||||
# --enable-auto-tool-choice; the 2339fae trio does not serve tools).
|
||||
# 2026-10-02 correction of the 2026-10-01 BF16 cutover): the serve
|
||||
# script drives `podman run` DIRECTLY (the image entrypoint execs
|
||||
# `vllm serve`), with every engine flag qualified by boot probe and carried
|
||||
# in the repo env file.
|
||||
# The seam pins what the flags alone do not say: the
|
||||
# KV qualification law (the night ladder: 262144/245760 OOM at graph
|
||||
# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
|
||||
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
|
||||
# clean cache + 262k windows), the automatic-offload law (the fork plans
|
||||
# the tiered-expert offload itself; no flag exists to set), the self-healing
|
||||
# law (the run returns before the engine is up, so the serve script must
|
||||
# wait on the container for Restart=on-failure to mean anything), the
|
||||
# memlock law that makes the pinned host rows legal, the deployer's
|
||||
# image/payload gates with the false-green MainPID check, and the measured
|
||||
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
|
||||
# no-spec family measured fastest once the prefetch tax was removed; MTP
|
||||
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
|
||||
# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
|
||||
# tool parser qwen3_xml, preserve_thinking on).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -43,7 +43,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# dank session's own hard limit is 8 MiB and cannot be raised without
|
||||
# restarting user@1000.service).
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
# Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
|
||||
# server exit CLEAN, so ExecStart returns 0 and on-failure never fires
|
||||
# (measured 2026-10-01 — the endpoint stayed down until a human noticed).
|
||||
grep -q '^Restart=always$' "$UNIT"
|
||||
! grep -q '^Restart=on-failure$' "$UNIT"
|
||||
# Exiting the engine takes as long as vLLM's drain: ExecStop runs
|
||||
# `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
|
||||
# kills the stop and records 'Failed with result timeout' (measured
|
||||
# 2026-10-01). Guard the relationship, not just the literal.
|
||||
stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
|
||||
[ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
|
||||
grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
|
||||
# StartLimitIntervalSec belongs to [Unit]; a copy in [Service] is an
|
||||
# 'Unknown key' warning systemd logs on every load.
|
||||
[ "$(grep -c '^StartLimitIntervalSec=' "$UNIT")" = "1" ]
|
||||
awk '/^\[Service\]/{in_svc=1} /^StartLimitIntervalSec=/{if(in_svc) exit 1}' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
@@ -76,41 +91,94 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES
|
||||
# 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and
|
||||
# reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is
|
||||
# PER RANK — sizing it to the global requirement OOMs the worker
|
||||
# (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4
|
||||
# full-length 200k contexts do NOT fit: four clients share the ~730k
|
||||
# token pool and each runs below 200k when all are deep.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
# 2026-10-01 night ladder (Frank's method), then a runtime-survivability
|
||||
# correction, then Frank's 2026-10-02 decision to give the pool back to
|
||||
# the expert arena. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
|
||||
# SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
|
||||
# free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
|
||||
# bytes free at runtime and a real client request (995-token prompt with
|
||||
# max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
|
||||
# ... 0 bytes free`. 200704 sits below the 241664 rung while leaving a
|
||||
# full 2.95 GiB/rank to cut loose for the arena. UTIL stays 0.98
|
||||
# (lowering it shrinks the engine's own budget instead of buying slack).
|
||||
# FINAL (2026-10-02, Frank): TWO windows at 200704 with FP8 KV and a pool
|
||||
# cut for the arena. fp8 costs 7,701 B/token (measured 6,138 attention +
|
||||
# 1,563 mamba/side), so 2 x 200704 = 401,408 tokens needs 3.091e9 B;
|
||||
# pinned at 3.15e9 (~405k tokens, ~44 MB margin). The arena is what
|
||||
# remains — the planner sizes it AFTER the KV override — so the 6,248
|
||||
# slots of the 6.1e9 shape become roughly 7,500 here.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
|
||||
# --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the
|
||||
# boot 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident)
|
||||
# and crash-looped at graph capture. The serve script keeps the opt-in
|
||||
# flag; the smaller pool here is the arena it would have been paying for.
|
||||
! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the pool is sized as the arena floor, not the context floor" {
|
||||
# THE LAW this seam pins: the fork's expert planner sizes the arena
|
||||
# AFTER the KV override, so every byte cut out of KV_BYTES becomes
|
||||
# arena. A context-size change must move the pool with it, or the
|
||||
# reduction buys nothing. 2 x 200704 = 401,408 tokens at fp8's 7,701
|
||||
# B/token is 3.091e9 B; the pin is 3.15e9, ~44 MB of margin above the
|
||||
# floor — the same convention the 6.1e9 shape used.
|
||||
floor=$(awk -v n="401408" -v b="7701" 'BEGIN{printf "%.0f", n*b}')
|
||||
[ "$floor" = "3091243008" ]
|
||||
kv=$(grep '^FRANK_PRIMARY_KV_BYTES=' "$ENV_FILE" | cut -d= -f2)
|
||||
[ "$kv" = "3150000000" ]
|
||||
margin=$(( kv - floor ))
|
||||
[ "$margin" -gt 40000000 ]
|
||||
[ "$margin" -lt 100000000 ]
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
|
||||
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
|
||||
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
||||
grep -q -- '--enable-prefix-caching' "$SERVE"
|
||||
grep -q -- '--trust-remote-code' "$SERVE"
|
||||
# DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
|
||||
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
|
||||
grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
|
||||
# --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
|
||||
# default (7.60 GiB = fitted + compile/graph-capture headroom), which only
|
||||
# moves the plan's `room` term and therefore the expert arena: ~1,025
|
||||
# more slots at 2.344 MiB each.
|
||||
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the endpoint serves NO tool dialect — tool agents route elsewhere" {
|
||||
# The 2339fae trio does NOT wire --enable-auto-tool-choice /
|
||||
# --tool-call-parser (the GLM era added glm47 dialect flags; the rollback
|
||||
# dropped them). tool_choice "auto" gets a 400 here — tool-using agents
|
||||
# must route at frank-small or frank-large until a parser is qualified
|
||||
# for this arm (a behavior change beyond the rollback, not a silent fix).
|
||||
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
! grep -q -- '--tool-call-parser' "$SERVE"
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
# 2026-10-01: the rollback-era trio served NO tool dialect and left
|
||||
# thinking inline in `content` with a dangling `</think>` and a null
|
||||
# `reasoning` field (measured: content 'We need answer...Brief.\n</think>
|
||||
# \n\n391') — the malformed thinking block clients choke on. The working
|
||||
# values are the ones ecaad36 shipped for this model and this fork's
|
||||
# registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
|
||||
# preserve_thinking keeps prior thinking blocks across turns.
|
||||
grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
|
||||
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
|
||||
grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
|
||||
grep -q 'preserve_thinking' "$ENV_FILE"
|
||||
grep -q 'enable_thinking' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
|
||||
@@ -125,28 +193,51 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
|
||||
}
|
||||
|
||||
@test "MTP4 spec decode is PRESENT BY MEASUREMENT" {
|
||||
# 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57
|
||||
# tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec
|
||||
# baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured
|
||||
# 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses
|
||||
# (17.4% acceptance). The earlier "MTP buys zero decode" reading carried
|
||||
# the prefetch tax and is retired with it. ngram-mod never booted on this
|
||||
# model (fork bug: PLE inputs prepared only on the MTP path).
|
||||
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
|
||||
grep -q -- '--speculative-config' "$SERVE"
|
||||
@test "speculative decode is ABSENT BY MEASUREMENT" {
|
||||
# The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
|
||||
# spec family against no-spec at matched geometry: no-spec measured
|
||||
# 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
|
||||
# the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
|
||||
# the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
|
||||
# never completed; until one shows a real win, no spec is the incumbent.
|
||||
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
}
|
||||
|
||||
@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" {
|
||||
# --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes
|
||||
# them together; a V-only quant needs a fork patch). The GLM-era MLA
|
||||
# concern (sparse-triton MLA decode path refusing fp8 KV) does not apply
|
||||
# to this arm: qwen's attention path accepted the dtype and a completion
|
||||
# probe qualified it 2026-09-29.
|
||||
@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
|
||||
# --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
|
||||
# bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
|
||||
# attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
|
||||
# 1,563. That halves the pool capacity per byte, which is why the contract
|
||||
# is two full-size windows instead of three; in exchange the KV is exact
|
||||
# (no quantization error) and measured +4% decode at shallow depth, where
|
||||
# fp8's dequant cost dominates over the extra bytes read.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
|
||||
# sweep's biggest lever, +18.7% decode vs default geometry, and the win
|
||||
# GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
|
||||
# g64x128 is past the peak; g8x16 mild. They are container ENV (the
|
||||
# fork's knobs), not engine flags — passed through with -e.
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "MNBT is at Frank's 1024 with the arena intact" {
|
||||
# MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
|
||||
# interchangeable on decode AND identical on arena (the sweep measured
|
||||
# 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
|
||||
# 1024 is the 2026-10-02 setting — the pool was cut to feed the expert
|
||||
# arena, so the workspace floor matters more than it did at 2048: the
|
||||
# arena budget is what is left, and 1024-4096 pay nothing for it.
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expert prefetch is absent BY LAW, not by omission" {
|
||||
# Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
|
||||
# prefetched slabs (36% precision) — roughly half the gather traffic was
|
||||
|
||||
+168
-75
@@ -1,12 +1,13 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
|
||||
# went back to the halogen engine on the iGPU. This seam pins what the flags
|
||||
# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
|
||||
# is involved), the repo-relative paths the unit executes, the separate client
|
||||
# port, the MTP contract (the head is a separate sidecar file loaded as the
|
||||
# draft model, chained behind ngram-mod), the vision handoff from frank-large,
|
||||
# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
|
||||
# Seam: the frank-small llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint; since 2026-10-01 night it serves
|
||||
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via a CUDA (sm_86) build of
|
||||
# llama.cpp master, chosen for the ASYMMETRIC KV Frank asked for: K f16
|
||||
# (unquantized) + V q8_0, which only llama.cpp exposes per-tensor. Since
|
||||
# 2026-10-02 it runs THREE contexts and carries a vision projector, and the
|
||||
# card must keep 2-3 GiB free. This seam runs the serve script against a stub
|
||||
# engine and asserts on the argv it actually execs, so what is pinned is the
|
||||
# command the operator gets.
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -15,6 +16,52 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
|
||||
TARGET_STUB="MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf"
|
||||
# Same basename as the shipped projector: the test env repoints it into $STAGE.
|
||||
MM_STUB="mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf"
|
||||
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
RUNTIME_DIR="$STAGE/llama"
|
||||
MODEL_DIR="$STAGE/models"
|
||||
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
|
||||
cat > "$RUNTIME_DIR/bin/llama-server" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$@"
|
||||
STUB
|
||||
chmod +x "$RUNTIME_DIR/bin/llama-server" "$RUNTIME_DIR/bin" 2>/dev/null || chmod +x "$RUNTIME_DIR/bin/llama-server"
|
||||
touch "$MODEL_DIR/$TARGET_STUB" "$MODEL_DIR/$MM_STUB"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$STAGE"
|
||||
}
|
||||
|
||||
# serve_argv [env-file] — run the serve script and capture the engine's
|
||||
# argv, one token per line, in $output.
|
||||
serve_argv() {
|
||||
local envf="${1:-$STAGE/test.env}"
|
||||
local mmproj
|
||||
mmproj="$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cat > "$envf" <<EOF
|
||||
LLAMA_BIN=$RUNTIME_DIR/bin/llama-server
|
||||
MODEL_FILE=$MODEL_DIR/$TARGET_STUB
|
||||
MMPROJ_FILE=$MODEL_DIR/${mmproj##*/}
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
CTX_SIZE=$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
|
||||
PARALLEL=$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")
|
||||
CACHE_TYPE_K=$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")
|
||||
CACHE_TYPE_V=$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")
|
||||
NGL=$(sed -nE 's/^NGL=([0-9]+)$/\1/p' "$ENV_FILE")
|
||||
EOF
|
||||
run env FRANK_SMALL_ENV_FILE="$envf" nvidia-modprobe=true bash "$SERVE"
|
||||
}
|
||||
|
||||
argv_value() {
|
||||
grep -A1 -xF -- "$1" <<<"$output" | tail -1
|
||||
}
|
||||
|
||||
@test "frank-small unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
@@ -28,94 +75,140 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
|
||||
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small offloads every layer to the 3090 with flash attention" {
|
||||
grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
|
||||
grep -q '^NGL=999$' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint; without --metrics there is no /metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
@test "the shipped env points at the llama.cpp runtime and the Q5_K_M GGUF" {
|
||||
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
|
||||
grep -qE "^MODEL_FILE=/home/dank/models/MiMo-Q5KM/$TARGET_STUB$" "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-small caps its CPU RAM cache below the build default" {
|
||||
# Without the flag the 8192 MiB default returns and the env cap is a
|
||||
# silent no-op; this memory-tight host needs the cap to hold.
|
||||
grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
|
||||
@test "the deployer gates on exactly the payload paths the env file ships" {
|
||||
# Anti-drift: the deploy handler checks files by its own defaults. If
|
||||
# those ever differ from what the unit actually loads, a green deploy
|
||||
# would be gating on a file nobody reads — and a missing projector would
|
||||
# ship a blind endpoint.
|
||||
local env_v dep_v path pair
|
||||
for pair in "MODEL_FILE LLM_SMALL_MODEL_FILE" "MMPROJ_FILE LLM_SMALL_MMPROJ"; do
|
||||
read -r env_v dep_v <<<"$pair"
|
||||
path="$(sed -nE "s/^${env_v}=(.+)\$/\1/p" "$ENV_FILE" | tail -1)"
|
||||
[ -n "$path" ]
|
||||
grep -qF -- "${dep_v}=\"\${${dep_v}:-${path}}\"" "$REMOTE_DEPLOY"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
|
||||
grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
# The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
|
||||
# nextn layers); it loads as --model-draft and shares the target's last
|
||||
# layers' KV. ngram-mod is draftless, so the chain is legal.
|
||||
grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
|
||||
@test "the shipped env serves the fleet shape: three 180224-token contexts" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --alias)" = "frank-small" ]
|
||||
[ "$(argv_value --port)" = "7771" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
|
||||
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
||||
# llama.cpp divides --ctx-size across slots and a mamba hybrid cannot
|
||||
# share one cache, so the observable contract is the per-slot window.
|
||||
[ "$(argv_value --parallel)" -eq 3 ]
|
||||
ctx="$(argv_value --ctx-size)"
|
||||
par="$(argv_value --parallel)"
|
||||
[ $((ctx % par)) -eq 0 ]
|
||||
[ $((ctx / par)) -eq 180224 ]
|
||||
[ $((ctx / par)) -le 262144 ]
|
||||
}
|
||||
|
||||
@test "frank-small carries the fleet's image path, and only it" {
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE"
|
||||
@test "the KV contract is asymmetric: unquantized K, q8_0 V, flash-attn on" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --cache-type-k)" = "f16" ]
|
||||
[ "$(argv_value --cache-type-v)" = "q8_0" ]
|
||||
# q8_0 on V only exists on the flash-attention path; without this flag
|
||||
# pair the KV contract in the env file is a lie the engine never sees.
|
||||
[ "$(argv_value --flash-attn)" = "on" ]
|
||||
}
|
||||
|
||||
@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
|
||||
local parallel cache_k cache_v ctx fixed per_slot
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
@test "the KV budget leaves 2-3 GiB of the 3090 free" {
|
||||
# Rate: K f16 (16 KiB/token) + V q8_0 (~8.5, budgeted 25 with slack).
|
||||
# The non-KV half is MEASURED on this card on 2026-10-02: with the
|
||||
# projector loaded the engine reported 22,972 MiB used against 14,113
|
||||
# MiB of KV, so weights + vision tower + DeltaNet state + compute
|
||||
# buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, both about
|
||||
# the output the operator sees: the pool must still fit with 2 GiB
|
||||
# spare, and it must be large enough that 5 GiB spare is NOT reachable
|
||||
# (an undersized pool is as wrong as an oversized one). Arithmetic, not
|
||||
# a boot log: change CTX_SIZE and this bites before the deploy does —
|
||||
# it rejects the 3 x 196,608 pool that measured only 1.1 GiB free. The
|
||||
# same pool with f16 V (34 KiB/token) lands past the card, which is why
|
||||
# the contract is asymmetric.
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
|
||||
# Every slot must be the model's FULL trained window (262144), not a
|
||||
# divided one -- CTX_SIZE is the total pool and --parallel divides it.
|
||||
[ $((ctx / parallel)) -eq 262144 ]
|
||||
[ $((ctx % parallel)) -eq 0 ]
|
||||
[ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
|
||||
# Fit budget from the measured boot arithmetic on the 3090 (24576
|
||||
# MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
|
||||
# so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
|
||||
# (weights, projector, sidecar draft, compute buffers, CUDA context)
|
||||
# is 11996-3450 = 8546 MiB. If the config asks for more slots than
|
||||
# the card holds, this inequality fails before deploy instead of the
|
||||
# KV allocation failing at boot.
|
||||
fixed=8546
|
||||
per_slot=3450
|
||||
[ $((fixed + parallel * per_slot)) -le 24576 ]
|
||||
# Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
|
||||
# without which prefill collapses about 12x (see the CUDA lesson in
|
||||
# AGENTS.md). The build-cuda tree must carry it.
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
kv_gib=$((ctx * 25 / 1048576))
|
||||
[ $((kv_gib + 9 + 2)) -le 24 ]
|
||||
[ $((kv_gib + 9 + 5)) -gt 24 ]
|
||||
f16v_gib=$((ctx * 34 / 1048576))
|
||||
[ $((f16v_gib + 9)) -gt 24 ]
|
||||
}
|
||||
|
||||
@test "frank-small runs on its own port beside frank-large's" {
|
||||
grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-small$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
# The two endpoints cannot share a port: frank-large holds 7770, this one
|
||||
# 7771.
|
||||
grep -q '^PORT=7770$' "$ENV_LARGE"
|
||||
! grep -q '^PORT=7771$' "$ENV_LARGE"
|
||||
@test "no speculative draft ships by default" {
|
||||
# The qwen35 GGUF has no MTP tensors, and the cross-model DFlash draft
|
||||
# is a measured decision the harness makes later, not a default.
|
||||
! grep -q -- '--model-draft' "$SERVE"
|
||||
! grep -q -- '--spec-type' "$SERVE"
|
||||
! grep -q '^DRAFT' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing model" {
|
||||
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
|
||||
# The Q5_K_M GGUF carries no `v.*` tensors, so the projector argument IS
|
||||
# the vision capability: no flag, no eyes.
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
|
||||
|
||||
# The documented opt-out must reach the engine as ABSENCE of the flag,
|
||||
# not as an argument pointing at an empty string.
|
||||
sed -i 's|^MMPROJ_FILE=.*|MMPROJ_FILE=|' "$STAGE/test.env"
|
||||
run env FRANK_SMALL_ENV_FILE="$STAGE/test.env" nvidia-modprobe=true bash "$SERVE"
|
||||
[ "$status" -eq 0 ]
|
||||
! grep -qxF -- '--mmproj' <<<"$output"
|
||||
}
|
||||
|
||||
@test "the fleet reasoning contract survives the reslice to three contexts" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# deepseek format = thinking lands in reasoning_content, the field the
|
||||
# harness reads on this endpoint; --jinja is what makes
|
||||
# chat_template_kwargs reach the template at all.
|
||||
[ "$(argv_value --reasoning-format)" = "deepseek" ]
|
||||
argv_value --jinja >/dev/null
|
||||
grep -qxF -- '--jinja' <<<"$output"
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing runtime, GGUF or projector" {
|
||||
rm -f "$MODEL_DIR/$MM_STUB"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-small: missing"* ]]
|
||||
|
||||
rm -f "$RUNTIME_DIR/bin/llama-server"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"frank-small: missing"* ]]
|
||||
}
|
||||
|
||||
@test "the deploy handler gates on binary, GGUF and projector, then MainPID-hardens health" {
|
||||
grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_SMALL_RUNTIME' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_SMALL_MODEL_FILE' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_SMALL_MMPROJ' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
|
||||
grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-small.service has no MainPID' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "frank-small's serve script refuses to start on a half-finished download" {
|
||||
grep -q 'frank-small: missing' "$SERVE"
|
||||
@test "the two endpoints keep separate ports and the large handler never pkills llama-server" {
|
||||
# frank-large's gufo sweep must not take down frank-small, whose
|
||||
# process is llama-server on the SAME host since this swap.
|
||||
grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
}
|
||||
Reference in new issue
Block a user