frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frank
omp
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
|
||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Gemma4-26B-A4B-QAT-Uncensored **Q4_K_M** (25.2 B-param MoE, ~4 B active, 16,780,192,888 B reported by the engine; 30 layers = 25 sliding-window (`n_head_kv` 8, window 1024) + 5 full-attention (indices 5/11/17/23/29, `n_head_kv` 2)) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **ONE 200,704-token slot** (`--parallel 1`; two 200,704 slots do not fit 24 GiB), **f16 K + q8_0 V** under flash attention, batch 4096 / **ubatch 1024**, `--log-verbosity 4`, **MTP OFF** (the assistant is a SEPARATE file and its draft KV is built against the target's full window — 23,806 / 319 MiB free with it loaded). Measured boots: **q8_0 K/ubatch 2048 = 21,890 used / 2,235 free; f16 K/ubatch 1024 = 21,868 / 2,257** (the +2,161 MiB of f16-K KV is paid by the ubatch cut). `attention_k_eq_v` is a compute-graph mapping only — the cache bills the K+V pair **twice** (6.125 B/elem at f16 K + q8_0 V; verified exactly against the engine's own `llama_kv_cache` lines). Cutover 2026-10-07 from Swift-1.5 RCO-IQ3_XXS 2×114,688. All three endpoints route harness **compaction** here. | 7771 |
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -308,12 +308,13 @@ image]`) — `frank-large` returned to vision with the flash-server v2 swap
|
||||
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
||||
are the engine core; `--mm-max-patches` available),
|
||||
and `frank-secondary` through
|
||||
the publisher's own BF16 mmproj passed as `--mmproj` — the Gemma4
|
||||
GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is
|
||||
**200704** (the one-slot window since the 2026-10-07 Gemma4 swap; `--parallel 1`,
|
||||
so the pool IS the conversation), the
|
||||
per-slot window, not the model's native 262144: the harness must compact
|
||||
before llama.cpp answers a longer prompt with a 400. Note also
|
||||
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
|
||||
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
|
||||
Gemma4 era used). `frank-secondary`'s `contextWindow` is
|
||||
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
|
||||
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
|
||||
the harness must compact before llama.cpp answers a longer prompt with a 400.
|
||||
Note also
|
||||
that
|
||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||
@@ -322,7 +323,7 @@ There is **no audio**
|
||||
input anywhere in the fleet: that capability left with the FLM NPU
|
||||
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(the MiMo tower is vision-only).
|
||||
(neither the MiMo nor the Qwen3.5 tower carries audio).
|
||||
|
||||
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -419,67 +420,65 @@ that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
|
||||
(`scripts/frank-secondary.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP` at
|
||||
**Q4_K_M** (16,796,015,520 B on disk; the engine reports
|
||||
16,780,192,888 B, 25,233,142,046 params, vocab 262,144) **plus the
|
||||
publisher's own BF16 mmproj** (1,194,827,776 B), both
|
||||
`/home/dank/models/Gemma4-26B-A4B/`, on port 7771. Cutover 2026-10-07
|
||||
from the Swift-1.5 RCO-IQ3_XXS 2×114,688 shape. The MTP assistant
|
||||
(`mtp-gemma-4-26B-A4B-it.gguf`, 251,937,728 B) is on disk, **not loaded**.
|
||||
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
|
||||
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
|
||||
LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
|
||||
`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
|
||||
2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).
|
||||
|
||||
**Architecture: gemma4 MoE hybrid** — 30 layers, 128 experts (8 active),
|
||||
n_embd 2816, native 262,144 window. Attention is hybrid per the file's
|
||||
`sliding_window_pattern`: **25 sliding-window layers (window 1024,
|
||||
`n_head_kv = 8`) + 5 full-attention layers (indices 5, 11, 17, 23, 29,
|
||||
`n_head_kv = 2`)**, head_dim 256. Two KV caches: the full-context cache
|
||||
scales with the 5 layers, the SWA cache is a fixed 1024-token ring per
|
||||
sliding layer. `attention_k_eq_v: True` is a **compute-graph** mapping
|
||||
(`gemma4.cpp` maps the absent `wv` to `Vcur = Kcur`) — the cache reserves K
|
||||
and V and, with `k_eq_v`, reserves the pair TWICE: 6.125 B/elem at f16 K +
|
||||
q8_0 V, 2.125 B/elem at q8_0/q8_0. The engine's own `llama_kv_cache` buffer
|
||||
sizes reproduce that rate exactly, so bill the pair, never K alone.
|
||||
**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
|
||||
the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
|
||||
layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
|
||||
heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
|
||||
group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
|
||||
248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
|
||||
the stock master build loads it (arch `qwen35` is registered in this build's
|
||||
`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
|
||||
layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
|
||||
bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
|
||||
layers bill a fixed recurrent state per SLOT, not per token, which is exactly
|
||||
why two 200k slots fit where the 26B MoE fit one.
|
||||
|
||||
**The shape (2026-10-07):** `--ctx-size 200704 --parallel 1` = ONE
|
||||
conversation (Frank's 200k cap; the harness `contextWindow` quotes
|
||||
200704), `--flash-attn on`, **f16 K + q8_0 V** (exact K, FP8-class V),
|
||||
batch 4096 / **ubatch 1024** (cut from 2048 to fund the f16-K raise),
|
||||
`--log-verbosity 4` (debug — the KV/buffer forensics land in the journal),
|
||||
**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
|
||||
conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
|
||||
the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
|
||||
batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
|
||||
decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
|
||||
`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).
|
||||
|
||||
**Why MTP is off**: Gemma4's MTP is a SEPARATE assistant file (arch
|
||||
`gemma4-assistant`), not an in-file head, and the draft context is built
|
||||
against the target's FULL window — measured cost **~1.9 GiB** (23,806 used
|
||||
/ 319 free with the assistant loaded, vs 21,890 / 2,235 without). The
|
||||
assistant also warns `n_ctx_seq (200704) > n_ctx_train (131072)` — the
|
||||
draft runs 53 % past its trained window. The q8_0-K boot leaves 2,235 MiB,
|
||||
which the f16-K raise (+2,161 MiB of KV) spends together with the ubatch
|
||||
cut; MTP and exact-K cannot coexist on this card.
|
||||
**Why MTP is off**: it is not a choice — **this Q6_K file carries no
|
||||
`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
|
||||
property of the file, not of the flags. The publisher card advertises "MTP
|
||||
support" for the base architecture; the quantized export dropped the head.
|
||||
|
||||
**Measured boots** (nvidia-smi, the referee): q8_0 K + q8_0 V, ubatch 2048,
|
||||
MTP off: **21,890 used / 2,235 free**; same shape with the MTP assistant:
|
||||
**23,806 / 319**; **f16 K + q8_0 V, ubatch 1024, MTP off (deployed):
|
||||
21,868 / 2,257**. The verbosity-4 ledger for the deployed boot: weights
|
||||
16,002.91 MiB CUDA0 + 577.50 MiB CPU-mapped, KV 3,001.25 (5 full layers) +
|
||||
306.25 (25 SWA rings) MiB, compute buffers 926.39 + 150.63 MiB. Decode is
|
||||
expected ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4).
|
||||
**Measured boots** (nvidia-smi, the referee): the pricing boots are the
|
||||
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry — 2 × 200,704 f16 K +
|
||||
f16 V: **21,276 used / 2,849 free** (ubatch 1024) and **21,846 / 2,279**
|
||||
(ubatch 2048). This file is 415 MiB lighter in weights ⇒ projected ~21,430 /
|
||||
~2,700. The deployed boot ledger: see the runbook. Decode is ubatch-invariant
|
||||
(Swift-era 3090 law). The Q6_K rung costs ~10 % decode vs Q5_K_M (weight
|
||||
bandwidth; MiMo-measured 94.4 vs 104.8 t/s).
|
||||
|
||||
**Vision**: the publisher's own BF16 mmproj via `--mmproj`. Probe traps
|
||||
carry over from the Swift era: ≥256 px probe images or the model answers
|
||||
"video", and only the OpenAI `image_url` content part reaches the model.
|
||||
**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
|
||||
natively multimodal; the Q6_K file carries no vision tensors). Probe traps
|
||||
carry over: ≥256 px probe images or the model answers "video", and only the
|
||||
OpenAI `image_url` content part reaches the model.
|
||||
|
||||
**Compaction hub**: `models.yml` routes `compactionModel` for
|
||||
`frank-primary`, `frank-large` and `frank-secondary` here — 200,704 is the
|
||||
only window in the fleet that fits a worst-case compaction input (~133k
|
||||
tokens). Consequence of `--parallel 1`: compaction requests serialize
|
||||
behind live generation on this slot.
|
||||
tokens). With `--parallel 2` a compaction request no longer serializes behind
|
||||
a live generation (the Gemma4 era's `--parallel 1` law).
|
||||
|
||||
Runbook:
|
||||
`serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md`.
|
||||
Rollback: revert the swap commit — the Swift payload
|
||||
(`/home/dank/models/Swift15-UncIQ3XXS/`), the MiMo Q6_K payload
|
||||
(`/home/dank/models/MiMo-Q6K/`) and the Ornith Q8_0 payload
|
||||
`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
|
||||
Rollback: revert the swap commit — the Gemma4 payload
|
||||
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
|
||||
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
|
||||
(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
|
||||
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
|
||||
|
||||
|
||||
### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
@@ -1001,7 +1000,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
|
||||
| frank-secondary (llama.cpp CUDA, Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md` |
|
||||
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
|
||||
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -1010,7 +1009,7 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` |
|
||||
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
|
||||
|
||||
@@ -1225,13 +1225,14 @@ deploy_llm_large() {
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's engine runs on the gfx1151 iGPU.
|
||||
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||
LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
# The projector IS the vision capability here: the GGUF carries no vision
|
||||
# tensors, and the publisher ships its own BF16 projector in the same repo, so
|
||||
# a deploy that starts without this file silently ships a text-only
|
||||
# endpoint. Checked unconditionally, like frank-large's.
|
||||
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
# The projector IS the vision capability here: the model is natively
|
||||
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
|
||||
# its own BF16 projector in the same repo, so a deploy that starts without
|
||||
# this file silently ships a text-only endpoint. Checked unconditionally, like
|
||||
# frank-large's.
|
||||
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||
|
||||
deploy_llm_secondary() {
|
||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||
|
||||
@@ -1,27 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-secondary payload for llama.cpp on the RTX 3090:
|
||||
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
|
||||
# **with the MTP head in the file** (the -mtp variant), plus the
|
||||
# PUBLISHER's OWN vision projector (mmproj-BF16.gguf, same repo). Public
|
||||
# repo — no auth header, and never a token on the command line (it is
|
||||
# world-readable in /proc/<pid>/cmdline).
|
||||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
|
||||
# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
|
||||
# multimodal, so this file is the endpoint's eyes). Public repo — no auth
|
||||
# header, and never a token on the command line (it is world-readable in
|
||||
# /proc/<pid>/cmdline).
|
||||
#
|
||||
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
|
||||
# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
|
||||
# success, then byte count and sha256. This repo is on Xet storage, so
|
||||
# the Hub tree/blob APIs expose no LFS oid: the sha256 values below were
|
||||
# computed locally after the first verified fetch (byte counts match the
|
||||
# Hub's declared sizes). Idempotent: complete files are skipped. The
|
||||
# projector is stored under a self-documenting dest name; the URL name
|
||||
# and sha stay pinned in the table below.
|
||||
# success, then byte count and sha256. The Hub exposes no LFS oid for this
|
||||
# repo (Xet storage), so the sha256 values below were computed locally after
|
||||
# the first verified fetch — byte counts match the Hub's declared sizes.
|
||||
# A sha of "-" means "first fetch: size-verified, sha printed for pinning".
|
||||
# Idempotent: complete files are skipped.
|
||||
#
|
||||
# 2026-10-06 (Frank, evening): the payload moved to the UNCELERSED
|
||||
# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (1.56 GiB lighter than the
|
||||
# IQ3_S-mtp it replaced; the lighter quant funds the MTP draft ctx plus
|
||||
# the second 131,072 slot). Rollback payload stays on disk:
|
||||
# /home/dank/models/Swift15-IQ3S/ (the IQ3_S-mtp that was live until this
|
||||
# swap); earlier payloads Ornith-Q8/ and MiMo-Q6K/ as well. Their
|
||||
# re-fetch recipes are in git history (predecessors of this script).
|
||||
# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at
|
||||
# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated
|
||||
# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk:
|
||||
# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being
|
||||
# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
|
||||
@@ -35,15 +32,14 @@ for arg in "$@"; do
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/Swift15-UncIQ3XXS}"
|
||||
DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}"
|
||||
|
||||
# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
|
||||
FILES=(
|
||||
"ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|RCO/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|10442827392|e13d3e7400b6c7f980ebd30c0edd6be43eb1865b50be8562a9db042706c7d803"
|
||||
"ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|mmproj-BF16.gguf|mmproj-BF16.gguf|931145888|19acd7fcf4ee504328566eddf0f8e603ef853a205b4830209d2a4908213abdea"
|
||||
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986"
|
||||
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r repo hubname destname size sha <<<"$entry"
|
||||
@@ -51,7 +47,9 @@ for entry in "${FILES[@]}"; do
|
||||
f="$DEST/$destname"
|
||||
have=0
|
||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
if [[ "$sha" == "-" ]]; then
|
||||
have=1
|
||||
elif sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
have=1
|
||||
fi
|
||||
fi
|
||||
@@ -73,6 +71,10 @@ for entry in "${FILES[@]}"; do
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
if [[ "$sha" == "-" ]]; then
|
||||
echo "VERIFIED $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)"
|
||||
continue
|
||||
fi
|
||||
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||
echo "SHA FAIL $destname"
|
||||
status=1
|
||||
|
||||
+42
-32
@@ -1,46 +1,56 @@
|
||||
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
||||
# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP
|
||||
# (Q4_K_M) on the RTX 3090, :7771.
|
||||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
|
||||
# :7771.
|
||||
#
|
||||
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
|
||||
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V,
|
||||
# MTP DISABLED (SPEC_TYPE=none), log verbosity 4.
|
||||
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
|
||||
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
|
||||
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
|
||||
#
|
||||
# Architecture (read from the file's own header at load): gemma4 MoE — 30
|
||||
# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding
|
||||
# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29,
|
||||
# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window.
|
||||
# Architecture — read from the GGUF's own header (HTTP range read, not the
|
||||
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
|
||||
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
|
||||
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
|
||||
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
|
||||
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
|
||||
# F32×177 — standard types only, so the stock master build loads it (no
|
||||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||||
#
|
||||
# KV rate law (measured, exact): the cache bills the K+V pair TWICE under
|
||||
# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at
|
||||
# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x
|
||||
# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the
|
||||
# FP8 class.
|
||||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||||
# where the 26B MoE fit only one.
|
||||
#
|
||||
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
|
||||
# (spec flags move as a set). The assistant file
|
||||
# (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB) stays on disk as the drafter
|
||||
# payload if MTP is ever re-enabled. Measured VRAM: MTP ON booted
|
||||
# 23,806 used / 319 free; MTP OFF boots 21,890 used / 2,235 free.
|
||||
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
|
||||
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
|
||||
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
|
||||
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
|
||||
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
|
||||
# The verbosity-4 boot ledger is the referee.
|
||||
#
|
||||
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
|
||||
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
|
||||
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||
MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
|
||||
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
|
||||
ALIAS=frank-secondary
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
CTX_SIZE=200704
|
||||
PARALLEL=1
|
||||
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||||
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||||
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||||
# answered with a 400, never truncated.
|
||||
CTX_SIZE=401408
|
||||
PARALLEL=2
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=1024
|
||||
UBATCH_SIZE=2048
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=q8_0
|
||||
CACHE_TYPE_V=f16
|
||||
SPEC_TYPE=none
|
||||
NGL=99
|
||||
|
||||
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph
|
||||
# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of
|
||||
# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) +
|
||||
# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63
|
||||
# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The
|
||||
# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235.
|
||||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||||
# read. Levels above 4 flood the journal.
|
||||
VERBOSITY=4
|
||||
|
||||
@@ -1,27 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS/Gemma4-26B-A4B-QAT-Uncensored (QAT **Q4_K_M**) as
|
||||
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
|
||||
# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
#
|
||||
# 2026-10-07 (Frank): Gemma4-26B-A4B swap with MTP OFF and ONE 200,704-token
|
||||
# conversation (--parallel 1) — the 24 GiB card cannot fund a second slot at
|
||||
# this weight, and the MTP assistant's KV (built against the target's full
|
||||
# window, ~1.9 GiB measured) plus the f16-K raise do not coexist on it.
|
||||
# Verbosity 4 ships so the VRAM-fit forensics land in the journal.
|
||||
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
|
||||
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
|
||||
# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
|
||||
# this GGUF carries no MTP tensors at all (verified from its tensor-name
|
||||
# list), so there is no in-file head to draft with. Verbosity 4 ships so the
|
||||
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
||||
#
|
||||
# Architecture: gemma4_26b_a4b — 30-layer MoE (active ~4B), hybrid attention:
|
||||
# 25 sliding-window layers (fixed 1024-token ring each) + 5 full-attention
|
||||
# layers (5, 11, 17, 23, 29), 8 KV heads x 256 dim, n_ctx_train 262,144. KV
|
||||
# therefore scales ONLY with the 5 full layers: ~30.7 KiB/token at f16 K +
|
||||
# q8_0 V (~6.2 GiB at 200,704). attention_k_eq_v is declared by the file and
|
||||
# honoured by the build (bill K-side accounting; the measured boot is the
|
||||
# referee: 21,890 used / 2,235 free at q8_0/q8_0, MTP off).
|
||||
# Vision comes from the projector file (the GGUF carries no vision
|
||||
# tensors; the publisher ships its own BF16 mmproj in the same repo), so
|
||||
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the
|
||||
# documented text-only opt-out.
|
||||
# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
|
||||
# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
|
||||
# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
|
||||
# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
|
||||
# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
|
||||
# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
|
||||
# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
|
||||
# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
|
||||
# Vision comes from the publisher's own BF16 projector (the model is natively
|
||||
# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
|
||||
# the documented text-only opt-out.
|
||||
#
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends its
|
||||
# recipe from models.yml); greedy is the harness default.
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends the
|
||||
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
|
||||
#
|
||||
# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
|
||||
set -euo pipefail
|
||||
@@ -37,32 +37,35 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||
: "${ALIAS:=frank-secondary}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots; with --parallel 1 the
|
||||
# pool IS the conversation window. 200,704 is inside the model's trained
|
||||
# 262,144; the harness contextWindow MUST quote this number — a longer prompt
|
||||
# is answered with a 400, never truncated.
|
||||
: "${CTX_SIZE:=200704}"
|
||||
: "${PARALLEL:=1}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||||
# per conversation, inside the model's trained 262,144. The harness
|
||||
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
|
||||
# with a 400, never truncated.
|
||||
: "${CTX_SIZE:=401408}"
|
||||
: "${PARALLEL:=2}"
|
||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||||
# 2026-10-07 (Frank): 1024 — cut from 2048 to fund the f16-K raise. The
|
||||
# logical batch is VRAM- and throughput-neutral at a fixed ubatch.
|
||||
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
|
||||
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
|
||||
# throughput-neutral at a fixed ubatch.
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits;
|
||||
# there is no true fp8 KV dtype). Explicit either way so the contract is
|
||||
# visible in the argv.
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
# Speculative decoding is OFF by default since the Gemma4 swap (Frank,
|
||||
# 2026-10-07): Gemma4's MTP is a SEPARATE assistant file (arch
|
||||
# gemma4-assistant), not an in-file head, and its KV is built against the
|
||||
# target's full window — ~1.9 GiB measured, which the f16-K raise cannot
|
||||
# afford. SPEC_TYPE=none strips BOTH speculative flags (flags move as a
|
||||
# set); naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE.
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
|
||||
# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
|
||||
# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
|
||||
# pool. Explicit either way so the contract is visible in the argv, and
|
||||
# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
|
||||
# FP8 class; no true fp8 KV dtype exists in this build).
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
|
||||
# there is no in-file head (the law: MTP is a property of the file, not of the
|
||||
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
|
||||
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||||
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||||
: "${SPEC_TYPE:=none}"
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${SPEC_DRAFT_FILE:=}"
|
||||
@@ -73,7 +76,7 @@ fi
|
||||
: "${VERBOSITY:=3}"
|
||||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||||
: "${MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||
|
||||
# Fail loudly rather than half-loading: a missing file is a download that
|
||||
# has not finished, and Restart=always would turn it into a crash loop that
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
|
||||
|
||||
**ACTIVE since 2026-10-07.** RTX 3090, port 7771, root systemd unit
|
||||
`frank-secondary.service`, served by `scripts/serve-frank-secondary.sh` +
|
||||
`scripts/frank-secondary.env`. Supersedes
|
||||
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] (Swift-1.5-Qwen3.8-27B
|
||||
RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP).
|
||||
**RETIRED 2026-10-07** — same day it went live. Superseded by
|
||||
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
|
||||
2 × 200,704 with exact KV); the payload stays at
|
||||
`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the
|
||||
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
|
||||
the drafter-cost boots, both of which outlive the model.
|
||||
|
||||
## Payload
|
||||
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
timestamp: 2026-10-07
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
Live since 2026-10-07, replacing the
|
||||
[[frank-secondary-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
|
||||
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
|
||||
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
|
||||
conversation slot and unquantized KV back.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
|
||||
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine) |
|
||||
|
||||
## Payload geometry (read from the GGUF header, not the model card)
|
||||
|
||||
Range-read of the file's own metadata + tensor-info section:
|
||||
|
||||
|Key|Value|
|
||||
|---|---|
|
||||
|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)|
|
||||
|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**|
|
||||
|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256|
|
||||
|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096|
|
||||
|context / vocab|262,144 native / 248,320|
|
||||
|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)|
|
||||
|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export|
|
||||
|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)|
|
||||
|
||||
## Shape
|
||||
|
||||
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
|
||||
so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness
|
||||
`contextWindow` quotes the per-slot number). `--flash-attn on`,
|
||||
**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 /
|
||||
**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto
|
||||
--reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF**
|
||||
(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors).
|
||||
|
||||
**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV
|
||||
heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills
|
||||
32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24
|
||||
DeltaNet layers bill a fixed recurrent state **per slot**, not per token —
|
||||
that is what makes two 200k slots affordable where the 26B MoE fit only one.
|
||||
|
||||
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
|
||||
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
|
||||
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
|
||||
2,279** at ubatch 2048 — see [[frank-secondary-llama-mimo-q6k-2x200704]].
|
||||
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
|
||||
so the projection is ~21,430 used / ~2,700 free.
|
||||
|
||||
Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
|
||||
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
|
||||
fallback rung if a future shape lands under the 2 GiB floor.
|
||||
|
||||
## Measured boot (2026-10-07)
|
||||
|
||||
TO BE FILLED BY THE DEPLOY VERIFICATION.
|
||||
|
||||
## Client contract
|
||||
|
||||
llama.cpp surface: model id `frank-secondary`, native tools via `--jinja`,
|
||||
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
|
||||
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
|
||||
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
|
||||
`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is
|
||||
0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve
|
||||
thinking" is satisfied by the 200,704 slot.
|
||||
|
||||
**Compaction hub**: `models.yml` routes `compactionModel` for all three
|
||||
endpoints here. With `--parallel 2` a compaction request no longer serializes
|
||||
behind a live generation (the Gemma4 era's `--parallel 1` law), and the
|
||||
200,704 slot still fits the worst-case compaction input (~133k tokens).
|
||||
|
||||
## Rollback
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-secondary-llama-gemma4-q4km-200704]]|
|
||||
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|
||||
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
|
||||
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
|
||||
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
|
||||
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
|
||||
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
|
||||
flags — the engine prints the exact figures the shape needs.
|
||||
@@ -102,13 +102,20 @@ frank-secondary, port 7771]
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — **ACTIVE since 2026-10-07**
|
||||
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP **Q4_K_M**
|
||||
on the RTX 3090 — ONE 200,704-token conversation, q8_0 K + q8_0 V,
|
||||
vision via the publisher's mmproj. MTP is OFF (the gemma4-assistant
|
||||
drafter costs ~1.9 GiB: 23,806 used/319 free with it, 21,890/2,235
|
||||
without). Supersedes the Swift-1.5 era (below).
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
|
||||
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
|
||||
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
|
||||
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
|
||||
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||||
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
|
||||
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
|
||||
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
|
||||
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
|
||||
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
|
||||
it, 21,890/2,235 without).
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
|
||||
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
|
||||
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
|
||||
|
||||
@@ -372,10 +372,16 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — the **ACTIVE** frank-secondary
|
||||
runbook since 2026-10-07: llama.cpp master (CUDA sm_86) serving
|
||||
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — the **ACTIVE** frank-secondary
|
||||
runbook since 2026-10-07 (late): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
|
||||
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
|
||||
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
|
||||
BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day;
|
||||
payload on disk as rollback): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
|
||||
200,704-token conversation, q8_0 K + q8_0 V, MTP **disabled** (the
|
||||
200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the
|
||||
`gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
|
||||
with it, 21,890/2,235 without), vision through the publisher's mmproj
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# Seam: the frank-secondary llama.cpp root unit on dankclaw.
|
||||
# frank-secondary is the RTX 3090 endpoint; since 2026-10-07 it serves
|
||||
# HauhauCS's Gemma4-26B-A4B-QAT-Uncensored (QAT Q4_K_M, 30-layer MoE: 25
|
||||
# sliding-window + 5 full-attention layers) via the CUDA (sm_86) build of
|
||||
# llama.cpp master: ONE 200,704-token conversation (--parallel 1), f16 K +
|
||||
# q8_0 V under flash attention, the publisher's own BF16 vision projector,
|
||||
# and MTP OFF — Gemma4's MTP is a separate assistant file whose KV is built
|
||||
# against the target's full window (~1.9 GiB measured), which the f16-K raise
|
||||
# cannot afford on a 24 GiB card. Log verbosity 4 ships so the VRAM-fit
|
||||
# forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
||||
# HauhauCS's Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K, qwen35 hybrid:
|
||||
# 32 layers, 8 full-attention + 24 Gated DeltaNet, no MTP tensors in the file)
|
||||
# via the CUDA (sm_86) build of llama.cpp master: TWO 200,704-token
|
||||
# conversations (--ctx-size 401408 --parallel 2), exact f16 K + f16 V under
|
||||
# flash attention, the publisher's own BF16 vision projector, and MTP off —
|
||||
# the file carries no nextn/mtp tensors, so there is no in-file head to draft
|
||||
# with. Log verbosity 4 ships so the VRAM-fit forensics (KV layout, buffer
|
||||
# sizes, graph capture) land in the journal.
|
||||
# This seam runs the serve script against a stub engine and asserts on the
|
||||
# argv it actually execs, so what is pinned is the command the operator gets.
|
||||
|
||||
@@ -34,7 +34,7 @@ STUB
|
||||
# (the env) instead of silently forking the test's filename constants.
|
||||
MODEL_STUB="$(basename "$(sed -nE 's/^MODEL_FILE=(.+)$/\1/p' "$ENV_FILE")")"
|
||||
MM_STUB="$(basename "$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")")"
|
||||
DRAFT_STUB="mtp-gemma-4-26B-A4B-it.gguf"
|
||||
DRAFT_STUB="draft-assistant.gguf"
|
||||
touch "$MODEL_DIR/$MODEL_STUB" "$MODEL_DIR/$MM_STUB" "$MODEL_DIR/$DRAFT_STUB"
|
||||
}
|
||||
|
||||
@@ -95,10 +95,10 @@ run_stage() {
|
||||
grep -q 'FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the shipped env points at the llama.cpp runtime and the Gemma4-26B-A4B QAT Q4_K_M GGUF" {
|
||||
@test "the shipped env points at the llama.cpp runtime and the Qwen3.5-9B Q6_K GGUF" {
|
||||
# The payload path IS the endpoint's identity.
|
||||
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
|
||||
grep -qE '^MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M\.gguf$' "$ENV_FILE"
|
||||
grep -qE '^MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3\.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K\.gguf$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the deployer gates on exactly the payload paths the env file ships" {
|
||||
@@ -115,7 +115,7 @@ run_stage() {
|
||||
done
|
||||
}
|
||||
|
||||
@test "one conversation owns the whole 200704 window" {
|
||||
@test "two conversations share the pool at a 200704 window each" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --alias)" = "frank-secondary" ]
|
||||
@@ -124,26 +124,30 @@ run_stage() {
|
||||
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
|
||||
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
||||
# llama.cpp divides --ctx-size across slots, so the observable contract
|
||||
# is the per-slot window. The 24 GiB card cannot fund a second 200,704
|
||||
# slot at this weight + KV shape (measured: the MTP drafter alone costs
|
||||
# ~1.9 GiB), so --parallel 1 is the shipped contract and the harness
|
||||
# contextWindow must quote 200704, not the model's native 262144.
|
||||
[ "$(argv_value --parallel)" -eq 1 ]
|
||||
[ "$(argv_value --ctx-size)" -eq 200704 ]
|
||||
# is the per-slot window. Frank's 200k cap is a per-conversation cap, so
|
||||
# the pair is 401,408 / 2 — and the harness contextWindow must quote
|
||||
# 200704, not the pool and not the model's native 262,144.
|
||||
[ "$(argv_value --parallel)" -eq 2 ]
|
||||
[ "$(argv_value --ctx-size)" -eq 401408 ]
|
||||
local pool slots
|
||||
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
|
||||
[ $((pool % slots)) -eq 0 ]
|
||||
[ $((pool / slots)) -eq 200704 ]
|
||||
}
|
||||
|
||||
@test "the KV contract is f16 K + q8_0 V, flash-attn on" {
|
||||
@test "the KV contract is exact f16 K + f16 V, flash-attn on" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# Frank, 2026-10-07: K raised to f16 (exact), V stays q8_0 — llama.cpp's
|
||||
# FP8 class (1.0625 B/elem; no true fp8 KV dtype exists). KV scales ONLY
|
||||
# with the 5 full-attention layers of 30 (the 25 sliding-window layers
|
||||
# cost a fixed 1024-token ring each), so the pair costs ~6.2 GiB at
|
||||
# 200,704 — the boot ledger, not this file, is the referee. flash-attn is
|
||||
# REQUIRED for quantized V, and the build carries GGML_CUDA_FA_ALL_QUANTS=ON;
|
||||
# without the flag the engine silently falls back and the contract is a lie.
|
||||
# Frank, 2026-10-07: the 9B rung funds unquantized KV (the 26B MoE era
|
||||
# shipped q8_0 V because the card was full). qwen35 bills the pool on the
|
||||
# 8 full-attention layers only (8 × 4 KV heads × 256 dim = 8,192
|
||||
# elements/token ⇒ 32 KiB/token at f16/f16, 12.25 GiB at the 401,408
|
||||
# pool); the 24 DeltaNet layers cost a fixed recurrent state per slot.
|
||||
# The boot ledger, not this file, is the referee. flash-attn stays pinned:
|
||||
# any quantized-V fallback (q8_0 = this build's FP8 class) requires it, and
|
||||
# the build carries GGML_CUDA_FA_ALL_QUANTS=ON.
|
||||
[ "$(argv_value --cache-type-k)" = "f16" ]
|
||||
[ "$(argv_value --cache-type-v)" = "q8_0" ]
|
||||
[ "$(argv_value --cache-type-v)" = "f16" ]
|
||||
[ "$(argv_value --flash-attn)" = "on" ]
|
||||
}
|
||||
|
||||
@@ -190,12 +194,12 @@ run_stage() {
|
||||
}
|
||||
|
||||
@test "speculation is off: no drafter flag reaches the argv" {
|
||||
# Frank, 2026-10-07: MTP OFF on Gemma4. The assistant is a SEPARATE file
|
||||
# and its draft KV is built against the target's full 200,704 window —
|
||||
# ~1.9 GiB measured (23,806/319 free), a knife-edge once K is f16. The
|
||||
# flags move as a set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max
|
||||
# AND --spec-draft-model together, so a no-spec boot cannot carry half a
|
||||
# spec pair.
|
||||
# Frank, 2026-10-07: MTP is off because it CANNOT be on — the Q6_K file
|
||||
# carries no nextn/mtp tensors (verified from its tensor-name list), and
|
||||
# MTP is a property of the file, not of the flags. The flags move as a
|
||||
# set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
|
||||
# --spec-draft-model together, so a no-spec boot cannot carry half a
|
||||
# spec pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
! grep -qxF -- '--spec-type' <<<"$output"
|
||||
@@ -205,10 +209,10 @@ run_stage() {
|
||||
}
|
||||
|
||||
@test "naming a drafter ships the full MTP trio" {
|
||||
# The opt-back-in path: SPEC_TYPE + SPEC_DRAFT_FILE (the separate
|
||||
# gemma4-assistant gguf) must reach the engine as a complete set — a
|
||||
# draft-mtp flag without its model file boots a drafter against the main
|
||||
# weights and drafts nothing on this architecture.
|
||||
# The opt-back-in path (e.g. a future Qwen3.5 -mtp build): SPEC_TYPE +
|
||||
# SPEC_DRAFT_FILE must reach the engine as a complete set — a draft-mtp
|
||||
# flag without its model file boots a drafter against the main weights and
|
||||
# drafts nothing.
|
||||
serve_argv
|
||||
{
|
||||
echo 'SPEC_TYPE=draft-mtp'
|
||||
@@ -222,8 +226,9 @@ run_stage() {
|
||||
}
|
||||
|
||||
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
|
||||
# The Gemma4 GGUF carries no vision tensors, so the projector argument IS
|
||||
# the vision capability: no flag, no eyes.
|
||||
# Qwen3.5-9B is natively multimodal, but the Q6_K file carries no vision
|
||||
# tensors — the publisher's projector argument IS the vision capability:
|
||||
# no flag, no eyes.
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
|
||||
|
||||
Reference in New Issue
Block a user