From 1d062f06a3e67725ec92e86812a32cdc8fa5b303 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 7 Oct 2026 14:47:32 -0700 Subject: [PATCH] frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704 Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp --- AGENTS.md | 113 +++++++++--------- scripts/deploy-remote.sh | 13 +- scripts/download-frank-secondary.sh | 48 ++++---- scripts/frank-secondary.env | 74 +++++++----- scripts/serve-frank-secondary.sh | 87 +++++++------- ...rank-secondary-llama-gemma4-q4km-200704.md | 11 +- ...-secondary-llama-qwen35-9b-q6k-2x200704.md | 113 ++++++++++++++++++ serverAppConfigs-docs/LLM/index.md | 19 ++- serverAppConfigs-docs/index.md | 12 +- tests/frank_secondary_unit.bats | 83 +++++++------ 10 files changed, 360 insertions(+), 213 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md diff --git a/AGENTS.md b/AGENTS.md index 21bc822..e61dcbc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 | -| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Gemma4-26B-A4B-QAT-Uncensored **Q4_K_M** (25.2 B-param MoE, ~4 B active, 16,780,192,888 B reported by the engine; 30 layers = 25 sliding-window (`n_head_kv` 8, window 1024) + 5 full-attention (indices 5/11/17/23/29, `n_head_kv` 2)) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **ONE 200,704-token slot** (`--parallel 1`; two 200,704 slots do not fit 24 GiB), **f16 K + q8_0 V** under flash attention, batch 4096 / **ubatch 1024**, `--log-verbosity 4`, **MTP OFF** (the assistant is a SEPARATE file and its draft KV is built against the target's full window — 23,806 / 319 MiB free with it loaded). Measured boots: **q8_0 K/ubatch 2048 = 21,890 used / 2,235 free; f16 K/ubatch 1024 = 21,868 / 2,257** (the +2,161 MiB of f16-K KV is paid by the ubatch cut). `attention_k_eq_v` is a compute-graph mapping only — the cache bills the K+V pair **twice** (6.125 B/elem at f16 K + q8_0 V; verified exactly against the engine's own `llama_kv_cache` lines). Cutover 2026-10-07 from Swift-1.5 RCO-IQ3_XXS 2×114,688. All three endpoints route harness **compaction** here. | 7771 | +| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -308,12 +308,13 @@ image]`) — `frank-large` returned to vision with the flash-server v2 swap 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; `--mm-max-patches` available), and `frank-secondary` through -the publisher's own BF16 mmproj passed as `--mmproj` — the Gemma4 -GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is -**200704** (the one-slot window since the 2026-10-07 Gemma4 swap; `--parallel 1`, -so the pool IS the conversation), the -per-slot window, not the model's native 262144: the harness must compact -before llama.cpp answers a longer prompt with a 400. Note also +the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively +multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the +Gemma4 era used). `frank-secondary`'s `contextWindow` is +**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the +2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144: +the harness must compact before llama.cpp answers a longer prompt with a 400. +Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — @@ -322,7 +323,7 @@ There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 -(the MiMo tower is vision-only). +(neither the MiMo nor the Qwen3.5 tower carries audio). ### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE) @@ -419,67 +420,65 @@ that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched. `GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (`scripts/frank-secondary.service`, `relink_unit`ed into `/etc/systemd/system/`), serving -`HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP` at -**Q4_K_M** (16,796,015,520 B on disk; the engine reports -16,780,192,888 B, 25,233,142,046 params, vocab 262,144) **plus the -publisher's own BF16 mmproj** (1,194,827,776 B), both -`/home/dank/models/Gemma4-26B-A4B/`, on port 7771. Cutover 2026-10-07 -from the Swift-1.5 RCO-IQ3_XXS 2×114,688 shape. The MTP assistant -(`mtp-gemma-4-26B-A4B-it.gguf`, 251,937,728 B) is on disk, **not loaded**. +`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K** +(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no +LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both +`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover +2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day). -**Architecture: gemma4 MoE hybrid** — 30 layers, 128 experts (8 active), -n_embd 2816, native 262,144 window. Attention is hybrid per the file's -`sliding_window_pattern`: **25 sliding-window layers (window 1024, -`n_head_kv = 8`) + 5 full-attention layers (indices 5, 11, 17, 23, 29, -`n_head_kv = 2`)**, head_dim 256. Two KV caches: the full-context cache -scales with the 5 layers, the SWA cache is a fixed 1024-token ring per -sliding layer. `attention_k_eq_v: True` is a **compute-graph** mapping -(`gemma4.cpp` maps the absent `wv` to `Vcur = Kcur`) — the cache reserves K -and V and, with `k_eq_v`, reserves the pair TWICE: 6.125 B/elem at f16 K + -q8_0 V, 2.125 B/elem at q8_0/q8_0. The engine's own `llama_kv_cache` buffer -sizes reproduce that rate exactly, so bill the pair, never K alone. +**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not +the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention +layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention +heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 / +group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab +248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so +the stock master build loads it (arch `qwen35` is registered in this build's +`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full +layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V +bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet +layers bill a fixed recurrent state per SLOT, not per token, which is exactly +why two 200k slots fit where the 26B MoE fit one. -**The shape (2026-10-07):** `--ctx-size 200704 --parallel 1` = ONE -conversation (Frank's 200k cap; the harness `contextWindow` quotes -200704), `--flash-attn on`, **f16 K + q8_0 V** (exact K, FP8-class V), -batch 4096 / **ubatch 1024** (cut from 2048 to fund the f16-K raise), -`--log-verbosity 4` (debug — the KV/buffer forensics land in the journal), +**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO +conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes +the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV), +batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill, +decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`, `--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`). -**Why MTP is off**: Gemma4's MTP is a SEPARATE assistant file (arch -`gemma4-assistant`), not an in-file head, and the draft context is built -against the target's FULL window — measured cost **~1.9 GiB** (23,806 used -/ 319 free with the assistant loaded, vs 21,890 / 2,235 without). The -assistant also warns `n_ctx_seq (200704) > n_ctx_train (131072)` — the -draft runs 53 % past its trained window. The q8_0-K boot leaves 2,235 MiB, -which the f16-K raise (+2,161 MiB of KV) spends together with the ubatch -cut; MTP and exact-K cannot coexist on this card. +**Why MTP is off**: it is not a choice — **this Q6_K file carries no +`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a +property of the file, not of the flags. The publisher card advertises "MTP +support" for the base architecture; the quantized export dropped the head. -**Measured boots** (nvidia-smi, the referee): q8_0 K + q8_0 V, ubatch 2048, -MTP off: **21,890 used / 2,235 free**; same shape with the MTP assistant: -**23,806 / 319**; **f16 K + q8_0 V, ubatch 1024, MTP off (deployed): -21,868 / 2,257**. The verbosity-4 ledger for the deployed boot: weights -16,002.91 MiB CUDA0 + 577.50 MiB CPU-mapped, KV 3,001.25 (5 full layers) + -306.25 (25 SWA rings) MiB, compute buffers 926.39 + 150.63 MiB. Decode is -expected ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4). +**Measured boots** (nvidia-smi, the referee): the pricing boots are the +MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry — 2 × 200,704 f16 K + +f16 V: **21,276 used / 2,849 free** (ubatch 1024) and **21,846 / 2,279** +(ubatch 2048). This file is 415 MiB lighter in weights ⇒ projected ~21,430 / +~2,700. The deployed boot ledger: see the runbook. Decode is ubatch-invariant +(Swift-era 3090 law). The Q6_K rung costs ~10 % decode vs Q5_K_M (weight +bandwidth; MiMo-measured 94.4 vs 104.8 t/s). -**Vision**: the publisher's own BF16 mmproj via `--mmproj`. Probe traps -carry over from the Swift era: ≥256 px probe images or the model answers -"video", and only the OpenAI `image_url` content part reaches the model. +**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is +natively multimodal; the Q6_K file carries no vision tensors). Probe traps +carry over: ≥256 px probe images or the model answers "video", and only the +OpenAI `image_url` content part reaches the model. **Compaction hub**: `models.yml` routes `compactionModel` for `frank-primary`, `frank-large` and `frank-secondary` here — 200,704 is the only window in the fleet that fits a worst-case compaction input (~133k -tokens). Consequence of `--parallel 1`: compaction requests serialize -behind live generation on this slot. +tokens). With `--parallel 2` a compaction request no longer serializes behind +a live generation (the Gemma4 era's `--parallel 1` law). Runbook: -`serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md`. -Rollback: revert the swap commit — the Swift payload -(`/home/dank/models/Swift15-UncIQ3XXS/`), the MiMo Q6_K payload -(`/home/dank/models/MiMo-Q6K/`) and the Ornith Q8_0 payload +`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`. +Rollback: revert the swap commit — the Gemma4 payload +(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload +(`/home/dank/models/MiMo-Q6K/`), the Swift payload +(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload (`/home/dank/models/Ornith-Q8/`) all remain on disk. + ### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE) The **radiance engine** (StillDeadcode, Apache-2.0; image @@ -1001,7 +1000,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` | -| frank-secondary (llama.cpp CUDA, Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md` | +| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` | | frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -1010,7 +1009,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| -| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` | +| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` | | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 7571b37..a11fa1d 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1225,13 +1225,14 @@ deploy_llm_large() { # # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is # not shared: frank-large's engine runs on the gfx1151 iGPU. -LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}" +LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}" LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}" -# The projector IS the vision capability here: the GGUF carries no vision -# tensors, and the publisher ships its own BF16 projector in the same repo, so -# a deploy that starts without this file silently ships a text-only -# endpoint. Checked unconditionally, like frank-large's. -LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}" +# The projector IS the vision capability here: the model is natively +# multimodal and the Q6_K file carries no vision tensors — the publisher ships +# its own BF16 projector in the same repo, so a deploy that starts without +# this file silently ships a text-only endpoint. Checked unconditionally, like +# frank-large's. +LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}" deploy_llm_secondary() { # Refuse rather than restart-loop: a unit whose runtime or payload is diff --git a/scripts/download-frank-secondary.sh b/scripts/download-frank-secondary.sh index 183b7cf..0d326e1 100755 --- a/scripts/download-frank-secondary.sh +++ b/scripts/download-frank-secondary.sh @@ -1,27 +1,24 @@ #!/usr/bin/env bash # Download the frank-secondary payload for llama.cpp on the RTX 3090: -# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS -# **with the MTP head in the file** (the -mtp variant), plus the -# PUBLISHER's OWN vision projector (mmproj-BF16.gguf, same repo). Public -# repo — no auth header, and never a token on the command line (it is -# world-readable in /proc//cmdline). +# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the +# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively +# multimodal, so this file is the endpoint's eyes). Public repo — no auth +# header, and never a token on the command line (it is world-readable in +# /proc//cmdline). # # Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized # file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a -# success, then byte count and sha256. This repo is on Xet storage, so -# the Hub tree/blob APIs expose no LFS oid: the sha256 values below were -# computed locally after the first verified fetch (byte counts match the -# Hub's declared sizes). Idempotent: complete files are skipped. The -# projector is stored under a self-documenting dest name; the URL name -# and sha stay pinned in the table below. +# success, then byte count and sha256. The Hub exposes no LFS oid for this +# repo (Xet storage), so the sha256 values below were computed locally after +# the first verified fetch — byte counts match the Hub's declared sizes. +# A sha of "-" means "first fetch: size-verified, sha printed for pinning". +# Idempotent: complete files are skipped. # -# 2026-10-06 (Frank, evening): the payload moved to the UNCELERSED -# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (1.56 GiB lighter than the -# IQ3_S-mtp it replaced; the lighter quant funds the MTP draft ctx plus -# the second 131,072 slot). Rollback payload stays on disk: -# /home/dank/models/Swift15-IQ3S/ (the IQ3_S-mtp that was live until this -# swap); earlier payloads Ornith-Q8/ and MiMo-Q6K/ as well. Their -# re-fetch recipes are in git history (predecessors of this script). +# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at +# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated +# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk: +# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being +# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/. # # Usage: # scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only] @@ -35,15 +32,14 @@ for arg in "$@"; do *) DEST="$arg" ;; esac done -DEST="${DEST:-/home/dank/models/Swift15-UncIQ3XXS}" +DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}" # repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo) FILES=( - "ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|RCO/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|10442827392|e13d3e7400b6c7f980ebd30c0edd6be43eb1865b50be8562a9db042706c7d803" - "ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|mmproj-BF16.gguf|mmproj-BF16.gguf|931145888|19acd7fcf4ee504328566eddf0f8e603ef853a205b4830209d2a4908213abdea" + "HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986" + "HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934" ) -mkdir -p "$DEST" status=0 for entry in "${FILES[@]}"; do IFS='|' read -r repo hubname destname size sha <<<"$entry" @@ -51,7 +47,9 @@ for entry in "${FILES[@]}"; do f="$DEST/$destname" have=0 if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then - if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then + if [[ "$sha" == "-" ]]; then + have=1 + elif sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then have=1 fi fi @@ -73,6 +71,10 @@ for entry in "${FILES[@]}"; do status=1 continue fi + if [[ "$sha" == "-" ]]; then + echo "VERIFIED $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)" + continue + fi if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then echo "SHA FAIL $destname" status=1 diff --git a/scripts/frank-secondary.env b/scripts/frank-secondary.env index 360a57e..bc0da57 100644 --- a/scripts/frank-secondary.env +++ b/scripts/frank-secondary.env @@ -1,46 +1,56 @@ # frank-secondary — llama.cpp (master, CUDA sm_86) serving -# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP -# (Q4_K_M) on the RTX 3090, :7771. +# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090, +# :7771. # -# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to -# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V, -# MTP DISABLED (SPEC_TYPE=none), log verbosity 4. +# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot) +# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV, +# MTP disabled (SPEC_TYPE=none), log verbosity 4. # -# Architecture (read from the file's own header at load): gemma4 MoE — 30 -# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding -# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29, -# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window. +# Architecture — read from the GGUF's own header (HTTP range read, not the +# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4 +# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers, +# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size +# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, +# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 + +# F32×177 — standard types only, so the stock master build loads it (no +# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so +# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload. # -# KV rate law (measured, exact): the cache bills the K+V pair TWICE under -# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at -# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x -# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the -# FP8 class. +# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per +# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the +# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed +# recurrent state per SLOT (not per token) — that is why two 200k slots fit +# where the 26B MoE fit only one. # -# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model -# (spec flags move as a set). The assistant file -# (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB) stays on disk as the drafter -# payload if MTP is ever re-enabled. Measured VRAM: MTP ON booted -# 23,806 used / 319 free; MTP OFF boots 21,890 used / 2,235 free. +# Feasibility comes from the predecessor's MEASURED boots, not a formula: +# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at +# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024 +# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights, +# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free. +# The verbosity-4 boot ledger is the referee. +# +# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor +# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost +# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor. LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server -MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf -MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf +MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf +MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf ALIAS=frank-secondary BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -CTX_SIZE=200704 -PARALLEL=1 +# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704 +# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and +# the harness contextWindow quotes the PER-SLOT number — a longer prompt is +# answered with a 400, never truncated. +CTX_SIZE=401408 +PARALLEL=2 BATCH_SIZE=4096 -UBATCH_SIZE=1024 +UBATCH_SIZE=2048 CACHE_TYPE_K=f16 -CACHE_TYPE_V=q8_0 +CACHE_TYPE_V=f16 SPEC_TYPE=none NGL=99 - -# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph -# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of -# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) + -# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63 -# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The -# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235. +# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes +# and graph capture land in the journal, which is what the VRAM-fit forensics +# read. Levels above 4 flood the journal. VERBOSITY=4 diff --git a/scripts/serve-frank-secondary.sh b/scripts/serve-frank-secondary.sh index 88ccb55..03c939f 100755 --- a/scripts/serve-frank-secondary.sh +++ b/scripts/serve-frank-secondary.sh @@ -1,27 +1,27 @@ -#!/usr/bin/env bash -# Serve HauhauCS/Gemma4-26B-A4B-QAT-Uncensored (QAT **Q4_K_M**) as +# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as # frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86). # -# 2026-10-07 (Frank): Gemma4-26B-A4B swap with MTP OFF and ONE 200,704-token -# conversation (--parallel 1) — the 24 GiB card cannot fund a second slot at -# this weight, and the MTP assistant's KV (built against the target's full -# window, ~1.9 GiB measured) plus the f16-K raise do not coexist on it. -# Verbosity 4 ships so the VRAM-fit forensics land in the journal. +# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations +# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the +# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off: +# this GGUF carries no MTP tensors at all (verified from its tensor-name +# list), so there is no in-file head to draft with. Verbosity 4 ships so the +# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal. # -# Architecture: gemma4_26b_a4b — 30-layer MoE (active ~4B), hybrid attention: -# 25 sliding-window layers (fixed 1024-token ring each) + 5 full-attention -# layers (5, 11, 17, 23, 29), 8 KV heads x 256 dim, n_ctx_train 262,144. KV -# therefore scales ONLY with the 5 full layers: ~30.7 KiB/token at f16 K + -# q8_0 V (~6.2 GiB at 200,704). attention_k_eq_v is declared by the file and -# honoured by the build (bill K-side accounting; the measured boot is the -# referee: 21,890 used / 2,235 free at q8_0/q8_0, MTP off). -# Vision comes from the projector file (the GGUF carries no vision -# tensors; the publisher ships its own BF16 mmproj in the same repo), so -# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the -# documented text-only opt-out. +# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4 +# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers), +# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool +# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at +# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per +# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same +# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 +# at ubatch 2048 — the boot ledger, not arithmetic, is the referee. +# Vision comes from the publisher's own BF16 projector (the model is natively +# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is +# the documented text-only opt-out. # -# Sampling: llama.cpp takes it per request (the fleet harness sends its -# recipe from models.yml); greedy is the harness default. +# Sampling: llama.cpp takes it per request (the fleet harness sends the +# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20). # # Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides. set -euo pipefail @@ -37,32 +37,35 @@ fi # --- defaults (used when the env file does not set them) ------------------- : "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}" -: "${MODEL_FILE:=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}" +: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}" : "${ALIAS:=frank-secondary}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7771}" -# ctx-size is the POOL llama.cpp divides across slots; with --parallel 1 the -# pool IS the conversation window. 200,704 is inside the model's trained -# 262,144; the harness contextWindow MUST quote this number — a longer prompt -# is answered with a 400, never truncated. -: "${CTX_SIZE:=200704}" -: "${PARALLEL:=1}" +# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704 +# per conversation, inside the model's trained 262,144. The harness +# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered +# with a 400, never truncated. +: "${CTX_SIZE:=401408}" +: "${PARALLEL:=2}" # Compute-buffer geometry. Ubatch is a prefill knob (decode never moves). -# 2026-10-07 (Frank): 1024 — cut from 2048 to fund the f16-K raise. The -# logical batch is VRAM- and throughput-neutral at a fixed ubatch. +# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill +# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and +# throughput-neutral at a fixed ubatch. : "${BATCH_SIZE:=4096}" -: "${UBATCH_SIZE:=1024}" -# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits; -# there is no true fp8 KV dtype). Explicit either way so the contract is -# visible in the argv. -: "${CACHE_TYPE_K:=q8_0}" -: "${CACHE_TYPE_V:=q8_0}" -# Speculative decoding is OFF by default since the Gemma4 swap (Frank, -# 2026-10-07): Gemma4's MTP is a SEPARATE assistant file (arch -# gemma4-assistant), not an in-file head, and its KV is built against the -# target's full window — ~1.9 GiB measured, which the f16-K raise cannot -# afford. SPEC_TYPE=none strips BOTH speculative flags (flags move as a -# set); naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE. +: "${UBATCH_SIZE:=2048}" +# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that +# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry +# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408 +# pool. Explicit either way so the contract is visible in the argv, and +# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's +# FP8 class; no true fp8 KV dtype exists in this build). +: "${CACHE_TYPE_K:=f16}" +: "${CACHE_TYPE_V:=f16}" +# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so +# there is no in-file head (the law: MTP is a property of the file, not of the +# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set); +# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4 +# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35. : "${SPEC_TYPE:=none}" : "${SPEC_DRAFT_N_MAX:=3}" : "${SPEC_DRAFT_FILE:=}" @@ -73,7 +76,7 @@ fi : "${VERBOSITY:=3}" # Vision projector. No colon on purpose: an explicitly empty value means # "text-only, on purpose" and must NOT be replaced by the default path. -: "${MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}" +: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}" # Fail loudly rather than half-loading: a missing file is a download that # has not finished, and Restart=always would turn it into a crash loop that diff --git a/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md index 7b53eb1..e1c4cd8 100644 --- a/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md @@ -1,10 +1,11 @@ # frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw) -**ACTIVE since 2026-10-07.** RTX 3090, port 7771, root systemd unit -`frank-secondary.service`, served by `scripts/serve-frank-secondary.sh` + -`scripts/frank-secondary.env`. Supersedes -[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] (Swift-1.5-Qwen3.8-27B -RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP). +**RETIRED 2026-10-07** — same day it went live. Superseded by +[[frank-secondary-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at +2 × 200,704 with exact KV); the payload stays at +`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the +Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and +the drafter-cost boots, both of which outlive the model. ## Payload diff --git a/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md new file mode 100644 index 0000000..a377d93 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md @@ -0,0 +1,113 @@ +--- +type: reference +title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)" +description: "The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape." +tags: + - llm + - frank-secondary + - dankclaw + - llama-cpp + - gguf + - rtx3090 +timestamp: 2026-10-07 +--- + +# frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV + +Live since 2026-10-07, replacing the +[[frank-secondary-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704 +slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of +weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second +conversation slot and unquantized KV back. + +## Identity + +| | | +|---|---| +| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | +| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface | +| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit | +| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | +| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) | +| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` | +| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability | +| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` | +| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine) | + +## Payload geometry (read from the GGUF header, not the model card) + +Range-read of the file's own metadata + tensor-info section: + +|Key|Value| +|---|---| +|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)| +|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**| +|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256| +|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096| +|context / vocab|262,144 native / 248,320| +|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)| +|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export| +|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)| + +## Shape + +`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots, +so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness +`contextWindow` quotes the per-slot number). `--flash-attn on`, +**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 / +**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto +--reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF** +(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors). + +**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV +heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills +32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24 +DeltaNet layers bill a fixed recurrent state **per slot**, not per token — +that is what makes two 200k slots affordable where the 26B MoE fit only one. + +**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical +qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K + +f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 / +2,279** at ubatch 2048 — see [[frank-secondary-llama-mimo-q6k-2x200704]]. +This file is 415 MiB lighter in weights and its projector is 3 MiB heavier, +so the projection is ~21,430 used / ~2,700 free. + +Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 % +prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented +fallback rung if a future shape lands under the 2 GiB floor. + +## Measured boot (2026-10-07) + +TO BE FILLED BY THE DEPLOY VERIFICATION. + +## Client contract + +llama.cpp surface: model id `frank-secondary`, native tools via `--jinja`, +thinking in **`reasoning_content`**, vision via OpenAI `image_url` content +parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video"). +The publisher's recipe ships in `models.yml` `extraBody`: thinking mode +`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is +0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve +thinking" is satisfied by the 200,704 slot. + +**Compaction hub**: `models.yml` routes `compactionModel` for all three +endpoints here. With `--parallel 2` a compaction request no longer serializes +behind a live generation (the Gemma4 era's `--parallel 1` law), and the +200,704 slot still fits the worst-case compaction input (~133k tokens). + +## Rollback + +|Target|How| +|---|---| +|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-secondary-llama-gemma4-q4km-200704]]| +|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048| +|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`| + +## When it misbehaves + +Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒ +`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew +one slot, not the pool; queue behind two long generations ⇒ the slot ceiling +is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in +`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching +flags — the engine prints the exact figures the shape needs. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index f80a24b..88a1ba6 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -102,13 +102,20 @@ frank-secondary, port 7771] (`slow`/`advisor` → frank-primary) carried forward. Superseded by the MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the engine moved with the model -- [[frank-secondary-llama-gemma4-q4km-200704]] — **ACTIVE since 2026-10-07** +- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving - HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP **Q4_K_M** - on the RTX 3090 — ONE 200,704-token conversation, q8_0 K + q8_0 V, - vision via the publisher's mmproj. MTP is OFF (the gemma4-assistant - drafter costs ~1.9 GiB: 23,806 used/319 free with it, 21,890/2,235 - without). Supersedes the Swift-1.5 era (below). + HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 + — TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact + f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because + the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same + qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below). +- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same + day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp + master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored- + HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token + conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was + OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with + it, 21,890/2,235 without). - [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED 2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback); ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 9c8fd3f..8076442 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -372,10 +372,16 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-secondary-llama-gemma4-q4km-200704]] — the **ACTIVE** frank-secondary - runbook since 2026-10-07: llama.cpp master (CUDA sm_86) serving +- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — the **ACTIVE** frank-secondary + runbook since 2026-10-07 (late): llama.cpp master (CUDA sm_86) serving + HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 — + TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible** + (the file carries no `nextn`/`mtp` tensors), vision through the publisher's + BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry) +- [[frank-secondary-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day; + payload on disk as rollback): llama.cpp master (CUDA sm_86) serving HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE - 200,704-token conversation, q8_0 K + q8_0 V, MTP **disabled** (the + 200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the `gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free with it, 21,890/2,235 without), vision through the publisher's mmproj - [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07 diff --git a/tests/frank_secondary_unit.bats b/tests/frank_secondary_unit.bats index 2dc6583..521dc0d 100755 --- a/tests/frank_secondary_unit.bats +++ b/tests/frank_secondary_unit.bats @@ -1,13 +1,13 @@ # Seam: the frank-secondary llama.cpp root unit on dankclaw. # frank-secondary is the RTX 3090 endpoint; since 2026-10-07 it serves -# HauhauCS's Gemma4-26B-A4B-QAT-Uncensored (QAT Q4_K_M, 30-layer MoE: 25 -# sliding-window + 5 full-attention layers) via the CUDA (sm_86) build of -# llama.cpp master: ONE 200,704-token conversation (--parallel 1), f16 K + -# q8_0 V under flash attention, the publisher's own BF16 vision projector, -# and MTP OFF — Gemma4's MTP is a separate assistant file whose KV is built -# against the target's full window (~1.9 GiB measured), which the f16-K raise -# cannot afford on a 24 GiB card. Log verbosity 4 ships so the VRAM-fit -# forensics (KV layout, buffer sizes, graph capture) land in the journal. +# HauhauCS's Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K, qwen35 hybrid: +# 32 layers, 8 full-attention + 24 Gated DeltaNet, no MTP tensors in the file) +# via the CUDA (sm_86) build of llama.cpp master: TWO 200,704-token +# conversations (--ctx-size 401408 --parallel 2), exact f16 K + f16 V under +# flash attention, the publisher's own BF16 vision projector, and MTP off — +# the file carries no nextn/mtp tensors, so there is no in-file head to draft +# with. Log verbosity 4 ships so the VRAM-fit forensics (KV layout, buffer +# sizes, graph capture) land in the journal. # This seam runs the serve script against a stub engine and asserts on the # argv it actually execs, so what is pinned is the command the operator gets. @@ -34,7 +34,7 @@ STUB # (the env) instead of silently forking the test's filename constants. MODEL_STUB="$(basename "$(sed -nE 's/^MODEL_FILE=(.+)$/\1/p' "$ENV_FILE")")" MM_STUB="$(basename "$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")")" - DRAFT_STUB="mtp-gemma-4-26B-A4B-it.gguf" + DRAFT_STUB="draft-assistant.gguf" touch "$MODEL_DIR/$MODEL_STUB" "$MODEL_DIR/$MM_STUB" "$MODEL_DIR/$DRAFT_STUB" } @@ -95,10 +95,10 @@ run_stage() { grep -q 'FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env' "$UNIT" } -@test "the shipped env points at the llama.cpp runtime and the Gemma4-26B-A4B QAT Q4_K_M GGUF" { +@test "the shipped env points at the llama.cpp runtime and the Qwen3.5-9B Q6_K GGUF" { # The payload path IS the endpoint's identity. grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE" - grep -qE '^MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M\.gguf$' "$ENV_FILE" + grep -qE '^MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3\.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K\.gguf$' "$ENV_FILE" } @test "the deployer gates on exactly the payload paths the env file ships" { @@ -115,7 +115,7 @@ run_stage() { done } -@test "one conversation owns the whole 200704 window" { +@test "two conversations share the pool at a 200704 window each" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --alias)" = "frank-secondary" ] @@ -124,26 +124,30 @@ run_stage() { [ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ] [ "$(argv_value --n-gpu-layers)" -gt 0 ] # llama.cpp divides --ctx-size across slots, so the observable contract - # is the per-slot window. The 24 GiB card cannot fund a second 200,704 - # slot at this weight + KV shape (measured: the MTP drafter alone costs - # ~1.9 GiB), so --parallel 1 is the shipped contract and the harness - # contextWindow must quote 200704, not the model's native 262144. - [ "$(argv_value --parallel)" -eq 1 ] - [ "$(argv_value --ctx-size)" -eq 200704 ] + # is the per-slot window. Frank's 200k cap is a per-conversation cap, so + # the pair is 401,408 / 2 — and the harness contextWindow must quote + # 200704, not the pool and not the model's native 262,144. + [ "$(argv_value --parallel)" -eq 2 ] + [ "$(argv_value --ctx-size)" -eq 401408 ] + local pool slots + pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)" + [ $((pool % slots)) -eq 0 ] + [ $((pool / slots)) -eq 200704 ] } -@test "the KV contract is f16 K + q8_0 V, flash-attn on" { +@test "the KV contract is exact f16 K + f16 V, flash-attn on" { serve_argv [ "$status" -eq 0 ] - # Frank, 2026-10-07: K raised to f16 (exact), V stays q8_0 — llama.cpp's - # FP8 class (1.0625 B/elem; no true fp8 KV dtype exists). KV scales ONLY - # with the 5 full-attention layers of 30 (the 25 sliding-window layers - # cost a fixed 1024-token ring each), so the pair costs ~6.2 GiB at - # 200,704 — the boot ledger, not this file, is the referee. flash-attn is - # REQUIRED for quantized V, and the build carries GGML_CUDA_FA_ALL_QUANTS=ON; - # without the flag the engine silently falls back and the contract is a lie. + # Frank, 2026-10-07: the 9B rung funds unquantized KV (the 26B MoE era + # shipped q8_0 V because the card was full). qwen35 bills the pool on the + # 8 full-attention layers only (8 × 4 KV heads × 256 dim = 8,192 + # elements/token ⇒ 32 KiB/token at f16/f16, 12.25 GiB at the 401,408 + # pool); the 24 DeltaNet layers cost a fixed recurrent state per slot. + # The boot ledger, not this file, is the referee. flash-attn stays pinned: + # any quantized-V fallback (q8_0 = this build's FP8 class) requires it, and + # the build carries GGML_CUDA_FA_ALL_QUANTS=ON. [ "$(argv_value --cache-type-k)" = "f16" ] - [ "$(argv_value --cache-type-v)" = "q8_0" ] + [ "$(argv_value --cache-type-v)" = "f16" ] [ "$(argv_value --flash-attn)" = "on" ] } @@ -190,12 +194,12 @@ run_stage() { } @test "speculation is off: no drafter flag reaches the argv" { - # Frank, 2026-10-07: MTP OFF on Gemma4. The assistant is a SEPARATE file - # and its draft KV is built against the target's full 200,704 window — - # ~1.9 GiB measured (23,806/319 free), a knife-edge once K is f16. The - # flags move as a set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max - # AND --spec-draft-model together, so a no-spec boot cannot carry half a - # spec pair. + # Frank, 2026-10-07: MTP is off because it CANNOT be on — the Q6_K file + # carries no nextn/mtp tensors (verified from its tensor-name list), and + # MTP is a property of the file, not of the flags. The flags move as a + # set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND + # --spec-draft-model together, so a no-spec boot cannot carry half a + # spec pair (the ROCmFPX-era baseline arm segfaulted at load doing that). serve_argv [ "$status" -eq 0 ] ! grep -qxF -- '--spec-type' <<<"$output" @@ -205,10 +209,10 @@ run_stage() { } @test "naming a drafter ships the full MTP trio" { - # The opt-back-in path: SPEC_TYPE + SPEC_DRAFT_FILE (the separate - # gemma4-assistant gguf) must reach the engine as a complete set — a - # draft-mtp flag without its model file boots a drafter against the main - # weights and drafts nothing on this architecture. + # The opt-back-in path (e.g. a future Qwen3.5 -mtp build): SPEC_TYPE + + # SPEC_DRAFT_FILE must reach the engine as a complete set — a draft-mtp + # flag without its model file boots a drafter against the main weights and + # drafts nothing. serve_argv { echo 'SPEC_TYPE=draft-mtp' @@ -222,8 +226,9 @@ run_stage() { } @test "vision ships by default and MMPROJ_FILE empty opts out of the image path" { - # The Gemma4 GGUF carries no vision tensors, so the projector argument IS - # the vision capability: no flag, no eyes. + # Qwen3.5-9B is natively multimodal, but the Q6_K file carries no vision + # tensors — the publisher's projector argument IS the vision capability: + # no flag, no eyes. serve_argv [ "$status" -eq 0 ] [ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]