dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704

Browse Source 
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention
+ 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less,
natively multimodal) costs ~7 GiB of weights where the 26B MoE cost
~16.6 GiB. The freed room buys back the second conversation slot and
exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K +
f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable
is impossible), vision via the publisher's BF16 mmproj, ubatch 1024
(the CUDA decode-optimal value), verbosity 4, publisher sampler recipe
(0.6/0.95/20/0) in models.yml.

Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB
used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch
2048. Compaction stays routed here; window math is unchanged (maxTokens
16,384 keeps the worst-case harness request inside the slot).

Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the
rollback; its runbook is retired in place, keeping the attention_k_eq_v
pair-billing law.

Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frankandomp committed 2026-10-07 14:47:32 -07:00
1 parent b51e078e36
commit 1d062f06a3
10 files changed
+360 -213
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_secondary_unit.bats
+44 -39
View File
@@ -1,13 +1,13 @@
# Seam: the frank-secondary llama.cpp root unit on dankclaw.
# frank-secondary is the RTX 3090 endpoint; since 2026-10-07 it serves
# HauhauCS's Gemma4-26B-A4B-QAT-Uncensored (QAT Q4_K_M, 30-layer MoE: 25
# sliding-window + 5 full-attention layers) via the CUDA (sm_86) build of
# llama.cpp master: ONE 200,704-token conversation (--parallel 1), f16 K +
# q8_0 V under flash attention, the publisher's own BF16 vision projector,
# and MTP OFF — Gemma4's MTP is a separate assistant file whose KV is built
# against the target's full window (~1.9 GiB measured), which the f16-K raise
# cannot afford on a 24 GiB card. Log verbosity 4 ships so the VRAM-fit
# forensics (KV layout, buffer sizes, graph capture) land in the journal.
# HauhauCS's Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K, qwen35 hybrid:
# 32 layers, 8 full-attention + 24 Gated DeltaNet, no MTP tensors in the file)
# via the CUDA (sm_86) build of llama.cpp master: TWO 200,704-token
# conversations (--ctx-size 401408 --parallel 2), exact f16 K + f16 V under
# flash attention, the publisher's own BF16 vision projector, and MTP off —
# the file carries no nextn/mtp tensors, so there is no in-file head to draft
# with. Log verbosity 4 ships so the VRAM-fit forensics (KV layout, buffer
# sizes, graph capture) land in the journal.
# This seam runs the serve script against a stub engine and asserts on the
# argv it actually execs, so what is pinned is the command the operator gets.
@@ -34,7 +34,7 @@ STUB
# (the env) instead of silently forking the test's filename constants.
MODEL_STUB="$(basename "$(sed -nE 's/^MODEL_FILE=(.+)$/\1/p' "$ENV_FILE")")"
MM_STUB="$(basename "$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")")"
DRAFT_STUB="mtp-gemma-4-26B-A4B-it.gguf"
DRAFT_STUB="draft-assistant.gguf"
touch "$MODEL_DIR/$MODEL_STUB" "$MODEL_DIR/$MM_STUB" "$MODEL_DIR/$DRAFT_STUB"
}
@@ -95,10 +95,10 @@ run_stage() {
grep -q 'FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env' "$UNIT"
}
@test "the shipped env points at the llama.cpp runtime and the Gemma4-26B-A4B QAT Q4_K_M GGUF" {
@test "the shipped env points at the llama.cpp runtime and the Qwen3.5-9B Q6_K GGUF" {
# The payload path IS the endpoint's identity.
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
grep -qE '^MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M\.gguf$' "$ENV_FILE"
grep -qE '^MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3\.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K\.gguf$' "$ENV_FILE"
}
@test "the deployer gates on exactly the payload paths the env file ships" {
@@ -115,7 +115,7 @@ run_stage() {
done
}
@test "one conversation owns the whole 200704 window" {
@test "two conversations share the pool at a 200704 window each" {
serve_argv
[ "$status" -eq 0 ]
[ "$(argv_value --alias)" = "frank-secondary" ]
@@ -124,26 +124,30 @@ run_stage() {
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
# llama.cpp divides --ctx-size across slots, so the observable contract
# is the per-slot window. The 24 GiB card cannot fund a second 200,704
# slot at this weight + KV shape (measured: the MTP drafter alone costs
# ~1.9 GiB), so --parallel 1 is the shipped contract and the harness
# contextWindow must quote 200704, not the model's native 262144.
[ "$(argv_value --parallel)" -eq 1 ]
[ "$(argv_value --ctx-size)" -eq 200704 ]
# is the per-slot window. Frank's 200k cap is a per-conversation cap, so
# the pair is 401,408 / 2 — and the harness contextWindow must quote
# 200704, not the pool and not the model's native 262,144.
[ "$(argv_value --parallel)" -eq 2 ]
[ "$(argv_value --ctx-size)" -eq 401408 ]
local pool slots
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
[ $((pool % slots)) -eq 0 ]
[ $((pool / slots)) -eq 200704 ]
}
@test "the KV contract is f16 K + q8_0 V, flash-attn on" {
@test "the KV contract is exact f16 K + f16 V, flash-attn on" {
serve_argv
[ "$status" -eq 0 ]
# Frank, 2026-10-07: K raised to f16 (exact), V stays q8_0 — llama.cpp's
# FP8 class (1.0625 B/elem; no true fp8 KV dtype exists). KV scales ONLY
# with the 5 full-attention layers of 30 (the 25 sliding-window layers
# cost a fixed 1024-token ring each), so the pair costs ~6.2 GiB at
# 200,704 — the boot ledger, not this file, is the referee. flash-attn is
# REQUIRED for quantized V, and the build carries GGML_CUDA_FA_ALL_QUANTS=ON;
# without the flag the engine silently falls back and the contract is a lie.
# Frank, 2026-10-07: the 9B rung funds unquantized KV (the 26B MoE era
# shipped q8_0 V because the card was full). qwen35 bills the pool on the
# 8 full-attention layers only (8 × 4 KV heads × 256 dim = 8,192
# elements/token ⇒ 32 KiB/token at f16/f16, 12.25 GiB at the 401,408
# pool); the 24 DeltaNet layers cost a fixed recurrent state per slot.
# The boot ledger, not this file, is the referee. flash-attn stays pinned:
# any quantized-V fallback (q8_0 = this build's FP8 class) requires it, and
# the build carries GGML_CUDA_FA_ALL_QUANTS=ON.
[ "$(argv_value --cache-type-k)" = "f16" ]
[ "$(argv_value --cache-type-v)" = "q8_0" ]
[ "$(argv_value --cache-type-v)" = "f16" ]
[ "$(argv_value --flash-attn)" = "on" ]
}
@@ -190,12 +194,12 @@ run_stage() {
}
@test "speculation is off: no drafter flag reaches the argv" {
# Frank, 2026-10-07: MTP OFF on Gemma4. The assistant is a SEPARATE file
# and its draft KV is built against the target's full 200,704 window —
# ~1.9 GiB measured (23,806/319 free), a knife-edge once K is f16. The
# flags move as a set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max
# AND --spec-draft-model together, so a no-spec boot cannot carry half a
# spec pair.
# Frank, 2026-10-07: MTP is off because it CANNOT be on — the Q6_K file
# carries no nextn/mtp tensors (verified from its tensor-name list), and
# MTP is a property of the file, not of the flags. The flags move as a
# set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
# --spec-draft-model together, so a no-spec boot cannot carry half a
# spec pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
serve_argv
[ "$status" -eq 0 ]
! grep -qxF -- '--spec-type' <<<"$output"
@@ -205,10 +209,10 @@ run_stage() {
}
@test "naming a drafter ships the full MTP trio" {
# The opt-back-in path: SPEC_TYPE + SPEC_DRAFT_FILE (the separate
# gemma4-assistant gguf) must reach the engine as a complete set — a
# draft-mtp flag without its model file boots a drafter against the main
# weights and drafts nothing on this architecture.
# The opt-back-in path (e.g. a future Qwen3.5 -mtp build): SPEC_TYPE +
# SPEC_DRAFT_FILE must reach the engine as a complete set — a draft-mtp
# flag without its model file boots a drafter against the main weights and
# drafts nothing.
serve_argv
{
echo 'SPEC_TYPE=draft-mtp'
@@ -222,8 +226,9 @@ run_stage() {
}
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
# The Gemma4 GGUF carries no vision tensors, so the projector argument IS
# the vision capability: no flag, no eyes.
# Qwen3.5-9B is natively multimodal, but the Q6_K file carries no vision
# tensors — the publisher's projector argument IS the vision capability:
# no flag, no eyes.
serve_argv
[ "$status" -eq 0 ]
[ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]