dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: swap Swift-1.5 RCO-IQ3_XXS -> Gemma4-26B-A4B QAT Q4_K_M

Browse Source 
Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B)
+ the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/.

Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the
24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch
1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk,
not loaded: its draft KV is built against the target's full window,
measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise).

Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off =
21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319.

Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5
full-attention layers) so KV scales with 5 layers only, and
attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the
measured q8_0 pool = 2.05 GiB = the K-only figure).

Serve script gained VERBOSITY (+ --log-verbosity in the argv), the
SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4.
deploy-remote.sh payload gates follow the env. The frank-secondary bats
seam is rewritten for the new contract (argv-based) and fixes a
pre-existing vacuity: serve_argv regenerates the test env, so
mutations made before it were silently undone - tests that mutate now
exec through run_stage. frank-secondary is the fleet compaction hub
(200,704 is the only window that fits a worst-case compaction input).
This commit is contained in:
Frank
2026-10-07 13:48:54 -07:00
parent 5b7b3ccb90
commit 68a525740d
8 changed files with 335 additions and 222 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/index.md
+10 -2
View File
@@ -372,8 +372,16 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the ACTIVE
frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
- [[frank-secondary-llama-gemma4-q4km-200704]] — the **ACTIVE** frank-secondary
runbook since 2026-10-07: llama.cpp master (CUDA sm_86) serving
HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
200,704-token conversation, q8_0 K + q8_0 V, MTP **disabled** (the
`gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
with it, 21,890/2,235 without), vision through the publisher's mmproj
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
(superseded by gemma4; payload on disk as the rollback): the
frank-secondary runbook 2026-10-05 (evening) → 2026-10-07
(**UNCECERSED RCO-IQ3_XXS
with MTP back on since 2026-10-06 evening; renamed from frank-small and
reshaped to TWO 114,688-token conversations at Frank's 112k cap with
the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA