rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
This commit is contained in:
Frank committed
1 parent
c71a3066c0
commit
87721ab0fe
87 files changed
+460
-460
No files matched your search
@@ -0,0 +1,120 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
|
||||
description: "The active frank-secondary runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
- gemma
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-09-17
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
|
||||
|
||||
> **ACTIVE since 2026-09-17.** Replaces [[frank-secondary-qwen38-gsq-rco]]
|
||||
> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
|
||||
> [[frank-secondary-vllm-gemma4]] (the same-day vLLM experiment, also the
|
||||
> rollback target). Same port (7771), same alias (`frank-secondary`), same
|
||||
> client contract surface.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
|
||||
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
|
||||
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
|
||||
collapses prefill ~12x on CUDA.
|
||||
- **Service shape:** root systemd unit `scripts/frank-secondary.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-secondary.sh` with tunables from
|
||||
`scripts/frank-secondary.env`.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
|
||||
Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
|
||||
x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
|
||||
token embeddings (PLE) inside the file, and a sliding-window attention
|
||||
pattern (`attention.sliding_window = 1024`) — most layers cache only a
|
||||
1024-position window, which is why the KV pool is a fraction of the
|
||||
full-ctx x all-layers number.
|
||||
- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
|
||||
`PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
|
||||
maximises the TOTAL pool, which would shrink each slot below the full window
|
||||
once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
|
||||
full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
|
||||
the card. 4 is the measured max.
|
||||
- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
|
||||
only tokenizes and samples (its CPU belongs to the halogen engine serving
|
||||
frank-large).
|
||||
|
||||
## Spec method
|
||||
|
||||
The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
|
||||
`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
|
||||
with `--spec-type draft-mtp`. The same `--spec-type` value chains the
|
||||
draftless **ngram-mod** matcher, which attaches unconditionally; two
|
||||
context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
|
||||
the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
|
||||
depth 4 + poll 100 decoded WORSE at long context — the longer verification
|
||||
round costs more attention re-read than the extra accepted token earns).
|
||||
|
||||
## KV
|
||||
|
||||
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
|
||||
(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
|
||||
MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
|
||||
3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
|
||||
the >12.8 GiB floor does not apply.
|
||||
|
||||
## Vision + audio
|
||||
|
||||
`--mmproj` loads the BF16 projector
|
||||
(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
|
||||
52M params), which also carries an audio encoder. The fleet's image path
|
||||
lives here. Audio works: `input_audio` content parts transcribe verbatim
|
||||
(verified 2026-09-17 with a 16k mono WAV).
|
||||
|
||||
## Payload
|
||||
|
||||
The main GGUF, the BF16 mmproj, and the MTP sidecar live under
|
||||
`/home/dank/models/Gemma4-12B-QAT`. The serve script refuses to start if any
|
||||
is missing (a missing file is an unfinished download, and the unit's restart
|
||||
policy would otherwise turn it into a crash loop that reads like a runtime
|
||||
fault).
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
|
||||
deepseek`, unlimited budget).
|
||||
- **Tools:** supported (`--jinja` is pinned even though this build defaults
|
||||
it on, so a future build that flips the default cannot silently take tools
|
||||
away).
|
||||
- **Vision + audio:** supported.
|
||||
- **Sampling:** greedy by default (fleet contract); a client that sends a
|
||||
temperature gets its own, and the harness applies per-model overrides from
|
||||
`models.yml`.
|
||||
|
||||
## Measured (max length)
|
||||
|
||||
At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
|
||||
pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
|
||||
(2048) OOM the MTP draft context's pp compute buffer and must not be tried.
|
||||
|
||||
## Rollback
|
||||
|
||||
Payload and runtime for [[frank-secondary-qwen38-gsq-rco]] (ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-secondary-vllm-gemma4]]
|
||||
is the same-day vLLM experiment (also a rollback target).
|
||||
|
||||
# Citations
|
||||
|
||||
- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
|
||||
- llama.cpp: https://github.com/ggml-org/llama.cpp
|
||||
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
|
||||
Reference in new issue
Block a user