0 Fork 0

Files

T

Frank 87721ab0fe rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)

Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:

- Service trio: scripts/{frank-small.env,frank-small.service,
  serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
  frank-secondary-*.md (9 files); the FreeToken runbook stays
  freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
  LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
  3090, and the Dankserver service that became frank-medium) are
  preserved verbatim — they describe what the service was called
  at the time.

2026-10-06 15:37:13 -07:00

5.4 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

ACTIVE since 2026-09-17. Replaces frank-secondary-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and frank-secondary-vllm-gemma4 (the same-day vLLM experiment, also the rollback target). Same port (7771), same alias (frank-secondary), same client contract surface.

Identity

Endpoint: frank-secondary at 192.168.0.143:7771 (Dankclaw).
Runtime: mainline llama.cpp master 37b53fd, CUDA build for sm_86 with -DGGML_CUDA_FA_ALL_QUANTS=ON (/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server). The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention collapses prefill ~12x on CUDA.
Service shape: root systemd unit scripts/frank-secondary.service (User=dank), launched by scripts/serve-frank-secondary.sh with tunables from scripts/frank-secondary.env.

Shape

Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (6.87 GiB). Architecture gemma4: 48 layers, MHA with 16 kv heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer token embeddings (PLE) inside the file, and a sliding-window attention pattern (attention.sliding_window = 1024) — most layers cache only a 1024-position window, which is why the KV pool is a fraction of the full-ctx x all-layers number.
Context: four FULL 262,144-token slots (CTX_SIZE=1048576, PARALLEL=4). The explicit CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, which would shrink each slot below the full window once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past the card. 4 is the measured max.
Placement: the whole model in VRAM (--gpu-layers 999); the host CPU only tokenizes and samples (its CPU belongs to the halogen engine serving frank-large).

Spec method

The MTP head ships as a separate file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 GiB), loaded via --model-draft with --spec-type draft-mtp. The same --spec-type value chains the draftless ngram-mod matcher, which attaches unconditionally; two context-backed drafters cannot share one process, so draft-mtp,ngram-mod is the only legal shape that carries both. Draft depth 3 + poll 50 (measured: depth 4 + poll 100 decoded WORSE at long context — the longer verification round costs more attention re-read than the extra accepted token earns).

KV

q8_0 (--cache-type-k q8_0 --cache-type-v q8_0), with flash attention (--flash-attn on, required for quantized KV). CPU-side cache capped at 2048 MiB (CACHE_RAM=2048) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the 3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so the >12.8 GiB floor does not apply.

Vision + audio

--mmproj loads the BF16 projector (mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf, 11 tensors, 52M params), which also carries an audio encoder. The fleet's image path lives here. Audio works: input_audio content parts transcribe verbatim (verified 2026-09-17 with a 16k mono WAV).

Payload

The main GGUF, the BF16 mmproj, and the MTP sidecar live under /home/dank/models/Gemma4-12B-QAT. The serve script refuses to start if any is missing (a missing file is an unfinished download, and the unit's restart policy would otherwise turn it into a crash loop that reads like a runtime fault).

Client contract

Alias: frank-secondary (the id at /v1/models).
Reasoning: extracted to reasoning_content (--reasoning-format deepseek, unlimited budget).
Tools: supported (--jinja is pinned even though this build defaults it on, so a future build that flips the default cannot silently take tools away).
Vision + audio: supported.
Sampling: greedy by default (fleet contract); a client that sends a temperature gets its own, and the harness applies per-model overrides from models.yml.

Measured (max length)

At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024: pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches (2048) OOM the MTP draft context's pp compute buffer and must not be tried.

Rollback

Payload and runtime for frank-secondary-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. frank-secondary-vllm-gemma4 is the same-day vLLM experiment (also a rollback target).

Citations

Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
llama.cpp: https://github.com/ggml-org/llama.cpp
Serve script: scripts/serve-frank-secondary.sh; tunables: scripts/frank-secondary.env

5.4 KiB Raw Blame History Unescape Escape