frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)

Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
  into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
  assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
  path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
  manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
  full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
  tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
  env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
  top docs index updated; old runbook moved to retired
This commit is contained in:
Frank committed 2026-09-18 21:06:46 -07:00
1 parent cf06c4eab8
commit a2b39b179a
15 files changed
+821 -377

No files matched your search

+10 -4
View File
@@ -166,11 +166,17 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
2026-09-17: mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
2026-09-19: vLLM serving
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
full-context sessions, native text+image+audio) on
Dankclaw's RTX 3090, port 7771
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
(MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
Dankclaw's RTX 3090
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector