frank-small: revert vLLM cutover — back to llama.cpp Gemma4-12B Q4_K_M

The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
This commit is contained in:
Frank committed 2026-09-19 01:42:37 -07:00
1 parent bab8f7f839
commit 374041cd7a
15 files changed
+495 -637

No files matched your search

+8 -10
View File
@@ -166,17 +166,15 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
2026-09-19: vLLM serving
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
full-context sessions, native text+image+audio) on
Dankclaw's RTX 3090, port 7771
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
mainline llama.cpp serving
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
2026-09-17: mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
(MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
Dankclaw's RTX 3090
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
Dankclaw's RTX 3090, port 7771
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
token pool, 2 concurrent full-context sessions, native text+image+audio
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector