frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
This commit is contained in:
1 parent
cf06c4eab8
commit
a2b39b179a
15 files changed
+821
-377
No files matched your search
@@ -166,11 +166,17 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
|
||||
2026-09-17: mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
|
||||
- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
|
||||
2026-09-19: vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
|
||||
full-context sessions, native text+image+audio) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
|
||||
Dankclaw's RTX 3090
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
|
||||
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
|
||||
|
||||
Reference in new issue
Block a user