frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)

Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
  into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
  assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
  path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
  manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
  full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
  tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
  env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
  top docs index updated; old runbook moved to retired
This commit is contained in:
Frank
2026-09-18 21:06:46 -07:00
parent cf06c4eab8
commit a2b39b179a
15 changed files with 821 additions and 377 deletions
+13 -5
View File
@@ -319,6 +319,12 @@ def services_for_path(path: str, target: str) -> set[str]:
# its unit and fleet settings live in one .container file.
_add(services, "frank-large")
return services
if len(parts) >= 2 and parts[1] == "frank-small":
# The service is a rootless Quadlet container since 2026-09-18
# (vLLM); its unit lives in one .container file. The old root
# unit's trio is in scripts/ and maps there.
_add(services, "frank-small")
return services
if len(parts) >= 2 and parts[1] == "observability":
# The five observability containers became Quadlet units on
# 2026-09-11. A unit change selects exactly its own service;
@@ -402,11 +408,13 @@ def services_for_path(path: str, target: str) -> set[str]:
# change to it redeploying alertmanager is the only path that
# reaches it on the host.
_add(services, "alertmanager")
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-small only. The ik_llama.cpp
# trio that lived here for two days is in scripts/.AGENTTRASH/,
# which is archive material and plans nothing.
if name in {"frank-small.env", "serve-frank-small.sh",
"download-frank-small-vllm.sh"}:
# vLLM frank-small (2026-09-18) keeps its env, entrypoint and
# fetch script here; each one selects frank-small only. The
# unit is the quadlet under dankclaw/frank-small/, and the old
# root unit file is in scripts/.AGENTTRASH/, which is archive
# material and plans nothing.
_add(services, "frank-small")
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve