frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)

Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
  into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
  assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
  path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
  manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
  full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
  tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
  env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
  top docs index updated; old runbook moved to retired
This commit is contained in:
Frank committed 2026-09-18 21:06:46 -07:00
1 parent cf06c4eab8
commit a2b39b179a
15 files changed
+821 -377

No files matched your search

+4 -2
View File
@@ -64,7 +64,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
run python3 "$PLANNER" --target dankclaw --paths \
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
[ "$status" -eq 0 ]
[ "$output" = "frank-small" ]
@@ -72,7 +73,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
# that name was retired with the FreeToken runtime on 2026-09-15 and must
# not come back through a planner mapping.
run python3 "$PLANNER" --target dankserver --paths \
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
[ "$status" -eq 0 ]
[ -z "$output" ]
}