frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)

Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
  into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
  assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
  path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
  manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
  full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
  tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
  env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
  top docs index updated; old runbook moved to retired
This commit is contained in:
Frank committed 2026-09-18 21:06:46 -07:00
1 parent cf06c4eab8
commit a2b39b179a
15 files changed
+821 -377

No files matched your search

+5 -1
View File
@@ -72,7 +72,11 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "frank-large loads no vision tower: the fleet's image path is frank-small" {
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET"
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
# frank-small's vLLM checkpoint (2026-09-18) is natively multimodal —
# image content is ingested by the model itself, so the seam that used
# to pin --mmproj now pins the projector's absence.
! grep -q 'mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
grep -q '^MODEL_DIR=/models/main$' "$REPO_ROOT/scripts/frank-small.env"
}
@test "the deploy handler retires the previous ROOT unit without deleting it" {