frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
This commit is contained in:
1 parent
cf06c4eab8
commit
a2b39b179a
15 files changed
+821
-377
No files matched your search
@@ -64,7 +64,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
|
||||
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-small" ]
|
||||
|
||||
@@ -72,7 +73,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# that name was retired with the FreeToken runtime on 2026-09-15 and must
|
||||
# not come back through a planner mapping.
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
|
||||
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user