frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
This commit is contained in:
+13
-5
@@ -319,6 +319,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# its unit and fleet settings live in one .container file.
|
||||
_add(services, "frank-large")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-small":
|
||||
# The service is a rootless Quadlet container since 2026-09-18
|
||||
# (vLLM); its unit lives in one .container file. The old root
|
||||
# unit's trio is in scripts/ and maps there.
|
||||
_add(services, "frank-small")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# The five observability containers became Quadlet units on
|
||||
# 2026-09-11. A unit change selects exactly its own service;
|
||||
@@ -402,11 +408,13 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-small only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
if name in {"frank-small.env", "serve-frank-small.sh",
|
||||
"download-frank-small-vllm.sh"}:
|
||||
# vLLM frank-small (2026-09-18) keeps its env, entrypoint and
|
||||
# fetch script here; each one selects frank-small only. The
|
||||
# unit is the quadlet under dankclaw/frank-small/, and the old
|
||||
# root unit file is in scripts/.AGENTTRASH/, which is archive
|
||||
# material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
|
||||
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
|
||||
|
||||
Reference in New Issue
Block a user