Frank
a2b39b179a
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
23 lines
1.3 KiB
Docker
23 lines
1.3 KiB
Docker
# frank-small runtime image: the digest-pinned vLLM gemma-x86_64-cu129 base
|
|
# (vLLM 0.22.1rc1.dev357+g74b5964f0 — the image that loads
|
|
# gemma-4-12B-it-qat-w4a16-ct, runs the Gemma4MTPModel drafter and carries
|
|
# the gemma4 reasoning/tool-call parsers) plus the vllm[audio] extra.
|
|
#
|
|
# The base image ships WITHOUT the audio extras, so an input_audio content
|
|
# part 500s with "Please install vllm[audio] for audio support" (measured
|
|
# 2026-09-19). The extra is just four PyPI packages (av, scipy, soundfile,
|
|
# mistral_common[audio]); installing the extras directly — not
|
|
# `pip install vllm[audio]` — keeps the pinned dev build in place: a bare
|
|
# `vllm[audio]` would make pip reach for the PyPI release and replace the
|
|
# 0.22.1rc1.dev build that knows gemma4.
|
|
#
|
|
# Built by scripts/deploy-remote.sh on dankclaw (rootless podman, network on
|
|
# for pip): the base is pulled by digest and re-tagged to the FROM ref below
|
|
# first, then `podman build` produces the :frank-small tag the quadlet runs.
|
|
FROM localhost/vllm/vllm-openai:gemma4-unified
|
|
|
|
# The vllm[audio] extra as raw packages (header explains why not
|
|
# `pip install vllm[audio]`): audio input needs av/scipy/soundfile plus
|
|
# the mistral_common audio protocol pieces.
|
|
RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]"
|