frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)

Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
2026-09-18 21:06:46 -07:00
parent cf06c4eab8
commit a2b39b179a
15 changed files with 821 additions and 377 deletions
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
+  the gfx1151 iGPU) and `frank-small` (vLLM on the RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 |--------|---------|---------|-------|------|
 |Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
 | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
+| Dankclaw | `frank-small` | vLLM `gemma-x86_64-cu129` dev build + audio extra (rootless Quadlet) | google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + Google's bf16 MTP assistant on the RTX 3090, 262144 window / 289,647-token bf16 KV pool / 2 full-context sessions, native text+image+audio — the fleet's multimodal endpoint | 7771 |
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly.
 The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
-their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
+their own files — frank-large and frank-small by their `dankclaw/`
-and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
+quadlet/Dockerfile directories, frank-medium by its unit/env/serve trio in `scripts/` — so unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
 runs on a host without passwordless sudo; the value never enters the repo.
@@ -232,8 +232,10 @@ merely wedged kernel usually still answers ARP.
     `~/.config/containers/systemd/frank-large.container` aside and
     `systemctl --user daemon-reload` — the quadlet generator re-creates that
     unit from the file on every reload, so the source is the lever, not `mask`.
-   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
+   - Dankclaw `frank-small` (rootless Quadlet since 2026-09-19, vLLM on the 3090):
-     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
+     `systemctl --user stop frank-small`, then move its
     `~/.config/containers/systemd/frank-small.container` aside and
     `systemctl --user daemon-reload` (same quadlet-source lever as frank-large).
   - Dankserver `frank-medium` (system unit):
     `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
 2. Bench.
@@ -263,10 +265,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
+of 2026-09-19 `frank-small` is the fleet's multimodal endpoint
-`frank-large` is text-only because its halogen deployment loads no vision
+(`input: [text, image, audio]`) — vLLM's native Gemma4 vision AND audio
-tower. There is **no audio** input anywhere in the fleet: that capability left
+towers, both verified live. `frank-large` is text-only because its halogen
-with the FLM NPU endpoint on 2026-09-11.
+deployment loads no vision tower.
 ### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
@@ -301,31 +303,37 @@ that as "tight and fragmented", not "out of memory".
 Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
 Rollback: [[frank-large-ik-uncensored]].
-### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+### frank-small — vLLM on the RTX 3090 (Dankclaw, ACTIVE)
-Mainline llama.cpp master `37b53fd` (CUDA build with
+vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (digest-pinned, audio
-`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
+extra pip-installed into `localhost/vllm/vllm-openai:frank-small` — see
-`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
+`dankclaw/frank-small/Dockerfile`) as a **rootless Quadlet**
-serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
+(`dankclaw/frank-small/quadlet/frank-small.container`), serving
-The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
+`google/gemma-4-12B-it-qat-w4a16-ct` (compressed-tensors w4a16, int4 Marlin
-nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
+weights, 10.26 GiB) with Google's **bf16** MTP assistant
-matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
+(`google/gemma-4-12B-it-assistant`, 845 MB). Port 7771, 262,144 window,
-port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
+bf16 KV pool of 289,647 tokens, **2 concurrent full-context sessions**
-(four FULL 262,144-token slots — the measured card ceiling), batch
+(the 3rd queues), CUDA graphs on, `ENFORCE_EAGER=0`.
 **8192/1024**, greedy default. Deployed flags are in
 `scripts/frank-small.env`; the serve script refuses to start if the
 weights, the projector, the sidecar or the binary are missing.
-Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
+The w4a16-ct assistant CANNOT draft in vLLM (the drafter path skips the
-chat completion returns content plus `reasoning_content`; a tool call
+Marlin kernel — packed int4 misread, acceptance 1.00); the bf16 assistant
-returns `tool_calls`; an image request is answered from the projector;
+measures acceptance 2.80–3.56. fp8 KV is hardware-dead on sm_86 (Ampere
-an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
+has no fp8 tensor cores), so bf16 is the only option.
-max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
+
-acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
+Dankclaw has **no nvidia-container-toolkit**: the unit wires CUDA by hand
-ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
+(`Device=` for the four nvidia nodes + host `libcuda`/`libnvidia-ml`
-100 is WORSE at long ctx) is tabled in the runbook.
+mounted read-only into two library paths). Host driver is
-Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
+`nvidia-driver-610` 610.57.04 on Ubuntu 26.04; `sm_86` is the correct
-Rollback: [[frank-small-qwen38-gsq-rco]].
+compute capability — no arch override.
 Verified at deploy (2026-09-19): `/v1/models` reports `frank-small`; chat
 returns content plus `reasoning` (vLLM's gemma4 parser field — the harness
 maps it via `compat.reasoningContentField`); a tool call returns
 `tool_calls`; an image is described; a 440 Hz WAV is described (native
 audio encoder). Measured: pp@67k 1,293 t/s, pp@213k 608 t/s,
 tg@213k ~13.6 t/s (bandwidth-bound), tg@85k ~17.6 t/s.
 Runbook: `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md`.
 Rollback: [[frank-small-gemma4-12b]] (llama.cpp, weights still on host).
 ### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
@@ -707,7 +715,7 @@ Active:
 | Service | Runbook |
 |---------|---------|
 | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
-| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
+| frank-small (vLLM Gemma4-12B w4a16, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md` |
 | frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -719,6 +727,7 @@ Retired (rollback targets and history):
 | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
 | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
 | llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 09-19 | `frank-small-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
@@ -0,0 +1,22 @@
 # frank-small runtime image: the digest-pinned vLLM gemma-x86_64-cu129 base
 # (vLLM 0.22.1rc1.dev357+g74b5964f0 — the image that loads
 # gemma-4-12B-it-qat-w4a16-ct, runs the Gemma4MTPModel drafter and carries
 # the gemma4 reasoning/tool-call parsers) plus the vllm[audio] extra.
 #
 # The base image ships WITHOUT the audio extras, so an input_audio content
 # part 500s with "Please install vllm[audio] for audio support" (measured
 # 2026-09-19). The extra is just four PyPI packages (av, scipy, soundfile,
 # mistral_common[audio]); installing the extras directly — not
 # `pip install vllm[audio]` — keeps the pinned dev build in place: a bare
 # `vllm[audio]` would make pip reach for the PyPI release and replace the
 # 0.22.1rc1.dev build that knows gemma4.
 #
 # Built by scripts/deploy-remote.sh on dankclaw (rootless podman, network on
 # for pip): the base is pulled by digest and re-tagged to the FROM ref below
 # first, then `podman build` produces the :frank-small tag the quadlet runs.
 FROM localhost/vllm/vllm-openai:gemma4-unified
 # The vllm[audio] extra as raw packages (header explains why not
 # `pip install vllm[audio]`): audio input needs av/scipy/soundfile plus
 # the mistral_common audio protocol pieces.
 RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]"
@@ -0,0 +1,100 @@
 # Quadlet unit: frank-small (rootless, boot-persistent).
 #
 # Install path on dankclaw: ~/.config/containers/systemd/frank-small.container
 # Unit name: frank-small.service (container name: frank-small)
 #
 # The small inference service. vLLM (since 2026-09-18) serving
 # google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16) with the
 # google bf16 MTP assistant for speculative decoding, on the RTX
 # 3090. It replaced the mainline-llama.cpp ROOT unit (the old
 # scripts/frank-small.service is retired into scripts/.AGENTTRASH/).
 #
 # SHAPE. The quadlet replaces the image's `vllm serve` entrypoint with
 # /scripts/serve-frank-small.sh (mounted read-only out of the repo checkout)
 # so the flag set stays in git. The model and drafter directories are bound
 # read-only: the payloads are fetched and sha256-verified by
 # scripts/download-frank-small-vllm.sh before first start, and the deploy
 # handler refuses to start the unit when either is absent.
 #
 # CUDA WITHOUT the nvidia-container-toolkit. This host has never run a CUDA
 # container (the 3090 served frank-small as a native binary, the iGPU runs
 # ROCm through /dev/kfd), so there is no nvidia hook to lean on. The recipe
 # that works (measured 2026-09-18, torch sees the 3090 at 25.3 GB):
 #   - the four /dev/nvidia* nodes via AddDevice (they are 0666 on this host,
 #     so the rootless user namespace can open them);
 #   - the HOST driver's libcuda.so.1 and libnvidia-ml.so.1 mounted read-only
 #     — the image carries its own CUDA 12.9 runtime, but vLLM's platform
 #     probe needs the driver's NVML stub, and without it `vllm serve` dies at
 #     parser construction with "Failed to infer device type";
 #   - NVIDIA_VISIBLE_DEVICES=all.
 # memlock must be unlimited: CUDA pins its staging buffers.
 #
 # SIZING. The tunables (full 262144 window, 0.95 of a dedicated 24 GiB card,
 # bf16 KV, 16-seq batch ceiling, 16k prefill chunk, 4-token MTP draft) live
 # in scripts/frank-small.env. The boot log's "Maximum concurrency for N
 # tokens per request" line is the naive pool ratio for a hybrid SWA model
 # and undercounts; verify the session count behaviourally on restart.
 #
 # IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
 # path cannot resolve registry-prefixed refs from local storage at create
 # time. The tag is a LOCAL BUILD, not a pull: deploy-remote.sh pulls the
 # digest-pinned docker.io base (the gemma-x86_64-cu129 build, vLLM
 # 0.22.1rc1.dev357+g74b5964f0 — the image that boots this checkpoint) and
 # re-tags it, then builds dankclaw/frank-small/Dockerfile on top, which
 # installs the vllm[audio] extra (av, scipy, soundfile,
 # mistral_common[audio]) so the model's native audio path works. Pull=never
 # keeps boots off the registry.
 [Unit]
 Description=frank-small — vLLM gemma4_unified w4a16 + MTP on RTX 3090
 After=network-online.target
 [Container]
 ContainerName=frank-small
 Image=localhost/vllm/vllm-openai:frank-small
 # Published port stays 7771: Hermes, the harness and the homepage probe all
 # point here. 8000 (vLLM's in-container port) is never reachable directly.
 PublishPort=7771:8000
 # CUDA device nodes (0666 on this host — rootless can open them).
 AddDevice=/dev/nvidia0
 AddDevice=/dev/nvidiactl
 AddDevice=/dev/nvidia-uvm
 AddDevice=/dev/nvidia-uvm-tools
 # Host driver stubs the image's CUDA runtime and vLLM's NVML probe need.
 # The image's LD_LIBRARY_PATH leads with /usr/local/nvidia/lib64 — the
 # nvidia-container-toolkit's conventional mount point, which this host does
 # not have — so the driver libs go there; the /usr/lib mounts cover torch
 # Inductor's own lookup.
 Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro
 Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so:ro
 Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:ro
 Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/local/nvidia/lib64/libcuda.so.1:ro
 Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/local/nvidia/lib64/libcuda.so:ro
 Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/local/nvidia/lib64/libnvidia-ml.so.1:ro
 Environment=NVIDIA_VISIBLE_DEVICES=all
 # Payloads, read-only: fetched and sha256-verified by
 # scripts/download-frank-small-vllm.sh before first start.
 Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro
 Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro
 # Serve script and env, read-only out of the repo checkout.
 Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro
 ContainerEntrypoint=/scripts/serve-frank-small.sh
 Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env
 Ulimit=memlock=-1:-1
 # /health answers 200 once the engine is up; vLLM 503s it while the model is
 # still loading, so a green check means ready-to-serve, not process-alive.
 # curl ships in the image.
 HealthCmd=curl -sf http://127.0.0.1:8000/health
 HealthInterval=30s
 HealthTimeout=10s
 HealthRetries=3
 HealthStartPeriod=600s
 Pull=never
 [Service]
 # First start reads ~10 GiB of weights and compiles the CUDA graphs.
 TimeoutStartSec=900
 Restart=on-failure
 RestartSec=10
 [Install]
 WantedBy=default.target
@@ -319,6 +319,12 @@ def services_for_path(path: str, target: str) -> set[str]:
            # its unit and fleet settings live in one .container file.
            _add(services, "frank-large")
            return services
        if len(parts) >= 2 and parts[1] == "frank-small":
            # The service is a rootless Quadlet container since 2026-09-18
            # (vLLM); its unit lives in one .container file. The old root
            # unit's trio is in scripts/ and maps there.
            _add(services, "frank-small")
            return services
        if len(parts) >= 2 and parts[1] == "observability":
            # The five observability containers became Quadlet units on
            # 2026-09-11. A unit change selects exactly its own service;
@@ -402,11 +408,13 @@ def services_for_path(path: str, target: str) -> set[str]:
            # change to it redeploying alertmanager is the only path that
            # reaches it on the host.
            _add(services, "alertmanager")
-        if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
+        if name in {"frank-small.env", "serve-frank-small.sh",
-            # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
+                    "download-frank-small-vllm.sh"}:
-            # files here; each one selects frank-small only. The ik_llama.cpp
+            # vLLM frank-small (2026-09-18) keeps its env, entrypoint and
-            # trio that lived here for two days is in scripts/.AGENTTRASH/,
+            # fetch script here; each one selects frank-small only. The
-            # which is archive material and plans nothing.
+            # unit is the quadlet under dankclaw/frank-small/, and the old
            # root unit file is in scripts/.AGENTTRASH/, which is archive
            # material and plans nothing.
            _add(services, "frank-small")
        if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
            # llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
@@ -1139,28 +1139,90 @@ wait_for_llm_large_ready() {
    return 0
 }
-# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
+# deploy_llm_small — frank-small is a rootless Quadlet running vLLM on the
-# ROOT unit: the runtime is a native CUDA binary, so there is no container
+# RTX 3090 since 2026-09-18: the runtime is the digest-pinned
-# image to install and no generated unit to avoid. relink_unit points
+# vllm/vllm-openai gemma4-unified image, and the unit is generated from
-# /etc/systemd/system/frank-small.service at the repo file and boot persistence
+# dankclaw/frank-small/quadlet/. It replaces the mainline-llama.cpp
-# is that unit's [Install] section.
+# deployment, which ran as a ROOT unit.
 #
-# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
+# TWO MANAGERS, ONE UNIT NAME. The llama.cpp incarnation ran as a ROOT unit
-# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
+# whose file is a symlink into this checkout; the vLLM unit is a USER
-LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
+# (quadlet) unit. Both must never exist at once — they would race for port
 # 7771 — so the root unit is retired first: stop, disable, move the /etc
 # symlink into the home trash (never `rm`), reload the root manager. This
 # host's `systemctl disable` has been observed to delete the unit file
 # itself (2026-08-26 incident), so the move tolerates a file that is
 # already gone.
 #
 # The payloads are fetched and sha256-verified by
 # scripts/download-frank-small-vllm.sh before first start; the handler
 # refuses to launch a unit that cannot load its model or its drafter.
 LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/gemma4-12b-qat-w4a16-ct/model.safetensors}"
 LLM_SMALL_DRAFT="${LLM_SMALL_DRAFT:-/home/dank/models/gemma4-12b-assistant-bf16/model.safetensors}"
 # Base image: the gemma-x86_64-cu129 build (vLLM 0.22.1rc1.dev357+
 # g74b5964f0) — the image that boots this checkpoint. The June 3
 # gemma4-unified build predates it and fails at weight load (missing
 # vision_embedder.patch_dense); the Sep 15 cu129-nightly fails at import
 # (torchvision ABI). The unit runs a LOCAL BUILD on top of this base —
 # dankclaw/frank-small/Dockerfile installs the vllm[audio] extra, which
 # the base ships without.
 VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:e24d3d8e367ebc2a6ccfc8cf3cecee858a30a6c66b5b1642d85363cd745e001e"
 VLLM_SMALL_BASE_ALIAS="localhost/vllm/vllm-openai:gemma4-unified"
 VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"
 retire_root_llm_small() {
    local trash="$HOME/.AGENTTRASH/20260918-vllm-cutover/host-units"
    run_privileged systemctl stop frank-small.service 2>/dev/null || true
    run_privileged systemctl disable frank-small.service 2>/dev/null || true
    if [[ -e /etc/systemd/system/frank-small.service || -L /etc/systemd/system/frank-small.service ]]; then
        mkdir -p "$trash"
        run_privileged mv -- /etc/systemd/system/frank-small.service \
            "$trash/frank-small.service"
        echo "retired root unit frank-small.service -> $trash"
    else
        # This host's `systemctl disable` is allowed to delete the unit file
        # outright (2026-08-26 incident), so there is often nothing left to
        # move. Say so: previously no message meant no evidence either way.
        echo "root unit frank-small.service was already removed by disable"
    fi
    run_privileged systemctl daemon-reload
 }
 deploy_llm_small() {
    if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
        echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
-        echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
+        echo "frank-small: run scripts/download-frank-small-vllm.sh first (main model + MTP assistant, sha256-verified);" >&2
        echo "frank-small: refusing to start a unit that cannot load." >&2
        return 1
    fi
-    relink_unit frank-small
+    if [[ ! -f "$LLM_SMALL_DRAFT" ]]; then
-    run_privileged systemctl daemon-reload
+        echo "ERROR: MTP assistant absent at $LLM_SMALL_DRAFT" >&2
-    run_privileged systemctl enable frank-small.service
+        echo "frank-small: run scripts/download-frank-small-vllm.sh first;" >&2
-    run_privileged systemctl restart frank-small.service
+        echo "frank-small: refusing to start a unit that cannot draft." >&2
-    wait_for_health frank-small http://127.0.0.1:7771/health 900
+        return 1
    fi
    retire_root_llm_small
    # The predecessor can also be THIS quadlet -- a redeploy of a running
    # frank-small -- so stop the user unit before waiting on the port. On a
    # first cutover it is not running and this is a no-op.
    systemctl --user stop frank-small.service 2>/dev/null || true
    # Pull the base by digest, re-tag it to the FROM ref the Dockerfile
    # expects, then build the derived image the quadlet runs: the quadlet
    # path cannot resolve registry-prefixed refs from local storage at
    # create time (same workaround as halogen), and Pull=never keeps boots
    # off the registry. The build needs network for pip (the vllm[audio]
    # extra); podman build has it on by default.
    podman pull "$VLLM_SMALL_BASE_IMAGE"
    podman tag "$VLLM_SMALL_BASE_IMAGE" "$VLLM_SMALL_BASE_ALIAS"
    podman build --pull=never -t "$VLLM_SMALL_IMAGE" \
        -f dankclaw/frank-small/Dockerfile dankclaw/frank-small
    # Nothing may start until the predecessor has released the port.
    wait_port_free 7771 || return 1
    install_quadlet frank-small dankclaw/frank-small/quadlet frank-small
    # No enable: generated units cannot be enabled on this host (v1 quadlet
    # generator); boot persistence comes from [Install] in the unit file.
    systemctl --user restart frank-small.service
    wait_for_health frank-small http://127.0.0.1:7771/health 1200
 }
 # deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a
@@ -0,0 +1,80 @@
 #!/usr/bin/env bash
 # Fetch the frank-small vLLM payloads on dankclaw (RTX 3090, since 2026-09-18):
 #
 #   google/gemma-4-12B-it-qat-w4a16-ct
 #       compressed-tensors w4a16 (int4, group 32) QAT checkpoint — the model
 #       vLLM's gemma4_unified implementation loads natively. One 9.56 GiB
 #       safetensors plus tokenizer/config.
 #
 #   google/gemma-4-12B-it-assistant
 #       the MTP draft/assistant (gemma4_unified_assistant, 4 layers) in
 #       bf16 — Google's original. 806 MiB. The w4a16-ct re-quant
 #       (kunhunjon) that was used first drafts at acceptance 1.00: vLLM's
 #       MTP drafter path does not apply the Marlin w4a16 kernel to the
 #       assistant's linears (no "Using MarlinLinearKernel" after the
 #       drafter load), so its packed int4 weights are misread. The bf16
 #       assistant accepts at 2.80 (measured 2026-09-19).
 #
 # Both repos are public and ungated; no token is needed. Every fetch is -f
 # (an HTTP error is a failure, not a 153-byte "artifact"), resumable, and the
 # two LFS payloads are sha256-verified against the Hub's declared oid — the
 # discipline from scripts/download-frank-medium-gguf.sh.
 #
 # The serve script refuses to start when either payload is absent, so a
 # half-finished run of this script reads as a missing file, not a runtime
 # fault.
 set -euo pipefail
 MODELS_DIR="${MODELS_DIR:-/home/dank/models}"
 LOG_DIR="$MODELS_DIR/logs"
 mkdir -p "$LOG_DIR"
 LOG="$LOG_DIR/$(basename "$0" .sh).log"
 log() { echo "$(date -Is) $*" | tee -a "$LOG"; }
 # fetch <repo> <dir> <file> [sha256] [size]
 fetch() {
    local repo="$1" dir="$2" file="$3" sha="${4:-}" size="${5:-}"
    mkdir -p "$dir"
    if [[ -n "$size" && -f "$dir/$file" ]] && [[ "$(stat -c%s "$dir/$file")" == "$size" ]]; then
        log "skip $dir/$file (size $size matches)"
    else
        log "fetch $repo/$file"
        curl -fsSL -C - "https://huggingface.co/$repo/resolve/main/$file" -o "$dir/$file"
        log "fetched $dir/$file ($(stat -c%s "$dir/$file") bytes)"
    fi
    if [[ -n "$sha" ]]; then
        local actual
        actual="$(sha256sum "$dir/$file" | cut -d' ' -f1)"
        if [[ "$actual" != "$sha" ]]; then
            log "SHA256 MISMATCH $dir/$file: $actual != $sha"
            return 1
        fi
        log "sha256 ok $dir/$file"
    fi
 }
 MAIN_REPO=google/gemma-4-12B-it-qat-w4a16-ct
 ASSISTANT_REPO=google/gemma-4-12B-it-assistant
 MAIN_DIR="$MODELS_DIR/gemma4-12b-qat-w4a16-ct"
 ASSISTANT_DIR="$MODELS_DIR/gemma4-12b-assistant-bf16"
 # Hub-declared LFS oids (sha256), 2026-09-18.
 fetch "$MAIN_REPO" "$MAIN_DIR" model.safetensors \
    60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3 10264229896
 fetch "$MAIN_REPO" "$MAIN_DIR" tokenizer.json \
    cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f 32169626
 for f in config.json generation_config.json chat_template.jinja \
         processor_config.json tokenizer_config.json recipe.yaml; do
    fetch "$MAIN_REPO" "$MAIN_DIR" "$f"
 done
 fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" model.safetensors \
    3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df 845719296
 fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" tokenizer.json \
    c001d9ada50af662c94f5ab17ec7e09f6438d1bec8246c47fee6510693d8de35 32169884
 for f in config.json generation_config.json tokenizer_config.json; do
    fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" "$f"
 done
 log "ALL FETCHED OK"
@@ -1,112 +1,81 @@
-# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
+# Tunables for frank-small (since 2026-09-18): vLLM serving
-# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
+# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
-# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
+# Google publishes "for native, optimized inference with vLLM") with the
 # google bf16 MTP assistant, on Dankclaw's RTX 3090.
 #
-# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
+# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
-#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
+# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
-#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
+# with the vllm[audio] extra installed on top (dankclaw/frank-small/
-#   cmake --build build-cuda --target llama-server -j
+# Dockerfile) so the model's native audio path works. That image carries
-# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
+# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
-# attention, and without the flag quantized-KV prefill collapses ~12x on
+# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
-# CUDA (see the CUDA lesson in AGENTS.md).
+# card. The older gemma4-unified build (June 3) predates this checkpoint
 # and fails at weight load; the cu129-nightly build (Sep 15) fails at
 # import (torchvision ABI); this one is the image that boots.
 #
-# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
+# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
-# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
+# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
-# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
+# embeddings, vision and audio. The fleet's image path lives here: the model
-# per-layer token embeddings (PLE) inside the file, and a sliding-window
+# ingests image (and audio) content natively, so no projector file exists
-# attention pattern (attention.sliding_window = 1024) -- most layers cache
+# any more and nothing external has to load.
 # only a 1024-position window, which is why the KV pool is a fraction of
 # the full-ctx x all-layers number.
 #
-# MTP: the Multi-Token Prediction head ships as a SEPARATE file
+# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
+# same int4 w4a16 compressed-tensors format — the card's compatibility rule
-# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
+# is that a QAT target must draft with a QAT drafter at the same precision.
-# is passed via --model-draft with --spec-type draft-mtp, so the draft
+# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
-# context loads the assistant model. The same --spec-type value chains the
+# MTP path, not a generic draft model.
 # draftless ngram-mod matcher, which attaches unconditionally; two
 # context-backed drafters cannot share one process, so draft-mtp,ngram-mod
 # is the only legal shape that carries both.
 #
-# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
+# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
-# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
+# for as many concurrent sessions as the card holds.
-# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
+#   MAX_MODEL_LEN=262144      the model's max_position_embeddings
-# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
+#   GPU_MEM_UTIL=0.95         the card is dedicated to this service
-# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
+#   KV_CACHE_DTYPE=auto       bf16 KV. fp8_e4m3 would halve the KV charge
-# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
+#                             (289,756 -> 623,666-token pool, one -> two
-# which would shrink each slot below the full 262144 window once
+#                             full-context sessions), but the 3090 is
-# parallel > 1.
+#                             sm_86 (Ampere) and has no native fp8: the
 #                             TRITON_ATTN backend Gemma4 forces emits
 #                             cvt.e4m3x2 instructions that exist only from
 #                             sm_89 (Ada), so fp8 KV cannot boot on this
 #                             card (measured 2026-09-19; the "sm_89
 #                             patch" experiment is in .AGENTTRASH).
 #   MAX_NUM_SEQS=16           the batch ceiling; the KV pool, not this,
 #                             caps full-context concurrency
 #   MAX_NUM_BATCHED_TOKENS=16384  the prefill chunk; larger chunks prefill
 #                             faster but grow the activation arena that
 #                             competes with the KV pool for the card
 # The boot log prints the KV pool size and "Maximum concurrency for N tokens
 # per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
 # max_model_len) and undercounts: the 42 sliding-window layers only retain a
 # 1024-token window per request, so the true full-context session count is
 # higher. Verify it behaviourally (concurrent long requests) on every
 # restart, not from these comments.
 #
-# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
+# Sampling: greedy by default (fleet contract) via
-# params; it carries an audio encoder too). The fleet's image path lives
+# --override-generation-config; a client that sends its own temperature gets
-# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
+# its own, and the harness applies per-model overrides from models.yml.
 # WORKS: input_audio content parts transcribe verbatim (verified
 # 2026-09-17 with a 16k mono WAV).
 #
-# Sampling: greedy by default (the fleet contract); a client that sends a
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
-# temperature gets its own, and the harness applies per-model overrides
+# ROCm hosts read those bare vars — see the 2026-08-13 incident in
 # from models.yml.
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
 # reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
 # AGENTS.md). This box is CUDA, but the rule keeps env files portable.
-# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
+# Container paths, bind-mounted by the quadlet. The host paths are declared
-# FA_ALL_QUANTS=ON.
+# in the quadlet and the deploy handler, not here: this file is read inside
-LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
+# the container.
 MODEL_DIR=/models/main
 DRAFT_DIR=/models/assistant
-# Model package. The MTP sidecar file is required for speculative decoding.
+# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
-MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
+# host 7771, the endpoint Hermes, the harness and the homepage probe have
-MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
+# always used. The alias is the id at /v1/models and what clients ask for.
 MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
 DRAFT_FILE=mtp-gemma-4-12B-it.gguf
 # Fleet identity and network. 7771 is this service's own endpoint on
 # Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
 # what clients ask for -- the harness entry in models.yml names it.
 ALIAS=frank-small
 BIND_ADDR=0.0.0.0
-LISTEN_PORT=7771
+LISTEN_PORT=8000
-# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
+# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling,
-# pool; --parallel divides it). See the SIZING note above for the fit
+# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
-# arithmetic; a fifth slot does not fit.
+MAX_MODEL_LEN=262144
-CTX_SIZE=1048576
+GPU_MEM_UTIL=0.95
-PARALLEL=4
+KV_CACHE_DTYPE=auto
-CACHE_TYPE_K=q8_0
+MAX_NUM_SEQS=16
-CACHE_TYPE_V=q8_0
+MAX_NUM_BATCHED_TOKENS=16384
-
+SPEC_NUM_TOKENS=4
 # Batch geometry: 8192 logical / 1024 physical. Measured on this card at
 # max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
 # ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
 # tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
 # compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
 # tried. Idle VRAM is 23,300 MiB at this geometry.
 BATCH_SIZE=8192
 UBATCH_SIZE=1024
 # CPU threads: this service runs entirely on the GPU (NGL=999), so threads
 # cover tokenization, sampling and the HTTP layer only. Deliberately modest
 # -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
 NUM_THREADS=8
 THREAD_BATCH=8
 # Every layer on the 3090, with flash attention (required for q8 KV).
 NGL=999
 FLASH_ATTN=on
 # Speculative decoding: the MTP sidecar (draft context) chained with the
 # draftless ngram-mod matcher. Draft depth and busy-wait level are made
 # explicit here (both were the implicit build defaults: 3 and 50).
 # Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
 # than depth 3 + poll 50 (257-302 t/s) — the longer verification round
 # costs more attention re-read at long context than the extra accepted
 # token earns. Keep 3/50.
 SPEC_TYPE=draft-mtp,ngram-mod
 SPEC_DRAFT_N_MAX=3
 POLL_LEVEL=50
 # Greedy default (fleet contract); reasoning is extracted into
 # reasoning_content with an unlimited budget (the model decides).
 TEMPERATURE=0.0
 REASONING_FORMAT=deepseek
 REASONING_BUDGET=-1
@@ -1,20 +0,0 @@
 [Unit]
 Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target
 [Service]
 Type=simple
 User=dank
 Group=dank
 # CUDA pins its staging buffers, so the process needs the memlock budget even
 # though the weights themselves are copied into VRAM. Parity with the halogen
 # quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
 LimitMEMLOCK=infinity
 Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
 Restart=on-failure
 RestartSec=10
 [Install]
 WantedBy=multi-user.target
@@ -1,43 +1,30 @@
 #!/usr/bin/env bash
-# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
+# Serve google/gemma-4-12B-it-qat-w4a16-ct as frank-small on Dankclaw's RTX
-# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
+# 3090 via vLLM (since 2026-09-18), with the google bf16 MTP assistant
 # for speculative decoding.
 #
-# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
+# This script is the CONTAINER ENTRYPOINT of the frank-small quadlet: the
-# weights plus the BF16 vision projector and q8_0 KV inside the card's
+# vllm/vllm-openai image's own entrypoint is `vllm serve`, and the quadlet
-# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
+# replaces it with this file (mounted read-only out of the repo checkout) so
-# belongs to the halogen engine that serves frank-large.
+# the flag set stays in git. The image pins the gemma4-unified vLLM build,
 # which is what loads gemma4_unified natively and runs the Gemma 4 MTP path.
 #
-# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
+# MTP: the drafter is a separate 4-layer assistant checkpoint in bf16
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
+# (Google's original gemma-4-12B-it-assistant). It goes in
-# passed via --model-draft with --spec-type draft-mtp: the draft context
+# --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path, not a
-# loads the assistant model. The same --spec-type value chains the
+# generic draft model. The w4a16-ct re-quant of the assistant was tried
-# draftless ngram-mod matcher, which attaches unconditionally; two
+# first and drafts at acceptance 1.00 (the drafter path does not apply the
-# context-backed drafters cannot share one process, so this chain is the
+# Marlin kernel to its linears); the bf16 assistant accepts at 2.80.
 # only shape that carries both.
 #
-# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
+# Sampling: greedy by default (fleet contract) via
-# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
+# --override-generation-config; clients that send their own temperature get
-# maximises the TOTAL pool instead, which would divide a smaller window
+# their own, and the harness overrides per model from models.yml.
 # across slots. The 4-slot layout is measured to fit the card (see the
 # SIZING note in frank-small.env); more slots would fail the KV allocation.
 #
-# Vision + audio: --mmproj loads the BF16 projector, which also carries
+# Tunables come from frank-small.env (mounted beside this file by the
-# an audio encoder — input_audio content parts transcribe verbatim
+# quadlet) unless FRANK_SMALL_ENV_FILE overrides the path.
 # (verified 2026-09-17). The fleet's image path lives here, which is why
 # halogen no longer sets HALOGEN_VISION_TOWER.
 #
 # Sampling: greedy by default (the fleet contract). Clients that send their own
 # temperature get their own values, and the harness overrides per model from
 # models.yml. --jinja is pinned even though this build defaults it on: the
 # model's own template supplies reasoning_effort, preserve_thinking and the
 # tool-call grammar, and a future build that flips that default must not
 # silently take tools away.
 #
 # Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
 set -euo pipefail
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_SMALL_ENV_FILE:-/scripts/frank-small.env}"
 ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
 if [[ -f "$ENV_FILE" ]]; then
    set -a
    # shellcheck disable=SC1090
@@ -46,93 +33,51 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
+: "${MODEL_DIR:=/models/main}"
-: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
+: "${DRAFT_DIR:=/models/assistant}"
 : "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
 : "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
 : "${DRAFT_FILE:=}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
-: "${LISTEN_PORT:=7771}"
+: "${LISTEN_PORT:=8000}"
-: "${CTX_SIZE:=}"
+: "${MAX_MODEL_LEN:=262144}"
-: "${PARALLEL:=1}"
+: "${GPU_MEM_UTIL:=0.95}"
-: "${CACHE_TYPE_K:=f16}"
+: "${KV_CACHE_DTYPE:=auto}"
-: "${CACHE_TYPE_V:=f16}"
+: "${MAX_NUM_SEQS:=16}"
-: "${BATCH_SIZE:=2048}"
+: "${MAX_NUM_BATCHED_TOKENS:=16384}"
-: "${UBATCH_SIZE:=512}"
+: "${SPEC_NUM_TOKENS:=4}"
-# Decode-shape knobs: the MTP drafter's per-round proposal length (build
+: "${ENFORCE_EAGER:=0}"
 # default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
 # busy-wait level (build default 50).
 : "${SPEC_DRAFT_N_MAX:=3}"
 : "${POLL_LEVEL:=50}"
 : "${NUM_THREADS:=8}"
 : "${THREAD_BATCH:=8}"
 : "${NGL:=999}"
 : "${FLASH_ATTN:=on}"
 : "${SPEC_TYPE:=draft-mtp}"
 : "${TEMPERATURE:=0.0}"
 : "${REASONING_FORMAT:=deepseek}"
 : "${REASONING_BUDGET:=-1}"
 # Fail loudly rather than half-loading a model into VRAM: a missing file here
 # is a download that has not finished, and the unit's restart policy would
 # otherwise turn that into a crash loop that reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
+for path in "$MODEL_DIR/model.safetensors" "$MODEL_DIR/config.json" \
            "$DRAFT_DIR/model.safetensors" "$DRAFT_DIR/config.json"; do
    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
-if [[ -n "${DRAFT_FILE:-}" ]]; then
+
-    [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
+SPEC_CONFIG="{\"method\":\"mtp\",\"model\":\"${DRAFT_DIR}\",\"num_speculative_tokens\":${SPEC_NUM_TOKENS}}"
 VLLM_FLAGS=(
    serve "$MODEL_DIR"
 --served-model-name "$ALIAS"
 --host "$BIND_ADDR"
 --port "$LISTEN_PORT"
 --max-model-len "$MAX_MODEL_LEN"
 --gpu-memory-utilization "$GPU_MEM_UTIL"
 --kv-cache-dtype "$KV_CACHE_DTYPE"
 --max-num-seqs "$MAX_NUM_SEQS"
 --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
 --speculative-config "$SPEC_CONFIG"
 --reasoning-parser gemma4
 --enable-auto-tool-choice
 --tool-call-parser gemma4
 --override-generation-config '{"temperature": 0}'
 )
 # ENFORCE_EAGER=1 skips Inductor/Triton compilation and CUDA graph capture
 # (slower steady-state decode, faster and more robust boot). The 3090 is
 # sm_86 (Ampere) — no native fp8 — so KV stays bf16 and the default
 # graphed path is the right one; the knob is a boot-debugging lever.
 if [[ "$ENFORCE_EAGER" == "1" ]]; then
    VLLM_FLAGS+=(--enforce-eager)
 fi
-# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
+exec vllm "${VLLM_FLAGS[@]}"
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 MMPROJ_ARGS=()
 if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
 fi
 SPEC_ARGS=()
 if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
    SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
 fi
 DRAFT_ARGS=()
 if [[ -n "${DRAFT_FILE:-}" ]]; then
    DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
 fi
 # CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
 # whatever the weights, projector, draft context and compute buffers leave
 # on the card; the engine prints the chosen value at startup.
 if [[ -n "${CTX_SIZE:-}" ]]; then
    CTX_ARGS=(--ctx-size "$CTX_SIZE")
 else
    CTX_ARGS=(--fit on)
 fi
 exec "$LLAMA_BIN" \
    --model "$MODEL_DIR/$MODEL_FILE" \
    "${MMPROJ_ARGS[@]}" \
    "${DRAFT_ARGS[@]}" \
    --alias "$ALIAS" \
    --host "$BIND_ADDR" \
    --port "$LISTEN_PORT" \
    --gpu-layers "$NGL" \
    --flash-attn "$FLASH_ATTN" \
    "${CTX_ARGS[@]}" \
    --parallel "$PARALLEL" \
    --batch-size "$BATCH_SIZE" \
    --ubatch-size "$UBATCH_SIZE" \
    --threads "$NUM_THREADS" \
    --threads-batch "$THREAD_BATCH" \
    --poll "$POLL_LEVEL" \
    --cache-type-k "$CACHE_TYPE_K" \
    --cache-type-v "$CACHE_TYPE_V" \
    "${SPEC_ARGS[@]}" \
    --jinja \
    --metrics \
    --temp "$TEMPERATURE" \
    --reasoning-format "$REASONING_FORMAT" \
    --reasoning-budget "$REASONING_BUDGET"
@@ -0,0 +1,211 @@
 ---
 type: reference
 title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
 description: The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant.
 tags:
  - llm
  - frank-small
  - dankclaw
  - vllm
  - rtx3090
  - gemma
  - mtp
  - vision
  - audio
 timestamp: 2026-09-19
 ---
 # frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
 > **ACTIVE since 2026-09-19.** Replaces [[frank-small-gemma4-12b]]
 > (llama.cpp, now the rollback target). Same port (7771), same alias
 > (`frank-small`), same client contract surface — the runtime, the model
 > payload, the drafter and the service shape (root systemd unit → rootless
 > Quadlet) all changed.
 ## Identity
 | what | value |
 |---|---|
 | runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d7…`) |
 | unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (root systemd unit retired 2026-09-19) |
 | model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
 | model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
 | drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
 | port | 7771 → 8000 (alias `frank-small`; `frank-large` keeps 7770) |
 | window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
 | multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
 ## Why vLLM
 - The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
  kernel is its native int4 GEMM path. llama.cpp could not load it at all.
 - Native audio encoder in the checkpoint — the fleet's only audio-capable
  endpoint moves here (llama.cpp's path needed the mmproj audio tower).
 - CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
  this host had no graph support.
 ## The drafter-Marlin bug (why the assistant is bf16)
 The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
 **cannot draft in vLLM**: the drafter's linear layers are loaded through a
 path that does not apply the Marlin kernel, so the packed int4 weights are
 misread as bf16 and every draft is garbage — measured acceptance **1.00**
 (pure pass-through). There is no "Using MarlinLinearKernel" log line after
 the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
 has no quantization to miss, and measures **acceptance 2.80–3.56**
 (per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
 (634 → 845 MB) is absorbed by pool rounding: both assistants yield the
 identical 289,647-token KV pool.
 ## KV arithmetic — why two full-context sessions
 Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
 16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
 per-token KV is dominated by the full-attention layers:
 6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
 289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
 sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
 shares the 1024-window ring across all concurrent requests, so the
 marginal cost of a *second* 213k session is ~6 full-attention GiB, not
 a second copy of everything.
 Measured (salted, 2026-09-19):
 | load | KV usage | result |
 |---|---|---|
 | 1 × 213k | ~35 % | runs |
 | 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
 | 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |
 **Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
 deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
 bytes, and vLLM's pool arithmetic is coarser. This is the accepted
 trade for the native w4a16 format + audio + CUDA graphs.)
 ## Measured throughput (salted, 2026-09-19)
 | probe | tokens | wall | rate |
 |---|---|---|---|
 | prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
 | prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
 | prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
 | 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
 | decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
 | decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |
 - **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
  layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
  as llama.cpp's flash-attn path here.
 - **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
  token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
  right at it. At normal depths decode is far faster (the 67k-depth probe
  is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
  numbers.
 - **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
  scale linearly in tokens — BPE compresses the repetition harder as the
  span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
  predicts). Size fillers with the model's own tokenizer
  (`AutoTokenizer.encode` on the exact prefix), not char counts.
 - `POST /flush_cache` is **404** in this build — salt every probe
  (uuid prefix) instead.
 ## The image
 `dankclaw/frank-small/Dockerfile` (repo) — 16 comment lines + `FROM
 localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
 --no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
 lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
 `sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
 after the driver swap, so no arch override is needed.
 ## No nvidia-container-toolkit — manual CUDA mounts
 Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
 by hand:
 - `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
  `/dev/nvidia-uvm-tools`
 - host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
  into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
 - `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`
 If the driver is upgraded on the host, these mount sources move with it
 (they are versionless symlinks) — no unit change needed.
 ## Host driver note
 Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
 Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
 595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
 the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
 not a hardware mystery.
 ## Deployed config
 Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`
 (mounted into the container at `/scripts/`, which is the entrypoint) +
 `dankclaw/frank-small/quadlet/frank-small.container`. Key values:
 `KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
 compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
 `ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
 `--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
 is **removed**. The serve script refuses to start if the weights, the
 assistant, or the vllm binary are missing.
 ## Client contract
 - **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
  parser), not `reasoning_content` — the harness maps it via
  `compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
  The field is empty when the model emits no thinking tag (behavior, not
  a parser bug).
 - `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
  accepted (200, not 400) — vLLM ignores the unknown key.
 - Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
 - Tools → `tool_calls` (verified `get_weather`).
 - Image → 200, described correctly (native vision tower).
 - Audio → 200, a 440 Hz WAV tone described correctly (native audio
  encoder; verified in the derived image).
 - Harness: `contextWindow: 262144`, `maxTokens: 32768`,
  `input: [text, image, audio]`.
 ## Boot ledger
 t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
 bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
 KV pool 289,647 tokens). Health is 503 until ready — do not kill the
 container while it loads.
 ## Troubleshooting
 - **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
  (drafter-Marlin bug). Confirm the assistant mount points at
  `gemma4-12b-assistant-bf16`.
 - **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
  must be absent. 8.6 is correct.
 - **Audio 400/500** — the container is running the *base* image, not the
  derived `:frank-small` (missing `av`). Check the deployed image digest
  is `8dad46d7…`.
 - **CUDA not visible in container** — a `Device=` line or a lib mount
  dropped out of the unit after an edit; the unit is the source of truth,
  the host-side generated unit is not.
 - Logs: `journalctl --user -u frank-small` (rootless Quadlet; `podman
  logs` covers only the current `--replace --rm` instance).
 ## Rollback
 [[frank-small-gemma4-12b]] — the llama.cpp deployment. Its weights,
 mmproj and MTP sidecar are still on the host under
 `/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
 `scripts/.AGENTTRASH/frank-small.service.llama.cpp`. Restoring it means
 reinstalling that unit and stopping the Quadlet (one owner per port).
 ## References
 - `scripts/serve-frank-small.sh` — the actual exec line
 - `dankclaw/frank-small/Dockerfile` — the audio-derived image
 - `dankclaw/frank-small/quadlet/frank-small.container` — the unit
 - [[frank-small-gemma4-12b]] — RETIRED predecessor (rollback)
 - [[halogen-flash-server]] — the sibling endpoint on the iGPU
 - [[VLLM Overview]] / [[VLLM Index]] — runtime background
@@ -5,9 +5,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-16):
 process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
 Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
 endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
-w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
+w4b on the gfx1151 iGPU, port 7770) and vLLM (frank-small,
-ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
+google/gemma-4-12B-it-qat-w4a16-ct with the bf16 MTP assistant on the RTX
-7771).
+3090, port 7771).
 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -35,10 +35,10 @@ gfx1151 iGPU 64 GiB unified memory]
 frank-large, port 7770]
        F[peonist-ai Qwen3.8-Flash-Next w4b
 2 x 200192 ctx over one 400384-position pool]
-        G[mainline llama.cpp 37b53fd
+        G[vLLM gemma-x86_64-cu129
 frank-small, port 7771]
-        H[Qwen3.8-27B GSQ-RCO IQ3_S
+        H[gemma-4-12B-it-qat-w4a16-ct
-131072 ctx / 1 slot, MTP + vision]
+262144 ctx / 2 full-context sessions, MTP + vision + audio]
        E2 --> E --> F
        E2 --> G --> H
    end
@@ -87,11 +87,15 @@ frank-small, port 7771]
  overlay) on the gfx1151 iGPU — two 200k conversations over one
  400,384-position pool, deployed flags, memory model, client contract,
  troubleshooting
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
+- [[frank-small-vllm-gemma4]] — ACTIVE frank-small runbook since 2026-09-19
-  (root systemd unit, port 7771): mainline llama.cpp serving
+  (rootless Quadlet, port 7771): vLLM serving
-  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
+  google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
-  3090, with the separate MTP assistant sidecar chained behind ngram-mod,
+  with Google's bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-token
-  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB
+  pool, 2 concurrent full-context sessions, native text+image+audio
 - [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): the
  prior frank-small runbook (2026-09-17 → 09-19), mainline llama.cpp serving
  HauhauCS Gemma4-12B-QAT-Uncensored Q4_K_M with the MTP sidecar, q8_0 KV,
  4 slots of 262k
 - [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
  prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
  Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
@@ -166,11 +166,17 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
+- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
-  2026-09-17: mainline llama.cpp serving
+  2026-09-19: vLLM serving
-  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
+  google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
-  (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
+  with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
  full-context sessions, native text+image+audio) on
  Dankclaw's RTX 3090, port 7771
 - [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
  (MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
  Dankclaw's RTX 3090
 - [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
  the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
  Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
@@ -64,7 +64,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
    run python3 "$PLANNER" --target dankclaw --paths \
-        scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
+        dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
        scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
    [ "$status" -eq 0 ]
    [ "$output" = "frank-small" ]
@@ -72,7 +73,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
    # that name was retired with the FreeToken runtime on 2026-09-15 and must
    # not come back through a planner mapping.
    run python3 "$PLANNER" --target dankserver --paths \
-        scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
+        dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
        scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
    [ "$status" -eq 0 ]
    [ -z "$output" ]
 }
@@ -72,7 +72,11 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "frank-large loads no vision tower: the fleet's image path is frank-small" {
    ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET"
-    grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
+    # frank-small's vLLM checkpoint (2026-09-18) is natively multimodal —
    # image content is ingested by the model itself, so the seam that used
    # to pin --mmproj now pins the projector's absence.
    ! grep -q 'mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
    grep -q '^MODEL_DIR=/models/main$' "$REPO_ROOT/scripts/frank-small.env"
 }
@test "the deploy handler retires the previous ROOT unit without deleting it" {
@@ -1,115 +1,157 @@
 #!/usr/bin/env bats
-# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
+# Seam: the frank-small vLLM quadlet on dankclaw (since 2026-09-18).
-# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
+# frank-small is the RTX 3090 endpoint. It now runs vLLM (the pinned
-# went back to the halogen engine on the iGPU. This seam pins what the flags
+# gemma4-unified image) serving google/gemma-4-12B-it-qat-w4a16-ct with the
-# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
+# google bf16 MTP assistant, replacing the mainline-llama.cpp root
-# is involved), the repo-relative paths the unit executes, the separate client
+# unit (retired into scripts/.AGENTTRASH/). This seam pins what the flags
-# port, the MTP contract (the head is a separate sidecar file loaded as the
+# alone do not say: boot persistence for the generated quadlet, the CUDA
-# draft model, chained behind ngram-mod), the vision handoff from frank-large,
+# wiring this host needs (no nvidia-container-toolkit), the repo-relative
-# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
+# entrypoint and env, the MTP contract (a separate QAT assistant checkpoint
 # in --speculative-config, method "mtp"), the fleet's image path (the model
 # is natively multimodal — no projector file), the full-window sizing, the
 # separate client port, and the deploy handler's cutover order.
 load helpers
-UNIT="$REPO_ROOT/scripts/frank-small.service"
+QUADLET="$REPO_ROOT/dankclaw/frank-small/quadlet/frank-small.container"
 DOCKERFILE="$REPO_ROOT/dankclaw/frank-small/Dockerfile"
 SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
 FETCH="$REPO_ROOT/scripts/download-frank-small-vllm.sh"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
-@test "frank-small unit, serve script and env exist in scripts/" {
+@test "frank-small quadlet, Dockerfile, entrypoint, env and fetch script exist" {
-    [ -f "$UNIT" ]
+    [ -f "$QUADLET" ]
    [ -f "$DOCKERFILE" ]
    [ -f "$SERVE" ]
    [ -f "$ENV_FILE" ]
    [ -f "$FETCH" ]
    # The old root unit is retired, not deleted: it must live in the trash,
    # not in scripts/, so the planner and relink_unit never see it.
    [ ! -e "$REPO_ROOT/scripts/frank-small.service" ]
    [ -e "$REPO_ROOT/scripts/.AGENTTRASH/frank-small.service.llama.cpp" ]
 }
-@test "frank-small is boot-persistent, self-healing and unprivileged" {
+@test "frank-small is boot-persistent, self-healing and registry-pinned" {
-    grep -q '^WantedBy=multi-user.target$' "$UNIT"
+    grep -q '^WantedBy=default.target$' "$QUADLET"
-    grep -q '^Restart=on-failure$' "$UNIT"
+    grep -q '^Restart=on-failure$' "$QUADLET"
-    grep -q '^RestartSec=' "$UNIT"
+    grep -q '^RestartSec=' "$QUADLET"
-    grep -q '^User=dank$' "$UNIT"
+    grep -q '^Pull=never$' "$QUADLET"
-    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
+    # The unit runs the localhost/-aliased LOCAL BUILD (base + vllm[audio]
    # extra); the digest-pinned docker.io base lives only in the deploy
    # handler, which pulls, re-tags and builds before restart.
    grep -q '^Image=localhost/vllm/vllm-openai:frank-small$' "$QUADLET"
    ! grep -q '^Image=docker.io' "$QUADLET"
 }
-@test "frank-small runs the repo's own serve script with the repo's env" {
+@test "frank-small wires CUDA without the nvidia-container-toolkit" {
-    grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
+    # The four device nodes (0666 on this host, so rootless can open them)...
-    grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
+    grep -q '^AddDevice=/dev/nvidia0$' "$QUADLET"
    grep -q '^AddDevice=/dev/nvidiactl$' "$QUADLET"
    grep -q '^AddDevice=/dev/nvidia-uvm$' "$QUADLET"
    grep -q '^AddDevice=/dev/nvidia-uvm-tools$' "$QUADLET"
    # ...plus the host driver's libcuda and NVML stubs. Without the NVML
    # mount vllm serve dies at parser construction with "Failed to infer
    # device type" (measured 2026-09-18).
    grep -q 'libcuda.so.1.*:ro' "$QUADLET"
    grep -q 'libnvidia-ml.so.1.*:ro' "$QUADLET"
    grep -q '^Environment=NVIDIA_VISIBLE_DEVICES=all$' "$QUADLET"
    # CUDA pins its staging buffers; memlock must be unlimited.
    grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET"
 }
-@test "frank-small offloads every layer to the 3090 with flash attention" {
+@test "frank-small runs the repo's own entrypoint with the repo's env" {
-    grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
+    grep -q '^ContainerEntrypoint=/scripts/serve-frank-small.sh$' "$QUADLET"
-    grep -q '^NGL=999$' "$ENV_FILE"
+    grep -q '^Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env$' "$QUADLET"
-    grep -q -- '--gpu-layers "$NGL"' "$SERVE"
+    grep -q 'Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro' "$QUADLET"
    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
    grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
    # Prometheus scrapes this endpoint; without --metrics there is no /metrics.
    grep -q -- '--metrics' "$SERVE"
 }
-@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
+@test "frank-small mounts the sha256-verified payloads read-only" {
-    grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
+    grep -q 'Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro' "$QUADLET"
-    grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
+    grep -q 'Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro' "$QUADLET"
-    # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
+    # The fetch script verifies both LFS payloads against the Hub oids.
-    # nextn layers); it loads as --model-draft and shares the target's last
+    grep -q '60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3' "$FETCH"
-    # layers' KV. ngram-mod is draftless, so the chain is legal.
+    grep -q '3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df' "$FETCH"
-    grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
+}
-    grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
+
@test "frank-small drafts with the QAT MTP assistant via method mtp" {
    # The drafter is a separate 4-layer assistant checkpoint in the same
    # int4 w4a16 compressed-tensors format (the card's compatibility rule:
    # a QAT target drafts with a QAT drafter at the same precision), passed
    # in --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path.
    grep -qF -- '\"method\":\"mtp\"' "$SERVE"
    grep -q -- '--speculative-config "$SPEC_CONFIG"' "$SERVE"
    grep -q '^SPEC_NUM_TOKENS=4$' "$ENV_FILE"
    # The gemma4 reasoning and tool-call parsers keep the fleet contract:
    # reasoning_content on the wire and tools answered.
    grep -q -- '--reasoning-parser gemma4' "$SERVE"
    grep -q -- '--tool-call-parser gemma4' "$SERVE"
    grep -q -- '--enable-auto-tool-choice' "$SERVE"
 }
@test "frank-small carries the fleet's image path, and only it" {
-    grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
+    # The model is natively multimodal (encoder-free unified): image content
-    grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
+    # is ingested by the checkpoint itself, so there is no projector file
    # any more — the seam that used to pin MMPROJ_PATH now pins its absence.
    ! grep -q '^MMPROJ' "$ENV_FILE"
    ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
 }
-@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
+@test "frank-small serves the full window with bf16 KV on a dedicated card" {
-    local parallel cache_k cache_v ctx fixed per_slot
+    local maxlen kvdtype util seqs
-    parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
+    maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")"
-    cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
+    kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
-    cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
+    util="$(sed -nE 's/^GPU_MEM_UTIL=([0-9.]+)$/\1/p' "$ENV_FILE")"
-    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")"
-    [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
+    # The model's FULL trained window (262144), not a divided one.
-    # Every slot must be the model's FULL trained window (262144), not a
+    [ "$maxlen" = "262144" ]
-    # divided one -- CTX_SIZE is the total pool and --parallel divides it.
+    # bf16 KV: fp8_e4m3 would halve the pool charge, but the 3090 is
-    [ $((ctx / parallel)) -eq 262144 ]
+    # sm_86 (Ampere) — no native fp8, and the TRITON_ATTN backend that
-    [ $((ctx % parallel)) -eq 0 ]
+    # Gemma4 forces emits cvt.e4m3x2 instructions that exist only from sm_89.
-    [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
+    [ "$kvdtype" = "auto" ]
-    # Fit budget from the measured boot arithmetic on the 3090 (24576
+    # The 3090 is dedicated to this service, so the engine may take most of
-    # MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
+    # it; the KV pool, not the seq ceiling, caps full-context concurrency.
-    # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
+    [ "$util" = "0.95" ]
-    # (weights, projector, sidecar draft, compute buffers, CUDA context)
+    [ "$seqs" -ge 4 ]
    # is 11996-3450 = 8546 MiB. If the config asks for more slots than
    # the card holds, this inequality fails before deploy instead of the
    # KV allocation failing at boot.
    fixed=8546
    per_slot=3450
    [ $((fixed + parallel * per_slot)) -le 24576 ]
    # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
    # without which prefill collapses about 12x (see the CUDA lesson in
    # AGENTS.md). The build-cuda tree must carry it.
    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
 }
@test "frank-small runs on its own port beside frank-large's" {
-    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
+    # 8000 is the in-container port; the quadlet publishes it as host 7771,
    # the endpoint Hermes, the harness and the homepage probe use.
    grep -q '^LISTEN_PORT=8000$' "$ENV_FILE"
    grep -q '^ALIAS=frank-small$' "$ENV_FILE"
    grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
    grep -q -- '--host "$BIND_ADDR"' "$SERVE"
-    # The two endpoints cannot share a port, and only the halogen unit publishes
+    grep -q '^PublishPort=7771:8000$' "$QUADLET"
-    # 7770.
+    # The two endpoints cannot share a port, and only the halogen unit
    # publishes 7770.
    grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE"
    ! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE"
 }
-@test "the deploy handler installs the root unit and refuses a missing model" {
+@test "the deploy handler cuts over from the root unit and gates on health" {
    grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
-    grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
+    # The digest-pinned docker.io base is pulled and re-tagged to the FROM
-    grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
+    # ref, then the Dockerfile is built into the localhost tag the quadlet
-    grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
+    # runs (the vllm[audio] extra the base ships without).
-    grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
+    grep -q 'VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:' "$REMOTE_DEPLOY"
    grep -q 'VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"' "$REMOTE_DEPLOY"
    grep -q 'podman tag "\$VLLM_SMALL_BASE_IMAGE" "\$VLLM_SMALL_BASE_ALIAS"' "$REMOTE_DEPLOY"
    grep -q 'podman build --pull=never -t "\$VLLM_SMALL_IMAGE"' "$REMOTE_DEPLOY"
    grep -q 'dankclaw/frank-small/Dockerfile' "$REMOTE_DEPLOY"
    # The old ROOT unit is retired (stop, disable, trash, reload) before the
    # user unit takes the port — two managers must never own frank-small.
    grep -q 'retire_root_llm_small' "$REMOTE_DEPLOY"
    grep -q 'wait_port_free 7771' "$REMOTE_DEPLOY"
    # Missing payloads refuse the start, and the health gate is the real
    # endpoint.
    grep -q 'gemma4-12b-qat-w4a16-ct/model.safetensors' "$REMOTE_DEPLOY"
    grep -q 'gemma4-12b-assistant-bf16/model.safetensors' "$REMOTE_DEPLOY"
    grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
    grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
 }
-@test "frank-small's serve script refuses to start on a half-finished download" {
+@test "frank-small's entrypoint refuses to start on a half-finished download" {
    grep -q 'frank-small: missing' "$SERVE"
 }