diff --git a/AGENTS.md b/AGENTS.md index 4369c19..cd688b5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on - the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on + the gfx1151 iGPU) and `frank-small` (vLLM on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| |Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770| | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | -| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | +| Dankclaw | `frank-small` | vLLM `gemma-x86_64-cu129` dev build + audio extra (rootless Quadlet) | google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + Google's bf16 MTP assistant on the RTX 3090, 262144 window / 289,647-token bf16 KV pool / 2 full-context sessions, native text+image+audio — the fleet's multimodal endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly. The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by -their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small -and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated +their own files — frank-large and frank-small by their `dankclaw/` +quadlet/Dockerfile directories, frank-medium by its unit/env/serve trio in `scripts/` — so unrelated container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo; the value never enters the repo. @@ -232,8 +232,10 @@ merely wedged kernel usually still answers ARP. `~/.config/containers/systemd/frank-large.container` aside and `systemctl --user daemon-reload` — the quadlet generator re-creates that unit from the file on every reload, so the source is the lever, not `mask`. - - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): - `sudo systemctl stop frank-small && sudo systemctl mask frank-small` + - Dankclaw `frank-small` (rootless Quadlet since 2026-09-19, vLLM on the 3090): + `systemctl --user stop frank-small`, then move its + `~/.config/containers/systemd/frank-small.container` aside and + `systemctl --user daemon-reload` (same quadlet-source lever as frank-large). - Dankserver `frank-medium` (system unit): `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium` 2. Bench. @@ -263,10 +265,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and -`frank-large` is text-only because its halogen deployment loads no vision -tower. There is **no audio** input anywhere in the fleet: that capability left -with the FLM NPU endpoint on 2026-09-11. +of 2026-09-19 `frank-small` is the fleet's multimodal endpoint +(`input: [text, image, audio]`) — vLLM's native Gemma4 vision AND audio +towers, both verified live. `frank-large` is text-only because its halogen +deployment loads no vision tower. ### frank-large — halogen-flash-server (Dankclaw, ACTIVE) @@ -301,31 +303,37 @@ that as "tight and fragmented", not "out of memory". Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. Rollback: [[frank-large-ik-uncensored]]. -### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) +### frank-small — vLLM on the RTX 3090 (Dankclaw, ACTIVE) -Mainline llama.cpp master `37b53fd` (CUDA build with -`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo -`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), -serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M. -The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4 -nextn layers) loaded via `--model-draft`; the draftless **ngram-mod** -matcher chains behind it in the same `--spec-type` value. BF16 mmproj, -port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4` -(four FULL 262,144-token slots — the measured card ceiling), batch -**8192/1024**, greedy default. Deployed flags are in -`scripts/frank-small.env`; the serve script refuses to start if the -weights, the projector, the sidecar or the binary are missing. +vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (digest-pinned, audio +extra pip-installed into `localhost/vllm/vllm-openai:frank-small` — see +`dankclaw/frank-small/Dockerfile`) as a **rootless Quadlet** +(`dankclaw/frank-small/quadlet/frank-small.container`), serving +`google/gemma-4-12B-it-qat-w4a16-ct` (compressed-tensors w4a16, int4 Marlin +weights, 10.26 GiB) with Google's **bf16** MTP assistant +(`google/gemma-4-12B-it-assistant`, 845 MB). Port 7771, 262,144 window, +bf16 KV pool of 289,647 tokens, **2 concurrent full-context sessions** +(the 3rd queues), CUDA graphs on, `ENFORCE_EAGER=0`. -Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a -chat completion returns content plus `reasoning_content`; a tool call -returns `tool_calls`; an image request is answered from the projector; -an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at -max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft -acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why -ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll -100 is WORSE at long ctx) is tabled in the runbook. -Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. -Rollback: [[frank-small-qwen38-gsq-rco]]. +The w4a16-ct assistant CANNOT draft in vLLM (the drafter path skips the +Marlin kernel — packed int4 misread, acceptance 1.00); the bf16 assistant +measures acceptance 2.80–3.56. fp8 KV is hardware-dead on sm_86 (Ampere +has no fp8 tensor cores), so bf16 is the only option. + +Dankclaw has **no nvidia-container-toolkit**: the unit wires CUDA by hand +(`Device=` for the four nvidia nodes + host `libcuda`/`libnvidia-ml` +mounted read-only into two library paths). Host driver is +`nvidia-driver-610` 610.57.04 on Ubuntu 26.04; `sm_86` is the correct +compute capability — no arch override. + +Verified at deploy (2026-09-19): `/v1/models` reports `frank-small`; chat +returns content plus `reasoning` (vLLM's gemma4 parser field — the harness +maps it via `compat.reasoningContentField`); a tool call returns +`tool_calls`; an image is described; a 440 Hz WAV is described (native +audio encoder). Measured: pp@67k 1,293 t/s, pp@213k 608 t/s, +tg@213k ~13.6 t/s (bandwidth-bound), tg@85k ~17.6 t/s. +Runbook: `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md`. +Rollback: [[frank-small-gemma4-12b]] (llama.cpp, weights still on host). ### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE) @@ -707,7 +715,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | -| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | +| frank-small (vLLM Gemma4-12B w4a16, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md` | | frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -719,6 +727,7 @@ Retired (rollback targets and history): | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | +| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 09-19 | `frank-small-gemma4-12b.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | diff --git a/dankclaw/frank-small/Dockerfile b/dankclaw/frank-small/Dockerfile new file mode 100644 index 0000000..6268bd6 --- /dev/null +++ b/dankclaw/frank-small/Dockerfile @@ -0,0 +1,22 @@ +# frank-small runtime image: the digest-pinned vLLM gemma-x86_64-cu129 base +# (vLLM 0.22.1rc1.dev357+g74b5964f0 — the image that loads +# gemma-4-12B-it-qat-w4a16-ct, runs the Gemma4MTPModel drafter and carries +# the gemma4 reasoning/tool-call parsers) plus the vllm[audio] extra. +# +# The base image ships WITHOUT the audio extras, so an input_audio content +# part 500s with "Please install vllm[audio] for audio support" (measured +# 2026-09-19). The extra is just four PyPI packages (av, scipy, soundfile, +# mistral_common[audio]); installing the extras directly — not +# `pip install vllm[audio]` — keeps the pinned dev build in place: a bare +# `vllm[audio]` would make pip reach for the PyPI release and replace the +# 0.22.1rc1.dev build that knows gemma4. +# +# Built by scripts/deploy-remote.sh on dankclaw (rootless podman, network on +# for pip): the base is pulled by digest and re-tagged to the FROM ref below +# first, then `podman build` produces the :frank-small tag the quadlet runs. +FROM localhost/vllm/vllm-openai:gemma4-unified + +# The vllm[audio] extra as raw packages (header explains why not +# `pip install vllm[audio]`): audio input needs av/scipy/soundfile plus +# the mistral_common audio protocol pieces. +RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]" diff --git a/dankclaw/frank-small/quadlet/frank-small.container b/dankclaw/frank-small/quadlet/frank-small.container new file mode 100644 index 0000000..770decc --- /dev/null +++ b/dankclaw/frank-small/quadlet/frank-small.container @@ -0,0 +1,100 @@ +# Quadlet unit: frank-small (rootless, boot-persistent). +# +# Install path on dankclaw: ~/.config/containers/systemd/frank-small.container +# Unit name: frank-small.service (container name: frank-small) +# +# The small inference service. vLLM (since 2026-09-18) serving +# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16) with the +# google bf16 MTP assistant for speculative decoding, on the RTX +# 3090. It replaced the mainline-llama.cpp ROOT unit (the old +# scripts/frank-small.service is retired into scripts/.AGENTTRASH/). +# +# SHAPE. The quadlet replaces the image's `vllm serve` entrypoint with +# /scripts/serve-frank-small.sh (mounted read-only out of the repo checkout) +# so the flag set stays in git. The model and drafter directories are bound +# read-only: the payloads are fetched and sha256-verified by +# scripts/download-frank-small-vllm.sh before first start, and the deploy +# handler refuses to start the unit when either is absent. +# +# CUDA WITHOUT the nvidia-container-toolkit. This host has never run a CUDA +# container (the 3090 served frank-small as a native binary, the iGPU runs +# ROCm through /dev/kfd), so there is no nvidia hook to lean on. The recipe +# that works (measured 2026-09-18, torch sees the 3090 at 25.3 GB): +# - the four /dev/nvidia* nodes via AddDevice (they are 0666 on this host, +# so the rootless user namespace can open them); +# - the HOST driver's libcuda.so.1 and libnvidia-ml.so.1 mounted read-only +# — the image carries its own CUDA 12.9 runtime, but vLLM's platform +# probe needs the driver's NVML stub, and without it `vllm serve` dies at +# parser construction with "Failed to infer device type"; +# - NVIDIA_VISIBLE_DEVICES=all. +# memlock must be unlimited: CUDA pins its staging buffers. +# +# SIZING. The tunables (full 262144 window, 0.95 of a dedicated 24 GiB card, +# bf16 KV, 16-seq batch ceiling, 16k prefill chunk, 4-token MTP draft) live +# in scripts/frank-small.env. The boot log's "Maximum concurrency for N +# tokens per request" line is the naive pool ratio for a hybrid SWA model +# and undercounts; verify the session count behaviourally on restart. +# +# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet +# path cannot resolve registry-prefixed refs from local storage at create +# time. The tag is a LOCAL BUILD, not a pull: deploy-remote.sh pulls the +# digest-pinned docker.io base (the gemma-x86_64-cu129 build, vLLM +# 0.22.1rc1.dev357+g74b5964f0 — the image that boots this checkpoint) and +# re-tags it, then builds dankclaw/frank-small/Dockerfile on top, which +# installs the vllm[audio] extra (av, scipy, soundfile, +# mistral_common[audio]) so the model's native audio path works. Pull=never +# keeps boots off the registry. +[Unit] +Description=frank-small — vLLM gemma4_unified w4a16 + MTP on RTX 3090 +After=network-online.target + +[Container] +ContainerName=frank-small +Image=localhost/vllm/vllm-openai:frank-small +# Published port stays 7771: Hermes, the harness and the homepage probe all +# point here. 8000 (vLLM's in-container port) is never reachable directly. +PublishPort=7771:8000 +# CUDA device nodes (0666 on this host — rootless can open them). +AddDevice=/dev/nvidia0 +AddDevice=/dev/nvidiactl +AddDevice=/dev/nvidia-uvm +AddDevice=/dev/nvidia-uvm-tools +# Host driver stubs the image's CUDA runtime and vLLM's NVML probe need. +# The image's LD_LIBRARY_PATH leads with /usr/local/nvidia/lib64 — the +# nvidia-container-toolkit's conventional mount point, which this host does +# not have — so the driver libs go there; the /usr/lib mounts cover torch +# Inductor's own lookup. +Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro +Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so:ro +Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:ro +Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/local/nvidia/lib64/libcuda.so.1:ro +Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/local/nvidia/lib64/libcuda.so:ro +Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/local/nvidia/lib64/libnvidia-ml.so.1:ro +Environment=NVIDIA_VISIBLE_DEVICES=all +# Payloads, read-only: fetched and sha256-verified by +# scripts/download-frank-small-vllm.sh before first start. +Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro +Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro +# Serve script and env, read-only out of the repo checkout. +Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro +ContainerEntrypoint=/scripts/serve-frank-small.sh +Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env +Ulimit=memlock=-1:-1 +# /health answers 200 once the engine is up; vLLM 503s it while the model is +# still loading, so a green check means ready-to-serve, not process-alive. +# curl ships in the image. +HealthCmd=curl -sf http://127.0.0.1:8000/health +HealthInterval=30s +HealthTimeout=10s +HealthRetries=3 +HealthStartPeriod=600s +Pull=never + +[Service] +# First start reads ~10 GiB of weights and compiles the CUDA graphs. +TimeoutStartSec=900 +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=default.target diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index 2fa5aea..accac4f 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -319,6 +319,12 @@ def services_for_path(path: str, target: str) -> set[str]: # its unit and fleet settings live in one .container file. _add(services, "frank-large") return services + if len(parts) >= 2 and parts[1] == "frank-small": + # The service is a rootless Quadlet container since 2026-09-18 + # (vLLM); its unit lives in one .container file. The old root + # unit's trio is in scripts/ and maps there. + _add(services, "frank-small") + return services if len(parts) >= 2 and parts[1] == "observability": # The five observability containers became Quadlet units on # 2026-09-11. A unit change selects exactly its own service; @@ -402,11 +408,13 @@ def services_for_path(path: str, target: str) -> set[str]: # change to it redeploying alertmanager is the only path that # reaches it on the host. _add(services, "alertmanager") - if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}: - # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve - # files here; each one selects frank-small only. The ik_llama.cpp - # trio that lived here for two days is in scripts/.AGENTTRASH/, - # which is archive material and plans nothing. + if name in {"frank-small.env", "serve-frank-small.sh", + "download-frank-small-vllm.sh"}: + # vLLM frank-small (2026-09-18) keeps its env, entrypoint and + # fetch script here; each one selects frank-small only. The + # unit is the quadlet under dankclaw/frank-small/, and the old + # root unit file is in scripts/.AGENTTRASH/, which is archive + # material and plans nothing. _add(services, "frank-small") if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}: # llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index c1216ac..1cae4f5 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1139,28 +1139,90 @@ wait_for_llm_large_ready() { return 0 } -# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a -# ROOT unit: the runtime is a native CUDA binary, so there is no container -# image to install and no generated unit to avoid. relink_unit points -# /etc/systemd/system/frank-small.service at the repo file and boot persistence -# is that unit's [Install] section. +# deploy_llm_small — frank-small is a rootless Quadlet running vLLM on the +# RTX 3090 since 2026-09-18: the runtime is the digest-pinned +# vllm/vllm-openai gemma4-unified image, and the unit is generated from +# dankclaw/frank-small/quadlet/. It replaces the mainline-llama.cpp +# deployment, which ran as a ROOT unit. # -# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is -# not shared: frank-large's halogen engine runs on the gfx1151 iGPU. -LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}" +# TWO MANAGERS, ONE UNIT NAME. The llama.cpp incarnation ran as a ROOT unit +# whose file is a symlink into this checkout; the vLLM unit is a USER +# (quadlet) unit. Both must never exist at once — they would race for port +# 7771 — so the root unit is retired first: stop, disable, move the /etc +# symlink into the home trash (never `rm`), reload the root manager. This +# host's `systemctl disable` has been observed to delete the unit file +# itself (2026-08-26 incident), so the move tolerates a file that is +# already gone. +# +# The payloads are fetched and sha256-verified by +# scripts/download-frank-small-vllm.sh before first start; the handler +# refuses to launch a unit that cannot load its model or its drafter. +LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/gemma4-12b-qat-w4a16-ct/model.safetensors}" +LLM_SMALL_DRAFT="${LLM_SMALL_DRAFT:-/home/dank/models/gemma4-12b-assistant-bf16/model.safetensors}" +# Base image: the gemma-x86_64-cu129 build (vLLM 0.22.1rc1.dev357+ +# g74b5964f0) — the image that boots this checkpoint. The June 3 +# gemma4-unified build predates it and fails at weight load (missing +# vision_embedder.patch_dense); the Sep 15 cu129-nightly fails at import +# (torchvision ABI). The unit runs a LOCAL BUILD on top of this base — +# dankclaw/frank-small/Dockerfile installs the vllm[audio] extra, which +# the base ships without. +VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:e24d3d8e367ebc2a6ccfc8cf3cecee858a30a6c66b5b1642d85363cd745e001e" +VLLM_SMALL_BASE_ALIAS="localhost/vllm/vllm-openai:gemma4-unified" +VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small" + +retire_root_llm_small() { + local trash="$HOME/.AGENTTRASH/20260918-vllm-cutover/host-units" + run_privileged systemctl stop frank-small.service 2>/dev/null || true + run_privileged systemctl disable frank-small.service 2>/dev/null || true + if [[ -e /etc/systemd/system/frank-small.service || -L /etc/systemd/system/frank-small.service ]]; then + mkdir -p "$trash" + run_privileged mv -- /etc/systemd/system/frank-small.service \ + "$trash/frank-small.service" + echo "retired root unit frank-small.service -> $trash" + else + # This host's `systemctl disable` is allowed to delete the unit file + # outright (2026-08-26 incident), so there is often nothing left to + # move. Say so: previously no message meant no evidence either way. + echo "root unit frank-small.service was already removed by disable" + fi + run_privileged systemctl daemon-reload +} deploy_llm_small() { if [[ ! -f "$LLM_SMALL_MODEL" ]]; then echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2 - echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2 + echo "frank-small: run scripts/download-frank-small-vllm.sh first (main model + MTP assistant, sha256-verified);" >&2 echo "frank-small: refusing to start a unit that cannot load." >&2 return 1 fi - relink_unit frank-small - run_privileged systemctl daemon-reload - run_privileged systemctl enable frank-small.service - run_privileged systemctl restart frank-small.service - wait_for_health frank-small http://127.0.0.1:7771/health 900 + if [[ ! -f "$LLM_SMALL_DRAFT" ]]; then + echo "ERROR: MTP assistant absent at $LLM_SMALL_DRAFT" >&2 + echo "frank-small: run scripts/download-frank-small-vllm.sh first;" >&2 + echo "frank-small: refusing to start a unit that cannot draft." >&2 + return 1 + fi + retire_root_llm_small + # The predecessor can also be THIS quadlet -- a redeploy of a running + # frank-small -- so stop the user unit before waiting on the port. On a + # first cutover it is not running and this is a no-op. + systemctl --user stop frank-small.service 2>/dev/null || true + # Pull the base by digest, re-tag it to the FROM ref the Dockerfile + # expects, then build the derived image the quadlet runs: the quadlet + # path cannot resolve registry-prefixed refs from local storage at + # create time (same workaround as halogen), and Pull=never keeps boots + # off the registry. The build needs network for pip (the vllm[audio] + # extra); podman build has it on by default. + podman pull "$VLLM_SMALL_BASE_IMAGE" + podman tag "$VLLM_SMALL_BASE_IMAGE" "$VLLM_SMALL_BASE_ALIAS" + podman build --pull=never -t "$VLLM_SMALL_IMAGE" \ + -f dankclaw/frank-small/Dockerfile dankclaw/frank-small + # Nothing may start until the predecessor has released the port. + wait_port_free 7771 || return 1 + install_quadlet frank-small dankclaw/frank-small/quadlet frank-small + # No enable: generated units cannot be enabled on this host (v1 quadlet + # generator); boot persistence comes from [Install] in the unit file. + systemctl --user restart frank-small.service + wait_for_health frank-small http://127.0.0.1:7771/health 1200 } # deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a diff --git a/scripts/download-frank-small-vllm.sh b/scripts/download-frank-small-vllm.sh new file mode 100755 index 0000000..4d811b0 --- /dev/null +++ b/scripts/download-frank-small-vllm.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Fetch the frank-small vLLM payloads on dankclaw (RTX 3090, since 2026-09-18): +# +# google/gemma-4-12B-it-qat-w4a16-ct +# compressed-tensors w4a16 (int4, group 32) QAT checkpoint — the model +# vLLM's gemma4_unified implementation loads natively. One 9.56 GiB +# safetensors plus tokenizer/config. +# +# google/gemma-4-12B-it-assistant +# the MTP draft/assistant (gemma4_unified_assistant, 4 layers) in +# bf16 — Google's original. 806 MiB. The w4a16-ct re-quant +# (kunhunjon) that was used first drafts at acceptance 1.00: vLLM's +# MTP drafter path does not apply the Marlin w4a16 kernel to the +# assistant's linears (no "Using MarlinLinearKernel" after the +# drafter load), so its packed int4 weights are misread. The bf16 +# assistant accepts at 2.80 (measured 2026-09-19). +# +# Both repos are public and ungated; no token is needed. Every fetch is -f +# (an HTTP error is a failure, not a 153-byte "artifact"), resumable, and the +# two LFS payloads are sha256-verified against the Hub's declared oid — the +# discipline from scripts/download-frank-medium-gguf.sh. +# +# The serve script refuses to start when either payload is absent, so a +# half-finished run of this script reads as a missing file, not a runtime +# fault. +set -euo pipefail + +MODELS_DIR="${MODELS_DIR:-/home/dank/models}" +LOG_DIR="$MODELS_DIR/logs" +mkdir -p "$LOG_DIR" +LOG="$LOG_DIR/$(basename "$0" .sh).log" + +log() { echo "$(date -Is) $*" | tee -a "$LOG"; } + +# fetch [sha256] [size] +fetch() { + local repo="$1" dir="$2" file="$3" sha="${4:-}" size="${5:-}" + mkdir -p "$dir" + if [[ -n "$size" && -f "$dir/$file" ]] && [[ "$(stat -c%s "$dir/$file")" == "$size" ]]; then + log "skip $dir/$file (size $size matches)" + else + log "fetch $repo/$file" + curl -fsSL -C - "https://huggingface.co/$repo/resolve/main/$file" -o "$dir/$file" + log "fetched $dir/$file ($(stat -c%s "$dir/$file") bytes)" + fi + if [[ -n "$sha" ]]; then + local actual + actual="$(sha256sum "$dir/$file" | cut -d' ' -f1)" + if [[ "$actual" != "$sha" ]]; then + log "SHA256 MISMATCH $dir/$file: $actual != $sha" + return 1 + fi + log "sha256 ok $dir/$file" + fi +} + +MAIN_REPO=google/gemma-4-12B-it-qat-w4a16-ct +ASSISTANT_REPO=google/gemma-4-12B-it-assistant +MAIN_DIR="$MODELS_DIR/gemma4-12b-qat-w4a16-ct" +ASSISTANT_DIR="$MODELS_DIR/gemma4-12b-assistant-bf16" + +# Hub-declared LFS oids (sha256), 2026-09-18. +fetch "$MAIN_REPO" "$MAIN_DIR" model.safetensors \ + 60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3 10264229896 +fetch "$MAIN_REPO" "$MAIN_DIR" tokenizer.json \ + cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f 32169626 +for f in config.json generation_config.json chat_template.jinja \ + processor_config.json tokenizer_config.json recipe.yaml; do + fetch "$MAIN_REPO" "$MAIN_DIR" "$f" +done + +fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" model.safetensors \ + 3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df 845719296 +fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" tokenizer.json \ + c001d9ada50af662c94f5ab17ec7e09f6438d1bec8246c47fee6510693d8de35 32169884 +for f in config.json generation_config.json tokenizer_config.json; do + fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" "$f" +done + +log "ALL FETCHED OK" diff --git a/scripts/frank-small.env b/scripts/frank-small.env index 42d8431..bbf71a0 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -1,112 +1,81 @@ -# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS- -# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP -# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17. +# Tunables for frank-small (since 2026-09-18): vLLM serving +# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format +# Google publishes "for native, optimized inference with vLLM") with the +# google bf16 MTP assistant, on Dankclaw's RTX 3090. # -# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86: -# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \ -# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON -# cmake --build build-cuda --target llama-server -j -# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash -# attention, and without the flag quantized-KV prefill collapses ~12x on -# CUDA (see the CUDA lesson in AGENTS.md). +# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM +# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh), +# with the vllm[audio] extra installed on top (dankclaw/frank-small/ +# Dockerfile) so the model's native audio path works. That image carries +# the native gemma4_unified implementation, the Gemma4MTPModel drafter, +# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada +# card. The older gemma4-unified build (June 3) predates this checkpoint +# and fails at weight load; the cu129-nightly build (Sep 15) fails at +# import (torchvision ABI); this one is the image that boots. # -# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M -# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv -# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, -# per-layer token embeddings (PLE) inside the file, and a sliding-window -# attention pattern (attention.sliding_window = 1024) -- most layers cache -# only a 1024-position window, which is why the KV pool is a fraction of -# the full-ctx x all-layers number. +# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window +# (1024-token) + 1 full-attention, int4 group-32 weights with bf16 +# embeddings, vision and audio. The fleet's image path lives here: the model +# ingests image (and audio) content natively, so no projector file exists +# any more and nothing external has to load. # -# MTP: the Multi-Token Prediction head ships as a SEPARATE file -# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 -# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one -# is passed via --model-draft with --spec-type draft-mtp, so the draft -# context loads the assistant model. The same --spec-type value chains the -# draftless ngram-mod matcher, which attaches unconditionally; two -# context-backed drafters cannot share one process, so draft-mtp,ngram-mod -# is the only legal shape that carries both. +# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the +# same int4 w4a16 compressed-tensors format — the card's compatibility rule +# is that a QAT target must draft with a QAT drafter at the same precision. +# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4 +# MTP path, not a generic draft model. # -# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four -# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346 -# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot -# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the -# card, and the KV pool allocates upfront -- so 4 is the max. Explicit -# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, -# which would shrink each slot below the full 262144 window once -# parallel > 1. +# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window +# for as many concurrent sessions as the card holds. +# MAX_MODEL_LEN=262144 the model's max_position_embeddings +# GPU_MEM_UTIL=0.95 the card is dedicated to this service +# KV_CACHE_DTYPE=auto bf16 KV. fp8_e4m3 would halve the KV charge +# (289,756 -> 623,666-token pool, one -> two +# full-context sessions), but the 3090 is +# sm_86 (Ampere) and has no native fp8: the +# TRITON_ATTN backend Gemma4 forces emits +# cvt.e4m3x2 instructions that exist only from +# sm_89 (Ada), so fp8 KV cannot boot on this +# card (measured 2026-09-19; the "sm_89 +# patch" experiment is in .AGENTTRASH). +# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this, +# caps full-context concurrency +# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill +# faster but grow the activation arena that +# competes with the KV pool for the card +# The boot log prints the KV pool size and "Maximum concurrency for N tokens +# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool / +# max_model_len) and undercounts: the 42 sliding-window layers only retain a +# 1024-token window per request, so the true full-context session count is +# higher. Verify it behaviourally (concurrent long requests) on every +# restart, not from these comments. # -# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M -# params; it carries an audio encoder too). The fleet's image path lives -# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio -# WORKS: input_audio content parts transcribe verbatim (verified -# 2026-09-17 with a 16k mono WAV). +# Sampling: greedy by default (fleet contract) via +# --override-generation-config; a client that sends its own temperature gets +# its own, and the harness applies per-model overrides from models.yml. # -# Sampling: greedy by default (the fleet contract); a client that sends a -# temperature gets its own, and the harness applies per-model overrides -# from models.yml. -# -# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP -# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in +# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the +# ROCm hosts read those bare vars — see the 2026-08-13 incident in # AGENTS.md). This box is CUDA, but the rule keeps env files portable. -# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86, -# FA_ALL_QUANTS=ON. -LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server +# Container paths, bind-mounted by the quadlet. The host paths are declared +# in the quadlet and the deploy handler, not here: this file is read inside +# the container. +MODEL_DIR=/models/main +DRAFT_DIR=/models/assistant -# Model package. The MTP sidecar file is required for speculative decoding. -MODEL_DIR=/home/dank/models/Gemma4-12B-QAT -MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf -MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf -DRAFT_FILE=mtp-gemma-4-12B-it.gguf - -# Fleet identity and network. 7771 is this service's own endpoint on -# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and -# what clients ask for -- the harness entry in models.yml names it. +# Fleet identity. 8000 is the in-container port; the quadlet publishes it as +# host 7771, the endpoint Hermes, the harness and the homepage probe have +# always used. The alias is the id at /v1/models and what clients ask for. ALIAS=frank-small BIND_ADDR=0.0.0.0 -LISTEN_PORT=7771 +LISTEN_PORT=8000 -# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total -# pool; --parallel divides it). See the SIZING note above for the fit -# arithmetic; a fifth slot does not fit. -CTX_SIZE=1048576 -PARALLEL=4 -CACHE_TYPE_K=q8_0 -CACHE_TYPE_V=q8_0 - -# Batch geometry: 8192 logical / 1024 physical. Measured on this card at -# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV): -# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962, -# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp -# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be -# tried. Idle VRAM is 23,300 MiB at this geometry. -BATCH_SIZE=8192 -UBATCH_SIZE=1024 - -# CPU threads: this service runs entirely on the GPU (NGL=999), so threads -# cover tokenization, sampling and the HTTP layer only. Deliberately modest -# -- this host's CPU is shared with halogen's engine, which is CPU-hungry. -NUM_THREADS=8 -THREAD_BATCH=8 - -# Every layer on the 3090, with flash attention (required for q8 KV). -NGL=999 -FLASH_ATTN=on - -# Speculative decoding: the MTP sidecar (draft context) chained with the -# draftless ngram-mod matcher. Draft depth and busy-wait level are made -# explicit here (both were the implicit build defaults: 3 and 50). -# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s) -# than depth 3 + poll 50 (257-302 t/s) — the longer verification round -# costs more attention re-read at long context than the extra accepted -# token earns. Keep 3/50. -SPEC_TYPE=draft-mtp,ngram-mod -SPEC_DRAFT_N_MAX=3 -POLL_LEVEL=50 - -# Greedy default (fleet contract); reasoning is extracted into -# reasoning_content with an unlimited budget (the model decides). -TEMPERATURE=0.0 -REASONING_FORMAT=deepseek -REASONING_BUDGET=-1 +# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling, +# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers). +MAX_MODEL_LEN=262144 +GPU_MEM_UTIL=0.95 +KV_CACHE_DTYPE=auto +MAX_NUM_SEQS=16 +MAX_NUM_BATCHED_TOKENS=16384 +SPEC_NUM_TOKENS=4 diff --git a/scripts/frank-small.service b/scripts/frank-small.service deleted file mode 100644 index 14ddd9c..0000000 --- a/scripts/frank-small.service +++ /dev/null @@ -1,20 +0,0 @@ -[Unit] -Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small) -After=network-online.target -Wants=network-online.target - -[Service] -Type=simple -User=dank -Group=dank -# CUDA pins its staging buffers, so the process needs the memlock budget even -# though the weights themselves are copied into VRAM. Parity with the halogen -# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit. -LimitMEMLOCK=infinity -Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env" -ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh -Restart=on-failure -RestartSec=10 - -[Install] -WantedBy=multi-user.target diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 0a9aade..c5fcb03 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,43 +1,30 @@ #!/usr/bin/env bash -# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as -# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp. +# Serve google/gemma-4-12B-it-qat-w4a16-ct as frank-small on Dankclaw's RTX +# 3090 via vLLM (since 2026-09-18), with the google bf16 MTP assistant +# for speculative decoding. # -# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M -# weights plus the BF16 vision projector and q8_0 KV inside the card's -# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU -# belongs to the halogen engine that serves frank-large. +# This script is the CONTAINER ENTRYPOINT of the frank-small quadlet: the +# vllm/vllm-openai image's own entrypoint is `vllm serve`, and the quadlet +# replaces it with this file (mounted read-only out of the repo checkout) so +# the flag set stays in git. The image pins the gemma4-unified vLLM build, +# which is what loads gemma4_unified natively and runs the Gemma 4 MTP path. # -# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file -# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is -# passed via --model-draft with --spec-type draft-mtp: the draft context -# loads the assistant model. The same --spec-type value chains the -# draftless ngram-mod matcher, which attaches unconditionally; two -# context-backed drafters cannot share one process, so this chain is the -# only shape that carries both. +# MTP: the drafter is a separate 4-layer assistant checkpoint in bf16 +# (Google's original gemma-4-12B-it-assistant). It goes in +# --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path, not a +# generic draft model. The w4a16-ct re-quant of the assistant was tried +# first and drafts at acceptance 1.00 (the drafter path does not apply the +# Marlin kernel to its linears); the bf16 assistant accepts at 2.80. # -# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot -# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below) -# maximises the TOTAL pool instead, which would divide a smaller window -# across slots. The 4-slot layout is measured to fit the card (see the -# SIZING note in frank-small.env); more slots would fail the KV allocation. +# Sampling: greedy by default (fleet contract) via +# --override-generation-config; clients that send their own temperature get +# their own, and the harness overrides per model from models.yml. # -# Vision + audio: --mmproj loads the BF16 projector, which also carries -# an audio encoder — input_audio content parts transcribe verbatim -# (verified 2026-09-17). The fleet's image path lives here, which is why -# halogen no longer sets HALOGEN_VISION_TOWER. -# -# Sampling: greedy by default (the fleet contract). Clients that send their own -# temperature get their own values, and the harness overrides per model from -# models.yml. --jinja is pinned even though this build defaults it on: the -# model's own template supplies reasoning_effort, preserve_thinking and the -# tool-call grammar, and a future build that flips that default must not -# silently take tools away. -# -# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it. +# Tunables come from frank-small.env (mounted beside this file by the +# quadlet) unless FRANK_SMALL_ENV_FILE overrides the path. set -euo pipefail -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}" +ENV_FILE="${FRANK_SMALL_ENV_FILE:-/scripts/frank-small.env}" if [[ -f "$ENV_FILE" ]]; then set -a # shellcheck disable=SC1090 @@ -46,93 +33,51 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}" -: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}" -: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}" -: "${DRAFT_FILE:=}" +: "${MODEL_DIR:=/models/main}" +: "${DRAFT_DIR:=/models/assistant}" : "${ALIAS:=frank-small}" : "${BIND_ADDR:=0.0.0.0}" -: "${LISTEN_PORT:=7771}" -: "${CTX_SIZE:=}" -: "${PARALLEL:=1}" -: "${CACHE_TYPE_K:=f16}" -: "${CACHE_TYPE_V:=f16}" -: "${BATCH_SIZE:=2048}" -: "${UBATCH_SIZE:=512}" -# Decode-shape knobs: the MTP drafter's per-round proposal length (build -# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's -# busy-wait level (build default 50). -: "${SPEC_DRAFT_N_MAX:=3}" -: "${POLL_LEVEL:=50}" -: "${NUM_THREADS:=8}" -: "${THREAD_BATCH:=8}" -: "${NGL:=999}" -: "${FLASH_ATTN:=on}" -: "${SPEC_TYPE:=draft-mtp}" -: "${TEMPERATURE:=0.0}" -: "${REASONING_FORMAT:=deepseek}" -: "${REASONING_BUDGET:=-1}" +: "${LISTEN_PORT:=8000}" +: "${MAX_MODEL_LEN:=262144}" +: "${GPU_MEM_UTIL:=0.95}" +: "${KV_CACHE_DTYPE:=auto}" +: "${MAX_NUM_SEQS:=16}" +: "${MAX_NUM_BATCHED_TOKENS:=16384}" +: "${SPEC_NUM_TOKENS:=4}" +: "${ENFORCE_EAGER:=0}" # Fail loudly rather than half-loading a model into VRAM: a missing file here # is a download that has not finished, and the unit's restart policy would # otherwise turn that into a crash loop that reads like a runtime fault. -for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do +for path in "$MODEL_DIR/model.safetensors" "$MODEL_DIR/config.json" \ + "$DRAFT_DIR/model.safetensors" "$DRAFT_DIR/config.json"; do [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; } done -if [[ -n "${DRAFT_FILE:-}" ]]; then - [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; } + +SPEC_CONFIG="{\"method\":\"mtp\",\"model\":\"${DRAFT_DIR}\",\"num_speculative_tokens\":${SPEC_NUM_TOKENS}}" + +VLLM_FLAGS=( + serve "$MODEL_DIR" +--served-model-name "$ALIAS" +--host "$BIND_ADDR" +--port "$LISTEN_PORT" +--max-model-len "$MAX_MODEL_LEN" +--gpu-memory-utilization "$GPU_MEM_UTIL" +--kv-cache-dtype "$KV_CACHE_DTYPE" +--max-num-seqs "$MAX_NUM_SEQS" +--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" +--speculative-config "$SPEC_CONFIG" +--reasoning-parser gemma4 +--enable-auto-tool-choice +--tool-call-parser gemma4 +--override-generation-config '{"temperature": 0}' +) +# ENFORCE_EAGER=1 skips Inductor/Triton compilation and CUDA graph capture +# (slower steady-state decode, faster and more robust boot). The 3090 is +# sm_86 (Ampere) — no native fp8 — so KV stays bf16 and the default +# graphed path is the right one; the knob is a boot-debugging lever. +if [[ "$ENFORCE_EAGER" == "1" ]]; then + VLLM_FLAGS+=(--enforce-eager) fi -# The UVM device node is created lazily by nvidia-modprobe on the first CUDA -# open after boot; make sure it exists before the server needs it. -nvidia-modprobe -u -c=0 2>/dev/null || true - -MMPROJ_ARGS=() -if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then - MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH") -fi - -SPEC_ARGS=() -if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then - SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX") -fi - -DRAFT_ARGS=() -if [[ -n "${DRAFT_FILE:-}" ]]; then - DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE") -fi - -# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against -# whatever the weights, projector, draft context and compute buffers leave -# on the card; the engine prints the chosen value at startup. -if [[ -n "${CTX_SIZE:-}" ]]; then - CTX_ARGS=(--ctx-size "$CTX_SIZE") -else - CTX_ARGS=(--fit on) -fi - -exec "$LLAMA_BIN" \ - --model "$MODEL_DIR/$MODEL_FILE" \ - "${MMPROJ_ARGS[@]}" \ - "${DRAFT_ARGS[@]}" \ - --alias "$ALIAS" \ - --host "$BIND_ADDR" \ - --port "$LISTEN_PORT" \ - --gpu-layers "$NGL" \ - --flash-attn "$FLASH_ATTN" \ - "${CTX_ARGS[@]}" \ - --parallel "$PARALLEL" \ - --batch-size "$BATCH_SIZE" \ - --ubatch-size "$UBATCH_SIZE" \ - --threads "$NUM_THREADS" \ - --threads-batch "$THREAD_BATCH" \ - --poll "$POLL_LEVEL" \ - --cache-type-k "$CACHE_TYPE_K" \ - --cache-type-v "$CACHE_TYPE_V" \ - "${SPEC_ARGS[@]}" \ - --jinja \ - --metrics \ - --temp "$TEMPERATURE" \ - --reasoning-format "$REASONING_FORMAT" \ - --reasoning-budget "$REASONING_BUDGET" +exec vllm "${VLLM_FLAGS[@]}" diff --git a/serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md b/serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md new file mode 100644 index 0000000..615c9f4 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md @@ -0,0 +1,211 @@ +--- +type: reference +title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors) +description: The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. +tags: + - llm + - frank-small + - dankclaw + - vllm + - rtx3090 + - gemma + - mtp + - vision + - audio +timestamp: 2026-09-19 +--- + +# frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16) + +> **ACTIVE since 2026-09-19.** Replaces [[frank-small-gemma4-12b]] +> (llama.cpp, now the rollback target). Same port (7771), same alias +> (`frank-small`), same client contract surface — the runtime, the model +> payload, the drafter and the service shape (root systemd unit → rootless +> Quadlet) all changed. + +## Identity + +| what | value | +|---|---| +| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d7…`) | +| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (root systemd unit retired 2026-09-19) | +| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) | +| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` | +| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` | +| port | 7771 → 8000 (alias `frank-small`; `frank-large` keeps 7770) | +| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** | +| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) | + +## Why vLLM + +- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin + kernel is its native int4 GEMM path. llama.cpp could not load it at all. +- Native audio encoder in the checkpoint — the fleet's only audio-capable + endpoint moves here (llama.cpp's path needed the mmproj audio tower). +- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on + this host had no graph support. + +## The drafter-Marlin bug (why the assistant is bf16) + +The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`) +**cannot draft in vLLM**: the drafter's linear layers are loaded through a +path that does not apply the Marlin kernel, so the packed int4 weights are +misread as bf16 and every draft is garbage — measured acceptance **1.00** +(pure pass-through). There is no "Using MarlinLinearKernel" log line after +the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`) +has no quantization to miss, and measures **acceptance 2.80–3.56** +(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta +(634 → 845 MB) is absorbed by pool rounding: both assistants yield the +identical 289,647-token KV pool. + +## KV arithmetic — why two full-context sessions + +Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA +16 heads × 512 dim. The SWA layers cache only the 1024-token window, so +per-token KV is dominated by the full-attention layers: +6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16). +289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not +sound like it fits a 24 GiB card — it fits because vLLM's SWA pool +shares the 1024-window ring across all concurrent requests, so the +marginal cost of a *second* 213k session is ~6 full-attention GiB, not +a second copy of everything. + +Measured (salted, 2026-09-19): + +| load | KV usage | result | +|---|---|---| +| 1 × 213k | ~35 % | runs | +| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) | +| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues | + +**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp +deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the +bytes, and vLLM's pool arithmetic is coarser. This is the accepted +trade for the native w4a16 format + audio + CUDA graphs.) + +## Measured throughput (salted, 2026-09-19) + +| probe | tokens | wall | rate | +|---|---|---|---| +| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) | +| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** | +| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** | +| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream | +| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) | +| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** | + +- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention + layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient + as llama.cpp's flash-attn path here. +- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per + token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is + right at it. At normal depths decode is far faster (the 67k-depth probe + is in the bench log). MTP acceptance ~3.2–3.6 is folded into these + numbers. +- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT + scale linearly in tokens — BPE compresses the repetition harder as the + span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit + predicts). Size fillers with the model's own tokenizer + (`AutoTokenizer.encode` on the exact prefix), not char counts. +- `POST /flush_cache` is **404** in this build — salt every probe + (uuid prefix) instead. + +## The image + +`dankclaw/frank-small/Dockerfile` (repo) — 16 comment lines + `FROM +localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install +--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY +lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone: +`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed +after the driver swap, so no arch override is needed. + +## No nvidia-container-toolkit — manual CUDA mounts + +Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA +by hand: + +- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, + `/dev/nvidia-uvm-tools` +- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only + into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/` +- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1` + +If the driver is upgraded on the host, these mount sources move with it +(they are versionless symlinks) — no unit change needed. + +## Host driver note + +Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box). +Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces +595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for +the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact, +not a hardware mystery. + +## Deployed config + +Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh` +(mounted into the container at `/scripts/`, which is the entrypoint) + +`dankclaw/frank-small/quadlet/frank-small.container`. Key values: +`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by +compressed-tensors checkpoints and fp8 is hardware-dead on sm_86), +`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via +`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH` +is **removed**. The serve script refuses to start if the weights, the +assistant, or the vllm binary are missing. + +## Client contract + +- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning + parser), not `reasoning_content` — the harness maps it via + `compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`. + The field is empty when the model emits no thinking tag (behavior, not + a parser bug). +- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are + accepted (200, not 400) — vLLM ignores the unknown key. +- Every `reasoning_effort` level accepted (all seven probed 2026-09-17). +- Tools → `tool_calls` (verified `get_weather`). +- Image → 200, described correctly (native vision tower). +- Audio → 200, a 440 Hz WAV tone described correctly (native audio + encoder; verified in the derived image). +- Harness: `contextWindow: 262144`, `maxTokens: 32768`, + `input: [text, image, audio]`. + +## Boot ledger + +t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a +bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11; +KV pool 289,647 tokens). Health is 503 until ready — do not kill the +container while it loads. + +## Troubleshooting + +- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again + (drafter-Marlin bug). Confirm the assistant mount points at + `gemma4-12b-assistant-bf16`. +- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH` + must be absent. 8.6 is correct. +- **Audio 400/500** — the container is running the *base* image, not the + derived `:frank-small` (missing `av`). Check the deployed image digest + is `8dad46d7…`. +- **CUDA not visible in container** — a `Device=` line or a lib mount + dropped out of the unit after an edit; the unit is the source of truth, + the host-side generated unit is not. +- Logs: `journalctl --user -u frank-small` (rootless Quadlet; `podman + logs` covers only the current `--replace --rm` instance). + +## Rollback + +[[frank-small-gemma4-12b]] — the llama.cpp deployment. Its weights, +mmproj and MTP sidecar are still on the host under +`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at +`scripts/.AGENTTRASH/frank-small.service.llama.cpp`. Restoring it means +reinstalling that unit and stopping the Quadlet (one owner per port). + +## References + +- `scripts/serve-frank-small.sh` — the actual exec line +- `dankclaw/frank-small/Dockerfile` — the audio-derived image +- `dankclaw/frank-small/quadlet/frank-small.container` — the unit +- [[frank-small-gemma4-12b]] — RETIRED predecessor (rollback) +- [[halogen-flash-server]] — the sibling endpoint on the iGPU +- [[VLLM Overview]] / [[VLLM Index]] — runtime background diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index d60baa6..59bcd5c 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -5,9 +5,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-16): process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B- Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next -w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, -ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port -7771). +w4b on the gfx1151 iGPU, port 7770) and vLLM (frank-small, +google/gemma-4-12B-it-qat-w4a16-ct with the bf16 MTP assistant on the RTX +3090, port 7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -35,10 +35,10 @@ gfx1151 iGPU 64 GiB unified memory] frank-large, port 7770] F[peonist-ai Qwen3.8-Flash-Next w4b 2 x 200192 ctx over one 400384-position pool] - G[mainline llama.cpp 37b53fd + G[vLLM gemma-x86_64-cu129 frank-small, port 7771] - H[Qwen3.8-27B GSQ-RCO IQ3_S -131072 ctx / 1 slot, MTP + vision] + H[gemma-4-12B-it-qat-w4a16-ct +262144 ctx / 2 full-context sessions, MTP + vision + audio] E2 --> E --> F E2 --> G --> H end @@ -87,11 +87,15 @@ frank-small, port 7771] overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting -- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17 - (root systemd unit, port 7771): mainline llama.cpp serving - HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX - 3090, with the separate MTP assistant sidecar chained behind ngram-mod, - q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB +- [[frank-small-vllm-gemma4]] — ACTIVE frank-small runbook since 2026-09-19 + (rootless Quadlet, port 7771): vLLM serving + google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + with Google's bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-token + pool, 2 concurrent full-context sessions, native text+image+audio +- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): the + prior frank-small runbook (2026-09-17 → 09-19), mainline llama.cpp serving + HauhauCS Gemma4-12B-QAT-Uncensored Q4_K_M with the MTP sidecar, q8_0 KV, + 4 slots of 262k - [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index a2957b5..4eb4e74 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -166,11 +166,17 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since - 2026-09-17: mainline llama.cpp serving - HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M - (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on +- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since + 2026-09-19: vLLM serving + google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2 + full-context sessions, native text+image+audio) on Dankclaw's RTX 3090, port 7771 +- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): + mainline llama.cpp serving + HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M + (MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on + Dankclaw's RTX 3090 - [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index 7028dbc..6fcf600 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -64,7 +64,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" { run python3 "$PLANNER" --target dankclaw --paths \ - scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh + dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \ + scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh [ "$status" -eq 0 ] [ "$output" = "frank-small" ] @@ -72,7 +73,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" # that name was retired with the FreeToken runtime on 2026-09-15 and must # not come back through a planner mapping. run python3 "$PLANNER" --target dankserver --paths \ - scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh + dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \ + scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh [ "$status" -eq 0 ] [ -z "$output" ] } diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index 5991dd8..804215f 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -72,7 +72,11 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "frank-large loads no vision tower: the fleet's image path is frank-small" { ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET" - grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh" + # frank-small's vLLM checkpoint (2026-09-18) is natively multimodal — + # image content is ingested by the model itself, so the seam that used + # to pin --mmproj now pins the projector's absence. + ! grep -q 'mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh" + grep -q '^MODEL_DIR=/models/main$' "$REPO_ROOT/scripts/frank-small.env" } @test "the deploy handler retires the previous ROOT unit without deleting it" { diff --git a/tests/frank_small_unit.bats b/tests/frank_small_unit.bats index 01163b1..20651c5 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_small_unit.bats @@ -1,115 +1,157 @@ #!/usr/bin/env bats -# Seam: the frank-small mainline-llama.cpp root unit on dankclaw. -# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large -# went back to the halogen engine on the iGPU. This seam pins what the flags -# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet -# is involved), the repo-relative paths the unit executes, the separate client -# port, the MTP contract (the head is a separate sidecar file loaded as the -# draft model, chained behind ngram-mod), the vision handoff from frank-large, -# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card. +# Seam: the frank-small vLLM quadlet on dankclaw (since 2026-09-18). +# frank-small is the RTX 3090 endpoint. It now runs vLLM (the pinned +# gemma4-unified image) serving google/gemma-4-12B-it-qat-w4a16-ct with the +# google bf16 MTP assistant, replacing the mainline-llama.cpp root +# unit (retired into scripts/.AGENTTRASH/). This seam pins what the flags +# alone do not say: boot persistence for the generated quadlet, the CUDA +# wiring this host needs (no nvidia-container-toolkit), the repo-relative +# entrypoint and env, the MTP contract (a separate QAT assistant checkpoint +# in --speculative-config, method "mtp"), the fleet's image path (the model +# is natively multimodal — no projector file), the full-window sizing, the +# separate client port, and the deploy handler's cutover order. load helpers -UNIT="$REPO_ROOT/scripts/frank-small.service" +QUADLET="$REPO_ROOT/dankclaw/frank-small/quadlet/frank-small.container" +DOCKERFILE="$REPO_ROOT/dankclaw/frank-small/Dockerfile" SERVE="$REPO_ROOT/scripts/serve-frank-small.sh" ENV_FILE="$REPO_ROOT/scripts/frank-small.env" +FETCH="$REPO_ROOT/scripts/download-frank-small-vllm.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" -@test "frank-small unit, serve script and env exist in scripts/" { - [ -f "$UNIT" ] +@test "frank-small quadlet, Dockerfile, entrypoint, env and fetch script exist" { + [ -f "$QUADLET" ] + [ -f "$DOCKERFILE" ] [ -f "$SERVE" ] [ -f "$ENV_FILE" ] + [ -f "$FETCH" ] + # The old root unit is retired, not deleted: it must live in the trash, + # not in scripts/, so the planner and relink_unit never see it. + [ ! -e "$REPO_ROOT/scripts/frank-small.service" ] + [ -e "$REPO_ROOT/scripts/.AGENTTRASH/frank-small.service.llama.cpp" ] } -@test "frank-small is boot-persistent, self-healing and unprivileged" { - grep -q '^WantedBy=multi-user.target$' "$UNIT" - grep -q '^Restart=on-failure$' "$UNIT" - grep -q '^RestartSec=' "$UNIT" - grep -q '^User=dank$' "$UNIT" - grep -q '^LimitMEMLOCK=infinity$' "$UNIT" +@test "frank-small is boot-persistent, self-healing and registry-pinned" { + grep -q '^WantedBy=default.target$' "$QUADLET" + grep -q '^Restart=on-failure$' "$QUADLET" + grep -q '^RestartSec=' "$QUADLET" + grep -q '^Pull=never$' "$QUADLET" + # The unit runs the localhost/-aliased LOCAL BUILD (base + vllm[audio] + # extra); the digest-pinned docker.io base lives only in the deploy + # handler, which pulls, re-tags and builds before restart. + grep -q '^Image=localhost/vllm/vllm-openai:frank-small$' "$QUADLET" + ! grep -q '^Image=docker.io' "$QUADLET" } -@test "frank-small runs the repo's own serve script with the repo's env" { - grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT" - grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT" +@test "frank-small wires CUDA without the nvidia-container-toolkit" { + # The four device nodes (0666 on this host, so rootless can open them)... + grep -q '^AddDevice=/dev/nvidia0$' "$QUADLET" + grep -q '^AddDevice=/dev/nvidiactl$' "$QUADLET" + grep -q '^AddDevice=/dev/nvidia-uvm$' "$QUADLET" + grep -q '^AddDevice=/dev/nvidia-uvm-tools$' "$QUADLET" + # ...plus the host driver's libcuda and NVML stubs. Without the NVML + # mount vllm serve dies at parser construction with "Failed to infer + # device type" (measured 2026-09-18). + grep -q 'libcuda.so.1.*:ro' "$QUADLET" + grep -q 'libnvidia-ml.so.1.*:ro' "$QUADLET" + grep -q '^Environment=NVIDIA_VISIBLE_DEVICES=all$' "$QUADLET" + # CUDA pins its staging buffers; memlock must be unlimited. + grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET" } -@test "frank-small offloads every layer to the 3090 with flash attention" { - grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE" - grep -q '^NGL=999$' "$ENV_FILE" - grep -q -- '--gpu-layers "$NGL"' "$SERVE" - grep -q '^FLASH_ATTN=on$' "$ENV_FILE" - grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE" - # Prometheus scrapes this endpoint; without --metrics there is no /metrics. - grep -q -- '--metrics' "$SERVE" +@test "frank-small runs the repo's own entrypoint with the repo's env" { + grep -q '^ContainerEntrypoint=/scripts/serve-frank-small.sh$' "$QUADLET" + grep -q '^Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env$' "$QUADLET" + grep -q 'Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro' "$QUADLET" } -@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" { - grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE" - grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE" - # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4 - # nextn layers); it loads as --model-draft and shares the target's last - # layers' KV. ngram-mod is draftless, so the chain is legal. - grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE" - grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE" +@test "frank-small mounts the sha256-verified payloads read-only" { + grep -q 'Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro' "$QUADLET" + grep -q 'Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro' "$QUADLET" + # The fetch script verifies both LFS payloads against the Hub oids. + grep -q '60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3' "$FETCH" + grep -q '3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df' "$FETCH" +} + +@test "frank-small drafts with the QAT MTP assistant via method mtp" { + # The drafter is a separate 4-layer assistant checkpoint in the same + # int4 w4a16 compressed-tensors format (the card's compatibility rule: + # a QAT target drafts with a QAT drafter at the same precision), passed + # in --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path. + grep -qF -- '\"method\":\"mtp\"' "$SERVE" + grep -q -- '--speculative-config "$SPEC_CONFIG"' "$SERVE" + grep -q '^SPEC_NUM_TOKENS=4$' "$ENV_FILE" + # The gemma4 reasoning and tool-call parsers keep the fleet contract: + # reasoning_content on the wire and tools answered. + grep -q -- '--reasoning-parser gemma4' "$SERVE" + grep -q -- '--tool-call-parser gemma4' "$SERVE" + grep -q -- '--enable-auto-tool-choice' "$SERVE" } @test "frank-small carries the fleet's image path, and only it" { - grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE" - grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE" + # The model is natively multimodal (encoder-free unified): image content + # is ingested by the checkpoint itself, so there is no projector file + # any more — the seam that used to pin MMPROJ_PATH now pins its absence. + ! grep -q '^MMPROJ' "$ENV_FILE" ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE" } -@test "frank-small's slot grid is full-window and fits the 24 GiB card" { - local parallel cache_k cache_v ctx fixed per_slot - parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")" - cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")" - cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")" - ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ] - # Every slot must be the model's FULL trained window (262144), not a - # divided one -- CTX_SIZE is the total pool and --parallel divides it. - [ $((ctx / parallel)) -eq 262144 ] - [ $((ctx % parallel)) -eq 0 ] - [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ] - # Fit budget from the measured boot arithmetic on the 3090 (24576 - # MiB): one full-context slot idles at 11,996 MiB, four at 22,346, - # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part - # (weights, projector, sidecar draft, compute buffers, CUDA context) - # is 11996-3450 = 8546 MiB. If the config asks for more slots than - # the card holds, this inequality fails before deploy instead of the - # KV allocation failing at boot. - fixed=8546 - per_slot=3450 - [ $((fixed + parallel * per_slot)) -le 24576 ] - # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option, - # without which prefill collapses about 12x (see the CUDA lesson in - # AGENTS.md). The build-cuda tree must carry it. - grep -q '^FLASH_ATTN=on$' "$ENV_FILE" +@test "frank-small serves the full window with bf16 KV on a dedicated card" { + local maxlen kvdtype util seqs + maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")" + kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")" + util="$(sed -nE 's/^GPU_MEM_UTIL=([0-9.]+)$/\1/p' "$ENV_FILE")" + seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")" + # The model's FULL trained window (262144), not a divided one. + [ "$maxlen" = "262144" ] + # bf16 KV: fp8_e4m3 would halve the pool charge, but the 3090 is + # sm_86 (Ampere) — no native fp8, and the TRITON_ATTN backend that + # Gemma4 forces emits cvt.e4m3x2 instructions that exist only from sm_89. + [ "$kvdtype" = "auto" ] + # The 3090 is dedicated to this service, so the engine may take most of + # it; the KV pool, not the seq ceiling, caps full-context concurrency. + [ "$util" = "0.95" ] + [ "$seqs" -ge 4 ] } @test "frank-small runs on its own port beside frank-large's" { - grep -q '^LISTEN_PORT=7771$' "$ENV_FILE" + # 8000 is the in-container port; the quadlet publishes it as host 7771, + # the endpoint Hermes, the harness and the homepage probe use. + grep -q '^LISTEN_PORT=8000$' "$ENV_FILE" grep -q '^ALIAS=frank-small$' "$ENV_FILE" grep -q -- '--port "$LISTEN_PORT"' "$SERVE" grep -q -- '--host "$BIND_ADDR"' "$SERVE" - # The two endpoints cannot share a port, and only the halogen unit publishes - # 7770. + grep -q '^PublishPort=7771:8000$' "$QUADLET" + # The two endpoints cannot share a port, and only the halogen unit + # publishes 7770. grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE" ! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE" } -@test "the deploy handler installs the root unit and refuses a missing model" { +@test "the deploy handler cuts over from the root unit and gates on health" { grep -q 'deploy_llm_small' "$REMOTE_DEPLOY" - grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY" - grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY" - grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY" - grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY" + # The digest-pinned docker.io base is pulled and re-tagged to the FROM + # ref, then the Dockerfile is built into the localhost tag the quadlet + # runs (the vllm[audio] extra the base ships without). + grep -q 'VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:' "$REMOTE_DEPLOY" + grep -q 'VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"' "$REMOTE_DEPLOY" + grep -q 'podman tag "\$VLLM_SMALL_BASE_IMAGE" "\$VLLM_SMALL_BASE_ALIAS"' "$REMOTE_DEPLOY" + grep -q 'podman build --pull=never -t "\$VLLM_SMALL_IMAGE"' "$REMOTE_DEPLOY" + grep -q 'dankclaw/frank-small/Dockerfile' "$REMOTE_DEPLOY" + # The old ROOT unit is retired (stop, disable, trash, reload) before the + # user unit takes the port — two managers must never own frank-small. + grep -q 'retire_root_llm_small' "$REMOTE_DEPLOY" + grep -q 'wait_port_free 7771' "$REMOTE_DEPLOY" + # Missing payloads refuse the start, and the health gate is the real + # endpoint. + grep -q 'gemma4-12b-qat-w4a16-ct/model.safetensors' "$REMOTE_DEPLOY" + grep -q 'gemma4-12b-assistant-bf16/model.safetensors' "$REMOTE_DEPLOY" grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY" grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY" } -@test "frank-small's serve script refuses to start on a half-finished download" { +@test "frank-small's entrypoint refuses to start on a half-finished download" { grep -q 'frank-small: missing' "$SERVE" }