diff --git a/AGENTS.md b/AGENTS.md index cd688b5..227ef1f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on - the gfx1151 iGPU) and `frank-small` (vLLM on the RTX 3090). Caddy on + the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| |Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770| | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | -| Dankclaw | `frank-small` | vLLM `gemma-x86_64-cu129` dev build + audio extra (rootless Quadlet) | google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + Google's bf16 MTP assistant on the RTX 3090, 262144 window / 289,647-token bf16 KV pool / 2 full-context sessions, native text+image+audio — the fleet's multimodal endpoint | 7771 | +| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly. The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by -their own files — frank-large and frank-small by their `dankclaw/` -quadlet/Dockerfile directories, frank-medium by its unit/env/serve trio in `scripts/` — so unrelated +their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small +and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo; the value never enters the repo. @@ -232,10 +232,8 @@ merely wedged kernel usually still answers ARP. `~/.config/containers/systemd/frank-large.container` aside and `systemctl --user daemon-reload` — the quadlet generator re-creates that unit from the file on every reload, so the source is the lever, not `mask`. - - Dankclaw `frank-small` (rootless Quadlet since 2026-09-19, vLLM on the 3090): - `systemctl --user stop frank-small`, then move its - `~/.config/containers/systemd/frank-small.container` aside and - `systemctl --user daemon-reload` (same quadlet-source lever as frank-large). + - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): + `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - Dankserver `frank-medium` (system unit): `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium` 2. Bench. @@ -265,10 +263,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-09-19 `frank-small` is the fleet's multimodal endpoint -(`input: [text, image, audio]`) — vLLM's native Gemma4 vision AND audio -towers, both verified live. `frank-large` is text-only because its halogen -deployment loads no vision tower. +of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and +`frank-large` is text-only because its halogen deployment loads no vision +tower. There is **no audio** input anywhere in the fleet: that capability left +with the FLM NPU endpoint on 2026-09-11. ### frank-large — halogen-flash-server (Dankclaw, ACTIVE) @@ -303,37 +301,31 @@ that as "tight and fragmented", not "out of memory". Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. Rollback: [[frank-large-ik-uncensored]]. -### frank-small — vLLM on the RTX 3090 (Dankclaw, ACTIVE) +### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) -vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (digest-pinned, audio -extra pip-installed into `localhost/vllm/vllm-openai:frank-small` — see -`dankclaw/frank-small/Dockerfile`) as a **rootless Quadlet** -(`dankclaw/frank-small/quadlet/frank-small.container`), serving -`google/gemma-4-12B-it-qat-w4a16-ct` (compressed-tensors w4a16, int4 Marlin -weights, 10.26 GiB) with Google's **bf16** MTP assistant -(`google/gemma-4-12B-it-assistant`, 845 MB). Port 7771, 262,144 window, -bf16 KV pool of 289,647 tokens, **2 concurrent full-context sessions** -(the 3rd queues), CUDA graphs on, `ENFORCE_EAGER=0`. +Mainline llama.cpp master `37b53fd` (CUDA build with +`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo +`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), +serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M. +The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4 +nextn layers) loaded via `--model-draft`; the draftless **ngram-mod** +matcher chains behind it in the same `--spec-type` value. BF16 mmproj, +port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4` +(four FULL 262,144-token slots — the measured card ceiling), batch +**8192/1024**, greedy default. Deployed flags are in +`scripts/frank-small.env`; the serve script refuses to start if the +weights, the projector, the sidecar or the binary are missing. -The w4a16-ct assistant CANNOT draft in vLLM (the drafter path skips the -Marlin kernel — packed int4 misread, acceptance 1.00); the bf16 assistant -measures acceptance 2.80–3.56. fp8 KV is hardware-dead on sm_86 (Ampere -has no fp8 tensor cores), so bf16 is the only option. - -Dankclaw has **no nvidia-container-toolkit**: the unit wires CUDA by hand -(`Device=` for the four nvidia nodes + host `libcuda`/`libnvidia-ml` -mounted read-only into two library paths). Host driver is -`nvidia-driver-610` 610.57.04 on Ubuntu 26.04; `sm_86` is the correct -compute capability — no arch override. - -Verified at deploy (2026-09-19): `/v1/models` reports `frank-small`; chat -returns content plus `reasoning` (vLLM's gemma4 parser field — the harness -maps it via `compat.reasoningContentField`); a tool call returns -`tool_calls`; an image is described; a 440 Hz WAV is described (native -audio encoder). Measured: pp@67k 1,293 t/s, pp@213k 608 t/s, -tg@213k ~13.6 t/s (bandwidth-bound), tg@85k ~17.6 t/s. -Runbook: `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md`. -Rollback: [[frank-small-gemma4-12b]] (llama.cpp, weights still on host). +Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a +chat completion returns content plus `reasoning_content`; a tool call +returns `tool_calls`; an image request is answered from the projector; +an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at +max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft +acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why +ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll +100 is WORSE at long ctx) is tabled in the runbook. +Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. +Rollback: [[frank-small-qwen38-gsq-rco]]. ### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE) @@ -715,7 +707,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | -| frank-small (vLLM Gemma4-12B w4a16, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md` | +| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -727,7 +719,7 @@ Retired (rollback targets and history): | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | -| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 09-19 | `frank-small-gemma4-12b.md` | +| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | diff --git a/dankclaw/frank-small/Dockerfile b/dankclaw/frank-small/Dockerfile deleted file mode 100644 index 6268bd6..0000000 --- a/dankclaw/frank-small/Dockerfile +++ /dev/null @@ -1,22 +0,0 @@ -# frank-small runtime image: the digest-pinned vLLM gemma-x86_64-cu129 base -# (vLLM 0.22.1rc1.dev357+g74b5964f0 — the image that loads -# gemma-4-12B-it-qat-w4a16-ct, runs the Gemma4MTPModel drafter and carries -# the gemma4 reasoning/tool-call parsers) plus the vllm[audio] extra. -# -# The base image ships WITHOUT the audio extras, so an input_audio content -# part 500s with "Please install vllm[audio] for audio support" (measured -# 2026-09-19). The extra is just four PyPI packages (av, scipy, soundfile, -# mistral_common[audio]); installing the extras directly — not -# `pip install vllm[audio]` — keeps the pinned dev build in place: a bare -# `vllm[audio]` would make pip reach for the PyPI release and replace the -# 0.22.1rc1.dev build that knows gemma4. -# -# Built by scripts/deploy-remote.sh on dankclaw (rootless podman, network on -# for pip): the base is pulled by digest and re-tagged to the FROM ref below -# first, then `podman build` produces the :frank-small tag the quadlet runs. -FROM localhost/vllm/vllm-openai:gemma4-unified - -# The vllm[audio] extra as raw packages (header explains why not -# `pip install vllm[audio]`): audio input needs av/scipy/soundfile plus -# the mistral_common audio protocol pieces. -RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]" diff --git a/dankclaw/frank-small/quadlet/frank-small.container b/dankclaw/frank-small/quadlet/frank-small.container deleted file mode 100644 index ae9aa95..0000000 --- a/dankclaw/frank-small/quadlet/frank-small.container +++ /dev/null @@ -1,100 +0,0 @@ -# Quadlet unit: frank-small (rootless, boot-persistent). -# -# Install path on dankclaw: ~/.config/containers/systemd/frank-small.container -# Unit name: frank-small.service (container name: frank-small) -# -# The small inference service. vLLM (since 2026-09-18) serving -# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16) with the -# google bf16 MTP assistant for speculative decoding, on the RTX -# 3090. It replaced the mainline-llama.cpp ROOT unit (the old -# scripts/frank-small.service is retired into scripts/.AGENTTRASH/). -# -# SHAPE. The quadlet replaces the image's `vllm serve` entrypoint with -# /scripts/serve-frank-small.sh (mounted read-only out of the repo checkout) -# so the flag set stays in git. The model and drafter directories are bound -# read-only: the payloads are fetched and sha256-verified by -# scripts/download-frank-small-vllm.sh before first start, and the deploy -# handler refuses to start the unit when either is absent. -# -# CUDA WITHOUT the nvidia-container-toolkit. This host has never run a CUDA -# container (the 3090 served frank-small as a native binary, the iGPU runs -# ROCm through /dev/kfd), so there is no nvidia hook to lean on. The recipe -# that works (measured 2026-09-18, torch sees the 3090 at 25.3 GB): -# - the four /dev/nvidia* nodes via AddDevice (they are 0666 on this host, -# so the rootless user namespace can open them); -# - the HOST driver's libcuda.so.1 and libnvidia-ml.so.1 mounted read-only -# — the image carries its own CUDA 12.9 runtime, but vLLM's platform -# probe needs the driver's NVML stub, and without it `vllm serve` dies at -# parser construction with "Failed to infer device type"; -# - NVIDIA_VISIBLE_DEVICES=all. -# memlock must be unlimited: CUDA pins its staging buffers. -# -# SIZING. The tunables (full 262144 window, 0.95 of a dedicated 24 GiB card, -# bf16 KV, 16-seq batch ceiling, 16k prefill chunk, 4-token MTP draft) live -# in scripts/frank-small.env. The boot log's "Maximum concurrency for N -# tokens per request" line is the naive pool ratio for a hybrid SWA model -# and undercounts; verify the session count behaviourally on restart. -# -# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet -# path cannot resolve registry-prefixed refs from local storage at create -# time. The tag is a LOCAL BUILD, not a pull: deploy-remote.sh pulls the -# digest-pinned docker.io base (the gemma-x86_64-cu129 build, vLLM -# 0.22.1rc1.dev357+g74b5964f0 — the image that boots this checkpoint) and -# re-tags it, then builds dankclaw/frank-small/Dockerfile on top, which -# installs the vllm[audio] extra (av, scipy, soundfile, -# mistral_common[audio]) so the model's native audio path works. Pull=never -# keeps boots off the registry. -[Unit] -Description=frank-small — vLLM gemma4_unified w4a16 + MTP on RTX 3090 -After=network-online.target - -[Container] -ContainerName=frank-small -Image=localhost/vllm/vllm-openai:frank-small -# Published port stays 7771: Hermes, the harness and the homepage probe all -# point here. 8000 (vLLM's in-container port) is never reachable directly. -PublishPort=7771:8000 -# CUDA device nodes (0666 on this host — rootless can open them). -AddDevice=/dev/nvidia0 -AddDevice=/dev/nvidiactl -AddDevice=/dev/nvidia-uvm -AddDevice=/dev/nvidia-uvm-tools -# Host driver stubs the image's CUDA runtime and vLLM's NVML probe need. -# The image's LD_LIBRARY_PATH leads with /usr/local/nvidia/lib64 — the -# nvidia-container-toolkit's conventional mount point, which this host does -# not have — so the driver libs go there; the /usr/lib mounts cover torch -# Inductor's own lookup. -Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro -Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so:ro -Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:ro -Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/local/nvidia/lib64/libcuda.so.1:ro -Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/local/nvidia/lib64/libcuda.so:ro -Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/local/nvidia/lib64/libnvidia-ml.so.1:ro -Environment=NVIDIA_VISIBLE_DEVICES=all -# Payloads, read-only: fetched and sha256-verified by -# scripts/download-frank-small-vllm.sh before first start. -Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro -Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro -# Serve script and env, read-only out of the repo checkout. -Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro -Entrypoint=/scripts/serve-frank-small.sh -Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env -Ulimit=memlock=-1:-1 -# /health answers 200 once the engine is up; vLLM 503s it while the model is -# still loading, so a green check means ready-to-serve, not process-alive. -# curl ships in the image. -HealthCmd=curl -sf http://127.0.0.1:8000/health -HealthInterval=30s -HealthTimeout=10s -HealthRetries=3 -HealthStartPeriod=600s -Pull=never - -[Service] -# First start reads ~10 GiB of weights and compiles the CUDA graphs. -TimeoutStartSec=900 -Restart=on-failure -RestartSec=10 - -[Install] -WantedBy=default.target diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index accac4f..2fa5aea 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -319,12 +319,6 @@ def services_for_path(path: str, target: str) -> set[str]: # its unit and fleet settings live in one .container file. _add(services, "frank-large") return services - if len(parts) >= 2 and parts[1] == "frank-small": - # The service is a rootless Quadlet container since 2026-09-18 - # (vLLM); its unit lives in one .container file. The old root - # unit's trio is in scripts/ and maps there. - _add(services, "frank-small") - return services if len(parts) >= 2 and parts[1] == "observability": # The five observability containers became Quadlet units on # 2026-09-11. A unit change selects exactly its own service; @@ -408,13 +402,11 @@ def services_for_path(path: str, target: str) -> set[str]: # change to it redeploying alertmanager is the only path that # reaches it on the host. _add(services, "alertmanager") - if name in {"frank-small.env", "serve-frank-small.sh", - "download-frank-small-vllm.sh"}: - # vLLM frank-small (2026-09-18) keeps its env, entrypoint and - # fetch script here; each one selects frank-small only. The - # unit is the quadlet under dankclaw/frank-small/, and the old - # root unit file is in scripts/.AGENTTRASH/, which is archive - # material and plans nothing. + if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}: + # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve + # files here; each one selects frank-small only. The ik_llama.cpp + # trio that lived here for two days is in scripts/.AGENTTRASH/, + # which is archive material and plans nothing. _add(services, "frank-small") if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}: # llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 1cae4f5..c1216ac 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1139,90 +1139,28 @@ wait_for_llm_large_ready() { return 0 } -# deploy_llm_small — frank-small is a rootless Quadlet running vLLM on the -# RTX 3090 since 2026-09-18: the runtime is the digest-pinned -# vllm/vllm-openai gemma4-unified image, and the unit is generated from -# dankclaw/frank-small/quadlet/. It replaces the mainline-llama.cpp -# deployment, which ran as a ROOT unit. +# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a +# ROOT unit: the runtime is a native CUDA binary, so there is no container +# image to install and no generated unit to avoid. relink_unit points +# /etc/systemd/system/frank-small.service at the repo file and boot persistence +# is that unit's [Install] section. # -# TWO MANAGERS, ONE UNIT NAME. The llama.cpp incarnation ran as a ROOT unit -# whose file is a symlink into this checkout; the vLLM unit is a USER -# (quadlet) unit. Both must never exist at once — they would race for port -# 7771 — so the root unit is retired first: stop, disable, move the /etc -# symlink into the home trash (never `rm`), reload the root manager. This -# host's `systemctl disable` has been observed to delete the unit file -# itself (2026-08-26 incident), so the move tolerates a file that is -# already gone. -# -# The payloads are fetched and sha256-verified by -# scripts/download-frank-small-vllm.sh before first start; the handler -# refuses to launch a unit that cannot load its model or its drafter. -LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/gemma4-12b-qat-w4a16-ct/model.safetensors}" -LLM_SMALL_DRAFT="${LLM_SMALL_DRAFT:-/home/dank/models/gemma4-12b-assistant-bf16/model.safetensors}" -# Base image: the gemma-x86_64-cu129 build (vLLM 0.22.1rc1.dev357+ -# g74b5964f0) — the image that boots this checkpoint. The June 3 -# gemma4-unified build predates it and fails at weight load (missing -# vision_embedder.patch_dense); the Sep 15 cu129-nightly fails at import -# (torchvision ABI). The unit runs a LOCAL BUILD on top of this base — -# dankclaw/frank-small/Dockerfile installs the vllm[audio] extra, which -# the base ships without. -VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:e24d3d8e367ebc2a6ccfc8cf3cecee858a30a6c66b5b1642d85363cd745e001e" -VLLM_SMALL_BASE_ALIAS="localhost/vllm/vllm-openai:gemma4-unified" -VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small" - -retire_root_llm_small() { - local trash="$HOME/.AGENTTRASH/20260918-vllm-cutover/host-units" - run_privileged systemctl stop frank-small.service 2>/dev/null || true - run_privileged systemctl disable frank-small.service 2>/dev/null || true - if [[ -e /etc/systemd/system/frank-small.service || -L /etc/systemd/system/frank-small.service ]]; then - mkdir -p "$trash" - run_privileged mv -- /etc/systemd/system/frank-small.service \ - "$trash/frank-small.service" - echo "retired root unit frank-small.service -> $trash" - else - # This host's `systemctl disable` is allowed to delete the unit file - # outright (2026-08-26 incident), so there is often nothing left to - # move. Say so: previously no message meant no evidence either way. - echo "root unit frank-small.service was already removed by disable" - fi - run_privileged systemctl daemon-reload -} +# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is +# not shared: frank-large's halogen engine runs on the gfx1151 iGPU. +LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}" deploy_llm_small() { if [[ ! -f "$LLM_SMALL_MODEL" ]]; then echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2 - echo "frank-small: run scripts/download-frank-small-vllm.sh first (main model + MTP assistant, sha256-verified);" >&2 + echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2 echo "frank-small: refusing to start a unit that cannot load." >&2 return 1 fi - if [[ ! -f "$LLM_SMALL_DRAFT" ]]; then - echo "ERROR: MTP assistant absent at $LLM_SMALL_DRAFT" >&2 - echo "frank-small: run scripts/download-frank-small-vllm.sh first;" >&2 - echo "frank-small: refusing to start a unit that cannot draft." >&2 - return 1 - fi - retire_root_llm_small - # The predecessor can also be THIS quadlet -- a redeploy of a running - # frank-small -- so stop the user unit before waiting on the port. On a - # first cutover it is not running and this is a no-op. - systemctl --user stop frank-small.service 2>/dev/null || true - # Pull the base by digest, re-tag it to the FROM ref the Dockerfile - # expects, then build the derived image the quadlet runs: the quadlet - # path cannot resolve registry-prefixed refs from local storage at - # create time (same workaround as halogen), and Pull=never keeps boots - # off the registry. The build needs network for pip (the vllm[audio] - # extra); podman build has it on by default. - podman pull "$VLLM_SMALL_BASE_IMAGE" - podman tag "$VLLM_SMALL_BASE_IMAGE" "$VLLM_SMALL_BASE_ALIAS" - podman build --pull=never -t "$VLLM_SMALL_IMAGE" \ - -f dankclaw/frank-small/Dockerfile dankclaw/frank-small - # Nothing may start until the predecessor has released the port. - wait_port_free 7771 || return 1 - install_quadlet frank-small dankclaw/frank-small/quadlet frank-small - # No enable: generated units cannot be enabled on this host (v1 quadlet - # generator); boot persistence comes from [Install] in the unit file. - systemctl --user restart frank-small.service - wait_for_health frank-small http://127.0.0.1:7771/health 1200 + relink_unit frank-small + run_privileged systemctl daemon-reload + run_privileged systemctl enable frank-small.service + run_privileged systemctl restart frank-small.service + wait_for_health frank-small http://127.0.0.1:7771/health 900 } # deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a diff --git a/scripts/download-frank-large-orca.sh b/scripts/download-frank-large-orca.sh new file mode 100755 index 0000000..480f53e --- /dev/null +++ b/scripts/download-frank-large-orca.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-large +# model) on Dankclaw. +# +# bash scripts/download-frank-large-orca.sh +# bash scripts/download-frank-large-orca.sh --verify-only +# +# The package is 136 GB of Hub-sized transport parts (HF caps new uploads at +# 50 GB, so the 79 GB main GGUF, the 52 GB PLE payload and the 4 GB MTP head +# ship as .partNofM fragments). assemble.py (from the repo) concatenates the +# parts, verifies every part and final SHA-256, and consumes the parts. +# +# Integrity: every part is checked against the per-part sha256 in parts.json +# (fetched from the repo, not trusted from disk), then the assembled files +# against SHA256SUMS. The repo is public (gated: false); an HF token at +# ~/.cache/huggingface/token is used when present to clear the anonymous +# rate limit on a 136 GB transfer. +# +# Layout after a successful run (MODEL_DIR): +# Qwen3.8-Flash-CIRU-STRIX-Orca.gguf 79,397,818,912 main model +# mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf 4,135,893,440 draft +# ple/ple.payload.bin 52,429,053,952 external PLE +# ple/ple.manifest.json, ple/ple.scale.bf16 PLE metadata +# vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj 904,004,320 projector +# assemble.py, parts.json, SHA256SUMS (kept for --verify-only) +set -euo pipefail + +REPO="jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca" +MODEL_DIR="/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca" +BASE="https://huggingface.co/$REPO/resolve/main" + +mode="download" +[[ "${1:-}" == "--verify-only" ]] && mode="verify" + +token="" +auth=() +if [[ -r "$HOME/.cache/huggingface/token" ]]; then + token="$(cat "$HOME/.cache/huggingface/token")" + auth=(-H "Authorization: Bearer $token") +else + echo "no HF token at ~/.cache/huggingface/token; downloading anonymously" +fi + +dl() { # dl + local dest="$MODEL_DIR/$1" + mkdir -p "$(dirname "$dest")" + echo " fetching $1" + curl -fL -C - -o "$dest" --retry 10 --retry-delay 5 --retry-max-time 21600 \ + "${auth[@]}" "$BASE/$1" +} + +if [[ "$mode" == "download" ]]; then + # Manifest + small files first: the part hashes below come from the + # freshly fetched parts.json, so a stale/corrupt copy cannot pass. + dl "assemble.py" + dl "parts.json" + dl "SHA256SUMS" + dl "ple/ple.manifest.json" + dl "ple/ple.scale.bf16" + dl "vision/SHA256SUMS" + dl "vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj" + + echo "fetching the five transport parts (136 GB)" + while IFS= read -r part; do + dl "$part" + done < <(python3 -c " +import json +for item in json.load(open('$MODEL_DIR/parts.json'))['files']: + for p in item['parts']: + print(p['path']) +") +fi + +echo "verifying part sha256 (against the repo parts.json)" +python3 - "$MODEL_DIR" <<'PY' +import hashlib, json, sys +from pathlib import Path +root = Path(sys.argv[1]) +manifest = json.loads((root / 'parts.json').read_text()) +ok = True +def digest(p): + h = hashlib.sha256() + with p.open('rb') as f: + for data in iter(lambda: f.read(8 * 1024 * 1024), b''): + h.update(data) + return h.hexdigest() +for item in manifest['files']: + for p in item['parts']: + path = root / p['path'] + if not path.exists(): + print(f" MISSING {p['path']}"); ok = False; continue + if path.stat().st_size != p['bytes']: + print(f" BAD-SIZE {p['path']}"); ok = False; continue + if digest(path) != p['sha256']: + print(f" BAD-SHA {p['path']}"); ok = False; continue + print(f" OK {p['path']}") +sys.exit(0 if ok else 1) +PY + +if [[ "$mode" == "download" ]]; then + echo "assembling (consumes verified parts)" + (cd "$MODEL_DIR" && python3 assemble.py) +fi + +echo "verifying assembled files against SHA256SUMS" +(cd "$MODEL_DIR" && sha256sum -c SHA256SUMS) +(cd "$MODEL_DIR/vision" && sha256sum -c SHA256SUMS) +echo "frank-large payload complete: $MODEL_DIR" diff --git a/scripts/download-frank-small-vllm.sh b/scripts/download-frank-small-vllm.sh deleted file mode 100755 index 4d811b0..0000000 --- a/scripts/download-frank-small-vllm.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env bash -# Fetch the frank-small vLLM payloads on dankclaw (RTX 3090, since 2026-09-18): -# -# google/gemma-4-12B-it-qat-w4a16-ct -# compressed-tensors w4a16 (int4, group 32) QAT checkpoint — the model -# vLLM's gemma4_unified implementation loads natively. One 9.56 GiB -# safetensors plus tokenizer/config. -# -# google/gemma-4-12B-it-assistant -# the MTP draft/assistant (gemma4_unified_assistant, 4 layers) in -# bf16 — Google's original. 806 MiB. The w4a16-ct re-quant -# (kunhunjon) that was used first drafts at acceptance 1.00: vLLM's -# MTP drafter path does not apply the Marlin w4a16 kernel to the -# assistant's linears (no "Using MarlinLinearKernel" after the -# drafter load), so its packed int4 weights are misread. The bf16 -# assistant accepts at 2.80 (measured 2026-09-19). -# -# Both repos are public and ungated; no token is needed. Every fetch is -f -# (an HTTP error is a failure, not a 153-byte "artifact"), resumable, and the -# two LFS payloads are sha256-verified against the Hub's declared oid — the -# discipline from scripts/download-frank-medium-gguf.sh. -# -# The serve script refuses to start when either payload is absent, so a -# half-finished run of this script reads as a missing file, not a runtime -# fault. -set -euo pipefail - -MODELS_DIR="${MODELS_DIR:-/home/dank/models}" -LOG_DIR="$MODELS_DIR/logs" -mkdir -p "$LOG_DIR" -LOG="$LOG_DIR/$(basename "$0" .sh).log" - -log() { echo "$(date -Is) $*" | tee -a "$LOG"; } - -# fetch [sha256] [size] -fetch() { - local repo="$1" dir="$2" file="$3" sha="${4:-}" size="${5:-}" - mkdir -p "$dir" - if [[ -n "$size" && -f "$dir/$file" ]] && [[ "$(stat -c%s "$dir/$file")" == "$size" ]]; then - log "skip $dir/$file (size $size matches)" - else - log "fetch $repo/$file" - curl -fsSL -C - "https://huggingface.co/$repo/resolve/main/$file" -o "$dir/$file" - log "fetched $dir/$file ($(stat -c%s "$dir/$file") bytes)" - fi - if [[ -n "$sha" ]]; then - local actual - actual="$(sha256sum "$dir/$file" | cut -d' ' -f1)" - if [[ "$actual" != "$sha" ]]; then - log "SHA256 MISMATCH $dir/$file: $actual != $sha" - return 1 - fi - log "sha256 ok $dir/$file" - fi -} - -MAIN_REPO=google/gemma-4-12B-it-qat-w4a16-ct -ASSISTANT_REPO=google/gemma-4-12B-it-assistant -MAIN_DIR="$MODELS_DIR/gemma4-12b-qat-w4a16-ct" -ASSISTANT_DIR="$MODELS_DIR/gemma4-12b-assistant-bf16" - -# Hub-declared LFS oids (sha256), 2026-09-18. -fetch "$MAIN_REPO" "$MAIN_DIR" model.safetensors \ - 60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3 10264229896 -fetch "$MAIN_REPO" "$MAIN_DIR" tokenizer.json \ - cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f 32169626 -for f in config.json generation_config.json chat_template.jinja \ - processor_config.json tokenizer_config.json recipe.yaml; do - fetch "$MAIN_REPO" "$MAIN_DIR" "$f" -done - -fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" model.safetensors \ - 3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df 845719296 -fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" tokenizer.json \ - c001d9ada50af662c94f5ab17ec7e09f6438d1bec8246c47fee6510693d8de35 32169884 -for f in config.json generation_config.json tokenizer_config.json; do - fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" "$f" -done - -log "ALL FETCHED OK" diff --git a/scripts/frank-small.env b/scripts/frank-small.env index fa62f6e..42d8431 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -1,101 +1,112 @@ -# Tunables for frank-small (since 2026-09-18): vLLM serving -# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format -# Google publishes "for native, optimized inference with vLLM") with the -# google bf16 MTP assistant, on Dankclaw's RTX 3090. +# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS- +# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP +# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17. # -# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM -# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh), -# with the vllm[audio] extra installed on top (dankclaw/frank-small/ -# Dockerfile) so the model's native audio path works. That image carries -# the native gemma4_unified implementation, the Gemma4MTPModel drafter, -# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada -# card. The older gemma4-unified build (June 3) predates this checkpoint -# and fails at weight load; the cu129-nightly build (Sep 15) fails at -# import (torchvision ABI); this one is the image that boots. +# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86: +# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \ +# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON +# cmake --build build-cuda --target llama-server -j +# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash +# attention, and without the flag quantized-KV prefill collapses ~12x on +# CUDA (see the CUDA lesson in AGENTS.md). # -# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window -# (1024-token) + 1 full-attention, int4 group-32 weights with bf16 -# embeddings, vision and audio. The fleet's image path lives here: the model -# ingests image (and audio) content natively, so no projector file exists -# any more and nothing external has to load. +# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M +# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv +# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, +# per-layer token embeddings (PLE) inside the file, and a sliding-window +# attention pattern (attention.sliding_window = 1024) -- most layers cache +# only a 1024-position window, which is why the KV pool is a fraction of +# the full-ctx x all-layers number. # -# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the -# same int4 w4a16 compressed-tensors format — the card's compatibility rule -# is that a QAT target must draft with a QAT drafter at the same precision. -# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4 -# MTP path, not a generic draft model. +# MTP: the Multi-Token Prediction head ships as a SEPARATE file +# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 +# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one +# is passed via --model-draft with --spec-type draft-mtp, so the draft +# context loads the assistant model. The same --spec-type value chains the +# draftless ngram-mod matcher, which attaches unconditionally; two +# context-backed drafters cannot share one process, so draft-mtp,ngram-mod +# is the only legal shape that carries both. # -# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window -# for as many concurrent sessions as the card holds. -# MAX_MODEL_LEN=262144 the model's max_position_embeddings -# GPU_MEM_UTIL=0.95 the card is dedicated to this service -# KV_CACHE_DTYPE=turboquant_k8v4 TurboQuant KV: K at 8-bit, V at 4-bit, -# per-16-element block scales packed INSIDE the -# tensor (Hadamard-rotated, bf16 compute, -# dedicated TURBOQUANT attention backend with -# an explicit Ampere path). 2.67x smaller KV -# than bf16 (289,647 -> ~772k-token pool, two -# -> five-or-six full-context sessions). -# The cheaper int8/fp8 per-token-head dtypes -# CANNOT boot this model: they keep a separate -# fp32 scale tensor per (token, head), and -# Gemma4's hybrid head dims (SWA 256 / full -# 512) make the per-token page bytes 4160 vs -# 8256 — not an integer ratio, so vLLM's -# unify_kv_cache_spec_page_size raises -# NotImplementedError (measured 2026-09-19, -# nine crash-loop restarts). fp8 KV is -# hardware-dead on this card anyway (sm_86, -# no cvt.e4m3x2). -# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this, -# caps full-context concurrency -# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill -# faster but grow the activation arena that -# competes with the KV pool for the card -# The boot log prints the KV pool size and "Maximum concurrency for N tokens -# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool / -# max_model_len) and undercounts: the 42 sliding-window layers only retain a -# 1024-token window per request, so the true full-context session count is -# higher. Verify it behaviourally (concurrent long requests) on every -# restart, not from these comments. +# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four +# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346 +# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot +# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the +# card, and the KV pool allocates upfront -- so 4 is the max. Explicit +# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, +# which would shrink each slot below the full 262144 window once +# parallel > 1. # -# Sampling: greedy by default (fleet contract) via -# --override-generation-config; a client that sends its own temperature gets -# its own, and the harness applies per-model overrides from models.yml. +# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M +# params; it carries an audio encoder too). The fleet's image path lives +# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio +# WORKS: input_audio content parts transcribe verbatim (verified +# 2026-09-17 with a 16k mono WAV). # -# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the -# ROCm hosts read those bare vars — see the 2026-08-13 incident in +# Sampling: greedy by default (the fleet contract); a client that sends a +# temperature gets its own, and the harness applies per-model overrides +# from models.yml. +# +# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP +# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in # AGENTS.md). This box is CUDA, but the rule keeps env files portable. -# Container paths, bind-mounted by the quadlet. The host paths are declared -# in the quadlet and the deploy handler, not here: this file is read inside -# the container. -MODEL_DIR=/models/main -DRAFT_DIR=/models/assistant +# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86, +# FA_ALL_QUANTS=ON. +LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server -# Fleet identity. 8000 is the in-container port; the quadlet publishes it as -# host 7771, the endpoint Hermes, the harness and the homepage probe have -# always used. The alias is the id at /v1/models and what clients ask for. +# Model package. The MTP sidecar file is required for speculative decoding. +MODEL_DIR=/home/dank/models/Gemma4-12B-QAT +MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf +MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf +DRAFT_FILE=mtp-gemma-4-12B-it.gguf + +# Fleet identity and network. 7771 is this service's own endpoint on +# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and +# what clients ask for -- the harness entry in models.yml names it. ALIAS=frank-small BIND_ADDR=0.0.0.0 -LISTEN_PORT=8000 +LISTEN_PORT=7771 -# Full window, dedicated card, turboquant k8v4 KV (see above), roomy batch ceiling, -# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers). -MAX_MODEL_LEN=262144 -GPU_MEM_UTIL=0.95 -KV_CACHE_DTYPE=auto -# KV quant history (measured 2026-09-19, all on this image): bf16 (auto) -# is the ONLY dtype that boots Gemma4 here. int8_per_token_head dies in -# unify_kv_cache_spec_page_size — its separate fp32 scale tensor makes the -# per-token page bytes 4160 (SWA, head_dim 256) vs 8256 (full, head_dim -# 512), not an integer ratio. turboquant_k8v4 dies in backend validation — -# the TURBOQUANT backend rejects "partial multimodal token full attention" -# (the mm-prefix path Gemma4's vision/audio tokens require), and the -# VLLM_ATTENTION_BACKEND env var does not exist in this build (the pin -# goes through --attention-backend). fp8 is hardware-dead on sm_86 -# (cvt.e4m3x2 is sm_89+). Revisit on a vLLM release that documents TurboQuant -# mm-prefix support. -MAX_NUM_SEQS=16 -MAX_NUM_BATCHED_TOKENS=16384 -SPEC_NUM_TOKENS=4 +# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total +# pool; --parallel divides it). See the SIZING note above for the fit +# arithmetic; a fifth slot does not fit. +CTX_SIZE=1048576 +PARALLEL=4 +CACHE_TYPE_K=q8_0 +CACHE_TYPE_V=q8_0 + +# Batch geometry: 8192 logical / 1024 physical. Measured on this card at +# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV): +# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962, +# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp +# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be +# tried. Idle VRAM is 23,300 MiB at this geometry. +BATCH_SIZE=8192 +UBATCH_SIZE=1024 + +# CPU threads: this service runs entirely on the GPU (NGL=999), so threads +# cover tokenization, sampling and the HTTP layer only. Deliberately modest +# -- this host's CPU is shared with halogen's engine, which is CPU-hungry. +NUM_THREADS=8 +THREAD_BATCH=8 + +# Every layer on the 3090, with flash attention (required for q8 KV). +NGL=999 +FLASH_ATTN=on + +# Speculative decoding: the MTP sidecar (draft context) chained with the +# draftless ngram-mod matcher. Draft depth and busy-wait level are made +# explicit here (both were the implicit build defaults: 3 and 50). +# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s) +# than depth 3 + poll 50 (257-302 t/s) — the longer verification round +# costs more attention re-read at long context than the extra accepted +# token earns. Keep 3/50. +SPEC_TYPE=draft-mtp,ngram-mod +SPEC_DRAFT_N_MAX=3 +POLL_LEVEL=50 + +# Greedy default (fleet contract); reasoning is extracted into +# reasoning_content with an unlimited budget (the model decides). +TEMPERATURE=0.0 +REASONING_FORMAT=deepseek +REASONING_BUDGET=-1 diff --git a/scripts/frank-small.service b/scripts/frank-small.service new file mode 100644 index 0000000..14ddd9c --- /dev/null +++ b/scripts/frank-small.service @@ -0,0 +1,20 @@ +[Unit] +Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=dank +Group=dank +# CUDA pins its staging buffers, so the process needs the memlock budget even +# though the weights themselves are copied into VRAM. Parity with the halogen +# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit. +LimitMEMLOCK=infinity +Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env" +ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index c5fcb03..0a9aade 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,30 +1,43 @@ #!/usr/bin/env bash -# Serve google/gemma-4-12B-it-qat-w4a16-ct as frank-small on Dankclaw's RTX -# 3090 via vLLM (since 2026-09-18), with the google bf16 MTP assistant -# for speculative decoding. +# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as +# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp. # -# This script is the CONTAINER ENTRYPOINT of the frank-small quadlet: the -# vllm/vllm-openai image's own entrypoint is `vllm serve`, and the quadlet -# replaces it with this file (mounted read-only out of the repo checkout) so -# the flag set stays in git. The image pins the gemma4-unified vLLM build, -# which is what loads gemma4_unified natively and runs the Gemma 4 MTP path. +# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M +# weights plus the BF16 vision projector and q8_0 KV inside the card's +# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU +# belongs to the halogen engine that serves frank-large. # -# MTP: the drafter is a separate 4-layer assistant checkpoint in bf16 -# (Google's original gemma-4-12B-it-assistant). It goes in -# --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path, not a -# generic draft model. The w4a16-ct re-quant of the assistant was tried -# first and drafts at acceptance 1.00 (the drafter path does not apply the -# Marlin kernel to its linears); the bf16 assistant accepts at 2.80. +# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file +# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is +# passed via --model-draft with --spec-type draft-mtp: the draft context +# loads the assistant model. The same --spec-type value chains the +# draftless ngram-mod matcher, which attaches unconditionally; two +# context-backed drafters cannot share one process, so this chain is the +# only shape that carries both. # -# Sampling: greedy by default (fleet contract) via -# --override-generation-config; clients that send their own temperature get -# their own, and the harness overrides per model from models.yml. +# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot +# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below) +# maximises the TOTAL pool instead, which would divide a smaller window +# across slots. The 4-slot layout is measured to fit the card (see the +# SIZING note in frank-small.env); more slots would fail the KV allocation. # -# Tunables come from frank-small.env (mounted beside this file by the -# quadlet) unless FRANK_SMALL_ENV_FILE overrides the path. +# Vision + audio: --mmproj loads the BF16 projector, which also carries +# an audio encoder — input_audio content parts transcribe verbatim +# (verified 2026-09-17). The fleet's image path lives here, which is why +# halogen no longer sets HALOGEN_VISION_TOWER. +# +# Sampling: greedy by default (the fleet contract). Clients that send their own +# temperature get their own values, and the harness overrides per model from +# models.yml. --jinja is pinned even though this build defaults it on: the +# model's own template supplies reasoning_effort, preserve_thinking and the +# tool-call grammar, and a future build that flips that default must not +# silently take tools away. +# +# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it. set -euo pipefail -ENV_FILE="${FRANK_SMALL_ENV_FILE:-/scripts/frank-small.env}" +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}" if [[ -f "$ENV_FILE" ]]; then set -a # shellcheck disable=SC1090 @@ -33,51 +46,93 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${MODEL_DIR:=/models/main}" -: "${DRAFT_DIR:=/models/assistant}" +: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}" +: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}" +: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}" +: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}" +: "${DRAFT_FILE:=}" : "${ALIAS:=frank-small}" : "${BIND_ADDR:=0.0.0.0}" -: "${LISTEN_PORT:=8000}" -: "${MAX_MODEL_LEN:=262144}" -: "${GPU_MEM_UTIL:=0.95}" -: "${KV_CACHE_DTYPE:=auto}" -: "${MAX_NUM_SEQS:=16}" -: "${MAX_NUM_BATCHED_TOKENS:=16384}" -: "${SPEC_NUM_TOKENS:=4}" -: "${ENFORCE_EAGER:=0}" +: "${LISTEN_PORT:=7771}" +: "${CTX_SIZE:=}" +: "${PARALLEL:=1}" +: "${CACHE_TYPE_K:=f16}" +: "${CACHE_TYPE_V:=f16}" +: "${BATCH_SIZE:=2048}" +: "${UBATCH_SIZE:=512}" +# Decode-shape knobs: the MTP drafter's per-round proposal length (build +# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's +# busy-wait level (build default 50). +: "${SPEC_DRAFT_N_MAX:=3}" +: "${POLL_LEVEL:=50}" +: "${NUM_THREADS:=8}" +: "${THREAD_BATCH:=8}" +: "${NGL:=999}" +: "${FLASH_ATTN:=on}" +: "${SPEC_TYPE:=draft-mtp}" +: "${TEMPERATURE:=0.0}" +: "${REASONING_FORMAT:=deepseek}" +: "${REASONING_BUDGET:=-1}" # Fail loudly rather than half-loading a model into VRAM: a missing file here # is a download that has not finished, and the unit's restart policy would # otherwise turn that into a crash loop that reads like a runtime fault. -for path in "$MODEL_DIR/model.safetensors" "$MODEL_DIR/config.json" \ - "$DRAFT_DIR/model.safetensors" "$DRAFT_DIR/config.json"; do +for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; } done - -SPEC_CONFIG="{\"method\":\"mtp\",\"model\":\"${DRAFT_DIR}\",\"num_speculative_tokens\":${SPEC_NUM_TOKENS}}" - -VLLM_FLAGS=( - serve "$MODEL_DIR" ---served-model-name "$ALIAS" ---host "$BIND_ADDR" ---port "$LISTEN_PORT" ---max-model-len "$MAX_MODEL_LEN" ---gpu-memory-utilization "$GPU_MEM_UTIL" ---kv-cache-dtype "$KV_CACHE_DTYPE" ---max-num-seqs "$MAX_NUM_SEQS" ---max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" ---speculative-config "$SPEC_CONFIG" ---reasoning-parser gemma4 ---enable-auto-tool-choice ---tool-call-parser gemma4 ---override-generation-config '{"temperature": 0}' -) -# ENFORCE_EAGER=1 skips Inductor/Triton compilation and CUDA graph capture -# (slower steady-state decode, faster and more robust boot). The 3090 is -# sm_86 (Ampere) — no native fp8 — so KV stays bf16 and the default -# graphed path is the right one; the knob is a boot-debugging lever. -if [[ "$ENFORCE_EAGER" == "1" ]]; then - VLLM_FLAGS+=(--enforce-eager) +if [[ -n "${DRAFT_FILE:-}" ]]; then + [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; } fi -exec vllm "${VLLM_FLAGS[@]}" +# The UVM device node is created lazily by nvidia-modprobe on the first CUDA +# open after boot; make sure it exists before the server needs it. +nvidia-modprobe -u -c=0 2>/dev/null || true + +MMPROJ_ARGS=() +if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then + MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH") +fi + +SPEC_ARGS=() +if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then + SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX") +fi + +DRAFT_ARGS=() +if [[ -n "${DRAFT_FILE:-}" ]]; then + DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE") +fi + +# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against +# whatever the weights, projector, draft context and compute buffers leave +# on the card; the engine prints the chosen value at startup. +if [[ -n "${CTX_SIZE:-}" ]]; then + CTX_ARGS=(--ctx-size "$CTX_SIZE") +else + CTX_ARGS=(--fit on) +fi + +exec "$LLAMA_BIN" \ + --model "$MODEL_DIR/$MODEL_FILE" \ + "${MMPROJ_ARGS[@]}" \ + "${DRAFT_ARGS[@]}" \ + --alias "$ALIAS" \ + --host "$BIND_ADDR" \ + --port "$LISTEN_PORT" \ + --gpu-layers "$NGL" \ + --flash-attn "$FLASH_ATTN" \ + "${CTX_ARGS[@]}" \ + --parallel "$PARALLEL" \ + --batch-size "$BATCH_SIZE" \ + --ubatch-size "$UBATCH_SIZE" \ + --threads "$NUM_THREADS" \ + --threads-batch "$THREAD_BATCH" \ + --poll "$POLL_LEVEL" \ + --cache-type-k "$CACHE_TYPE_K" \ + --cache-type-v "$CACHE_TYPE_V" \ + "${SPEC_ARGS[@]}" \ + --jinja \ + --metrics \ + --temp "$TEMPERATURE" \ + --reasoning-format "$REASONING_FORMAT" \ + --reasoning-budget "$REASONING_BUDGET" diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 59bcd5c..d35dec6 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -5,9 +5,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-16): process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B- Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next -w4b on the gfx1151 iGPU, port 7770) and vLLM (frank-small, -google/gemma-4-12B-it-qat-w4a16-ct with the bf16 MTP assistant on the RTX -3090, port 7771). +w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, +ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port +7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -35,10 +35,10 @@ gfx1151 iGPU 64 GiB unified memory] frank-large, port 7770] F[peonist-ai Qwen3.8-Flash-Next w4b 2 x 200192 ctx over one 400384-position pool] - G[vLLM gemma-x86_64-cu129 + G[mainline llama.cpp 37b53fd frank-small, port 7771] - H[gemma-4-12B-it-qat-w4a16-ct -262144 ctx / 2 full-context sessions, MTP + vision + audio] + H[Qwen3.8-27B GSQ-RCO IQ3_S +131072 ctx / 1 slot, MTP + vision] E2 --> E --> F E2 --> G --> H end @@ -87,15 +87,16 @@ frank-small, port 7771] overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting -- [[frank-small-vllm-gemma4]] — ACTIVE frank-small runbook since 2026-09-19 - (rootless Quadlet, port 7771): vLLM serving - google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) - with Google's bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-token - pool, 2 concurrent full-context sessions, native text+image+audio -- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): the - prior frank-small runbook (2026-09-17 → 09-19), mainline llama.cpp serving - HauhauCS Gemma4-12B-QAT-Uncensored Q4_K_M with the MTP sidecar, q8_0 KV, - 4 slots of 262k +- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17 + (root systemd unit, port 7771): mainline llama.cpp serving + HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX + 3090, with the separate MTP assistant sidecar chained behind ngram-mod, + q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day + vLLM detour (2026-09-19) was reverted the same day; see below. +- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, + rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 + Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- + token pool, 2 concurrent full-context sessions, native text+image+audio. - [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 4eb4e74..15dfc34 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -166,17 +166,15 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since - 2026-09-19: vLLM serving - google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) - with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2 - full-context sessions, native text+image+audio) on - Dankclaw's RTX 3090, port 7771 -- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): - mainline llama.cpp serving +- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since + 2026-09-17: mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M - (MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on - Dankclaw's RTX 3090 + (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on + Dankclaw's RTX 3090, port 7771 +- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, + rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 + Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- + token pool, 2 concurrent full-context sessions, native text+image+audio - [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index 6fcf600..7028dbc 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -64,8 +64,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" { run python3 "$PLANNER" --target dankclaw --paths \ - dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \ - scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh + scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh [ "$status" -eq 0 ] [ "$output" = "frank-small" ] @@ -73,8 +72,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" # that name was retired with the FreeToken runtime on 2026-09-15 and must # not come back through a planner mapping. run python3 "$PLANNER" --target dankserver --paths \ - dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \ - scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh + scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh [ "$status" -eq 0 ] [ -z "$output" ] } diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index 804215f..5991dd8 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -72,11 +72,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "frank-large loads no vision tower: the fleet's image path is frank-small" { ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET" - # frank-small's vLLM checkpoint (2026-09-18) is natively multimodal — - # image content is ingested by the model itself, so the seam that used - # to pin --mmproj now pins the projector's absence. - ! grep -q 'mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh" - grep -q '^MODEL_DIR=/models/main$' "$REPO_ROOT/scripts/frank-small.env" + grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh" } @test "the deploy handler retires the previous ROOT unit without deleting it" { diff --git a/tests/frank_small_unit.bats b/tests/frank_small_unit.bats index c2bf275..01163b1 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_small_unit.bats @@ -1,164 +1,115 @@ #!/usr/bin/env bats -# Seam: the frank-small vLLM quadlet on dankclaw (since 2026-09-18). -# frank-small is the RTX 3090 endpoint. It now runs vLLM (the pinned -# gemma4-unified image) serving google/gemma-4-12B-it-qat-w4a16-ct with the -# google bf16 MTP assistant, replacing the mainline-llama.cpp root -# unit (retired into scripts/.AGENTTRASH/). This seam pins what the flags -# alone do not say: boot persistence for the generated quadlet, the CUDA -# wiring this host needs (no nvidia-container-toolkit), the repo-relative -# entrypoint and env, the MTP contract (a separate QAT assistant checkpoint -# in --speculative-config, method "mtp"), the fleet's image path (the model -# is natively multimodal — no projector file), the full-window sizing, the -# separate client port, and the deploy handler's cutover order. +# Seam: the frank-small mainline-llama.cpp root unit on dankclaw. +# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large +# went back to the halogen engine on the iGPU. This seam pins what the flags +# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet +# is involved), the repo-relative paths the unit executes, the separate client +# port, the MTP contract (the head is a separate sidecar file loaded as the +# draft model, chained behind ngram-mod), the vision handoff from frank-large, +# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card. load helpers -QUADLET="$REPO_ROOT/dankclaw/frank-small/quadlet/frank-small.container" -DOCKERFILE="$REPO_ROOT/dankclaw/frank-small/Dockerfile" +UNIT="$REPO_ROOT/scripts/frank-small.service" SERVE="$REPO_ROOT/scripts/serve-frank-small.sh" ENV_FILE="$REPO_ROOT/scripts/frank-small.env" -FETCH="$REPO_ROOT/scripts/download-frank-small-vllm.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" -@test "frank-small quadlet, Dockerfile, entrypoint, env and fetch script exist" { - [ -f "$QUADLET" ] - [ -f "$DOCKERFILE" ] +@test "frank-small unit, serve script and env exist in scripts/" { + [ -f "$UNIT" ] [ -f "$SERVE" ] [ -f "$ENV_FILE" ] - [ -f "$FETCH" ] - # The old root unit is retired, not deleted: it must live in the trash, - # not in scripts/, so the planner and relink_unit never see it. - [ ! -e "$REPO_ROOT/scripts/frank-small.service" ] - [ -e "$REPO_ROOT/scripts/.AGENTTRASH/frank-small.service.llama.cpp" ] } -@test "frank-small is boot-persistent, self-healing and registry-pinned" { - grep -q '^WantedBy=default.target$' "$QUADLET" - grep -q '^Restart=on-failure$' "$QUADLET" - grep -q '^RestartSec=' "$QUADLET" - grep -q '^Pull=never$' "$QUADLET" - # The unit runs the localhost/-aliased LOCAL BUILD (base + vllm[audio] - # extra); the digest-pinned docker.io base lives only in the deploy - # handler, which pulls, re-tags and builds before restart. - grep -q '^Image=localhost/vllm/vllm-openai:frank-small$' "$QUADLET" - ! grep -q '^Image=docker.io' "$QUADLET" +@test "frank-small is boot-persistent, self-healing and unprivileged" { + grep -q '^WantedBy=multi-user.target$' "$UNIT" + grep -q '^Restart=on-failure$' "$UNIT" + grep -q '^RestartSec=' "$UNIT" + grep -q '^User=dank$' "$UNIT" + grep -q '^LimitMEMLOCK=infinity$' "$UNIT" } -@test "frank-small wires CUDA without the nvidia-container-toolkit" { - # The four device nodes (0666 on this host, so rootless can open them)... - grep -q '^AddDevice=/dev/nvidia0$' "$QUADLET" - grep -q '^AddDevice=/dev/nvidiactl$' "$QUADLET" - grep -q '^AddDevice=/dev/nvidia-uvm$' "$QUADLET" - grep -q '^AddDevice=/dev/nvidia-uvm-tools$' "$QUADLET" - # ...plus the host driver's libcuda and NVML stubs. Without the NVML - # mount vllm serve dies at parser construction with "Failed to infer - # device type" (measured 2026-09-18). - grep -q 'libcuda.so.1.*:ro' "$QUADLET" - grep -q 'libnvidia-ml.so.1.*:ro' "$QUADLET" - grep -q '^Environment=NVIDIA_VISIBLE_DEVICES=all$' "$QUADLET" - # CUDA pins its staging buffers; memlock must be unlimited. - grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET" +@test "frank-small runs the repo's own serve script with the repo's env" { + grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT" + grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT" } -@test "frank-small runs the repo's own entrypoint with the repo's env" { - grep -q '^Entrypoint=/scripts/serve-frank-small.sh$' "$QUADLET" - grep -q '^Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env$' "$QUADLET" - grep -q 'Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro' "$QUADLET" +@test "frank-small offloads every layer to the 3090 with flash attention" { + grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE" + grep -q '^NGL=999$' "$ENV_FILE" + grep -q -- '--gpu-layers "$NGL"' "$SERVE" + grep -q '^FLASH_ATTN=on$' "$ENV_FILE" + grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE" + # Prometheus scrapes this endpoint; without --metrics there is no /metrics. + grep -q -- '--metrics' "$SERVE" } -@test "frank-small mounts the sha256-verified payloads read-only" { - grep -q 'Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro' "$QUADLET" - grep -q 'Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro' "$QUADLET" - # The fetch script verifies both LFS payloads against the Hub oids. - grep -q '60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3' "$FETCH" - grep -q '3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df' "$FETCH" -} - -@test "frank-small drafts with the QAT MTP assistant via method mtp" { - # The drafter is a separate 4-layer assistant checkpoint in the same - # int4 w4a16 compressed-tensors format (the card's compatibility rule: - # a QAT target drafts with a QAT drafter at the same precision), passed - # in --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path. - grep -qF -- '\"method\":\"mtp\"' "$SERVE" - grep -q -- '--speculative-config "$SPEC_CONFIG"' "$SERVE" - grep -q '^SPEC_NUM_TOKENS=4$' "$ENV_FILE" - # The gemma4 reasoning and tool-call parsers keep the fleet contract: - # reasoning_content on the wire and tools answered. - grep -q -- '--reasoning-parser gemma4' "$SERVE" - grep -q -- '--tool-call-parser gemma4' "$SERVE" - grep -q -- '--enable-auto-tool-choice' "$SERVE" +@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" { + grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE" + grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE" + # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4 + # nextn layers); it loads as --model-draft and shares the target's last + # layers' KV. ngram-mod is draftless, so the chain is legal. + grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE" + grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE" } @test "frank-small carries the fleet's image path, and only it" { - # The model is natively multimodal (encoder-free unified): image content - # is ingested by the checkpoint itself, so there is no projector file - # any more — the seam that used to pin MMPROJ_PATH now pins its absence. - ! grep -q '^MMPROJ' "$ENV_FILE" + grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE" + grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE" ! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE" } -@test "frank-small serves the full window with bf16 KV on a dedicated card" { - local maxlen kvdtype util seqs - maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")" - kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")" - util="$(sed -nE 's/^GPU_MEM_UTIL=([0-9.]+)$/\1/p' "$ENV_FILE")" - seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")" - # The model's FULL trained window (262144), not a divided one. - [ "$maxlen" = "262144" ] - # bf16 KV (auto): the only dtype that boots Gemma4 in this vLLM build. - # int8_per_token_head dies in unify_kv_cache_spec_page_size (separate - # fp32 scale tensor breaks the 256/512 head-dim page-size ratio), - # turboquant_k8v4 dies in backend validation (no mm-prefix attention), - # and fp8 is hardware-dead on sm_86 — see frank-small.env. - [ "$kvdtype" = "auto" ] - # No pinned attention backend: Gemma4's config hook selects TRITON_ATTN - # for the mixed head dims, and pinning any other backend breaks the KV - # dtype or mm-prefix validation. - ! grep -q '^ATTENTION_BACKEND' "$ENV_FILE" - ! grep -q -- '--attention-backend' "$SERVE" - # The 3090 is dedicated to this service, so the engine may take most of - # it; the KV pool, not the seq ceiling, caps full-context concurrency. - [ "$util" = "0.95" ] - [ "$seqs" -ge 4 ] +@test "frank-small's slot grid is full-window and fits the 24 GiB card" { + local parallel cache_k cache_v ctx fixed per_slot + parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")" + cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")" + cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")" + ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" + [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ] + # Every slot must be the model's FULL trained window (262144), not a + # divided one -- CTX_SIZE is the total pool and --parallel divides it. + [ $((ctx / parallel)) -eq 262144 ] + [ $((ctx % parallel)) -eq 0 ] + [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ] + # Fit budget from the measured boot arithmetic on the 3090 (24576 + # MiB): one full-context slot idles at 11,996 MiB, four at 22,346, + # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part + # (weights, projector, sidecar draft, compute buffers, CUDA context) + # is 11996-3450 = 8546 MiB. If the config asks for more slots than + # the card holds, this inequality fails before deploy instead of the + # KV allocation failing at boot. + fixed=8546 + per_slot=3450 + [ $((fixed + parallel * per_slot)) -le 24576 ] + # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option, + # without which prefill collapses about 12x (see the CUDA lesson in + # AGENTS.md). The build-cuda tree must carry it. + grep -q '^FLASH_ATTN=on$' "$ENV_FILE" } @test "frank-small runs on its own port beside frank-large's" { - # 8000 is the in-container port; the quadlet publishes it as host 7771, - # the endpoint Hermes, the harness and the homepage probe use. - grep -q '^LISTEN_PORT=8000$' "$ENV_FILE" + grep -q '^LISTEN_PORT=7771$' "$ENV_FILE" grep -q '^ALIAS=frank-small$' "$ENV_FILE" grep -q -- '--port "$LISTEN_PORT"' "$SERVE" grep -q -- '--host "$BIND_ADDR"' "$SERVE" - grep -q '^PublishPort=7771:8000$' "$QUADLET" - # The two endpoints cannot share a port, and only the halogen unit - # publishes 7770. + # The two endpoints cannot share a port, and only the halogen unit publishes + # 7770. grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE" ! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE" } -@test "the deploy handler cuts over from the root unit and gates on health" { +@test "the deploy handler installs the root unit and refuses a missing model" { grep -q 'deploy_llm_small' "$REMOTE_DEPLOY" - # The digest-pinned docker.io base is pulled and re-tagged to the FROM - # ref, then the Dockerfile is built into the localhost tag the quadlet - # runs (the vllm[audio] extra the base ships without). - grep -q 'VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:' "$REMOTE_DEPLOY" - grep -q 'VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"' "$REMOTE_DEPLOY" - grep -q 'podman tag "\$VLLM_SMALL_BASE_IMAGE" "\$VLLM_SMALL_BASE_ALIAS"' "$REMOTE_DEPLOY" - grep -q 'podman build --pull=never -t "\$VLLM_SMALL_IMAGE"' "$REMOTE_DEPLOY" - grep -q 'dankclaw/frank-small/Dockerfile' "$REMOTE_DEPLOY" - # The old ROOT unit is retired (stop, disable, trash, reload) before the - # user unit takes the port — two managers must never own frank-small. - grep -q 'retire_root_llm_small' "$REMOTE_DEPLOY" - grep -q 'wait_port_free 7771' "$REMOTE_DEPLOY" - # Missing payloads refuse the start, and the health gate is the real - # endpoint. - grep -q 'gemma4-12b-qat-w4a16-ct/model.safetensors' "$REMOTE_DEPLOY" - grep -q 'gemma4-12b-assistant-bf16/model.safetensors' "$REMOTE_DEPLOY" + grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY" + grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY" + grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY" + grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY" grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY" grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY" } -@test "frank-small's entrypoint refuses to start on a half-finished download" { +@test "frank-small's serve script refuses to start on a half-finished download" { grep -q 'frank-small: missing' "$SERVE" }