frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
This commit is contained in:
Frank
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
the gfx1151 iGPU) and `frank-small` (vLLM on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
|
||||
| Dankclaw | `frank-small` | vLLM `gemma-x86_64-cu129` dev build + audio extra (rootless Quadlet) | google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + Google's bf16 MTP assistant on the RTX 3090, 262144 window / 289,647-token bf16 KV pool / 2 full-context sessions, native text+image+audio — the fleet's multimodal endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
|
||||
and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
|
||||
their own files — frank-large and frank-small by their `dankclaw/`
|
||||
quadlet/Dockerfile directories, frank-medium by its unit/env/serve trio in `scripts/` — so unrelated
|
||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||
runs on a host without passwordless sudo; the value never enters the repo.
|
||||
|
||||
@@ -232,8 +232,10 @@ merely wedged kernel usually still answers ARP.
|
||||
`~/.config/containers/systemd/frank-large.container` aside and
|
||||
`systemctl --user daemon-reload` — the quadlet generator re-creates that
|
||||
unit from the file on every reload, so the source is the lever, not `mask`.
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankclaw `frank-small` (rootless Quadlet since 2026-09-19, vLLM on the 3090):
|
||||
`systemctl --user stop frank-small`, then move its
|
||||
`~/.config/containers/systemd/frank-small.container` aside and
|
||||
`systemctl --user daemon-reload` (same quadlet-source lever as frank-large).
|
||||
- Dankserver `frank-medium` (system unit):
|
||||
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
|
||||
2. Bench.
|
||||
@@ -263,10 +265,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
|
||||
`frank-large` is text-only because its halogen deployment loads no vision
|
||||
tower. There is **no audio** input anywhere in the fleet: that capability left
|
||||
with the FLM NPU endpoint on 2026-09-11.
|
||||
of 2026-09-19 `frank-small` is the fleet's multimodal endpoint
|
||||
(`input: [text, image, audio]`) — vLLM's native Gemma4 vision AND audio
|
||||
towers, both verified live. `frank-large` is text-only because its halogen
|
||||
deployment loads no vision tower.
|
||||
|
||||
### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -301,31 +303,37 @@ that as "tight and fragmented", not "out of memory".
|
||||
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
Rollback: [[frank-large-ik-uncensored]].
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
### frank-small — vLLM on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
Mainline llama.cpp master `37b53fd` (CUDA build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
|
||||
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
|
||||
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
|
||||
(four FULL 262,144-token slots — the measured card ceiling), batch
|
||||
**8192/1024**, greedy default. Deployed flags are in
|
||||
`scripts/frank-small.env`; the serve script refuses to start if the
|
||||
weights, the projector, the sidecar or the binary are missing.
|
||||
vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (digest-pinned, audio
|
||||
extra pip-installed into `localhost/vllm/vllm-openai:frank-small` — see
|
||||
`dankclaw/frank-small/Dockerfile`) as a **rootless Quadlet**
|
||||
(`dankclaw/frank-small/quadlet/frank-small.container`), serving
|
||||
`google/gemma-4-12B-it-qat-w4a16-ct` (compressed-tensors w4a16, int4 Marlin
|
||||
weights, 10.26 GiB) with Google's **bf16** MTP assistant
|
||||
(`google/gemma-4-12B-it-assistant`, 845 MB). Port 7771, 262,144 window,
|
||||
bf16 KV pool of 289,647 tokens, **2 concurrent full-context sessions**
|
||||
(the 3rd queues), CUDA graphs on, `ENFORCE_EAGER=0`.
|
||||
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; an image request is answered from the projector;
|
||||
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
|
||||
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
|
||||
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
|
||||
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
100 is WORSE at long ctx) is tabled in the runbook.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
The w4a16-ct assistant CANNOT draft in vLLM (the drafter path skips the
|
||||
Marlin kernel — packed int4 misread, acceptance 1.00); the bf16 assistant
|
||||
measures acceptance 2.80–3.56. fp8 KV is hardware-dead on sm_86 (Ampere
|
||||
has no fp8 tensor cores), so bf16 is the only option.
|
||||
|
||||
Dankclaw has **no nvidia-container-toolkit**: the unit wires CUDA by hand
|
||||
(`Device=` for the four nvidia nodes + host `libcuda`/`libnvidia-ml`
|
||||
mounted read-only into two library paths). Host driver is
|
||||
`nvidia-driver-610` 610.57.04 on Ubuntu 26.04; `sm_86` is the correct
|
||||
compute capability — no arch override.
|
||||
|
||||
Verified at deploy (2026-09-19): `/v1/models` reports `frank-small`; chat
|
||||
returns content plus `reasoning` (vLLM's gemma4 parser field — the harness
|
||||
maps it via `compat.reasoningContentField`); a tool call returns
|
||||
`tool_calls`; an image is described; a 440 Hz WAV is described (native
|
||||
audio encoder). Measured: pp@67k 1,293 t/s, pp@213k 608 t/s,
|
||||
tg@213k ~13.6 t/s (bandwidth-bound), tg@85k ~17.6 t/s.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md`.
|
||||
Rollback: [[frank-small-gemma4-12b]] (llama.cpp, weights still on host).
|
||||
|
||||
### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
|
||||
|
||||
@@ -707,7 +715,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-small (vLLM Gemma4-12B w4a16, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md` |
|
||||
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -719,6 +727,7 @@ Retired (rollback targets and history):
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 09-19 | `frank-small-gemma4-12b.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# frank-small runtime image: the digest-pinned vLLM gemma-x86_64-cu129 base
|
||||
# (vLLM 0.22.1rc1.dev357+g74b5964f0 — the image that loads
|
||||
# gemma-4-12B-it-qat-w4a16-ct, runs the Gemma4MTPModel drafter and carries
|
||||
# the gemma4 reasoning/tool-call parsers) plus the vllm[audio] extra.
|
||||
#
|
||||
# The base image ships WITHOUT the audio extras, so an input_audio content
|
||||
# part 500s with "Please install vllm[audio] for audio support" (measured
|
||||
# 2026-09-19). The extra is just four PyPI packages (av, scipy, soundfile,
|
||||
# mistral_common[audio]); installing the extras directly — not
|
||||
# `pip install vllm[audio]` — keeps the pinned dev build in place: a bare
|
||||
# `vllm[audio]` would make pip reach for the PyPI release and replace the
|
||||
# 0.22.1rc1.dev build that knows gemma4.
|
||||
#
|
||||
# Built by scripts/deploy-remote.sh on dankclaw (rootless podman, network on
|
||||
# for pip): the base is pulled by digest and re-tagged to the FROM ref below
|
||||
# first, then `podman build` produces the :frank-small tag the quadlet runs.
|
||||
FROM localhost/vllm/vllm-openai:gemma4-unified
|
||||
|
||||
# The vllm[audio] extra as raw packages (header explains why not
|
||||
# `pip install vllm[audio]`): audio input needs av/scipy/soundfile plus
|
||||
# the mistral_common audio protocol pieces.
|
||||
RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]"
|
||||
@@ -0,0 +1,100 @@
|
||||
# Quadlet unit: frank-small (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/frank-small.container
|
||||
# Unit name: frank-small.service (container name: frank-small)
|
||||
#
|
||||
# The small inference service. vLLM (since 2026-09-18) serving
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16) with the
|
||||
# google bf16 MTP assistant for speculative decoding, on the RTX
|
||||
# 3090. It replaced the mainline-llama.cpp ROOT unit (the old
|
||||
# scripts/frank-small.service is retired into scripts/.AGENTTRASH/).
|
||||
#
|
||||
# SHAPE. The quadlet replaces the image's `vllm serve` entrypoint with
|
||||
# /scripts/serve-frank-small.sh (mounted read-only out of the repo checkout)
|
||||
# so the flag set stays in git. The model and drafter directories are bound
|
||||
# read-only: the payloads are fetched and sha256-verified by
|
||||
# scripts/download-frank-small-vllm.sh before first start, and the deploy
|
||||
# handler refuses to start the unit when either is absent.
|
||||
#
|
||||
# CUDA WITHOUT the nvidia-container-toolkit. This host has never run a CUDA
|
||||
# container (the 3090 served frank-small as a native binary, the iGPU runs
|
||||
# ROCm through /dev/kfd), so there is no nvidia hook to lean on. The recipe
|
||||
# that works (measured 2026-09-18, torch sees the 3090 at 25.3 GB):
|
||||
# - the four /dev/nvidia* nodes via AddDevice (they are 0666 on this host,
|
||||
# so the rootless user namespace can open them);
|
||||
# - the HOST driver's libcuda.so.1 and libnvidia-ml.so.1 mounted read-only
|
||||
# — the image carries its own CUDA 12.9 runtime, but vLLM's platform
|
||||
# probe needs the driver's NVML stub, and without it `vllm serve` dies at
|
||||
# parser construction with "Failed to infer device type";
|
||||
# - NVIDIA_VISIBLE_DEVICES=all.
|
||||
# memlock must be unlimited: CUDA pins its staging buffers.
|
||||
#
|
||||
# SIZING. The tunables (full 262144 window, 0.95 of a dedicated 24 GiB card,
|
||||
# bf16 KV, 16-seq batch ceiling, 16k prefill chunk, 4-token MTP draft) live
|
||||
# in scripts/frank-small.env. The boot log's "Maximum concurrency for N
|
||||
# tokens per request" line is the naive pool ratio for a hybrid SWA model
|
||||
# and undercounts; verify the session count behaviourally on restart.
|
||||
#
|
||||
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
||||
# path cannot resolve registry-prefixed refs from local storage at create
|
||||
# time. The tag is a LOCAL BUILD, not a pull: deploy-remote.sh pulls the
|
||||
# digest-pinned docker.io base (the gemma-x86_64-cu129 build, vLLM
|
||||
# 0.22.1rc1.dev357+g74b5964f0 — the image that boots this checkpoint) and
|
||||
# re-tags it, then builds dankclaw/frank-small/Dockerfile on top, which
|
||||
# installs the vllm[audio] extra (av, scipy, soundfile,
|
||||
# mistral_common[audio]) so the model's native audio path works. Pull=never
|
||||
# keeps boots off the registry.
|
||||
[Unit]
|
||||
Description=frank-small — vLLM gemma4_unified w4a16 + MTP on RTX 3090
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-small
|
||||
Image=localhost/vllm/vllm-openai:frank-small
|
||||
# Published port stays 7771: Hermes, the harness and the homepage probe all
|
||||
# point here. 8000 (vLLM's in-container port) is never reachable directly.
|
||||
PublishPort=7771:8000
|
||||
# CUDA device nodes (0666 on this host — rootless can open them).
|
||||
AddDevice=/dev/nvidia0
|
||||
AddDevice=/dev/nvidiactl
|
||||
AddDevice=/dev/nvidia-uvm
|
||||
AddDevice=/dev/nvidia-uvm-tools
|
||||
# Host driver stubs the image's CUDA runtime and vLLM's NVML probe need.
|
||||
# The image's LD_LIBRARY_PATH leads with /usr/local/nvidia/lib64 — the
|
||||
# nvidia-container-toolkit's conventional mount point, which this host does
|
||||
# not have — so the driver libs go there; the /usr/lib mounts cover torch
|
||||
# Inductor's own lookup.
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/local/nvidia/lib64/libcuda.so.1:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/local/nvidia/lib64/libcuda.so:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/local/nvidia/lib64/libnvidia-ml.so.1:ro
|
||||
Environment=NVIDIA_VISIBLE_DEVICES=all
|
||||
# Payloads, read-only: fetched and sha256-verified by
|
||||
# scripts/download-frank-small-vllm.sh before first start.
|
||||
Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro
|
||||
Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro
|
||||
# Serve script and env, read-only out of the repo checkout.
|
||||
Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro
|
||||
ContainerEntrypoint=/scripts/serve-frank-small.sh
|
||||
Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env
|
||||
Ulimit=memlock=-1:-1
|
||||
# /health answers 200 once the engine is up; vLLM 503s it while the model is
|
||||
# still loading, so a green check means ready-to-serve, not process-alive.
|
||||
# curl ships in the image.
|
||||
HealthCmd=curl -sf http://127.0.0.1:8000/health
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=600s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start reads ~10 GiB of weights and compiles the CUDA graphs.
|
||||
TimeoutStartSec=900
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
+13
-5
@@ -319,6 +319,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# its unit and fleet settings live in one .container file.
|
||||
_add(services, "frank-large")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-small":
|
||||
# The service is a rootless Quadlet container since 2026-09-18
|
||||
# (vLLM); its unit lives in one .container file. The old root
|
||||
# unit's trio is in scripts/ and maps there.
|
||||
_add(services, "frank-small")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# The five observability containers became Quadlet units on
|
||||
# 2026-09-11. A unit change selects exactly its own service;
|
||||
@@ -402,11 +408,13 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-small only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
if name in {"frank-small.env", "serve-frank-small.sh",
|
||||
"download-frank-small-vllm.sh"}:
|
||||
# vLLM frank-small (2026-09-18) keeps its env, entrypoint and
|
||||
# fetch script here; each one selects frank-small only. The
|
||||
# unit is the quadlet under dankclaw/frank-small/, and the old
|
||||
# root unit file is in scripts/.AGENTTRASH/, which is archive
|
||||
# material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
|
||||
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
|
||||
|
||||
+76
-14
@@ -1139,28 +1139,90 @@ wait_for_llm_large_ready() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
|
||||
# ROOT unit: the runtime is a native CUDA binary, so there is no container
|
||||
# image to install and no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
|
||||
# is that unit's [Install] section.
|
||||
# deploy_llm_small — frank-small is a rootless Quadlet running vLLM on the
|
||||
# RTX 3090 since 2026-09-18: the runtime is the digest-pinned
|
||||
# vllm/vllm-openai gemma4-unified image, and the unit is generated from
|
||||
# dankclaw/frank-small/quadlet/. It replaces the mainline-llama.cpp
|
||||
# deployment, which ran as a ROOT unit.
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
# TWO MANAGERS, ONE UNIT NAME. The llama.cpp incarnation ran as a ROOT unit
|
||||
# whose file is a symlink into this checkout; the vLLM unit is a USER
|
||||
# (quadlet) unit. Both must never exist at once — they would race for port
|
||||
# 7771 — so the root unit is retired first: stop, disable, move the /etc
|
||||
# symlink into the home trash (never `rm`), reload the root manager. This
|
||||
# host's `systemctl disable` has been observed to delete the unit file
|
||||
# itself (2026-08-26 incident), so the move tolerates a file that is
|
||||
# already gone.
|
||||
#
|
||||
# The payloads are fetched and sha256-verified by
|
||||
# scripts/download-frank-small-vllm.sh before first start; the handler
|
||||
# refuses to launch a unit that cannot load its model or its drafter.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/gemma4-12b-qat-w4a16-ct/model.safetensors}"
|
||||
LLM_SMALL_DRAFT="${LLM_SMALL_DRAFT:-/home/dank/models/gemma4-12b-assistant-bf16/model.safetensors}"
|
||||
# Base image: the gemma-x86_64-cu129 build (vLLM 0.22.1rc1.dev357+
|
||||
# g74b5964f0) — the image that boots this checkpoint. The June 3
|
||||
# gemma4-unified build predates it and fails at weight load (missing
|
||||
# vision_embedder.patch_dense); the Sep 15 cu129-nightly fails at import
|
||||
# (torchvision ABI). The unit runs a LOCAL BUILD on top of this base —
|
||||
# dankclaw/frank-small/Dockerfile installs the vllm[audio] extra, which
|
||||
# the base ships without.
|
||||
VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:e24d3d8e367ebc2a6ccfc8cf3cecee858a30a6c66b5b1642d85363cd745e001e"
|
||||
VLLM_SMALL_BASE_ALIAS="localhost/vllm/vllm-openai:gemma4-unified"
|
||||
VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"
|
||||
|
||||
retire_root_llm_small() {
|
||||
local trash="$HOME/.AGENTTRASH/20260918-vllm-cutover/host-units"
|
||||
run_privileged systemctl stop frank-small.service 2>/dev/null || true
|
||||
run_privileged systemctl disable frank-small.service 2>/dev/null || true
|
||||
if [[ -e /etc/systemd/system/frank-small.service || -L /etc/systemd/system/frank-small.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
run_privileged mv -- /etc/systemd/system/frank-small.service \
|
||||
"$trash/frank-small.service"
|
||||
echo "retired root unit frank-small.service -> $trash"
|
||||
else
|
||||
# This host's `systemctl disable` is allowed to delete the unit file
|
||||
# outright (2026-08-26 incident), so there is often nothing left to
|
||||
# move. Say so: previously no message meant no evidence either way.
|
||||
echo "root unit frank-small.service was already removed by disable"
|
||||
fi
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_small() {
|
||||
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
|
||||
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
|
||||
echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
|
||||
echo "frank-small: run scripts/download-frank-small-vllm.sh first (main model + MTP assistant, sha256-verified);" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
relink_unit frank-small
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-small.service
|
||||
run_privileged systemctl restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
if [[ ! -f "$LLM_SMALL_DRAFT" ]]; then
|
||||
echo "ERROR: MTP assistant absent at $LLM_SMALL_DRAFT" >&2
|
||||
echo "frank-small: run scripts/download-frank-small-vllm.sh first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot draft." >&2
|
||||
return 1
|
||||
fi
|
||||
retire_root_llm_small
|
||||
# The predecessor can also be THIS quadlet -- a redeploy of a running
|
||||
# frank-small -- so stop the user unit before waiting on the port. On a
|
||||
# first cutover it is not running and this is a no-op.
|
||||
systemctl --user stop frank-small.service 2>/dev/null || true
|
||||
# Pull the base by digest, re-tag it to the FROM ref the Dockerfile
|
||||
# expects, then build the derived image the quadlet runs: the quadlet
|
||||
# path cannot resolve registry-prefixed refs from local storage at
|
||||
# create time (same workaround as halogen), and Pull=never keeps boots
|
||||
# off the registry. The build needs network for pip (the vllm[audio]
|
||||
# extra); podman build has it on by default.
|
||||
podman pull "$VLLM_SMALL_BASE_IMAGE"
|
||||
podman tag "$VLLM_SMALL_BASE_IMAGE" "$VLLM_SMALL_BASE_ALIAS"
|
||||
podman build --pull=never -t "$VLLM_SMALL_IMAGE" \
|
||||
-f dankclaw/frank-small/Dockerfile dankclaw/frank-small
|
||||
# Nothing may start until the predecessor has released the port.
|
||||
wait_port_free 7771 || return 1
|
||||
install_quadlet frank-small dankclaw/frank-small/quadlet frank-small
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence comes from [Install] in the unit file.
|
||||
systemctl --user restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 1200
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a
|
||||
|
||||
Executable
+80
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetch the frank-small vLLM payloads on dankclaw (RTX 3090, since 2026-09-18):
|
||||
#
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct
|
||||
# compressed-tensors w4a16 (int4, group 32) QAT checkpoint — the model
|
||||
# vLLM's gemma4_unified implementation loads natively. One 9.56 GiB
|
||||
# safetensors plus tokenizer/config.
|
||||
#
|
||||
# google/gemma-4-12B-it-assistant
|
||||
# the MTP draft/assistant (gemma4_unified_assistant, 4 layers) in
|
||||
# bf16 — Google's original. 806 MiB. The w4a16-ct re-quant
|
||||
# (kunhunjon) that was used first drafts at acceptance 1.00: vLLM's
|
||||
# MTP drafter path does not apply the Marlin w4a16 kernel to the
|
||||
# assistant's linears (no "Using MarlinLinearKernel" after the
|
||||
# drafter load), so its packed int4 weights are misread. The bf16
|
||||
# assistant accepts at 2.80 (measured 2026-09-19).
|
||||
#
|
||||
# Both repos are public and ungated; no token is needed. Every fetch is -f
|
||||
# (an HTTP error is a failure, not a 153-byte "artifact"), resumable, and the
|
||||
# two LFS payloads are sha256-verified against the Hub's declared oid — the
|
||||
# discipline from scripts/download-frank-medium-gguf.sh.
|
||||
#
|
||||
# The serve script refuses to start when either payload is absent, so a
|
||||
# half-finished run of this script reads as a missing file, not a runtime
|
||||
# fault.
|
||||
set -euo pipefail
|
||||
|
||||
MODELS_DIR="${MODELS_DIR:-/home/dank/models}"
|
||||
LOG_DIR="$MODELS_DIR/logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
LOG="$LOG_DIR/$(basename "$0" .sh).log"
|
||||
|
||||
log() { echo "$(date -Is) $*" | tee -a "$LOG"; }
|
||||
|
||||
# fetch <repo> <dir> <file> [sha256] [size]
|
||||
fetch() {
|
||||
local repo="$1" dir="$2" file="$3" sha="${4:-}" size="${5:-}"
|
||||
mkdir -p "$dir"
|
||||
if [[ -n "$size" && -f "$dir/$file" ]] && [[ "$(stat -c%s "$dir/$file")" == "$size" ]]; then
|
||||
log "skip $dir/$file (size $size matches)"
|
||||
else
|
||||
log "fetch $repo/$file"
|
||||
curl -fsSL -C - "https://huggingface.co/$repo/resolve/main/$file" -o "$dir/$file"
|
||||
log "fetched $dir/$file ($(stat -c%s "$dir/$file") bytes)"
|
||||
fi
|
||||
if [[ -n "$sha" ]]; then
|
||||
local actual
|
||||
actual="$(sha256sum "$dir/$file" | cut -d' ' -f1)"
|
||||
if [[ "$actual" != "$sha" ]]; then
|
||||
log "SHA256 MISMATCH $dir/$file: $actual != $sha"
|
||||
return 1
|
||||
fi
|
||||
log "sha256 ok $dir/$file"
|
||||
fi
|
||||
}
|
||||
|
||||
MAIN_REPO=google/gemma-4-12B-it-qat-w4a16-ct
|
||||
ASSISTANT_REPO=google/gemma-4-12B-it-assistant
|
||||
MAIN_DIR="$MODELS_DIR/gemma4-12b-qat-w4a16-ct"
|
||||
ASSISTANT_DIR="$MODELS_DIR/gemma4-12b-assistant-bf16"
|
||||
|
||||
# Hub-declared LFS oids (sha256), 2026-09-18.
|
||||
fetch "$MAIN_REPO" "$MAIN_DIR" model.safetensors \
|
||||
60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3 10264229896
|
||||
fetch "$MAIN_REPO" "$MAIN_DIR" tokenizer.json \
|
||||
cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f 32169626
|
||||
for f in config.json generation_config.json chat_template.jinja \
|
||||
processor_config.json tokenizer_config.json recipe.yaml; do
|
||||
fetch "$MAIN_REPO" "$MAIN_DIR" "$f"
|
||||
done
|
||||
|
||||
fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" model.safetensors \
|
||||
3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df 845719296
|
||||
fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" tokenizer.json \
|
||||
c001d9ada50af662c94f5ab17ec7e09f6438d1bec8246c47fee6510693d8de35 32169884
|
||||
for f in config.json generation_config.json tokenizer_config.json; do
|
||||
fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" "$f"
|
||||
done
|
||||
|
||||
log "ALL FETCHED OK"
|
||||
+69
-100
@@ -1,112 +1,81 @@
|
||||
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
||||
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
||||
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
||||
# Tunables for frank-small (since 2026-09-18): vLLM serving
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
|
||||
# Google publishes "for native, optimized inference with vLLM") with the
|
||||
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
|
||||
#
|
||||
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
||||
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
||||
# CUDA (see the CUDA lesson in AGENTS.md).
|
||||
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
|
||||
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
|
||||
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
|
||||
# Dockerfile) so the model's native audio path works. That image carries
|
||||
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
|
||||
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
|
||||
# card. The older gemma4-unified build (June 3) predates this checkpoint
|
||||
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
|
||||
# import (torchvision ABI); this one is the image that boots.
|
||||
#
|
||||
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
||||
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
||||
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
||||
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
||||
# only a 1024-position window, which is why the KV pool is a fraction of
|
||||
# the full-ctx x all-layers number.
|
||||
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
|
||||
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
|
||||
# embeddings, vision and audio. The fleet's image path lives here: the model
|
||||
# ingests image (and audio) content natively, so no projector file exists
|
||||
# any more and nothing external has to load.
|
||||
#
|
||||
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
||||
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
||||
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
||||
# context loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
||||
# is the only legal shape that carries both.
|
||||
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
|
||||
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
|
||||
# is that a QAT target must draft with a QAT drafter at the same precision.
|
||||
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
|
||||
# MTP path, not a generic draft model.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
||||
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
||||
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
||||
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
||||
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
||||
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
||||
# which would shrink each slot below the full 262144 window once
|
||||
# parallel > 1.
|
||||
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
|
||||
# for as many concurrent sessions as the card holds.
|
||||
# MAX_MODEL_LEN=262144 the model's max_position_embeddings
|
||||
# GPU_MEM_UTIL=0.95 the card is dedicated to this service
|
||||
# KV_CACHE_DTYPE=auto bf16 KV. fp8_e4m3 would halve the KV charge
|
||||
# (289,756 -> 623,666-token pool, one -> two
|
||||
# full-context sessions), but the 3090 is
|
||||
# sm_86 (Ampere) and has no native fp8: the
|
||||
# TRITON_ATTN backend Gemma4 forces emits
|
||||
# cvt.e4m3x2 instructions that exist only from
|
||||
# sm_89 (Ada), so fp8 KV cannot boot on this
|
||||
# card (measured 2026-09-19; the "sm_89
|
||||
# patch" experiment is in .AGENTTRASH).
|
||||
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
|
||||
# caps full-context concurrency
|
||||
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
|
||||
# faster but grow the activation arena that
|
||||
# competes with the KV pool for the card
|
||||
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
|
||||
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
|
||||
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
|
||||
# 1024-token window per request, so the true full-context session count is
|
||||
# higher. Verify it behaviourally (concurrent long requests) on every
|
||||
# restart, not from these comments.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
||||
# params; it carries an audio encoder too). The fleet's image path lives
|
||||
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
||||
# WORKS: input_audio content parts transcribe verbatim (verified
|
||||
# 2026-09-17 with a 16k mono WAV).
|
||||
# Sampling: greedy by default (fleet contract) via
|
||||
# --override-generation-config; a client that sends its own temperature gets
|
||||
# its own, and the harness applies per-model overrides from models.yml.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
# temperature gets its own, and the harness applies per-model overrides
|
||||
# from models.yml.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
||||
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
|
||||
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
||||
# FA_ALL_QUANTS=ON.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
# Container paths, bind-mounted by the quadlet. The host paths are declared
|
||||
# in the quadlet and the deploy handler, not here: this file is read inside
|
||||
# the container.
|
||||
MODEL_DIR=/models/main
|
||||
DRAFT_DIR=/models/assistant
|
||||
|
||||
# Model package. The MTP sidecar file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
||||
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
||||
# what clients ask for -- the harness entry in models.yml names it.
|
||||
# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
|
||||
# host 7771, the endpoint Hermes, the harness and the homepage probe have
|
||||
# always used. The alias is the id at /v1/models and what clients ask for.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
LISTEN_PORT=8000
|
||||
|
||||
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
||||
# pool; --parallel divides it). See the SIZING note above for the fit
|
||||
# arithmetic; a fifth slot does not fit.
|
||||
CTX_SIZE=1048576
|
||||
PARALLEL=4
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
|
||||
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
|
||||
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
|
||||
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
|
||||
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
|
||||
# tried. Idle VRAM is 23,300 MiB at this geometry.
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
||||
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention (required for q8 KV).
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
||||
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
||||
# explicit here (both were the implicit build defaults: 3 and 50).
|
||||
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
|
||||
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
|
||||
# costs more attention re-read at long context than the extra accepted
|
||||
# token earns. Keep 3/50.
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
POLL_LEVEL=50
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling,
|
||||
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
|
||||
MAX_MODEL_LEN=262144
|
||||
GPU_MEM_UTIL=0.95
|
||||
KV_CACHE_DTYPE=auto
|
||||
MAX_NUM_SEQS=16
|
||||
MAX_NUM_BATCHED_TOKENS=16384
|
||||
SPEC_NUM_TOKENS=4
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||
# though the weights themselves are copied into VRAM. Parity with the halogen
|
||||
# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
+58
-113
@@ -1,43 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
|
||||
# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
|
||||
# Serve google/gemma-4-12B-it-qat-w4a16-ct as frank-small on Dankclaw's RTX
|
||||
# 3090 via vLLM (since 2026-09-18), with the google bf16 MTP assistant
|
||||
# for speculative decoding.
|
||||
#
|
||||
# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
|
||||
# weights plus the BF16 vision projector and q8_0 KV inside the card's
|
||||
# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
|
||||
# belongs to the halogen engine that serves frank-large.
|
||||
# This script is the CONTAINER ENTRYPOINT of the frank-small quadlet: the
|
||||
# vllm/vllm-openai image's own entrypoint is `vllm serve`, and the quadlet
|
||||
# replaces it with this file (mounted read-only out of the repo checkout) so
|
||||
# the flag set stays in git. The image pins the gemma4-unified vLLM build,
|
||||
# which is what loads gemma4_unified natively and runs the Gemma 4 MTP path.
|
||||
#
|
||||
# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
|
||||
# passed via --model-draft with --spec-type draft-mtp: the draft context
|
||||
# loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so this chain is the
|
||||
# only shape that carries both.
|
||||
# MTP: the drafter is a separate 4-layer assistant checkpoint in bf16
|
||||
# (Google's original gemma-4-12B-it-assistant). It goes in
|
||||
# --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path, not a
|
||||
# generic draft model. The w4a16-ct re-quant of the assistant was tried
|
||||
# first and drafts at acceptance 1.00 (the drafter path does not apply the
|
||||
# Marlin kernel to its linears); the bf16 assistant accepts at 2.80.
|
||||
#
|
||||
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
|
||||
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
|
||||
# maximises the TOTAL pool instead, which would divide a smaller window
|
||||
# across slots. The 4-slot layout is measured to fit the card (see the
|
||||
# SIZING note in frank-small.env); more slots would fail the KV allocation.
|
||||
# Sampling: greedy by default (fleet contract) via
|
||||
# --override-generation-config; clients that send their own temperature get
|
||||
# their own, and the harness overrides per model from models.yml.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector, which also carries
|
||||
# an audio encoder — input_audio content parts transcribe verbatim
|
||||
# (verified 2026-09-17). The fleet's image path lives here, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
# models.yml. --jinja is pinned even though this build defaults it on: the
|
||||
# model's own template supplies reasoning_effort, preserve_thinking and the
|
||||
# tool-call grammar, and a future build that flips that default must not
|
||||
# silently take tools away.
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
|
||||
# Tunables come from frank-small.env (mounted beside this file by the
|
||||
# quadlet) unless FRANK_SMALL_ENV_FILE overrides the path.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-/scripts/frank-small.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
@@ -46,93 +33,51 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
: "${DRAFT_FILE:=}"
|
||||
: "${MODEL_DIR:=/models/main}"
|
||||
: "${DRAFT_DIR:=/models/assistant}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
: "${CTX_SIZE:=}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
|
||||
# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
|
||||
# busy-wait level (build default 50).
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${POLL_LEVEL:=50}"
|
||||
: "${NUM_THREADS:=8}"
|
||||
: "${THREAD_BATCH:=8}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
: "${LISTEN_PORT:=8000}"
|
||||
: "${MAX_MODEL_LEN:=262144}"
|
||||
: "${GPU_MEM_UTIL:=0.95}"
|
||||
: "${KV_CACHE_DTYPE:=auto}"
|
||||
: "${MAX_NUM_SEQS:=16}"
|
||||
: "${MAX_NUM_BATCHED_TOKENS:=16384}"
|
||||
: "${SPEC_NUM_TOKENS:=4}"
|
||||
: "${ENFORCE_EAGER:=0}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
# is a download that has not finished, and the unit's restart policy would
|
||||
# otherwise turn that into a crash loop that reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
|
||||
for path in "$MODEL_DIR/model.safetensors" "$MODEL_DIR/config.json" \
|
||||
"$DRAFT_DIR/model.safetensors" "$DRAFT_DIR/config.json"; do
|
||||
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
||||
done
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
[[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
|
||||
|
||||
SPEC_CONFIG="{\"method\":\"mtp\",\"model\":\"${DRAFT_DIR}\",\"num_speculative_tokens\":${SPEC_NUM_TOKENS}}"
|
||||
|
||||
VLLM_FLAGS=(
|
||||
serve "$MODEL_DIR"
|
||||
--served-model-name "$ALIAS"
|
||||
--host "$BIND_ADDR"
|
||||
--port "$LISTEN_PORT"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--gpu-memory-utilization "$GPU_MEM_UTIL"
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--speculative-config "$SPEC_CONFIG"
|
||||
--reasoning-parser gemma4
|
||||
--enable-auto-tool-choice
|
||||
--tool-call-parser gemma4
|
||||
--override-generation-config '{"temperature": 0}'
|
||||
)
|
||||
# ENFORCE_EAGER=1 skips Inductor/Triton compilation and CUDA graph capture
|
||||
# (slower steady-state decode, faster and more robust boot). The 3090 is
|
||||
# sm_86 (Ampere) — no native fp8 — so KV stays bf16 and the default
|
||||
# graphed path is the right one; the knob is a boot-debugging lever.
|
||||
if [[ "$ENFORCE_EAGER" == "1" ]]; then
|
||||
VLLM_FLAGS+=(--enforce-eager)
|
||||
fi
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
fi
|
||||
|
||||
DRAFT_ARGS=()
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
|
||||
fi
|
||||
|
||||
# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
|
||||
# whatever the weights, projector, draft context and compute buffers leave
|
||||
# on the card; the engine prints the chosen value at startup.
|
||||
if [[ -n "${CTX_SIZE:-}" ]]; then
|
||||
CTX_ARGS=(--ctx-size "$CTX_SIZE")
|
||||
else
|
||||
CTX_ARGS=(--fit on)
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
"${DRAFT_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
"${CTX_ARGS[@]}" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--poll "$POLL_LEVEL" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET"
|
||||
exec vllm "${VLLM_FLAGS[@]}"
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||||
description: The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant.
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- vllm
|
||||
- rtx3090
|
||||
- gemma
|
||||
- mtp
|
||||
- vision
|
||||
- audio
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
|
||||
# frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||||
|
||||
> **ACTIVE since 2026-09-19.** Replaces [[frank-small-gemma4-12b]]
|
||||
> (llama.cpp, now the rollback target). Same port (7771), same alias
|
||||
> (`frank-small`), same client contract surface — the runtime, the model
|
||||
> payload, the drafter and the service shape (root systemd unit → rootless
|
||||
> Quadlet) all changed.
|
||||
|
||||
## Identity
|
||||
|
||||
| what | value |
|
||||
|---|---|
|
||||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d7…`) |
|
||||
| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (root systemd unit retired 2026-09-19) |
|
||||
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
|
||||
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
|
||||
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
|
||||
| port | 7771 → 8000 (alias `frank-small`; `frank-large` keeps 7770) |
|
||||
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
|
||||
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
|
||||
|
||||
## Why vLLM
|
||||
|
||||
- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
|
||||
kernel is its native int4 GEMM path. llama.cpp could not load it at all.
|
||||
- Native audio encoder in the checkpoint — the fleet's only audio-capable
|
||||
endpoint moves here (llama.cpp's path needed the mmproj audio tower).
|
||||
- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
|
||||
this host had no graph support.
|
||||
|
||||
## The drafter-Marlin bug (why the assistant is bf16)
|
||||
|
||||
The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
|
||||
**cannot draft in vLLM**: the drafter's linear layers are loaded through a
|
||||
path that does not apply the Marlin kernel, so the packed int4 weights are
|
||||
misread as bf16 and every draft is garbage — measured acceptance **1.00**
|
||||
(pure pass-through). There is no "Using MarlinLinearKernel" log line after
|
||||
the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
|
||||
has no quantization to miss, and measures **acceptance 2.80–3.56**
|
||||
(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
|
||||
(634 → 845 MB) is absorbed by pool rounding: both assistants yield the
|
||||
identical 289,647-token KV pool.
|
||||
|
||||
## KV arithmetic — why two full-context sessions
|
||||
|
||||
Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
|
||||
16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
|
||||
per-token KV is dominated by the full-attention layers:
|
||||
6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
|
||||
289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
|
||||
sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
|
||||
shares the 1024-window ring across all concurrent requests, so the
|
||||
marginal cost of a *second* 213k session is ~6 full-attention GiB, not
|
||||
a second copy of everything.
|
||||
|
||||
Measured (salted, 2026-09-19):
|
||||
|
||||
| load | KV usage | result |
|
||||
|---|---|---|
|
||||
| 1 × 213k | ~35 % | runs |
|
||||
| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
|
||||
| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |
|
||||
|
||||
**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
|
||||
deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
|
||||
bytes, and vLLM's pool arithmetic is coarser. This is the accepted
|
||||
trade for the native w4a16 format + audio + CUDA graphs.)
|
||||
|
||||
## Measured throughput (salted, 2026-09-19)
|
||||
|
||||
| probe | tokens | wall | rate |
|
||||
|---|---|---|---|
|
||||
| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
|
||||
| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
|
||||
| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
|
||||
| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
|
||||
| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
|
||||
| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |
|
||||
|
||||
- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
|
||||
layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
|
||||
as llama.cpp's flash-attn path here.
|
||||
- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
|
||||
token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
|
||||
right at it. At normal depths decode is far faster (the 67k-depth probe
|
||||
is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
|
||||
numbers.
|
||||
- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
|
||||
scale linearly in tokens — BPE compresses the repetition harder as the
|
||||
span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
|
||||
predicts). Size fillers with the model's own tokenizer
|
||||
(`AutoTokenizer.encode` on the exact prefix), not char counts.
|
||||
- `POST /flush_cache` is **404** in this build — salt every probe
|
||||
(uuid prefix) instead.
|
||||
|
||||
## The image
|
||||
|
||||
`dankclaw/frank-small/Dockerfile` (repo) — 16 comment lines + `FROM
|
||||
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
|
||||
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
|
||||
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
|
||||
`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
|
||||
after the driver swap, so no arch override is needed.
|
||||
|
||||
## No nvidia-container-toolkit — manual CUDA mounts
|
||||
|
||||
Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
|
||||
by hand:
|
||||
|
||||
- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
|
||||
`/dev/nvidia-uvm-tools`
|
||||
- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
|
||||
into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
|
||||
- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`
|
||||
|
||||
If the driver is upgraded on the host, these mount sources move with it
|
||||
(they are versionless symlinks) — no unit change needed.
|
||||
|
||||
## Host driver note
|
||||
|
||||
Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
|
||||
Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
|
||||
595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
|
||||
the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
|
||||
not a hardware mystery.
|
||||
|
||||
## Deployed config
|
||||
|
||||
Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`
|
||||
(mounted into the container at `/scripts/`, which is the entrypoint) +
|
||||
`dankclaw/frank-small/quadlet/frank-small.container`. Key values:
|
||||
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
|
||||
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
|
||||
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
|
||||
`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
|
||||
is **removed**. The serve script refuses to start if the weights, the
|
||||
assistant, or the vllm binary are missing.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
|
||||
parser), not `reasoning_content` — the harness maps it via
|
||||
`compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
|
||||
The field is empty when the model emits no thinking tag (behavior, not
|
||||
a parser bug).
|
||||
- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
|
||||
accepted (200, not 400) — vLLM ignores the unknown key.
|
||||
- Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
|
||||
- Tools → `tool_calls` (verified `get_weather`).
|
||||
- Image → 200, described correctly (native vision tower).
|
||||
- Audio → 200, a 440 Hz WAV tone described correctly (native audio
|
||||
encoder; verified in the derived image).
|
||||
- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
|
||||
`input: [text, image, audio]`.
|
||||
|
||||
## Boot ledger
|
||||
|
||||
t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
|
||||
bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
|
||||
KV pool 289,647 tokens). Health is 503 until ready — do not kill the
|
||||
container while it loads.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
|
||||
(drafter-Marlin bug). Confirm the assistant mount points at
|
||||
`gemma4-12b-assistant-bf16`.
|
||||
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
|
||||
must be absent. 8.6 is correct.
|
||||
- **Audio 400/500** — the container is running the *base* image, not the
|
||||
derived `:frank-small` (missing `av`). Check the deployed image digest
|
||||
is `8dad46d7…`.
|
||||
- **CUDA not visible in container** — a `Device=` line or a lib mount
|
||||
dropped out of the unit after an edit; the unit is the source of truth,
|
||||
the host-side generated unit is not.
|
||||
- Logs: `journalctl --user -u frank-small` (rootless Quadlet; `podman
|
||||
logs` covers only the current `--replace --rm` instance).
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||||
mmproj and MTP sidecar are still on the host under
|
||||
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
|
||||
`scripts/.AGENTTRASH/frank-small.service.llama.cpp`. Restoring it means
|
||||
reinstalling that unit and stopping the Quadlet (one owner per port).
|
||||
|
||||
## References
|
||||
|
||||
- `scripts/serve-frank-small.sh` — the actual exec line
|
||||
- `dankclaw/frank-small/Dockerfile` — the audio-derived image
|
||||
- `dankclaw/frank-small/quadlet/frank-small.container` — the unit
|
||||
- [[frank-small-gemma4-12b]] — RETIRED predecessor (rollback)
|
||||
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
|
||||
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
|
||||
@@ -5,9 +5,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-16):
|
||||
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
|
||||
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
|
||||
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
|
||||
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
|
||||
7771).
|
||||
w4b on the gfx1151 iGPU, port 7770) and vLLM (frank-small,
|
||||
google/gemma-4-12B-it-qat-w4a16-ct with the bf16 MTP assistant on the RTX
|
||||
3090, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -35,10 +35,10 @@ gfx1151 iGPU 64 GiB unified memory]
|
||||
frank-large, port 7770]
|
||||
F[peonist-ai Qwen3.8-Flash-Next w4b
|
||||
2 x 200192 ctx over one 400384-position pool]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
G[vLLM gemma-x86_64-cu129
|
||||
frank-small, port 7771]
|
||||
H[Qwen3.8-27B GSQ-RCO IQ3_S
|
||||
131072 ctx / 1 slot, MTP + vision]
|
||||
H[gemma-4-12B-it-qat-w4a16-ct
|
||||
262144 ctx / 2 full-context sessions, MTP + vision + audio]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -87,11 +87,15 @@ frank-small, port 7771]
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB
|
||||
- [[frank-small-vllm-gemma4]] — ACTIVE frank-small runbook since 2026-09-19
|
||||
(rootless Quadlet, port 7771): vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-token
|
||||
pool, 2 concurrent full-context sessions, native text+image+audio
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): the
|
||||
prior frank-small runbook (2026-09-17 → 09-19), mainline llama.cpp serving
|
||||
HauhauCS Gemma4-12B-QAT-Uncensored Q4_K_M with the MTP sidecar, q8_0 KV,
|
||||
4 slots of 262k
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
|
||||
@@ -166,11 +166,17 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
|
||||
2026-09-17: mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
|
||||
- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
|
||||
2026-09-19: vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
|
||||
full-context sessions, native text+image+audio) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
|
||||
Dankclaw's RTX 3090
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
|
||||
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
|
||||
|
||||
@@ -64,7 +64,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
|
||||
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-small" ]
|
||||
|
||||
@@ -72,7 +73,8 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# that name was retired with the FreeToken runtime on 2026-09-15 and must
|
||||
# not come back through a planner mapping.
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
|
||||
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@@ -72,7 +72,11 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "frank-large loads no vision tower: the fleet's image path is frank-small" {
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET"
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
# frank-small's vLLM checkpoint (2026-09-18) is natively multimodal —
|
||||
# image content is ingested by the model itself, so the seam that used
|
||||
# to pin --mmproj now pins the projector's absence.
|
||||
! grep -q 'mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
grep -q '^MODEL_DIR=/models/main$' "$REPO_ROOT/scripts/frank-small.env"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the previous ROOT unit without deleting it" {
|
||||
|
||||
+115
-73
@@ -1,115 +1,157 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
|
||||
# went back to the halogen engine on the iGPU. This seam pins what the flags
|
||||
# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
|
||||
# is involved), the repo-relative paths the unit executes, the separate client
|
||||
# port, the MTP contract (the head is a separate sidecar file loaded as the
|
||||
# draft model, chained behind ngram-mod), the vision handoff from frank-large,
|
||||
# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
|
||||
# Seam: the frank-small vLLM quadlet on dankclaw (since 2026-09-18).
|
||||
# frank-small is the RTX 3090 endpoint. It now runs vLLM (the pinned
|
||||
# gemma4-unified image) serving google/gemma-4-12B-it-qat-w4a16-ct with the
|
||||
# google bf16 MTP assistant, replacing the mainline-llama.cpp root
|
||||
# unit (retired into scripts/.AGENTTRASH/). This seam pins what the flags
|
||||
# alone do not say: boot persistence for the generated quadlet, the CUDA
|
||||
# wiring this host needs (no nvidia-container-toolkit), the repo-relative
|
||||
# entrypoint and env, the MTP contract (a separate QAT assistant checkpoint
|
||||
# in --speculative-config, method "mtp"), the fleet's image path (the model
|
||||
# is natively multimodal — no projector file), the full-window sizing, the
|
||||
# separate client port, and the deploy handler's cutover order.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-small.service"
|
||||
QUADLET="$REPO_ROOT/dankclaw/frank-small/quadlet/frank-small.container"
|
||||
DOCKERFILE="$REPO_ROOT/dankclaw/frank-small/Dockerfile"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-small-vllm.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
|
||||
@test "frank-small unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
@test "frank-small quadlet, Dockerfile, entrypoint, env and fetch script exist" {
|
||||
[ -f "$QUADLET" ]
|
||||
[ -f "$DOCKERFILE" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$FETCH" ]
|
||||
# The old root unit is retired, not deleted: it must live in the trash,
|
||||
# not in scripts/, so the planner and relink_unit never see it.
|
||||
[ ! -e "$REPO_ROOT/scripts/frank-small.service" ]
|
||||
[ -e "$REPO_ROOT/scripts/.AGENTTRASH/frank-small.service.llama.cpp" ]
|
||||
}
|
||||
|
||||
@test "frank-small is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
@test "frank-small is boot-persistent, self-healing and registry-pinned" {
|
||||
grep -q '^WantedBy=default.target$' "$QUADLET"
|
||||
grep -q '^Restart=on-failure$' "$QUADLET"
|
||||
grep -q '^RestartSec=' "$QUADLET"
|
||||
grep -q '^Pull=never$' "$QUADLET"
|
||||
# The unit runs the localhost/-aliased LOCAL BUILD (base + vllm[audio]
|
||||
# extra); the digest-pinned docker.io base lives only in the deploy
|
||||
# handler, which pulls, re-tags and builds before restart.
|
||||
grep -q '^Image=localhost/vllm/vllm-openai:frank-small$' "$QUADLET"
|
||||
! grep -q '^Image=docker.io' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-small runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
|
||||
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
|
||||
@test "frank-small wires CUDA without the nvidia-container-toolkit" {
|
||||
# The four device nodes (0666 on this host, so rootless can open them)...
|
||||
grep -q '^AddDevice=/dev/nvidia0$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/nvidiactl$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/nvidia-uvm$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/nvidia-uvm-tools$' "$QUADLET"
|
||||
# ...plus the host driver's libcuda and NVML stubs. Without the NVML
|
||||
# mount vllm serve dies at parser construction with "Failed to infer
|
||||
# device type" (measured 2026-09-18).
|
||||
grep -q 'libcuda.so.1.*:ro' "$QUADLET"
|
||||
grep -q 'libnvidia-ml.so.1.*:ro' "$QUADLET"
|
||||
grep -q '^Environment=NVIDIA_VISIBLE_DEVICES=all$' "$QUADLET"
|
||||
# CUDA pins its staging buffers; memlock must be unlimited.
|
||||
grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-small offloads every layer to the 3090 with flash attention" {
|
||||
grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
|
||||
grep -q '^NGL=999$' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint; without --metrics there is no /metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
@test "frank-small runs the repo's own entrypoint with the repo's env" {
|
||||
grep -q '^ContainerEntrypoint=/scripts/serve-frank-small.sh$' "$QUADLET"
|
||||
grep -q '^Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env$' "$QUADLET"
|
||||
grep -q 'Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
|
||||
grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
# The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
|
||||
# nextn layers); it loads as --model-draft and shares the target's last
|
||||
# layers' KV. ngram-mod is draftless, so the chain is legal.
|
||||
grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
|
||||
@test "frank-small mounts the sha256-verified payloads read-only" {
|
||||
grep -q 'Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro' "$QUADLET"
|
||||
grep -q 'Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro' "$QUADLET"
|
||||
# The fetch script verifies both LFS payloads against the Hub oids.
|
||||
grep -q '60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3' "$FETCH"
|
||||
grep -q '3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df' "$FETCH"
|
||||
}
|
||||
|
||||
@test "frank-small drafts with the QAT MTP assistant via method mtp" {
|
||||
# The drafter is a separate 4-layer assistant checkpoint in the same
|
||||
# int4 w4a16 compressed-tensors format (the card's compatibility rule:
|
||||
# a QAT target drafts with a QAT drafter at the same precision), passed
|
||||
# in --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path.
|
||||
grep -qF -- '\"method\":\"mtp\"' "$SERVE"
|
||||
grep -q -- '--speculative-config "$SPEC_CONFIG"' "$SERVE"
|
||||
grep -q '^SPEC_NUM_TOKENS=4$' "$ENV_FILE"
|
||||
# The gemma4 reasoning and tool-call parsers keep the fleet contract:
|
||||
# reasoning_content on the wire and tools answered.
|
||||
grep -q -- '--reasoning-parser gemma4' "$SERVE"
|
||||
grep -q -- '--tool-call-parser gemma4' "$SERVE"
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small carries the fleet's image path, and only it" {
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
# The model is natively multimodal (encoder-free unified): image content
|
||||
# is ingested by the checkpoint itself, so there is no projector file
|
||||
# any more — the seam that used to pin MMPROJ_PATH now pins its absence.
|
||||
! grep -q '^MMPROJ' "$ENV_FILE"
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
|
||||
local parallel cache_k cache_v ctx fixed per_slot
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
|
||||
# Every slot must be the model's FULL trained window (262144), not a
|
||||
# divided one -- CTX_SIZE is the total pool and --parallel divides it.
|
||||
[ $((ctx / parallel)) -eq 262144 ]
|
||||
[ $((ctx % parallel)) -eq 0 ]
|
||||
[ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
|
||||
# Fit budget from the measured boot arithmetic on the 3090 (24576
|
||||
# MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
|
||||
# so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
|
||||
# (weights, projector, sidecar draft, compute buffers, CUDA context)
|
||||
# is 11996-3450 = 8546 MiB. If the config asks for more slots than
|
||||
# the card holds, this inequality fails before deploy instead of the
|
||||
# KV allocation failing at boot.
|
||||
fixed=8546
|
||||
per_slot=3450
|
||||
[ $((fixed + parallel * per_slot)) -le 24576 ]
|
||||
# Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
|
||||
# without which prefill collapses about 12x (see the CUDA lesson in
|
||||
# AGENTS.md). The build-cuda tree must carry it.
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
@test "frank-small serves the full window with bf16 KV on a dedicated card" {
|
||||
local maxlen kvdtype util seqs
|
||||
maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
|
||||
util="$(sed -nE 's/^GPU_MEM_UTIL=([0-9.]+)$/\1/p' "$ENV_FILE")"
|
||||
seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
# The model's FULL trained window (262144), not a divided one.
|
||||
[ "$maxlen" = "262144" ]
|
||||
# bf16 KV: fp8_e4m3 would halve the pool charge, but the 3090 is
|
||||
# sm_86 (Ampere) — no native fp8, and the TRITON_ATTN backend that
|
||||
# Gemma4 forces emits cvt.e4m3x2 instructions that exist only from sm_89.
|
||||
[ "$kvdtype" = "auto" ]
|
||||
# The 3090 is dedicated to this service, so the engine may take most of
|
||||
# it; the KV pool, not the seq ceiling, caps full-context concurrency.
|
||||
[ "$util" = "0.95" ]
|
||||
[ "$seqs" -ge 4 ]
|
||||
}
|
||||
|
||||
@test "frank-small runs on its own port beside frank-large's" {
|
||||
grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
|
||||
# 8000 is the in-container port; the quadlet publishes it as host 7771,
|
||||
# the endpoint Hermes, the harness and the homepage probe use.
|
||||
grep -q '^LISTEN_PORT=8000$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-small$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
# The two endpoints cannot share a port, and only the halogen unit publishes
|
||||
# 7770.
|
||||
grep -q '^PublishPort=7771:8000$' "$QUADLET"
|
||||
# The two endpoints cannot share a port, and only the halogen unit
|
||||
# publishes 7770.
|
||||
grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE"
|
||||
! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing model" {
|
||||
@test "the deploy handler cuts over from the root unit and gates on health" {
|
||||
grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
|
||||
# The digest-pinned docker.io base is pulled and re-tagged to the FROM
|
||||
# ref, then the Dockerfile is built into the localhost tag the quadlet
|
||||
# runs (the vllm[audio] extra the base ships without).
|
||||
grep -q 'VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:' "$REMOTE_DEPLOY"
|
||||
grep -q 'VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"' "$REMOTE_DEPLOY"
|
||||
grep -q 'podman tag "\$VLLM_SMALL_BASE_IMAGE" "\$VLLM_SMALL_BASE_ALIAS"' "$REMOTE_DEPLOY"
|
||||
grep -q 'podman build --pull=never -t "\$VLLM_SMALL_IMAGE"' "$REMOTE_DEPLOY"
|
||||
grep -q 'dankclaw/frank-small/Dockerfile' "$REMOTE_DEPLOY"
|
||||
# The old ROOT unit is retired (stop, disable, trash, reload) before the
|
||||
# user unit takes the port — two managers must never own frank-small.
|
||||
grep -q 'retire_root_llm_small' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7771' "$REMOTE_DEPLOY"
|
||||
# Missing payloads refuse the start, and the health gate is the real
|
||||
# endpoint.
|
||||
grep -q 'gemma4-12b-qat-w4a16-ct/model.safetensors' "$REMOTE_DEPLOY"
|
||||
grep -q 'gemma4-12b-assistant-bf16/model.safetensors' "$REMOTE_DEPLOY"
|
||||
grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "frank-small's serve script refuses to start on a half-finished download" {
|
||||
@test "frank-small's entrypoint refuses to start on a half-finished download" {
|
||||
grep -q 'frank-small: missing' "$SERVE"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user