frank-small: revert vLLM cutover — back to llama.cpp Gemma4-12B Q4_K_M
The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
This commit is contained in:
Frank
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
|
||||
the gfx1151 iGPU) and `frank-small` (vLLM on the RTX 3090). Caddy on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | vLLM `gemma-x86_64-cu129` dev build + audio extra (rootless Quadlet) | google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + Google's bf16 MTP assistant on the RTX 3090, 262144 window / 289,647-token bf16 KV pool / 2 full-context sessions, native text+image+audio — the fleet's multimodal endpoint | 7771 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — frank-large and frank-small by their `dankclaw/`
|
||||
quadlet/Dockerfile directories, frank-medium by its unit/env/serve trio in `scripts/` — so unrelated
|
||||
their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
|
||||
and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
|
||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||
runs on a host without passwordless sudo; the value never enters the repo.
|
||||
|
||||
@@ -232,10 +232,8 @@ merely wedged kernel usually still answers ARP.
|
||||
`~/.config/containers/systemd/frank-large.container` aside and
|
||||
`systemctl --user daemon-reload` — the quadlet generator re-creates that
|
||||
unit from the file on every reload, so the source is the lever, not `mask`.
|
||||
- Dankclaw `frank-small` (rootless Quadlet since 2026-09-19, vLLM on the 3090):
|
||||
`systemctl --user stop frank-small`, then move its
|
||||
`~/.config/containers/systemd/frank-small.container` aside and
|
||||
`systemctl --user daemon-reload` (same quadlet-source lever as frank-large).
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-medium` (system unit):
|
||||
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
|
||||
2. Bench.
|
||||
@@ -265,10 +263,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-19 `frank-small` is the fleet's multimodal endpoint
|
||||
(`input: [text, image, audio]`) — vLLM's native Gemma4 vision AND audio
|
||||
towers, both verified live. `frank-large` is text-only because its halogen
|
||||
deployment loads no vision tower.
|
||||
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
|
||||
`frank-large` is text-only because its halogen deployment loads no vision
|
||||
tower. There is **no audio** input anywhere in the fleet: that capability left
|
||||
with the FLM NPU endpoint on 2026-09-11.
|
||||
|
||||
### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -303,37 +301,31 @@ that as "tight and fragmented", not "out of memory".
|
||||
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
Rollback: [[frank-large-ik-uncensored]].
|
||||
|
||||
### frank-small — vLLM on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (digest-pinned, audio
|
||||
extra pip-installed into `localhost/vllm/vllm-openai:frank-small` — see
|
||||
`dankclaw/frank-small/Dockerfile`) as a **rootless Quadlet**
|
||||
(`dankclaw/frank-small/quadlet/frank-small.container`), serving
|
||||
`google/gemma-4-12B-it-qat-w4a16-ct` (compressed-tensors w4a16, int4 Marlin
|
||||
weights, 10.26 GiB) with Google's **bf16** MTP assistant
|
||||
(`google/gemma-4-12B-it-assistant`, 845 MB). Port 7771, 262,144 window,
|
||||
bf16 KV pool of 289,647 tokens, **2 concurrent full-context sessions**
|
||||
(the 3rd queues), CUDA graphs on, `ENFORCE_EAGER=0`.
|
||||
Mainline llama.cpp master `37b53fd` (CUDA build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
|
||||
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
|
||||
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
|
||||
(four FULL 262,144-token slots — the measured card ceiling), batch
|
||||
**8192/1024**, greedy default. Deployed flags are in
|
||||
`scripts/frank-small.env`; the serve script refuses to start if the
|
||||
weights, the projector, the sidecar or the binary are missing.
|
||||
|
||||
The w4a16-ct assistant CANNOT draft in vLLM (the drafter path skips the
|
||||
Marlin kernel — packed int4 misread, acceptance 1.00); the bf16 assistant
|
||||
measures acceptance 2.80–3.56. fp8 KV is hardware-dead on sm_86 (Ampere
|
||||
has no fp8 tensor cores), so bf16 is the only option.
|
||||
|
||||
Dankclaw has **no nvidia-container-toolkit**: the unit wires CUDA by hand
|
||||
(`Device=` for the four nvidia nodes + host `libcuda`/`libnvidia-ml`
|
||||
mounted read-only into two library paths). Host driver is
|
||||
`nvidia-driver-610` 610.57.04 on Ubuntu 26.04; `sm_86` is the correct
|
||||
compute capability — no arch override.
|
||||
|
||||
Verified at deploy (2026-09-19): `/v1/models` reports `frank-small`; chat
|
||||
returns content plus `reasoning` (vLLM's gemma4 parser field — the harness
|
||||
maps it via `compat.reasoningContentField`); a tool call returns
|
||||
`tool_calls`; an image is described; a 440 Hz WAV is described (native
|
||||
audio encoder). Measured: pp@67k 1,293 t/s, pp@213k 608 t/s,
|
||||
tg@213k ~13.6 t/s (bandwidth-bound), tg@85k ~17.6 t/s.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md`.
|
||||
Rollback: [[frank-small-gemma4-12b]] (llama.cpp, weights still on host).
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; an image request is answered from the projector;
|
||||
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
|
||||
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
|
||||
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
|
||||
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
100 is WORSE at long ctx) is tabled in the runbook.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
|
||||
|
||||
@@ -715,7 +707,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-small (vLLM Gemma4-12B w4a16, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -727,7 +719,7 @@ Retired (rollback targets and history):
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 09-19 | `frank-small-gemma4-12b.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
# frank-small runtime image: the digest-pinned vLLM gemma-x86_64-cu129 base
|
||||
# (vLLM 0.22.1rc1.dev357+g74b5964f0 — the image that loads
|
||||
# gemma-4-12B-it-qat-w4a16-ct, runs the Gemma4MTPModel drafter and carries
|
||||
# the gemma4 reasoning/tool-call parsers) plus the vllm[audio] extra.
|
||||
#
|
||||
# The base image ships WITHOUT the audio extras, so an input_audio content
|
||||
# part 500s with "Please install vllm[audio] for audio support" (measured
|
||||
# 2026-09-19). The extra is just four PyPI packages (av, scipy, soundfile,
|
||||
# mistral_common[audio]); installing the extras directly — not
|
||||
# `pip install vllm[audio]` — keeps the pinned dev build in place: a bare
|
||||
# `vllm[audio]` would make pip reach for the PyPI release and replace the
|
||||
# 0.22.1rc1.dev build that knows gemma4.
|
||||
#
|
||||
# Built by scripts/deploy-remote.sh on dankclaw (rootless podman, network on
|
||||
# for pip): the base is pulled by digest and re-tagged to the FROM ref below
|
||||
# first, then `podman build` produces the :frank-small tag the quadlet runs.
|
||||
FROM localhost/vllm/vllm-openai:gemma4-unified
|
||||
|
||||
# The vllm[audio] extra as raw packages (header explains why not
|
||||
# `pip install vllm[audio]`): audio input needs av/scipy/soundfile plus
|
||||
# the mistral_common audio protocol pieces.
|
||||
RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]"
|
||||
@@ -1,100 +0,0 @@
|
||||
# Quadlet unit: frank-small (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/frank-small.container
|
||||
# Unit name: frank-small.service (container name: frank-small)
|
||||
#
|
||||
# The small inference service. vLLM (since 2026-09-18) serving
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16) with the
|
||||
# google bf16 MTP assistant for speculative decoding, on the RTX
|
||||
# 3090. It replaced the mainline-llama.cpp ROOT unit (the old
|
||||
# scripts/frank-small.service is retired into scripts/.AGENTTRASH/).
|
||||
#
|
||||
# SHAPE. The quadlet replaces the image's `vllm serve` entrypoint with
|
||||
# /scripts/serve-frank-small.sh (mounted read-only out of the repo checkout)
|
||||
# so the flag set stays in git. The model and drafter directories are bound
|
||||
# read-only: the payloads are fetched and sha256-verified by
|
||||
# scripts/download-frank-small-vllm.sh before first start, and the deploy
|
||||
# handler refuses to start the unit when either is absent.
|
||||
#
|
||||
# CUDA WITHOUT the nvidia-container-toolkit. This host has never run a CUDA
|
||||
# container (the 3090 served frank-small as a native binary, the iGPU runs
|
||||
# ROCm through /dev/kfd), so there is no nvidia hook to lean on. The recipe
|
||||
# that works (measured 2026-09-18, torch sees the 3090 at 25.3 GB):
|
||||
# - the four /dev/nvidia* nodes via AddDevice (they are 0666 on this host,
|
||||
# so the rootless user namespace can open them);
|
||||
# - the HOST driver's libcuda.so.1 and libnvidia-ml.so.1 mounted read-only
|
||||
# — the image carries its own CUDA 12.9 runtime, but vLLM's platform
|
||||
# probe needs the driver's NVML stub, and without it `vllm serve` dies at
|
||||
# parser construction with "Failed to infer device type";
|
||||
# - NVIDIA_VISIBLE_DEVICES=all.
|
||||
# memlock must be unlimited: CUDA pins its staging buffers.
|
||||
#
|
||||
# SIZING. The tunables (full 262144 window, 0.95 of a dedicated 24 GiB card,
|
||||
# bf16 KV, 16-seq batch ceiling, 16k prefill chunk, 4-token MTP draft) live
|
||||
# in scripts/frank-small.env. The boot log's "Maximum concurrency for N
|
||||
# tokens per request" line is the naive pool ratio for a hybrid SWA model
|
||||
# and undercounts; verify the session count behaviourally on restart.
|
||||
#
|
||||
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
||||
# path cannot resolve registry-prefixed refs from local storage at create
|
||||
# time. The tag is a LOCAL BUILD, not a pull: deploy-remote.sh pulls the
|
||||
# digest-pinned docker.io base (the gemma-x86_64-cu129 build, vLLM
|
||||
# 0.22.1rc1.dev357+g74b5964f0 — the image that boots this checkpoint) and
|
||||
# re-tags it, then builds dankclaw/frank-small/Dockerfile on top, which
|
||||
# installs the vllm[audio] extra (av, scipy, soundfile,
|
||||
# mistral_common[audio]) so the model's native audio path works. Pull=never
|
||||
# keeps boots off the registry.
|
||||
[Unit]
|
||||
Description=frank-small — vLLM gemma4_unified w4a16 + MTP on RTX 3090
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-small
|
||||
Image=localhost/vllm/vllm-openai:frank-small
|
||||
# Published port stays 7771: Hermes, the harness and the homepage probe all
|
||||
# point here. 8000 (vLLM's in-container port) is never reachable directly.
|
||||
PublishPort=7771:8000
|
||||
# CUDA device nodes (0666 on this host — rootless can open them).
|
||||
AddDevice=/dev/nvidia0
|
||||
AddDevice=/dev/nvidiactl
|
||||
AddDevice=/dev/nvidia-uvm
|
||||
AddDevice=/dev/nvidia-uvm-tools
|
||||
# Host driver stubs the image's CUDA runtime and vLLM's NVML probe need.
|
||||
# The image's LD_LIBRARY_PATH leads with /usr/local/nvidia/lib64 — the
|
||||
# nvidia-container-toolkit's conventional mount point, which this host does
|
||||
# not have — so the driver libs go there; the /usr/lib mounts cover torch
|
||||
# Inductor's own lookup.
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/local/nvidia/lib64/libcuda.so.1:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libcuda.so:/usr/local/nvidia/lib64/libcuda.so:ro
|
||||
Volume=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:/usr/local/nvidia/lib64/libnvidia-ml.so.1:ro
|
||||
Environment=NVIDIA_VISIBLE_DEVICES=all
|
||||
# Payloads, read-only: fetched and sha256-verified by
|
||||
# scripts/download-frank-small-vllm.sh before first start.
|
||||
Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro
|
||||
Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro
|
||||
# Serve script and env, read-only out of the repo checkout.
|
||||
Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro
|
||||
Entrypoint=/scripts/serve-frank-small.sh
|
||||
Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env
|
||||
Ulimit=memlock=-1:-1
|
||||
# /health answers 200 once the engine is up; vLLM 503s it while the model is
|
||||
# still loading, so a green check means ready-to-serve, not process-alive.
|
||||
# curl ships in the image.
|
||||
HealthCmd=curl -sf http://127.0.0.1:8000/health
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=600s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start reads ~10 GiB of weights and compiles the CUDA graphs.
|
||||
TimeoutStartSec=900
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
+5
-13
@@ -319,12 +319,6 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# its unit and fleet settings live in one .container file.
|
||||
_add(services, "frank-large")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-small":
|
||||
# The service is a rootless Quadlet container since 2026-09-18
|
||||
# (vLLM); its unit lives in one .container file. The old root
|
||||
# unit's trio is in scripts/ and maps there.
|
||||
_add(services, "frank-small")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# The five observability containers became Quadlet units on
|
||||
# 2026-09-11. A unit change selects exactly its own service;
|
||||
@@ -408,13 +402,11 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"frank-small.env", "serve-frank-small.sh",
|
||||
"download-frank-small-vllm.sh"}:
|
||||
# vLLM frank-small (2026-09-18) keeps its env, entrypoint and
|
||||
# fetch script here; each one selects frank-small only. The
|
||||
# unit is the quadlet under dankclaw/frank-small/, and the old
|
||||
# root unit file is in scripts/.AGENTTRASH/, which is archive
|
||||
# material and plans nothing.
|
||||
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-small only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
|
||||
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
|
||||
|
||||
+14
-76
@@ -1139,90 +1139,28 @@ wait_for_llm_large_ready() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# deploy_llm_small — frank-small is a rootless Quadlet running vLLM on the
|
||||
# RTX 3090 since 2026-09-18: the runtime is the digest-pinned
|
||||
# vllm/vllm-openai gemma4-unified image, and the unit is generated from
|
||||
# dankclaw/frank-small/quadlet/. It replaces the mainline-llama.cpp
|
||||
# deployment, which ran as a ROOT unit.
|
||||
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
|
||||
# ROOT unit: the runtime is a native CUDA binary, so there is no container
|
||||
# image to install and no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
|
||||
# is that unit's [Install] section.
|
||||
#
|
||||
# TWO MANAGERS, ONE UNIT NAME. The llama.cpp incarnation ran as a ROOT unit
|
||||
# whose file is a symlink into this checkout; the vLLM unit is a USER
|
||||
# (quadlet) unit. Both must never exist at once — they would race for port
|
||||
# 7771 — so the root unit is retired first: stop, disable, move the /etc
|
||||
# symlink into the home trash (never `rm`), reload the root manager. This
|
||||
# host's `systemctl disable` has been observed to delete the unit file
|
||||
# itself (2026-08-26 incident), so the move tolerates a file that is
|
||||
# already gone.
|
||||
#
|
||||
# The payloads are fetched and sha256-verified by
|
||||
# scripts/download-frank-small-vllm.sh before first start; the handler
|
||||
# refuses to launch a unit that cannot load its model or its drafter.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/gemma4-12b-qat-w4a16-ct/model.safetensors}"
|
||||
LLM_SMALL_DRAFT="${LLM_SMALL_DRAFT:-/home/dank/models/gemma4-12b-assistant-bf16/model.safetensors}"
|
||||
# Base image: the gemma-x86_64-cu129 build (vLLM 0.22.1rc1.dev357+
|
||||
# g74b5964f0) — the image that boots this checkpoint. The June 3
|
||||
# gemma4-unified build predates it and fails at weight load (missing
|
||||
# vision_embedder.patch_dense); the Sep 15 cu129-nightly fails at import
|
||||
# (torchvision ABI). The unit runs a LOCAL BUILD on top of this base —
|
||||
# dankclaw/frank-small/Dockerfile installs the vllm[audio] extra, which
|
||||
# the base ships without.
|
||||
VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:e24d3d8e367ebc2a6ccfc8cf3cecee858a30a6c66b5b1642d85363cd745e001e"
|
||||
VLLM_SMALL_BASE_ALIAS="localhost/vllm/vllm-openai:gemma4-unified"
|
||||
VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"
|
||||
|
||||
retire_root_llm_small() {
|
||||
local trash="$HOME/.AGENTTRASH/20260918-vllm-cutover/host-units"
|
||||
run_privileged systemctl stop frank-small.service 2>/dev/null || true
|
||||
run_privileged systemctl disable frank-small.service 2>/dev/null || true
|
||||
if [[ -e /etc/systemd/system/frank-small.service || -L /etc/systemd/system/frank-small.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
run_privileged mv -- /etc/systemd/system/frank-small.service \
|
||||
"$trash/frank-small.service"
|
||||
echo "retired root unit frank-small.service -> $trash"
|
||||
else
|
||||
# This host's `systemctl disable` is allowed to delete the unit file
|
||||
# outright (2026-08-26 incident), so there is often nothing left to
|
||||
# move. Say so: previously no message meant no evidence either way.
|
||||
echo "root unit frank-small.service was already removed by disable"
|
||||
fi
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
|
||||
deploy_llm_small() {
|
||||
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
|
||||
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
|
||||
echo "frank-small: run scripts/download-frank-small-vllm.sh first (main model + MTP assistant, sha256-verified);" >&2
|
||||
echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$LLM_SMALL_DRAFT" ]]; then
|
||||
echo "ERROR: MTP assistant absent at $LLM_SMALL_DRAFT" >&2
|
||||
echo "frank-small: run scripts/download-frank-small-vllm.sh first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot draft." >&2
|
||||
return 1
|
||||
fi
|
||||
retire_root_llm_small
|
||||
# The predecessor can also be THIS quadlet -- a redeploy of a running
|
||||
# frank-small -- so stop the user unit before waiting on the port. On a
|
||||
# first cutover it is not running and this is a no-op.
|
||||
systemctl --user stop frank-small.service 2>/dev/null || true
|
||||
# Pull the base by digest, re-tag it to the FROM ref the Dockerfile
|
||||
# expects, then build the derived image the quadlet runs: the quadlet
|
||||
# path cannot resolve registry-prefixed refs from local storage at
|
||||
# create time (same workaround as halogen), and Pull=never keeps boots
|
||||
# off the registry. The build needs network for pip (the vllm[audio]
|
||||
# extra); podman build has it on by default.
|
||||
podman pull "$VLLM_SMALL_BASE_IMAGE"
|
||||
podman tag "$VLLM_SMALL_BASE_IMAGE" "$VLLM_SMALL_BASE_ALIAS"
|
||||
podman build --pull=never -t "$VLLM_SMALL_IMAGE" \
|
||||
-f dankclaw/frank-small/Dockerfile dankclaw/frank-small
|
||||
# Nothing may start until the predecessor has released the port.
|
||||
wait_port_free 7771 || return 1
|
||||
install_quadlet frank-small dankclaw/frank-small/quadlet frank-small
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence comes from [Install] in the unit file.
|
||||
systemctl --user restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 1200
|
||||
relink_unit frank-small
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-small.service
|
||||
run_privileged systemctl restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a
|
||||
|
||||
Executable
+108
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-large
|
||||
# model) on Dankclaw.
|
||||
#
|
||||
# bash scripts/download-frank-large-orca.sh
|
||||
# bash scripts/download-frank-large-orca.sh --verify-only
|
||||
#
|
||||
# The package is 136 GB of Hub-sized transport parts (HF caps new uploads at
|
||||
# 50 GB, so the 79 GB main GGUF, the 52 GB PLE payload and the 4 GB MTP head
|
||||
# ship as .partNofM fragments). assemble.py (from the repo) concatenates the
|
||||
# parts, verifies every part and final SHA-256, and consumes the parts.
|
||||
#
|
||||
# Integrity: every part is checked against the per-part sha256 in parts.json
|
||||
# (fetched from the repo, not trusted from disk), then the assembled files
|
||||
# against SHA256SUMS. The repo is public (gated: false); an HF token at
|
||||
# ~/.cache/huggingface/token is used when present to clear the anonymous
|
||||
# rate limit on a 136 GB transfer.
|
||||
#
|
||||
# Layout after a successful run (MODEL_DIR):
|
||||
# Qwen3.8-Flash-CIRU-STRIX-Orca.gguf 79,397,818,912 main model
|
||||
# mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf 4,135,893,440 draft
|
||||
# ple/ple.payload.bin 52,429,053,952 external PLE
|
||||
# ple/ple.manifest.json, ple/ple.scale.bf16 PLE metadata
|
||||
# vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj 904,004,320 projector
|
||||
# assemble.py, parts.json, SHA256SUMS (kept for --verify-only)
|
||||
set -euo pipefail
|
||||
|
||||
REPO="jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca"
|
||||
MODEL_DIR="/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca"
|
||||
BASE="https://huggingface.co/$REPO/resolve/main"
|
||||
|
||||
mode="download"
|
||||
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
||||
|
||||
token=""
|
||||
auth=()
|
||||
if [[ -r "$HOME/.cache/huggingface/token" ]]; then
|
||||
token="$(cat "$HOME/.cache/huggingface/token")"
|
||||
auth=(-H "Authorization: Bearer $token")
|
||||
else
|
||||
echo "no HF token at ~/.cache/huggingface/token; downloading anonymously"
|
||||
fi
|
||||
|
||||
dl() { # dl <relpath>
|
||||
local dest="$MODEL_DIR/$1"
|
||||
mkdir -p "$(dirname "$dest")"
|
||||
echo " fetching $1"
|
||||
curl -fL -C - -o "$dest" --retry 10 --retry-delay 5 --retry-max-time 21600 \
|
||||
"${auth[@]}" "$BASE/$1"
|
||||
}
|
||||
|
||||
if [[ "$mode" == "download" ]]; then
|
||||
# Manifest + small files first: the part hashes below come from the
|
||||
# freshly fetched parts.json, so a stale/corrupt copy cannot pass.
|
||||
dl "assemble.py"
|
||||
dl "parts.json"
|
||||
dl "SHA256SUMS"
|
||||
dl "ple/ple.manifest.json"
|
||||
dl "ple/ple.scale.bf16"
|
||||
dl "vision/SHA256SUMS"
|
||||
dl "vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj"
|
||||
|
||||
echo "fetching the five transport parts (136 GB)"
|
||||
while IFS= read -r part; do
|
||||
dl "$part"
|
||||
done < <(python3 -c "
|
||||
import json
|
||||
for item in json.load(open('$MODEL_DIR/parts.json'))['files']:
|
||||
for p in item['parts']:
|
||||
print(p['path'])
|
||||
")
|
||||
fi
|
||||
|
||||
echo "verifying part sha256 (against the repo parts.json)"
|
||||
python3 - "$MODEL_DIR" <<'PY'
|
||||
import hashlib, json, sys
|
||||
from pathlib import Path
|
||||
root = Path(sys.argv[1])
|
||||
manifest = json.loads((root / 'parts.json').read_text())
|
||||
ok = True
|
||||
def digest(p):
|
||||
h = hashlib.sha256()
|
||||
with p.open('rb') as f:
|
||||
for data in iter(lambda: f.read(8 * 1024 * 1024), b''):
|
||||
h.update(data)
|
||||
return h.hexdigest()
|
||||
for item in manifest['files']:
|
||||
for p in item['parts']:
|
||||
path = root / p['path']
|
||||
if not path.exists():
|
||||
print(f" MISSING {p['path']}"); ok = False; continue
|
||||
if path.stat().st_size != p['bytes']:
|
||||
print(f" BAD-SIZE {p['path']}"); ok = False; continue
|
||||
if digest(path) != p['sha256']:
|
||||
print(f" BAD-SHA {p['path']}"); ok = False; continue
|
||||
print(f" OK {p['path']}")
|
||||
sys.exit(0 if ok else 1)
|
||||
PY
|
||||
|
||||
if [[ "$mode" == "download" ]]; then
|
||||
echo "assembling (consumes verified parts)"
|
||||
(cd "$MODEL_DIR" && python3 assemble.py)
|
||||
fi
|
||||
|
||||
echo "verifying assembled files against SHA256SUMS"
|
||||
(cd "$MODEL_DIR" && sha256sum -c SHA256SUMS)
|
||||
(cd "$MODEL_DIR/vision" && sha256sum -c SHA256SUMS)
|
||||
echo "frank-large payload complete: $MODEL_DIR"
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetch the frank-small vLLM payloads on dankclaw (RTX 3090, since 2026-09-18):
|
||||
#
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct
|
||||
# compressed-tensors w4a16 (int4, group 32) QAT checkpoint — the model
|
||||
# vLLM's gemma4_unified implementation loads natively. One 9.56 GiB
|
||||
# safetensors plus tokenizer/config.
|
||||
#
|
||||
# google/gemma-4-12B-it-assistant
|
||||
# the MTP draft/assistant (gemma4_unified_assistant, 4 layers) in
|
||||
# bf16 — Google's original. 806 MiB. The w4a16-ct re-quant
|
||||
# (kunhunjon) that was used first drafts at acceptance 1.00: vLLM's
|
||||
# MTP drafter path does not apply the Marlin w4a16 kernel to the
|
||||
# assistant's linears (no "Using MarlinLinearKernel" after the
|
||||
# drafter load), so its packed int4 weights are misread. The bf16
|
||||
# assistant accepts at 2.80 (measured 2026-09-19).
|
||||
#
|
||||
# Both repos are public and ungated; no token is needed. Every fetch is -f
|
||||
# (an HTTP error is a failure, not a 153-byte "artifact"), resumable, and the
|
||||
# two LFS payloads are sha256-verified against the Hub's declared oid — the
|
||||
# discipline from scripts/download-frank-medium-gguf.sh.
|
||||
#
|
||||
# The serve script refuses to start when either payload is absent, so a
|
||||
# half-finished run of this script reads as a missing file, not a runtime
|
||||
# fault.
|
||||
set -euo pipefail
|
||||
|
||||
MODELS_DIR="${MODELS_DIR:-/home/dank/models}"
|
||||
LOG_DIR="$MODELS_DIR/logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
LOG="$LOG_DIR/$(basename "$0" .sh).log"
|
||||
|
||||
log() { echo "$(date -Is) $*" | tee -a "$LOG"; }
|
||||
|
||||
# fetch <repo> <dir> <file> [sha256] [size]
|
||||
fetch() {
|
||||
local repo="$1" dir="$2" file="$3" sha="${4:-}" size="${5:-}"
|
||||
mkdir -p "$dir"
|
||||
if [[ -n "$size" && -f "$dir/$file" ]] && [[ "$(stat -c%s "$dir/$file")" == "$size" ]]; then
|
||||
log "skip $dir/$file (size $size matches)"
|
||||
else
|
||||
log "fetch $repo/$file"
|
||||
curl -fsSL -C - "https://huggingface.co/$repo/resolve/main/$file" -o "$dir/$file"
|
||||
log "fetched $dir/$file ($(stat -c%s "$dir/$file") bytes)"
|
||||
fi
|
||||
if [[ -n "$sha" ]]; then
|
||||
local actual
|
||||
actual="$(sha256sum "$dir/$file" | cut -d' ' -f1)"
|
||||
if [[ "$actual" != "$sha" ]]; then
|
||||
log "SHA256 MISMATCH $dir/$file: $actual != $sha"
|
||||
return 1
|
||||
fi
|
||||
log "sha256 ok $dir/$file"
|
||||
fi
|
||||
}
|
||||
|
||||
MAIN_REPO=google/gemma-4-12B-it-qat-w4a16-ct
|
||||
ASSISTANT_REPO=google/gemma-4-12B-it-assistant
|
||||
MAIN_DIR="$MODELS_DIR/gemma4-12b-qat-w4a16-ct"
|
||||
ASSISTANT_DIR="$MODELS_DIR/gemma4-12b-assistant-bf16"
|
||||
|
||||
# Hub-declared LFS oids (sha256), 2026-09-18.
|
||||
fetch "$MAIN_REPO" "$MAIN_DIR" model.safetensors \
|
||||
60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3 10264229896
|
||||
fetch "$MAIN_REPO" "$MAIN_DIR" tokenizer.json \
|
||||
cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f 32169626
|
||||
for f in config.json generation_config.json chat_template.jinja \
|
||||
processor_config.json tokenizer_config.json recipe.yaml; do
|
||||
fetch "$MAIN_REPO" "$MAIN_DIR" "$f"
|
||||
done
|
||||
|
||||
fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" model.safetensors \
|
||||
3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df 845719296
|
||||
fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" tokenizer.json \
|
||||
c001d9ada50af662c94f5ab17ec7e09f6438d1bec8246c47fee6510693d8de35 32169884
|
||||
for f in config.json generation_config.json tokenizer_config.json; do
|
||||
fetch "$ASSISTANT_REPO" "$ASSISTANT_DIR" "$f"
|
||||
done
|
||||
|
||||
log "ALL FETCHED OK"
|
||||
+100
-89
@@ -1,101 +1,112 @@
|
||||
# Tunables for frank-small (since 2026-09-18): vLLM serving
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
|
||||
# Google publishes "for native, optimized inference with vLLM") with the
|
||||
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
|
||||
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
||||
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
||||
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
||||
#
|
||||
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
|
||||
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
|
||||
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
|
||||
# Dockerfile) so the model's native audio path works. That image carries
|
||||
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
|
||||
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
|
||||
# card. The older gemma4-unified build (June 3) predates this checkpoint
|
||||
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
|
||||
# import (torchvision ABI); this one is the image that boots.
|
||||
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
||||
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
||||
# CUDA (see the CUDA lesson in AGENTS.md).
|
||||
#
|
||||
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
|
||||
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
|
||||
# embeddings, vision and audio. The fleet's image path lives here: the model
|
||||
# ingests image (and audio) content natively, so no projector file exists
|
||||
# any more and nothing external has to load.
|
||||
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
||||
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
||||
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
||||
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
||||
# only a 1024-position window, which is why the KV pool is a fraction of
|
||||
# the full-ctx x all-layers number.
|
||||
#
|
||||
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
|
||||
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
|
||||
# is that a QAT target must draft with a QAT drafter at the same precision.
|
||||
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
|
||||
# MTP path, not a generic draft model.
|
||||
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
||||
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
||||
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
||||
# context loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
||||
# is the only legal shape that carries both.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
|
||||
# for as many concurrent sessions as the card holds.
|
||||
# MAX_MODEL_LEN=262144 the model's max_position_embeddings
|
||||
# GPU_MEM_UTIL=0.95 the card is dedicated to this service
|
||||
# KV_CACHE_DTYPE=turboquant_k8v4 TurboQuant KV: K at 8-bit, V at 4-bit,
|
||||
# per-16-element block scales packed INSIDE the
|
||||
# tensor (Hadamard-rotated, bf16 compute,
|
||||
# dedicated TURBOQUANT attention backend with
|
||||
# an explicit Ampere path). 2.67x smaller KV
|
||||
# than bf16 (289,647 -> ~772k-token pool, two
|
||||
# -> five-or-six full-context sessions).
|
||||
# The cheaper int8/fp8 per-token-head dtypes
|
||||
# CANNOT boot this model: they keep a separate
|
||||
# fp32 scale tensor per (token, head), and
|
||||
# Gemma4's hybrid head dims (SWA 256 / full
|
||||
# 512) make the per-token page bytes 4160 vs
|
||||
# 8256 — not an integer ratio, so vLLM's
|
||||
# unify_kv_cache_spec_page_size raises
|
||||
# NotImplementedError (measured 2026-09-19,
|
||||
# nine crash-loop restarts). fp8 KV is
|
||||
# hardware-dead on this card anyway (sm_86,
|
||||
# no cvt.e4m3x2).
|
||||
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
|
||||
# caps full-context concurrency
|
||||
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
|
||||
# faster but grow the activation arena that
|
||||
# competes with the KV pool for the card
|
||||
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
|
||||
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
|
||||
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
|
||||
# 1024-token window per request, so the true full-context session count is
|
||||
# higher. Verify it behaviourally (concurrent long requests) on every
|
||||
# restart, not from these comments.
|
||||
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
||||
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
||||
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
||||
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
||||
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
||||
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
||||
# which would shrink each slot below the full 262144 window once
|
||||
# parallel > 1.
|
||||
#
|
||||
# Sampling: greedy by default (fleet contract) via
|
||||
# --override-generation-config; a client that sends its own temperature gets
|
||||
# its own, and the harness applies per-model overrides from models.yml.
|
||||
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
||||
# params; it carries an audio encoder too). The fleet's image path lives
|
||||
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
||||
# WORKS: input_audio content parts transcribe verbatim (verified
|
||||
# 2026-09-17 with a 16k mono WAV).
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
|
||||
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
# temperature gets its own, and the harness applies per-model overrides
|
||||
# from models.yml.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
||||
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
|
||||
# Container paths, bind-mounted by the quadlet. The host paths are declared
|
||||
# in the quadlet and the deploy handler, not here: this file is read inside
|
||||
# the container.
|
||||
MODEL_DIR=/models/main
|
||||
DRAFT_DIR=/models/assistant
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
||||
# FA_ALL_QUANTS=ON.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
|
||||
# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
|
||||
# host 7771, the endpoint Hermes, the harness and the homepage probe have
|
||||
# always used. The alias is the id at /v1/models and what clients ask for.
|
||||
# Model package. The MTP sidecar file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
||||
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
||||
# what clients ask for -- the harness entry in models.yml names it.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=8000
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# Full window, dedicated card, turboquant k8v4 KV (see above), roomy batch ceiling,
|
||||
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
|
||||
MAX_MODEL_LEN=262144
|
||||
GPU_MEM_UTIL=0.95
|
||||
KV_CACHE_DTYPE=auto
|
||||
# KV quant history (measured 2026-09-19, all on this image): bf16 (auto)
|
||||
# is the ONLY dtype that boots Gemma4 here. int8_per_token_head dies in
|
||||
# unify_kv_cache_spec_page_size — its separate fp32 scale tensor makes the
|
||||
# per-token page bytes 4160 (SWA, head_dim 256) vs 8256 (full, head_dim
|
||||
# 512), not an integer ratio. turboquant_k8v4 dies in backend validation —
|
||||
# the TURBOQUANT backend rejects "partial multimodal token full attention"
|
||||
# (the mm-prefix path Gemma4's vision/audio tokens require), and the
|
||||
# VLLM_ATTENTION_BACKEND env var does not exist in this build (the pin
|
||||
# goes through --attention-backend). fp8 is hardware-dead on sm_86
|
||||
# (cvt.e4m3x2 is sm_89+). Revisit on a vLLM release that documents TurboQuant
|
||||
# mm-prefix support.
|
||||
MAX_NUM_SEQS=16
|
||||
MAX_NUM_BATCHED_TOKENS=16384
|
||||
SPEC_NUM_TOKENS=4
|
||||
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
||||
# pool; --parallel divides it). See the SIZING note above for the fit
|
||||
# arithmetic; a fifth slot does not fit.
|
||||
CTX_SIZE=1048576
|
||||
PARALLEL=4
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
|
||||
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
|
||||
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
|
||||
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
|
||||
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
|
||||
# tried. Idle VRAM is 23,300 MiB at this geometry.
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
||||
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention (required for q8 KV).
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
||||
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
||||
# explicit here (both were the implicit build defaults: 3 and 50).
|
||||
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
|
||||
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
|
||||
# costs more attention re-read at long context than the extra accepted
|
||||
# token earns. Keep 3/50.
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
POLL_LEVEL=50
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||
# though the weights themselves are copied into VRAM. Parity with the halogen
|
||||
# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
+113
-58
@@ -1,30 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve google/gemma-4-12B-it-qat-w4a16-ct as frank-small on Dankclaw's RTX
|
||||
# 3090 via vLLM (since 2026-09-18), with the google bf16 MTP assistant
|
||||
# for speculative decoding.
|
||||
# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
|
||||
# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
|
||||
#
|
||||
# This script is the CONTAINER ENTRYPOINT of the frank-small quadlet: the
|
||||
# vllm/vllm-openai image's own entrypoint is `vllm serve`, and the quadlet
|
||||
# replaces it with this file (mounted read-only out of the repo checkout) so
|
||||
# the flag set stays in git. The image pins the gemma4-unified vLLM build,
|
||||
# which is what loads gemma4_unified natively and runs the Gemma 4 MTP path.
|
||||
# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
|
||||
# weights plus the BF16 vision projector and q8_0 KV inside the card's
|
||||
# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
|
||||
# belongs to the halogen engine that serves frank-large.
|
||||
#
|
||||
# MTP: the drafter is a separate 4-layer assistant checkpoint in bf16
|
||||
# (Google's original gemma-4-12B-it-assistant). It goes in
|
||||
# --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path, not a
|
||||
# generic draft model. The w4a16-ct re-quant of the assistant was tried
|
||||
# first and drafts at acceptance 1.00 (the drafter path does not apply the
|
||||
# Marlin kernel to its linears); the bf16 assistant accepts at 2.80.
|
||||
# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
|
||||
# passed via --model-draft with --spec-type draft-mtp: the draft context
|
||||
# loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so this chain is the
|
||||
# only shape that carries both.
|
||||
#
|
||||
# Sampling: greedy by default (fleet contract) via
|
||||
# --override-generation-config; clients that send their own temperature get
|
||||
# their own, and the harness overrides per model from models.yml.
|
||||
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
|
||||
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
|
||||
# maximises the TOTAL pool instead, which would divide a smaller window
|
||||
# across slots. The 4-slot layout is measured to fit the card (see the
|
||||
# SIZING note in frank-small.env); more slots would fail the KV allocation.
|
||||
#
|
||||
# Tunables come from frank-small.env (mounted beside this file by the
|
||||
# quadlet) unless FRANK_SMALL_ENV_FILE overrides the path.
|
||||
# Vision + audio: --mmproj loads the BF16 projector, which also carries
|
||||
# an audio encoder — input_audio content parts transcribe verbatim
|
||||
# (verified 2026-09-17). The fleet's image path lives here, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
# models.yml. --jinja is pinned even though this build defaults it on: the
|
||||
# model's own template supplies reasoning_effort, preserve_thinking and the
|
||||
# tool-call grammar, and a future build that flips that default must not
|
||||
# silently take tools away.
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
|
||||
set -euo pipefail
|
||||
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-/scripts/frank-small.env}"
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
@@ -33,51 +46,93 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${MODEL_DIR:=/models/main}"
|
||||
: "${DRAFT_DIR:=/models/assistant}"
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
: "${DRAFT_FILE:=}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=8000}"
|
||||
: "${MAX_MODEL_LEN:=262144}"
|
||||
: "${GPU_MEM_UTIL:=0.95}"
|
||||
: "${KV_CACHE_DTYPE:=auto}"
|
||||
: "${MAX_NUM_SEQS:=16}"
|
||||
: "${MAX_NUM_BATCHED_TOKENS:=16384}"
|
||||
: "${SPEC_NUM_TOKENS:=4}"
|
||||
: "${ENFORCE_EAGER:=0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
: "${CTX_SIZE:=}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
|
||||
# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
|
||||
# busy-wait level (build default 50).
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${POLL_LEVEL:=50}"
|
||||
: "${NUM_THREADS:=8}"
|
||||
: "${THREAD_BATCH:=8}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
# is a download that has not finished, and the unit's restart policy would
|
||||
# otherwise turn that into a crash loop that reads like a runtime fault.
|
||||
for path in "$MODEL_DIR/model.safetensors" "$MODEL_DIR/config.json" \
|
||||
"$DRAFT_DIR/model.safetensors" "$DRAFT_DIR/config.json"; do
|
||||
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
|
||||
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
||||
done
|
||||
|
||||
SPEC_CONFIG="{\"method\":\"mtp\",\"model\":\"${DRAFT_DIR}\",\"num_speculative_tokens\":${SPEC_NUM_TOKENS}}"
|
||||
|
||||
VLLM_FLAGS=(
|
||||
serve "$MODEL_DIR"
|
||||
--served-model-name "$ALIAS"
|
||||
--host "$BIND_ADDR"
|
||||
--port "$LISTEN_PORT"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--gpu-memory-utilization "$GPU_MEM_UTIL"
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--speculative-config "$SPEC_CONFIG"
|
||||
--reasoning-parser gemma4
|
||||
--enable-auto-tool-choice
|
||||
--tool-call-parser gemma4
|
||||
--override-generation-config '{"temperature": 0}'
|
||||
)
|
||||
# ENFORCE_EAGER=1 skips Inductor/Triton compilation and CUDA graph capture
|
||||
# (slower steady-state decode, faster and more robust boot). The 3090 is
|
||||
# sm_86 (Ampere) — no native fp8 — so KV stays bf16 and the default
|
||||
# graphed path is the right one; the knob is a boot-debugging lever.
|
||||
if [[ "$ENFORCE_EAGER" == "1" ]]; then
|
||||
VLLM_FLAGS+=(--enforce-eager)
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
[[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
|
||||
fi
|
||||
|
||||
exec vllm "${VLLM_FLAGS[@]}"
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
fi
|
||||
|
||||
DRAFT_ARGS=()
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
|
||||
fi
|
||||
|
||||
# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
|
||||
# whatever the weights, projector, draft context and compute buffers leave
|
||||
# on the card; the engine prints the chosen value at startup.
|
||||
if [[ -n "${CTX_SIZE:-}" ]]; then
|
||||
CTX_ARGS=(--ctx-size "$CTX_SIZE")
|
||||
else
|
||||
CTX_ARGS=(--fit on)
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
"${DRAFT_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
"${CTX_ARGS[@]}" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--poll "$POLL_LEVEL" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET"
|
||||
|
||||
@@ -5,9 +5,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-16):
|
||||
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
|
||||
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
|
||||
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
|
||||
w4b on the gfx1151 iGPU, port 7770) and vLLM (frank-small,
|
||||
google/gemma-4-12B-it-qat-w4a16-ct with the bf16 MTP assistant on the RTX
|
||||
3090, port 7771).
|
||||
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
|
||||
7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -35,10 +35,10 @@ gfx1151 iGPU 64 GiB unified memory]
|
||||
frank-large, port 7770]
|
||||
F[peonist-ai Qwen3.8-Flash-Next w4b
|
||||
2 x 200192 ctx over one 400384-position pool]
|
||||
G[vLLM gemma-x86_64-cu129
|
||||
G[mainline llama.cpp 37b53fd
|
||||
frank-small, port 7771]
|
||||
H[gemma-4-12B-it-qat-w4a16-ct
|
||||
262144 ctx / 2 full-context sessions, MTP + vision + audio]
|
||||
H[Qwen3.8-27B GSQ-RCO IQ3_S
|
||||
131072 ctx / 1 slot, MTP + vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -87,15 +87,16 @@ frank-small, port 7771]
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting
|
||||
- [[frank-small-vllm-gemma4]] — ACTIVE frank-small runbook since 2026-09-19
|
||||
(rootless Quadlet, port 7771): vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-token
|
||||
pool, 2 concurrent full-context sessions, native text+image+audio
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): the
|
||||
prior frank-small runbook (2026-09-17 → 09-19), mainline llama.cpp serving
|
||||
HauhauCS Gemma4-12B-QAT-Uncensored Q4_K_M with the MTP sidecar, q8_0 KV,
|
||||
4 slots of 262k
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
|
||||
vLLM detour (2026-09-19) was reverted the same day; see below.
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
|
||||
@@ -166,17 +166,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
|
||||
2026-09-19: vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
|
||||
full-context sessions, native text+image+audio) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
|
||||
2026-09-17: mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
|
||||
Dankclaw's RTX 3090
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
token pool, 2 concurrent full-context sessions, native text+image+audio
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
|
||||
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
|
||||
|
||||
@@ -64,8 +64,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
|
||||
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-small" ]
|
||||
|
||||
@@ -73,8 +72,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# that name was retired with the FreeToken runtime on 2026-09-15 and must
|
||||
# not come back through a planner mapping.
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
dankclaw/frank-small/quadlet/frank-small.container scripts/frank-small.env \
|
||||
scripts/serve-frank-small.sh scripts/download-frank-small-vllm.sh
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@@ -72,11 +72,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "frank-large loads no vision tower: the fleet's image path is frank-small" {
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET"
|
||||
# frank-small's vLLM checkpoint (2026-09-18) is natively multimodal —
|
||||
# image content is ingested by the model itself, so the seam that used
|
||||
# to pin --mmproj now pins the projector's absence.
|
||||
! grep -q 'mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
grep -q '^MODEL_DIR=/models/main$' "$REPO_ROOT/scripts/frank-small.env"
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the previous ROOT unit without deleting it" {
|
||||
|
||||
+73
-122
@@ -1,164 +1,115 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-small vLLM quadlet on dankclaw (since 2026-09-18).
|
||||
# frank-small is the RTX 3090 endpoint. It now runs vLLM (the pinned
|
||||
# gemma4-unified image) serving google/gemma-4-12B-it-qat-w4a16-ct with the
|
||||
# google bf16 MTP assistant, replacing the mainline-llama.cpp root
|
||||
# unit (retired into scripts/.AGENTTRASH/). This seam pins what the flags
|
||||
# alone do not say: boot persistence for the generated quadlet, the CUDA
|
||||
# wiring this host needs (no nvidia-container-toolkit), the repo-relative
|
||||
# entrypoint and env, the MTP contract (a separate QAT assistant checkpoint
|
||||
# in --speculative-config, method "mtp"), the fleet's image path (the model
|
||||
# is natively multimodal — no projector file), the full-window sizing, the
|
||||
# separate client port, and the deploy handler's cutover order.
|
||||
# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
|
||||
# went back to the halogen engine on the iGPU. This seam pins what the flags
|
||||
# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
|
||||
# is involved), the repo-relative paths the unit executes, the separate client
|
||||
# port, the MTP contract (the head is a separate sidecar file loaded as the
|
||||
# draft model, chained behind ngram-mod), the vision handoff from frank-large,
|
||||
# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
|
||||
|
||||
load helpers
|
||||
|
||||
QUADLET="$REPO_ROOT/dankclaw/frank-small/quadlet/frank-small.container"
|
||||
DOCKERFILE="$REPO_ROOT/dankclaw/frank-small/Dockerfile"
|
||||
UNIT="$REPO_ROOT/scripts/frank-small.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-small-vllm.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
|
||||
@test "frank-small quadlet, Dockerfile, entrypoint, env and fetch script exist" {
|
||||
[ -f "$QUADLET" ]
|
||||
[ -f "$DOCKERFILE" ]
|
||||
@test "frank-small unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$FETCH" ]
|
||||
# The old root unit is retired, not deleted: it must live in the trash,
|
||||
# not in scripts/, so the planner and relink_unit never see it.
|
||||
[ ! -e "$REPO_ROOT/scripts/frank-small.service" ]
|
||||
[ -e "$REPO_ROOT/scripts/.AGENTTRASH/frank-small.service.llama.cpp" ]
|
||||
}
|
||||
|
||||
@test "frank-small is boot-persistent, self-healing and registry-pinned" {
|
||||
grep -q '^WantedBy=default.target$' "$QUADLET"
|
||||
grep -q '^Restart=on-failure$' "$QUADLET"
|
||||
grep -q '^RestartSec=' "$QUADLET"
|
||||
grep -q '^Pull=never$' "$QUADLET"
|
||||
# The unit runs the localhost/-aliased LOCAL BUILD (base + vllm[audio]
|
||||
# extra); the digest-pinned docker.io base lives only in the deploy
|
||||
# handler, which pulls, re-tags and builds before restart.
|
||||
grep -q '^Image=localhost/vllm/vllm-openai:frank-small$' "$QUADLET"
|
||||
! grep -q '^Image=docker.io' "$QUADLET"
|
||||
@test "frank-small is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small wires CUDA without the nvidia-container-toolkit" {
|
||||
# The four device nodes (0666 on this host, so rootless can open them)...
|
||||
grep -q '^AddDevice=/dev/nvidia0$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/nvidiactl$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/nvidia-uvm$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/nvidia-uvm-tools$' "$QUADLET"
|
||||
# ...plus the host driver's libcuda and NVML stubs. Without the NVML
|
||||
# mount vllm serve dies at parser construction with "Failed to infer
|
||||
# device type" (measured 2026-09-18).
|
||||
grep -q 'libcuda.so.1.*:ro' "$QUADLET"
|
||||
grep -q 'libnvidia-ml.so.1.*:ro' "$QUADLET"
|
||||
grep -q '^Environment=NVIDIA_VISIBLE_DEVICES=all$' "$QUADLET"
|
||||
# CUDA pins its staging buffers; memlock must be unlimited.
|
||||
grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET"
|
||||
@test "frank-small runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
|
||||
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small runs the repo's own entrypoint with the repo's env" {
|
||||
grep -q '^Entrypoint=/scripts/serve-frank-small.sh$' "$QUADLET"
|
||||
grep -q '^Environment=FRANK_SMALL_ENV_FILE=/scripts/frank-small.env$' "$QUADLET"
|
||||
grep -q 'Volume=/home/dank/serverAppConfigs/scripts:/scripts:ro' "$QUADLET"
|
||||
@test "frank-small offloads every layer to the 3090 with flash attention" {
|
||||
grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
|
||||
grep -q '^NGL=999$' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint; without --metrics there is no /metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small mounts the sha256-verified payloads read-only" {
|
||||
grep -q 'Volume=/home/dank/models/gemma4-12b-qat-w4a16-ct:/models/main:ro' "$QUADLET"
|
||||
grep -q 'Volume=/home/dank/models/gemma4-12b-assistant-bf16:/models/assistant:ro' "$QUADLET"
|
||||
# The fetch script verifies both LFS payloads against the Hub oids.
|
||||
grep -q '60b6e3989502969d8ae04185d72ecbbc7db63978d5af747a493d53895aa6bfa3' "$FETCH"
|
||||
grep -q '3279c173daddd7186e79d652ad94022415736d3a1370625696c898429b06d6df' "$FETCH"
|
||||
}
|
||||
|
||||
@test "frank-small drafts with the QAT MTP assistant via method mtp" {
|
||||
# The drafter is a separate 4-layer assistant checkpoint in the same
|
||||
# int4 w4a16 compressed-tensors format (the card's compatibility rule:
|
||||
# a QAT target drafts with a QAT drafter at the same precision), passed
|
||||
# in --speculative-config as method "mtp" — vLLM's Gemma 4 MTP path.
|
||||
grep -qF -- '\"method\":\"mtp\"' "$SERVE"
|
||||
grep -q -- '--speculative-config "$SPEC_CONFIG"' "$SERVE"
|
||||
grep -q '^SPEC_NUM_TOKENS=4$' "$ENV_FILE"
|
||||
# The gemma4 reasoning and tool-call parsers keep the fleet contract:
|
||||
# reasoning_content on the wire and tools answered.
|
||||
grep -q -- '--reasoning-parser gemma4' "$SERVE"
|
||||
grep -q -- '--tool-call-parser gemma4' "$SERVE"
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
|
||||
grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
# The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
|
||||
# nextn layers); it loads as --model-draft and shares the target's last
|
||||
# layers' KV. ngram-mod is draftless, so the chain is legal.
|
||||
grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small carries the fleet's image path, and only it" {
|
||||
# The model is natively multimodal (encoder-free unified): image content
|
||||
# is ingested by the checkpoint itself, so there is no projector file
|
||||
# any more — the seam that used to pin MMPROJ_PATH now pins its absence.
|
||||
! grep -q '^MMPROJ' "$ENV_FILE"
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "frank-small serves the full window with bf16 KV on a dedicated card" {
|
||||
local maxlen kvdtype util seqs
|
||||
maxlen="$(sed -nE 's/^MAX_MODEL_LEN=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
kvdtype="$(sed -nE 's/^KV_CACHE_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
|
||||
util="$(sed -nE 's/^GPU_MEM_UTIL=([0-9.]+)$/\1/p' "$ENV_FILE")"
|
||||
seqs="$(sed -nE 's/^MAX_NUM_SEQS=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
# The model's FULL trained window (262144), not a divided one.
|
||||
[ "$maxlen" = "262144" ]
|
||||
# bf16 KV (auto): the only dtype that boots Gemma4 in this vLLM build.
|
||||
# int8_per_token_head dies in unify_kv_cache_spec_page_size (separate
|
||||
# fp32 scale tensor breaks the 256/512 head-dim page-size ratio),
|
||||
# turboquant_k8v4 dies in backend validation (no mm-prefix attention),
|
||||
# and fp8 is hardware-dead on sm_86 — see frank-small.env.
|
||||
[ "$kvdtype" = "auto" ]
|
||||
# No pinned attention backend: Gemma4's config hook selects TRITON_ATTN
|
||||
# for the mixed head dims, and pinning any other backend breaks the KV
|
||||
# dtype or mm-prefix validation.
|
||||
! grep -q '^ATTENTION_BACKEND' "$ENV_FILE"
|
||||
! grep -q -- '--attention-backend' "$SERVE"
|
||||
# The 3090 is dedicated to this service, so the engine may take most of
|
||||
# it; the KV pool, not the seq ceiling, caps full-context concurrency.
|
||||
[ "$util" = "0.95" ]
|
||||
[ "$seqs" -ge 4 ]
|
||||
@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
|
||||
local parallel cache_k cache_v ctx fixed per_slot
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
|
||||
# Every slot must be the model's FULL trained window (262144), not a
|
||||
# divided one -- CTX_SIZE is the total pool and --parallel divides it.
|
||||
[ $((ctx / parallel)) -eq 262144 ]
|
||||
[ $((ctx % parallel)) -eq 0 ]
|
||||
[ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
|
||||
# Fit budget from the measured boot arithmetic on the 3090 (24576
|
||||
# MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
|
||||
# so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
|
||||
# (weights, projector, sidecar draft, compute buffers, CUDA context)
|
||||
# is 11996-3450 = 8546 MiB. If the config asks for more slots than
|
||||
# the card holds, this inequality fails before deploy instead of the
|
||||
# KV allocation failing at boot.
|
||||
fixed=8546
|
||||
per_slot=3450
|
||||
[ $((fixed + parallel * per_slot)) -le 24576 ]
|
||||
# Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
|
||||
# without which prefill collapses about 12x (see the CUDA lesson in
|
||||
# AGENTS.md). The build-cuda tree must carry it.
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-small runs on its own port beside frank-large's" {
|
||||
# 8000 is the in-container port; the quadlet publishes it as host 7771,
|
||||
# the endpoint Hermes, the harness and the homepage probe use.
|
||||
grep -q '^LISTEN_PORT=8000$' "$ENV_FILE"
|
||||
grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-small$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
grep -q '^PublishPort=7771:8000$' "$QUADLET"
|
||||
# The two endpoints cannot share a port, and only the halogen unit
|
||||
# publishes 7770.
|
||||
# The two endpoints cannot share a port, and only the halogen unit publishes
|
||||
# 7770.
|
||||
grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE"
|
||||
! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "the deploy handler cuts over from the root unit and gates on health" {
|
||||
@test "the deploy handler installs the root unit and refuses a missing model" {
|
||||
grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
|
||||
# The digest-pinned docker.io base is pulled and re-tagged to the FROM
|
||||
# ref, then the Dockerfile is built into the localhost tag the quadlet
|
||||
# runs (the vllm[audio] extra the base ships without).
|
||||
grep -q 'VLLM_SMALL_BASE_IMAGE="docker.io/vllm/vllm-openai@sha256:' "$REMOTE_DEPLOY"
|
||||
grep -q 'VLLM_SMALL_IMAGE="localhost/vllm/vllm-openai:frank-small"' "$REMOTE_DEPLOY"
|
||||
grep -q 'podman tag "\$VLLM_SMALL_BASE_IMAGE" "\$VLLM_SMALL_BASE_ALIAS"' "$REMOTE_DEPLOY"
|
||||
grep -q 'podman build --pull=never -t "\$VLLM_SMALL_IMAGE"' "$REMOTE_DEPLOY"
|
||||
grep -q 'dankclaw/frank-small/Dockerfile' "$REMOTE_DEPLOY"
|
||||
# The old ROOT unit is retired (stop, disable, trash, reload) before the
|
||||
# user unit takes the port — two managers must never own frank-small.
|
||||
grep -q 'retire_root_llm_small' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7771' "$REMOTE_DEPLOY"
|
||||
# Missing payloads refuse the start, and the health gate is the real
|
||||
# endpoint.
|
||||
grep -q 'gemma4-12b-qat-w4a16-ct/model.safetensors' "$REMOTE_DEPLOY"
|
||||
grep -q 'gemma4-12b-assistant-bf16/model.safetensors' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
|
||||
grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "frank-small's entrypoint refuses to start on a half-finished download" {
|
||||
@test "frank-small's serve script refuses to start on a half-finished download" {
|
||||
grep -q 'frank-small: missing' "$SERVE"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user