inference: frank-large returns to halogen, frank-small takes the 3090
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server quadlet again -- the 2026-09-13 unit file was never touched by the two-day ik_llama.cpp interlude, so the runtime was still in the repo and its image is still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192 with a 400,384-position pool and two slots, exactly two 200k conversations. frank-small is new: mainline llama.cpp on the RTX 3090 serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main model's own weights instead of a second 12 GiB copy) plus the BF16 vision projector, as one 131,072-token slot with f16 KV on port 7771. The host is shared now, and that is what the halogen retune pays for: HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to 20 to keep the fit check honest. Vision moved wholesale to frank-small. Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the Qwen-Sharp template only that runtime read, and four benchmark harnesses bound to retired runtimes. Because retiring a file makes any test that greps it pass by absence, the seams assert the archive exists AND the live path does not.
This commit is contained in:
Frank
@@ -3,11 +3,13 @@
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
||||
# Unit name: frank-large.service (container name: frank-large)
|
||||
#
|
||||
# The large inference service, since 2026-09-11 the halogen-flash-server
|
||||
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
||||
# The large inference service. The halogen-flash-server runtime (since
|
||||
# 2026-09-11, restored 2026-09-16 after a two-day ik_llama.cpp hybrid
|
||||
# interlude) serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
||||
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
||||
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
||||
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
||||
# prefill (the dominant cost of any real prompt) on this exact SKU. The RTX
|
||||
# 3090 it briefly shared with frank-large is now frank-small's card.
|
||||
#
|
||||
# SHAPE. The image runs both halves in one container: the engine on container
|
||||
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
||||
@@ -39,18 +41,23 @@
|
||||
# engine lowers it itself and prints what it settled on. Read that ledger on
|
||||
# every restart — it is the ground truth for sizing, not this comment.
|
||||
#
|
||||
# POOL TRIMMED 2026-09-13: 401,408 positions (~11.0 GiB of KV as reserved by
|
||||
# the engine), down from the 524,288 the 2026-09-12 doubling deployed. The
|
||||
# doubling fit only at reserve 15, which left the host 7.1 GiB / 310
|
||||
# contiguous 2 MiB blocks and made the engine print its co-tenant stall
|
||||
# warning on every boot. MEASURED AT 401,408 (deploy ledger): 68.0 weights +
|
||||
# 11.0 KV + 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB
|
||||
# blocks left, engine need printed ~31.5 vs the modelled ~33.4, and NO
|
||||
# stall-warning NOTE. Capacity: one full 262k conversation resident beside a
|
||||
# ~139k one, or three concurrent 131k chats -- the fleet's real shape at
|
||||
# three slots. Fit check budget: 121.2 - 67.7 - 15 = 38.5. The 09-12 doubling
|
||||
# story (silent clamp at reserve 24, the arena term, the reserve ceiling) is
|
||||
# preserved in the budget block below and the runbook.
|
||||
# POOL RETUNED 2026-09-16: 400,384 positions -- exactly two 200,192-token
|
||||
# conversations -- beside a per-conversation ceiling of 200,192 and two slots.
|
||||
# It replaces the 401,408 the 2026-09-13 trim left behind, which held one full
|
||||
# native-262k conversation beside a ~139k one, or three at 131k.
|
||||
#
|
||||
# MEASURED AT 401,408 (2026-09-13 deploy ledger): 68.0 weights + 11.0 KV +
|
||||
# 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB blocks left,
|
||||
# engine need printed ~31.5 against the modelled ~33.4, and NO stall-warning
|
||||
# NOTE. Fit-check budget here: 121.2 - 67.7 - 20 = 33.5. That boot is the
|
||||
# baseline the 09-16 retune is measured against: the vision tower (1.97 GiB)
|
||||
# and roughly 4.2 GiB of arena come back to the host, which is what pays for
|
||||
# the co-tenant below.
|
||||
#
|
||||
# CO-TENANT since 2026-09-16: this host also serves frank-small on the RTX
|
||||
# 3090 (llama.cpp, ~12.2 GiB of weights that load through the page cache and
|
||||
# then live in VRAM, plus a CUDA context). It is the first real co-tenant this
|
||||
# service has had, which is why the arena was halved and the reserve raised.
|
||||
#
|
||||
# BOOT NOTE: on the first request after a (re)start the engine can go silent
|
||||
# for ~45-60 s ("has not answered PING") while it first-touches the freshly
|
||||
@@ -67,9 +74,11 @@
|
||||
# tower and the tokenizer. The repo is public and ungated, so no token is
|
||||
# needed; later starts open no outbound connections at all.
|
||||
#
|
||||
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
||||
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
||||
# 2026-09-11; audio input has no replacement.
|
||||
# VISION. Deliberately NOT set since 2026-09-16. The tower is a 1.97 GiB
|
||||
# charge against the same device budget as the pool and the arena, and the
|
||||
# fleet's image path moved to frank-small (llama.cpp on the RTX 3090, loaded
|
||||
# with --mmproj mmproj-Qwen3.8-27B-BF16). This server refuses image content
|
||||
# parts, and models.yml declares frank-large text-only to match.
|
||||
#
|
||||
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
||||
# There are no server-side sampler defaults: a request that sends no
|
||||
@@ -101,83 +110,84 @@ Volume=/home/dank/models/halogen:/models
|
||||
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
||||
# Fleet identity: the id at /v1/models, /health and every response body.
|
||||
Environment=HALOGEN_MODEL_ID=frank-large
|
||||
# Native context of the model. Matches what both clients declare.
|
||||
Environment=HALOGEN_CTX=262144
|
||||
# Trimmed 2026-09-13 from the 524,288 the 09-12 doubling deployed: buys
|
||||
# ~3.4 GiB of co-tenant headroom, keeps one full 262k conversation resident
|
||||
# beside a ~139k one, or three concurrent 131k chats.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=401408
|
||||
# Concurrency cap. Each slot is 111.0 MiB of O(1) state; the KV is ONE
|
||||
# 401408-position pool carved into per-request regions, so slots cap how many
|
||||
# requests run at once while the pool binds how many positions exist. The
|
||||
# slots still OVERSUBSCRIBE the pool (3 x 262144 > 401408), so admission is
|
||||
# pool-bound -- one full-length conversation plus a ~139k second, or three
|
||||
# at 131k.
|
||||
# LOWERED 4 -> 3 on 2026-09-12: frees ~111 MiB on a host with 7.1 GiB left,
|
||||
# and three users is the real fleet size. A fourth concurrent request now
|
||||
# queues behind the three rather than contending.
|
||||
Environment=HALOGEN_KV_SLOTS=3
|
||||
# Sizing is budget arithmetic, all of it. ONE budget covers
|
||||
# the KV pool, the prefill arena and the vision tower together:
|
||||
# Per-conversation ceiling: exactly half the pool, so two conversations of the
|
||||
# full window are always resident. 200,192 rather than a round 200,000 is
|
||||
# pool/2. The model's native window is 262,144; a conversation past this
|
||||
# ceiling is refused, never silently truncated. Clients declare 200192 (see
|
||||
# the frank-large contextWindow in models.yml).
|
||||
Environment=HALOGEN_CTX=200192
|
||||
# The KV pool: 400,384 positions = 2 x 200,192 (the 2026-09-16 retune, down
|
||||
# from 401,408). ~28 KiB per position including block scratch, so ~11.0 GiB
|
||||
# held -- the 401,408 ledger entry measured exactly 11.0. The two slots still
|
||||
# OVERSUBSCRIBE the pool for any real request, because admission reserves
|
||||
# prompt + max_tokens: two full-window conversations each holding an output
|
||||
# budget are pool-bound, and the second waits in arrival order for the first
|
||||
# to release its region. Read the `kv pool:` lines on every boot;
|
||||
# HALOGEN_KV_POOL_FIT lowers the pool itself if the budget refuses the
|
||||
# configured one, and a green health check does not mean the pool you asked
|
||||
# for is the pool you got.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=400384
|
||||
# Concurrency cap: two, one per half of the pool. Each slot is 111.0 MiB of
|
||||
# O(1) state, so this hands ~111 MiB back against the three set on 2026-09-12
|
||||
# for the fleet size of the time.
|
||||
Environment=HALOGEN_KV_SLOTS=2
|
||||
# Sizing is budget arithmetic, all of it. ONE budget covers the KV pool, the
|
||||
# per-call prefill arena and the vision tower together:
|
||||
#
|
||||
# budget = 121.2 host - 67.7 resident weights - reserve
|
||||
# need = KV (~28 KiB/position) + arena + vision (1.97 measured)
|
||||
# need = KV (~28 KiB/position) + arena + vision
|
||||
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
||||
#
|
||||
# The arena term is linear in max_tok, and it reproduces five live restarts
|
||||
# AND upstream's three published "on the device" figures to within 0.1 GiB —
|
||||
# note those published figures are pool+arena totals, not pool alone, which is
|
||||
# the misreading that first made a doubling look affordable here. Every corner
|
||||
# this box offered at a reserve of 24:
|
||||
# the misreading that first made a doubling look affordable here. Corners this
|
||||
# box offered at a reserve of 24, at the old 262,144-position pool:
|
||||
#
|
||||
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
|
||||
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
|
||||
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
|
||||
#
|
||||
# None of those three gave both features; reserve 15 made the doubled pool
|
||||
# fit (38.5 vs 37.0) but at 7.1 GiB of host headroom. On 2026-09-13 the pool
|
||||
# came down to 401,408 to restore co-tenant headroom: need ~33.4 vs 38.5.
|
||||
# LOWERED 32768 -> 24576 on 2026-09-16 for the frank-small co-tenant on this
|
||||
# same host. The arena's variable term goes 8.40 x 2 -> 8.40 x 1.5, handing
|
||||
# ~4.2 GiB back to the host at roughly a 4-5 % prefill cost -- the same lever
|
||||
# at 16,384 costs 8-9 %. The vendor's own notes name 24,576 as the largest
|
||||
# arena that allocated with headroom to spare, so this stays inside the range
|
||||
# it has tested.
|
||||
#
|
||||
# The arena is a PER-CALL PREFILL CHUNK, not a prompt ceiling: the
|
||||
# 200,192-token conversation is prefilled in arena-sized pieces, so a smaller
|
||||
# arena means more pieces and a slower prefill, not a smaller conversation.
|
||||
#
|
||||
# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
|
||||
# the fit check's printed ledger on each boot is the arbiter, not these
|
||||
# comments.
|
||||
Environment=HALOGEN_MAX_TOK=32768
|
||||
# RAM left for the n-gram table the engine reads through the page cache.
|
||||
# LOWERED 2026-09-12 from 24 (vendor default is 20) to buy the doubled pool at
|
||||
# full prefill speed. The fit check's OWN requirement is the line to honour —
|
||||
# read from a clamped boot, not modelled:
|
||||
# "524288 positions need ~35.0 GiB (plus 1.5 of margin)" -> budget >= 36.5
|
||||
# budget = 121.2 host - 67.7 weights - reserve
|
||||
# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed). At the
|
||||
# old 524,288 pool, 18 -> 35.5 CLAMPED it to 262144, so the ceiling there was
|
||||
# 16; at 401,408 (need ~33.4) it loosens, but a higher reserve still buys
|
||||
# nothing.
|
||||
Environment=HALOGEN_MAX_TOK=24576
|
||||
# RAM the fit check refuses to count on, watched over by the engine's own
|
||||
# n-gram lookup table (47.7 GiB, read through the page cache and never held)
|
||||
# and, since 2026-09-16, by frank-small on the same host.
|
||||
#
|
||||
# AT THE DOUBLED POOL THE CEILING WAS 16, AND 16 BOUGHT NOTHING. Tested live at 524,288: 15 -> 16 changed the fit
|
||||
# margin only. The server held 103.5 GiB and reported "7.1 GiB left for
|
||||
# everything else" at BOTH settings, because this reserve feeds the fit-check
|
||||
# budget and is not an allocation — actual footprint is weights + KV + arena +
|
||||
# vision, none of which it touches. A higher reserve therefore cannot give the
|
||||
# host memory back; it can only take the pool away. 15 keeps the larger margin.
|
||||
# RAISED 15 -> 20 with the 2026-09-16 retune. This knob feeds the fit-check
|
||||
# budget and is NOT an allocation: raising it cannot hand memory back to the
|
||||
# host, because the actual footprint is weights + KV + arena + vision, none of
|
||||
# which it touches — it can only take the pool away. What it does is keep the
|
||||
# fit check honest at the smaller arena. With the arena at 24,576 and no
|
||||
# vision tower, the engine's printed need falls to roughly 25 GiB against a
|
||||
# budget of 121.2 - 67.7 - 20 = 33.5, leaving ~8 GiB of margin, so the pool is
|
||||
# not clamped.
|
||||
#
|
||||
# Measured at 15 before that test: pool reserved at 524288 with no clamp and no
|
||||
# WARNING, and the server held 103.5 GiB. The ledger
|
||||
# reads "host memory left for everything else: 310 contiguous 2 MiB blocks
|
||||
# (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve
|
||||
# baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set).
|
||||
# The 2026-09-13 profile for comparison: 103.5 GiB held and "7.1 GiB left for
|
||||
# everything else" (310 contiguous 2 MiB blocks) at the doubled 524,288 pool,
|
||||
# which is why the pool came down; at 401,408 the same ledger read 9.1 GiB /
|
||||
# 1396 blocks. The 09-16 retune gives back the vision tower (1.97) and ~4.2 of
|
||||
# arena on top of that for the co-tenant, so the expected figure at this boot
|
||||
# is ~15 GiB left. Read the ledger; do not trust this arithmetic.
|
||||
#
|
||||
# The price at 524,288 was that the engine volunteered this note on every boot:
|
||||
# when it runs out, BOTH they and this server can stall for minutes at a
|
||||
# time at full CPU with no disk activity and no output, which is not a
|
||||
# crash and needs no restart.
|
||||
# That is the failure mode to watch, and it is NOT the legible one predicted
|
||||
# above: a minutes-long stall that looks like a hang is harder to diagnose
|
||||
# than a slow prompt. The 09-13 trim to 401,408 was the first application of
|
||||
# the remedy. If it happens again, cut further:
|
||||
# HALOGEN_KV_POOL_POSITIONS=262144 (hands back another 3.6 GiB), or
|
||||
# HALOGEN_MAX_TOK=16384 (hands back 8.4) with reserve then free to rise to 24.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=15
|
||||
# Image input (the sidecar ships inside the weights repo).
|
||||
Environment=HALOGEN_VISION_TOWER=1
|
||||
# To free host memory, change the POOL or the ARENA, and the reserve then
|
||||
# becomes free to rise: HALOGEN_KV_POOL_POSITIONS=262144 hands back 3.6 GiB,
|
||||
# HALOGEN_MAX_TOK=16384 hands back another 4.2. Both are one-line edits the
|
||||
# fit check confirms on the next boot.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=20
|
||||
# Verbose engine logging: the vendor's first-line evidence for load,
|
||||
# allocation and admission questions. Shipping it on means an incident report
|
||||
# already has the full startup account instead of needing a restart-into-debug.
|
||||
|
||||
@@ -60,6 +60,7 @@ SERVICE_ORDER = (
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-large",
|
||||
"frank-small",
|
||||
"frank-medium",
|
||||
)
|
||||
|
||||
@@ -90,6 +91,7 @@ DANKCLAW_SERVICES = {
|
||||
"caddy",
|
||||
"homepage",
|
||||
"frank-large",
|
||||
"frank-small",
|
||||
}
|
||||
|
||||
DANKSERVER_SERVICES = {
|
||||
@@ -390,14 +392,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
name = p.name
|
||||
if name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}:
|
||||
_add(services, "homepage")
|
||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
||||
# The ik_llama.cpp runtime (2026-09-15 cutover) keeps its unit,
|
||||
# env and serve files here; each one selects frank-large only.
|
||||
_add(services, "frank-large")
|
||||
if name == "chat-template.jinja":
|
||||
# The shared chat template is read by frank-large's serve script
|
||||
# again; the vllm-radiance runtime on Dankserver ships its own.
|
||||
_add(services, "frank-large")
|
||||
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-small only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||
# restart a runtime service.
|
||||
return services
|
||||
|
||||
+91
-37
@@ -90,7 +90,7 @@ service_is_allowed() {
|
||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
||||
dankclaw:frank-large|\
|
||||
dankclaw:frank-large|dankclaw:frank-small|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
@@ -965,51 +965,104 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large runs as a ROOT unit again since 2026-09-15: ik_llama.cpp
|
||||
# (qwen4exp support merged upstream) replaced the halogen-flash-server
|
||||
# rootless Quadlet, and the ik server is a native binary — CUDA on the RTX
|
||||
# 3090 for dense/attention/KV, the Strix Halo CPU path for the MoE experts.
|
||||
# There is no container image to pull and no generated unit to avoid:
|
||||
# relink_unit points /etc/systemd/system/frank-large.service at the repo
|
||||
# file, and boot persistence is the unit's [Install] section.
|
||||
# frank-large runs as a rootless Quadlet unit again since 2026-09-16: the
|
||||
# halogen-flash-server runtime is back (it replaced ik_llama.cpp, which had
|
||||
# replaced it), and the RTX 3090 it shared during that interlude now belongs
|
||||
# to frank-small.
|
||||
#
|
||||
# The halogen Quadlet is retired first. Both managers must never know this
|
||||
# unit name at once (they would race for port 7770): stop the user unit,
|
||||
# poll until it is really gone, move the installed .container into the home
|
||||
# trash (never rm), then reload the user manager.
|
||||
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M/Qwen3.8-Flash-Next-Uncensored.i1-Q4_K_M.gguf}"
|
||||
# TWO MANAGERS, ONE UNIT NAME. The ik incarnation ran as a ROOT unit whose
|
||||
# file is a symlink into this checkout; the halogen unit is a USER (quadlet)
|
||||
# unit. Both must never exist at once — they would race for port 7770 — so the
|
||||
# root unit is retired first: stop, disable, move the /etc symlink into the
|
||||
# home trash (never `rm`), reload the root manager. This host's `systemctl
|
||||
# disable` has been observed to delete the unit file itself (2026-08-26
|
||||
# incident), so the move tolerates a file that is already gone.
|
||||
#
|
||||
# FIRST START fetches ~127 GB of weights, and /health cannot answer while it
|
||||
# does. wait_for_llm_large_ready recognises that (checkpoint absent) and waits
|
||||
# for the file instead of burning the health budget on a failure that is not
|
||||
# one.
|
||||
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/halogen/qwen38-flash-next-w4b.hgn}"
|
||||
|
||||
retire_quadlet_llm_large() {
|
||||
local trash="$HOME/.AGENTTRASH/20260915-ik-swap/halogen-quadlet"
|
||||
systemctl --user stop frank-large.service 2>/dev/null || true
|
||||
# A stopping quadlet runs podman stop (SIGTERM, then SIGKILL at the
|
||||
# timeout); llama-family engines never preempt an in-flight generation.
|
||||
# Poll until the user unit is really inactive before removing its file.
|
||||
local i
|
||||
for ((i = 0; i < 30; i++)); do
|
||||
systemctl --user is-active --quiet frank-large.service 2>/dev/null || break
|
||||
sleep 2
|
||||
done
|
||||
if [[ -e "$HOME/.config/containers/systemd/frank-large.container" || -L "$HOME/.config/containers/systemd/frank-large.container" ]]; then
|
||||
retire_root_llm_large() {
|
||||
local trash="$HOME/.AGENTTRASH/20260916-halogen-restore/host-units"
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
run_privileged systemctl disable frank-large.service 2>/dev/null || true
|
||||
if [[ -e /etc/systemd/system/frank-large.service || -L /etc/systemd/system/frank-large.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$HOME/.config/containers/systemd/frank-large.container" \
|
||||
"$trash/frank-large.container"
|
||||
echo "retired halogen quadlet -> $trash"
|
||||
run_privileged mv -- /etc/systemd/system/frank-large.service \
|
||||
"$trash/frank-large.service"
|
||||
echo "retired root unit frank-large.service -> $trash"
|
||||
fi
|
||||
systemctl --user daemon-reload
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_large() {
|
||||
retire_quadlet_llm_large
|
||||
if [[ ! -f "$LLM_LARGE_CHECKPOINT" ]]; then
|
||||
echo "ERROR: checkpoint absent at $LLM_LARGE_CHECKPOINT" >&2
|
||||
echo "frank-large: download the mradermacher i1-Q4_K_M GGUF first; refusing to start a unit that cannot load." >&2
|
||||
retire_root_llm_large
|
||||
# Pull once, then run the localhost/-aliased tag: this host's quadlet path
|
||||
# cannot resolve registry-prefixed refs from local storage at create time
|
||||
# (same workaround as anubis/hermes), and Pull=never keeps boots off the
|
||||
# registry.
|
||||
local image="ghcr.io/peonist-ai/halogen-flash-server:0.5.8"
|
||||
local alias="localhost/peonist-ai/halogen-flash-server:0.5.8"
|
||||
podman pull "$image"
|
||||
podman tag "$image" "$alias"
|
||||
# The weights volume must exist, and the container needs it read-write:
|
||||
# that is where a first start downloads into.
|
||||
mkdir -p /home/dank/models/halogen
|
||||
install_quadlet frank-large dankclaw/frank-large/quadlet frank-large
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence comes from [Install] in the unit file.
|
||||
systemctl --user restart frank-large.service
|
||||
wait_for_llm_large_ready
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
}
|
||||
|
||||
# Bootstrap-aware gate: if the checkpoint is absent the container is fetching
|
||||
# it; wait for the file (the fetch resumes, and the unit restarts it on
|
||||
# failure) rather than gating on a /health that cannot answer yet. Reports
|
||||
# success either way, so a first deploy does not look like a failure while the
|
||||
# engine does exactly what it was told to.
|
||||
wait_for_llm_large_ready() {
|
||||
[[ -f "$LLM_LARGE_CHECKPOINT" ]] && return 0
|
||||
echo "frank-large: checkpoint absent at $LLM_LARGE_CHECKPOINT"
|
||||
echo "frank-large: first start — the container is downloading ~127 GB from"
|
||||
echo "frank-large: peonist-ai/halogen-qwen3.8-flash-next (resumes if interrupted)."
|
||||
local i
|
||||
for ((i = 1; i <= 1200; i++)); do
|
||||
if [[ -f "$LLM_LARGE_CHECKPOINT" ]]; then
|
||||
echo "frank-large: weights landed after approximately $((i * 2))s; gating on /health"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "WARN: frank-large download still in progress after 40m — NOT a failure." >&2
|
||||
echo "WARN: the unit keeps fetching; re-run this deploy (or re-check /health) later." >&2
|
||||
return 0
|
||||
}
|
||||
|
||||
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
|
||||
# ROOT unit: the runtime is a native CUDA binary, so there is no container
|
||||
# image to install and no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
|
||||
# is that unit's [Install] section.
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
|
||||
deploy_llm_small() {
|
||||
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
|
||||
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
|
||||
echo "frank-small: fetch ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (the -mtp build plus the mmproj) first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
relink_unit frank-large
|
||||
run_privileged systemctl enable frank-large.service
|
||||
run_privileged systemctl restart frank-large.service
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
relink_unit frank-small
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-small.service
|
||||
run_privileged systemctl restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium runs as a rootless Quadlet since the
|
||||
@@ -1255,6 +1308,7 @@ deploy_service() {
|
||||
seerr) deploy_seerr ;;
|
||||
marreta) deploy_marreta ;;
|
||||
frank-large) deploy_llm_large ;;
|
||||
frank-small) deploy_llm_small ;;
|
||||
frank-medium) deploy_llm_medium ;;
|
||||
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
||||
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
||||
|
||||
+1
-1
@@ -140,7 +140,7 @@ valid_service() {
|
||||
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
|
||||
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
||||
dankclaw:frank-large|dankclaw:frank-npu|\
|
||||
dankclaw:frank-large|dankclaw:frank-small|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Tunables for frank-small (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the IQ3_S
|
||||
# build with the MTP head, plus the BF16 vision projector) on mainline
|
||||
# llama.cpp, Dankclaw's RTX 3090, since 2026-09-16.
|
||||
#
|
||||
# Runtime: llama.cpp master 37b53fd (2026-09-16), CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is deliberately OFF in that build: it is only needed
|
||||
# for QUANTIZED KV with flash attention, and this service runs f16 KV (below).
|
||||
#
|
||||
# Model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the `-mtp` build of IQ3_S.
|
||||
# GSQ-RCO assigns a separate quantization type to every tensor from a
|
||||
# gradient-based search under a total size budget, rather than applying one
|
||||
# type throughout; IQ3_S at 3.50 bpw is the release's task-lossless operating
|
||||
# point (AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
|
||||
# GPQA-Diamond 0.51 behind, at 11.8 GB against 53.8).
|
||||
#
|
||||
# WHY THAT FILE: the `-mtp` build is ~0.35 GB larger because it carries the
|
||||
# model's Multi-Token Prediction head as GGUF block `blk.64.nextn.*`
|
||||
# (qwen35.nextn_predict_layers = 1). llama.cpp loads it from the SAME file --
|
||||
# `--spec-type draft-mtp` sets load_mtp, and the draft context runs on the main
|
||||
# model's weights (common/common.cpp passes shares_model when there is no
|
||||
# --model-draft; speculative.cpp gives it LLAMA_CONTEXT_TYPE_MTP), so there is
|
||||
# no second 12 GB copy in VRAM. MTP is therefore a property of the FILE, not of
|
||||
# the flags: the plain IQ3_S build has no MTP tensors to load.
|
||||
#
|
||||
# Architecture (GGUF arch qwen35): 64 layers laid out as 16 x (3 x (Gated
|
||||
# DeltaNet -> FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention
|
||||
# layers with CONSTANT-SIZE per-sequence state and 16 full-attention layers,
|
||||
# 4 KV heads x 256 head_dim. Only the attention layers take KV, so a token
|
||||
# costs 2 (K,V) x 4 heads x 256 x 2 B = 64 KiB at f16 (34 KiB at q8_0, 18 KiB
|
||||
# at q4_0), and the DeltaNet layers add ~0.16 GiB per sequence in f32 at any
|
||||
# depth.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB) -- modelled, and the loader's own ledger
|
||||
# ("KV self size", "compute buffer size") outranks it:
|
||||
# weights 11.29 GiB (12,120,016,960 B)
|
||||
# mmproj 0.87 GiB (931,146,528 B, 27 blocks)
|
||||
# MTP draft KV 0.50 GiB (its context is n_ctx long
|
||||
# over one nextn layer: 4 KiB/token)
|
||||
# DeltaNet state 0.16 GiB (one sequence)
|
||||
# CUDA context + prefill/decode bufs ~1.5 GiB
|
||||
# KV at CTX_SIZE 8.00 GiB (131,072 x 64 KiB)
|
||||
# --------
|
||||
# ~22.3 GiB of 24, ~2 GiB of slack
|
||||
# The binding number is the KV: ~9.5 GiB is left after the fixed terms, which
|
||||
# at 64 KiB/token is ~155k tokens across all slots. The deployed shape is one
|
||||
# slot at 131,072, full f16 precision, no KV quantization risk, no rebuild.
|
||||
# Alternatives inside the same budget: 2 x 65536 at f16; 4 x 65536 or
|
||||
# 1 x 262144 at q8_0, both of which need a rebuild with
|
||||
# -DGGML_CUDA_FA_ALL_QUANTS=ON (without it quantized-KV prefill collapses
|
||||
# ~12x on CUDA -- see the CUDA lesson in AGENTS.md).
|
||||
# --parallel N DIVIDES --ctx-size between slots; it does not multiply. One slot
|
||||
# means a second concurrent request queues behind the first.
|
||||
#
|
||||
# Sampling: greedy by default, which is the fleet contract (a request that
|
||||
# sends no temperature decodes greedy; a client that sends one gets its own).
|
||||
# Qwen's own agentic evaluations used temperature 1.0 / top_p 0.95, and the
|
||||
# harness applies exactly that per model in models.yml (extraBody), the same
|
||||
# way it did for the earlier frank-large runtime.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||
# those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
|
||||
# Model package. The -mtp file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp
|
||||
MODEL_FILE=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/mmproj-Qwen3.8-27B-BF16.gguf
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on Dankclaw;
|
||||
# 7770 stays frank-large's. The alias is the id at /v1/models and what clients
|
||||
# ask for -- the harness entry in models.yml names it.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# One slot at 131,072 tokens with f16 KV (see the sizing note above).
|
||||
CTX_SIZE=131072
|
||||
PARALLEL=1
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical). The
|
||||
# prefill wall clock at this model size is set by the card, not by ubatch, and
|
||||
# the small ubatch keeps the DeltaNet chunk buffers small. Raise UBATCH_SIZE
|
||||
# only after reading the compute buffer size the loader reports.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest --
|
||||
# this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention.
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding from the model's own MTP head (see the model note).
|
||||
# Draft depth stays llama.cpp's default of 3 tokens.
|
||||
SPEC_TYPE=draft-mtp
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
@@ -0,0 +1,20 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||
# though the weights themselves are copied into VRAM. Parity with the halogen
|
||||
# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp as frank-small on Dankclaw's
|
||||
# RTX 3090 via mainline llama.cpp.
|
||||
#
|
||||
# Topology: the whole model lives in VRAM (-ngl 999) -- 11.29 GiB of IQ3_S-mtp
|
||||
# weights, 0.87 GiB of BF16 vision projector and ~8 GiB of f16 KV inside the
|
||||
# card's 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
|
||||
# belongs to the halogen engine that serves frank-large.
|
||||
#
|
||||
# MTP: the `-mtp` build carries the NextN block (blk.64.nextn.*), and
|
||||
# `--spec-type draft-mtp` makes llama.cpp load it and run the draft context on
|
||||
# the SAME weights. There is no --model-draft here on purpose: a separate draft
|
||||
# path would load a second 12 GB copy of a 12 GB model. Draft depth is
|
||||
# llama.cpp's default (--spec-draft-n-max 3).
|
||||
#
|
||||
# Vision: --mmproj loads the BF16 projector, so image content parts work. The
|
||||
# fleet's image path moved here from frank-large on 2026-09-16, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
# models.yml. --jinja is pinned even though this build defaults it on: the
|
||||
# model's own template supplies reasoning_effort, preserve_thinking and the
|
||||
# tool-call grammar, and a future build that flips that default must not
|
||||
# silently take tools away.
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
: "${CTX_SIZE:=131072}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${NUM_THREADS:=8}"
|
||||
: "${THREAD_BATCH:=8}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
# is a download that has not finished, and the unit's restart policy would
|
||||
# otherwise turn that into a crash loop that reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
|
||||
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE")
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET"
|
||||
+15
-4
@@ -145,11 +145,12 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# frank-large moved to a rootless quadlet (halogen-flash-server) on
|
||||
# 2026-09-11; its unit carries the fleet's GPU and podman posture, so
|
||||
# it gets its own seam rather than the generic fallback below.
|
||||
# frank-large is a rootless quadlet again since 2026-09-16 (the
|
||||
# halogen-flash-server restore). Its unit carries the fleet's GPU and
|
||||
# podman posture, so it gets its own seam rather than the generic
|
||||
# fallback below.
|
||||
dankclaw/frank-large/*)
|
||||
add_test tests/frank_large_quadlet.bats
|
||||
add_test tests/frank_large_unit.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
@@ -187,6 +188,16 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
# script all live in scripts/, which is why this case has to precede
|
||||
# the generic scripts/*.service and scripts/* fallbacks below.
|
||||
scripts/frank-small.*|scripts/serve-frank-small.sh)
|
||||
add_test tests/frank_small_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/*.service)
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
|
||||
@@ -18,7 +18,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "deployment planner maps large-model changes to the large model only" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-large.env scripts/frank-large.service scripts/serve-frank-large.sh
|
||||
dankclaw/frank-large/quadlet/frank-large.container
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-large" ]
|
||||
}
|
||||
@@ -28,14 +28,18 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "chat-template changes plan frank-large: the ik runtime reads the tracked template" {
|
||||
# ik_llama.cpp (2026-09-15 cutover) serves the Qwen-Sharp template from
|
||||
# the tracked scripts/chat-template.jinja again. The vllm-radiance
|
||||
# runtime on Dankserver ships its own qwen-fixed-v22.3.jinja out of the
|
||||
# pinned clone, so a template edit deploys nothing there.
|
||||
@test "the retired ik chat template plans nothing, and its archive is still here" {
|
||||
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
||||
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
||||
# llama.cpp frank-small runtime serves the template the GGUF itself
|
||||
# carries, which is the one with reasoning_effort, preserve_thinking and
|
||||
# the tool-call grammar. Editing the archive must deploy nothing -- and the
|
||||
# archive must still be present, so this cannot pass by the file having
|
||||
# vanished.
|
||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp" ]
|
||||
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-large" ]
|
||||
[ -z "$output" ]
|
||||
|
||||
run python3 "$PLANNER" --target dankserver --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
@@ -51,9 +55,19 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
[ "$output" = "frank-medium" ]
|
||||
}
|
||||
|
||||
@test "the retired small endpoint is absent from the deployment inventory" {
|
||||
! grep -q '"frank-small"' "$PLANNER"
|
||||
! grep -qE 'frank-small\) deploy|dankserver:frank-small' "$REMOTE_DEPLOY" "$REPO_ROOT/scripts/deploy.sh"
|
||||
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-small" ]
|
||||
|
||||
# frank-small runs on Dankclaw on the RTX 3090; the Dankserver endpoint of
|
||||
# that name was retired with the FreeToken runtime on 2026-09-15 and must
|
||||
# not come back through a planner mapping.
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "documentation-only changes plan no runtime deployment" {
|
||||
|
||||
+79
-61
@@ -1,86 +1,104 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-large ik_llama.cpp root unit on dankclaw.
|
||||
# ik_llama.cpp replaced the halogen rootless Quadlet on 2026-09-15. This seam
|
||||
# pins the deployment-policy properties that no reviewer can see by reading
|
||||
# the flags alone: a root unit whose files all live in this repo, the hybrid
|
||||
# placement contract (experts + PLE mmap'd on the CPU path — NEVER the
|
||||
# CUDA_Host pinned buffer), no pinned CUDA host memory, the single client
|
||||
# port every consumer already targets, the fleet alias, greedy-by-default
|
||||
# sampling, vision via the mtmd mmproj, and a deploy handler that retires
|
||||
# the halogen quadlet without deleting it.
|
||||
# Seam: the frank-large halogen-flash-server quadlet on dankclaw.
|
||||
# halogen-flash-server replaced ik_llama.cpp again on 2026-09-16, so this
|
||||
# service is once more a rootless Quadlet container on the gfx1151 iGPU. This
|
||||
# seam pins what a flag list cannot show: the boot-persistent podman posture
|
||||
# (both device nodes, memlock, the create-time image contract), the
|
||||
# matched ctx/pool/slots set that IS the two-conversation promise, the
|
||||
# co-tenancy budget the arena and reserve are chosen for, the image handoff to
|
||||
# frank-small, and a deploy handler that retires the previous ROOT unit without
|
||||
# deleting it.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
QUADLET="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "frank-large unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
@test "the frank-large quadlet exists where the deployer installs it from" {
|
||||
[ -f "$QUADLET" ]
|
||||
}
|
||||
|
||||
@test "frank-large unit is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
@test "frank-large is boot-persistent, self-healing and container-shaped" {
|
||||
grep -q '^WantedBy=default.target$' "$QUADLET"
|
||||
grep -q '^Restart=on-failure$' "$QUADLET"
|
||||
grep -q '^RestartSec=' "$QUADLET"
|
||||
grep -q '^Pull=never$' "$QUADLET"
|
||||
# The localhost/-aliased tag is what create-time resolution needs on this
|
||||
# host; the deployer pulls the ghcr.io ref and re-tags it.
|
||||
grep -q '^Image=localhost/peonist-ai/halogen-flash-server:' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-large unit points at the repo serve script and env" {
|
||||
grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh$' "$UNIT"
|
||||
grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
|
||||
@test "frank-large has the gfx1151 devices and the image's documented posture" {
|
||||
grep -q '^AddDevice=/dev/kfd$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/dri$' "$QUADLET"
|
||||
grep -q '^PodmanArgs=--ipc=host --security-opt seccomp=unconfined$' "$QUADLET"
|
||||
grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET"
|
||||
# The weights mount must stay read-write: the first start downloads into it.
|
||||
grep -q '^Volume=/home/dank/models/halogen:/models$' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-large places experts and PLE on the mmap'd CPU path, never the pinned buffer" {
|
||||
# --cpu-moe routes CPU-overridden weights through the CUDA_Host PINNED
|
||||
# buffer (108.12 GiB measured on 2026-09-15; it evicted every co-tenant
|
||||
# into swap and starved sshd into unreachability). The -ot ...=CPU plain
|
||||
# CPU backend buffer is the mmap-backed path.
|
||||
grep -q -- '-ot "exps=CPU" -ot "per_layer_token_embd=CPU"' "$SERVE"
|
||||
grep -q '^GGML_CUDA_NO_PINNED_WEIGHTS=1$' "$ENV_FILE"
|
||||
# The serve script refuses to run without the no-pinned policy.
|
||||
grep -q 'GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host' "$SERVE"
|
||||
@test "frank-large keeps the fleet alias and publishes only the API port" {
|
||||
grep -q '^Environment=HALOGEN_MODEL_ID=frank-large$' "$QUADLET"
|
||||
grep -q '^PublishPort=7770:8731$' "$QUADLET"
|
||||
# The engine's own listener stays inside the container: it is unauthenticated.
|
||||
! grep -q 'PublishPort=.*8730' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-large serves the ik runtime from the unified-memory hybrid layout" {
|
||||
grep -q 'ik_llama.cpp/build/bin/llama-server' "$ENV_FILE"
|
||||
grep -q 'mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
@test "the KV pool is exactly two conversations of the per-request ceiling" {
|
||||
local ctx pool slots
|
||||
ctx="$(sed -nE 's/^Environment=HALOGEN_CTX=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
pool="$(sed -nE 's/^Environment=HALOGEN_KV_POOL_POSITIONS=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
slots="$(sed -nE 's/^Environment=HALOGEN_KV_SLOTS=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
[ -n "$ctx" ] && [ -n "$pool" ] && [ -n "$slots" ]
|
||||
# The deployed promise is two resident conversations: the pool is twice the
|
||||
# ceiling, one slot per conversation. Slots still oversubscribe the pool for
|
||||
# any real request, because admission reserves prompt + max_tokens.
|
||||
[ "$pool" -eq $((ctx * 2)) ]
|
||||
[ "$slots" -eq 2 ]
|
||||
}
|
||||
|
||||
@test "frank-large keeps the fleet identity and the client port" {
|
||||
grep -q '^LISTEN_PORT=7770$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-large$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
@test "the prefill arena stays inside the range the vendor tested" {
|
||||
local max_tok reserve
|
||||
max_tok="$(sed -nE 's/^Environment=HALOGEN_MAX_TOK=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
reserve="$(sed -nE 's/^Environment=HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
[ -n "$max_tok" ] && [ -n "$reserve" ]
|
||||
# 32768 is the vendor default; 65536 cannot boot on this host because the
|
||||
# arena alone outgrows the device budget. The arena is a per-call prefill
|
||||
# chunk, so a smaller value costs prefill speed, not context.
|
||||
[ "$max_tok" -le 32768 ]
|
||||
[ "$max_tok" -ge 16384 ]
|
||||
[ "$reserve" -ge 15 ]
|
||||
}
|
||||
|
||||
@test "frank-large decodes greedy by default (halogen client contract)" {
|
||||
grep -q '^TEMPERATURE=0.0$' "$ENV_FILE"
|
||||
grep -q -- '--temp "$TEMPERATURE"' "$SERVE"
|
||||
@test "frank-large loads no vision tower: the fleet's image path is frank-small" {
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET"
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
}
|
||||
|
||||
@test "frank-large keeps image input via the mtmd mmproj" {
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-f16.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the halogen quadlet without deleting it" {
|
||||
grep -q 'retire_quadlet_llm_large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user stop frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-large.container' "$REMOTE_DEPLOY"
|
||||
@test "the deploy handler retires the previous ROOT unit without deleting it" {
|
||||
grep -q 'retire_root_llm_large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl stop frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl disable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing checkpoint" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'checkpoint absent' "$REMOTE_DEPLOY"
|
||||
! grep -q 'install_quadlet frank-large' "$REMOTE_DEPLOY"
|
||||
@test "the deploy handler installs the quadlet and gates the first start on weights" {
|
||||
grep -q 'install_quadlet frank-large dankclaw/frank-large/quadlet frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_for_llm_large_ready' "$REMOTE_DEPLOY"
|
||||
grep -q 'qwen38-flash-next-w4b.hgn' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the ik_llama.cpp runtime that held this service is archived, not live" {
|
||||
# Its files must not sit in scripts/ where a stale root unit could still be
|
||||
# pointed at them, and must not be gone either: .AGENTTRASH is the rollback.
|
||||
local retired
|
||||
for retired in frank-large.env.ik-llama.cpp frank-large.service.ik-llama.cpp \
|
||||
serve-frank-large.sh.ik-llama.cpp; do
|
||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/$retired" ]
|
||||
done
|
||||
[ ! -e "$REPO_ROOT/scripts/frank-large.env" ]
|
||||
[ ! -e "$REPO_ROOT/scripts/frank-large.service" ]
|
||||
[ ! -e "$REPO_ROOT/scripts/serve-frank-large.sh" ]
|
||||
}
|
||||
|
||||
Executable
+106
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
|
||||
# went back to the halogen engine on the iGPU. This seam pins what the flags
|
||||
# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
|
||||
# is involved), the repo-relative paths the unit executes, the separate client
|
||||
# port, the MTP contract (the head lives in the weights file, and no second copy
|
||||
# of the model is loaded for it), the vision handoff from frank-large, and the
|
||||
# KV budget that has to fit a 24 GiB card beside 12 GiB of weights.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-small.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
|
||||
@test "frank-small unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
}
|
||||
|
||||
@test "frank-small is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
|
||||
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small offloads every layer to the 3090 with flash attention" {
|
||||
grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
|
||||
grep -q '^NGL=999$' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint; without --metrics there is no /metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small's MTP head comes from the weights file, not a second copy" {
|
||||
grep -q '^SPEC_TYPE=draft-mtp$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
# The -mtp build is the one carrying blk.64.nextn.*; the plain IQ3_S has no
|
||||
# MTP tensors at all, so MTP is a property of the file, not of the flags.
|
||||
grep -q '^MODEL_FILE=.*-mtp\.gguf$' "$ENV_FILE"
|
||||
# A separate draft model would load a second 12 GiB copy of a 12 GiB model.
|
||||
! grep -q -- '--model-draft' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small carries the fleet's image path, and only it" {
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-Qwen3\.8-27B-BF16\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "frank-small's KV fits the card beside the weights it loads" {
|
||||
local ctx parallel parallel_slots cache_k cache_v
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads x
|
||||
# 256 dim x 2 B = 64 KiB per token at f16. 11.29 GiB of weights + 0.87 of
|
||||
# mmproj + ~2 GiB of context/compute/MTP leave under 10 GiB of the card's
|
||||
# 24 GiB for KV, and --parallel divides --ctx-size between the slots.
|
||||
[ $((ctx * 64)) -le $((10 * 1024 * 1024)) ]
|
||||
# f16 is the deployed type on purpose: a quantized KV needs the
|
||||
# GGML_CUDA_FA_ALL_QUANTS build option, without which prefill on CUDA
|
||||
# collapses about 12x (see the CUDA lesson in AGENTS.md).
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
}
|
||||
|
||||
@test "frank-small runs on its own port beside frank-large's" {
|
||||
grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-small$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
# The two endpoints cannot share a port, and only the halogen unit publishes
|
||||
# 7770.
|
||||
grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE"
|
||||
! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing model" {
|
||||
grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp' "$REMOTE_DEPLOY"
|
||||
grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "frank-small's serve script refuses to start on a half-finished download" {
|
||||
grep -q 'frank-small: missing' "$SERVE"
|
||||
}
|
||||
Reference in New Issue
Block a user