inference: frank-large returns to halogen, frank-small takes the 3090
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server quadlet again -- the 2026-09-13 unit file was never touched by the two-day ik_llama.cpp interlude, so the runtime was still in the repo and its image is still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192 with a 400,384-position pool and two slots, exactly two 200k conversations. frank-small is new: mainline llama.cpp on the RTX 3090 serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main model's own weights instead of a second 12 GiB copy) plus the BF16 vision projector, as one 131,072-token slot with f16 KV on port 7771. The host is shared now, and that is what the halogen retune pays for: HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to 20 to keep the fit check honest. Vision moved wholesale to frank-small. Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the Qwen-Sharp template only that runtime read, and four benchmark harnesses bound to retired runtimes. Because retiring a file makes any test that greps it pass by absence, the seams assert the archive exists AND the live path does not.
This commit is contained in:
Frank committed
1 parent
cc29875842
commit
43ec77fee6
19 files changed
+648
-198
No files matched your search
@@ -3,11 +3,13 @@
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
||||
# Unit name: frank-large.service (container name: frank-large)
|
||||
#
|
||||
# The large inference service, since 2026-09-11 the halogen-flash-server
|
||||
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
||||
# The large inference service. The halogen-flash-server runtime (since
|
||||
# 2026-09-11, restored 2026-09-16 after a two-day ik_llama.cpp hybrid
|
||||
# interlude) serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
||||
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
||||
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
||||
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
||||
# prefill (the dominant cost of any real prompt) on this exact SKU. The RTX
|
||||
# 3090 it briefly shared with frank-large is now frank-small's card.
|
||||
#
|
||||
# SHAPE. The image runs both halves in one container: the engine on container
|
||||
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
||||
@@ -39,18 +41,23 @@
|
||||
# engine lowers it itself and prints what it settled on. Read that ledger on
|
||||
# every restart — it is the ground truth for sizing, not this comment.
|
||||
#
|
||||
# POOL TRIMMED 2026-09-13: 401,408 positions (~11.0 GiB of KV as reserved by
|
||||
# the engine), down from the 524,288 the 2026-09-12 doubling deployed. The
|
||||
# doubling fit only at reserve 15, which left the host 7.1 GiB / 310
|
||||
# contiguous 2 MiB blocks and made the engine print its co-tenant stall
|
||||
# warning on every boot. MEASURED AT 401,408 (deploy ledger): 68.0 weights +
|
||||
# 11.0 KV + 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB
|
||||
# blocks left, engine need printed ~31.5 vs the modelled ~33.4, and NO
|
||||
# stall-warning NOTE. Capacity: one full 262k conversation resident beside a
|
||||
# ~139k one, or three concurrent 131k chats -- the fleet's real shape at
|
||||
# three slots. Fit check budget: 121.2 - 67.7 - 15 = 38.5. The 09-12 doubling
|
||||
# story (silent clamp at reserve 24, the arena term, the reserve ceiling) is
|
||||
# preserved in the budget block below and the runbook.
|
||||
# POOL RETUNED 2026-09-16: 400,384 positions -- exactly two 200,192-token
|
||||
# conversations -- beside a per-conversation ceiling of 200,192 and two slots.
|
||||
# It replaces the 401,408 the 2026-09-13 trim left behind, which held one full
|
||||
# native-262k conversation beside a ~139k one, or three at 131k.
|
||||
#
|
||||
# MEASURED AT 401,408 (2026-09-13 deploy ledger): 68.0 weights + 11.0 KV +
|
||||
# 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB blocks left,
|
||||
# engine need printed ~31.5 against the modelled ~33.4, and NO stall-warning
|
||||
# NOTE. Fit-check budget here: 121.2 - 67.7 - 20 = 33.5. That boot is the
|
||||
# baseline the 09-16 retune is measured against: the vision tower (1.97 GiB)
|
||||
# and roughly 4.2 GiB of arena come back to the host, which is what pays for
|
||||
# the co-tenant below.
|
||||
#
|
||||
# CO-TENANT since 2026-09-16: this host also serves frank-small on the RTX
|
||||
# 3090 (llama.cpp, ~12.2 GiB of weights that load through the page cache and
|
||||
# then live in VRAM, plus a CUDA context). It is the first real co-tenant this
|
||||
# service has had, which is why the arena was halved and the reserve raised.
|
||||
#
|
||||
# BOOT NOTE: on the first request after a (re)start the engine can go silent
|
||||
# for ~45-60 s ("has not answered PING") while it first-touches the freshly
|
||||
@@ -67,9 +74,11 @@
|
||||
# tower and the tokenizer. The repo is public and ungated, so no token is
|
||||
# needed; later starts open no outbound connections at all.
|
||||
#
|
||||
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
||||
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
||||
# 2026-09-11; audio input has no replacement.
|
||||
# VISION. Deliberately NOT set since 2026-09-16. The tower is a 1.97 GiB
|
||||
# charge against the same device budget as the pool and the arena, and the
|
||||
# fleet's image path moved to frank-small (llama.cpp on the RTX 3090, loaded
|
||||
# with --mmproj mmproj-Qwen3.8-27B-BF16). This server refuses image content
|
||||
# parts, and models.yml declares frank-large text-only to match.
|
||||
#
|
||||
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
||||
# There are no server-side sampler defaults: a request that sends no
|
||||
@@ -101,83 +110,84 @@ Volume=/home/dank/models/halogen:/models
|
||||
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
||||
# Fleet identity: the id at /v1/models, /health and every response body.
|
||||
Environment=HALOGEN_MODEL_ID=frank-large
|
||||
# Native context of the model. Matches what both clients declare.
|
||||
Environment=HALOGEN_CTX=262144
|
||||
# Trimmed 2026-09-13 from the 524,288 the 09-12 doubling deployed: buys
|
||||
# ~3.4 GiB of co-tenant headroom, keeps one full 262k conversation resident
|
||||
# beside a ~139k one, or three concurrent 131k chats.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=401408
|
||||
# Concurrency cap. Each slot is 111.0 MiB of O(1) state; the KV is ONE
|
||||
# 401408-position pool carved into per-request regions, so slots cap how many
|
||||
# requests run at once while the pool binds how many positions exist. The
|
||||
# slots still OVERSUBSCRIBE the pool (3 x 262144 > 401408), so admission is
|
||||
# pool-bound -- one full-length conversation plus a ~139k second, or three
|
||||
# at 131k.
|
||||
# LOWERED 4 -> 3 on 2026-09-12: frees ~111 MiB on a host with 7.1 GiB left,
|
||||
# and three users is the real fleet size. A fourth concurrent request now
|
||||
# queues behind the three rather than contending.
|
||||
Environment=HALOGEN_KV_SLOTS=3
|
||||
# Sizing is budget arithmetic, all of it. ONE budget covers
|
||||
# the KV pool, the prefill arena and the vision tower together:
|
||||
# Per-conversation ceiling: exactly half the pool, so two conversations of the
|
||||
# full window are always resident. 200,192 rather than a round 200,000 is
|
||||
# pool/2. The model's native window is 262,144; a conversation past this
|
||||
# ceiling is refused, never silently truncated. Clients declare 200192 (see
|
||||
# the frank-large contextWindow in models.yml).
|
||||
Environment=HALOGEN_CTX=200192
|
||||
# The KV pool: 400,384 positions = 2 x 200,192 (the 2026-09-16 retune, down
|
||||
# from 401,408). ~28 KiB per position including block scratch, so ~11.0 GiB
|
||||
# held -- the 401,408 ledger entry measured exactly 11.0. The two slots still
|
||||
# OVERSUBSCRIBE the pool for any real request, because admission reserves
|
||||
# prompt + max_tokens: two full-window conversations each holding an output
|
||||
# budget are pool-bound, and the second waits in arrival order for the first
|
||||
# to release its region. Read the `kv pool:` lines on every boot;
|
||||
# HALOGEN_KV_POOL_FIT lowers the pool itself if the budget refuses the
|
||||
# configured one, and a green health check does not mean the pool you asked
|
||||
# for is the pool you got.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=400384
|
||||
# Concurrency cap: two, one per half of the pool. Each slot is 111.0 MiB of
|
||||
# O(1) state, so this hands ~111 MiB back against the three set on 2026-09-12
|
||||
# for the fleet size of the time.
|
||||
Environment=HALOGEN_KV_SLOTS=2
|
||||
# Sizing is budget arithmetic, all of it. ONE budget covers the KV pool, the
|
||||
# per-call prefill arena and the vision tower together:
|
||||
#
|
||||
# budget = 121.2 host - 67.7 resident weights - reserve
|
||||
# need = KV (~28 KiB/position) + arena + vision (1.97 measured)
|
||||
# need = KV (~28 KiB/position) + arena + vision
|
||||
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
||||
#
|
||||
# The arena term is linear in max_tok, and it reproduces five live restarts
|
||||
# AND upstream's three published "on the device" figures to within 0.1 GiB —
|
||||
# note those published figures are pool+arena totals, not pool alone, which is
|
||||
# the misreading that first made a doubling look affordable here. Every corner
|
||||
# this box offered at a reserve of 24:
|
||||
# the misreading that first made a doubling look affordable here. Corners this
|
||||
# box offered at a reserve of 24, at the old 262,144-position pool:
|
||||
#
|
||||
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
|
||||
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
|
||||
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
|
||||
#
|
||||
# None of those three gave both features; reserve 15 made the doubled pool
|
||||
# fit (38.5 vs 37.0) but at 7.1 GiB of host headroom. On 2026-09-13 the pool
|
||||
# came down to 401,408 to restore co-tenant headroom: need ~33.4 vs 38.5.
|
||||
# LOWERED 32768 -> 24576 on 2026-09-16 for the frank-small co-tenant on this
|
||||
# same host. The arena's variable term goes 8.40 x 2 -> 8.40 x 1.5, handing
|
||||
# ~4.2 GiB back to the host at roughly a 4-5 % prefill cost -- the same lever
|
||||
# at 16,384 costs 8-9 %. The vendor's own notes name 24,576 as the largest
|
||||
# arena that allocated with headroom to spare, so this stays inside the range
|
||||
# it has tested.
|
||||
#
|
||||
# The arena is a PER-CALL PREFILL CHUNK, not a prompt ceiling: the
|
||||
# 200,192-token conversation is prefilled in arena-sized pieces, so a smaller
|
||||
# arena means more pieces and a slower prefill, not a smaller conversation.
|
||||
#
|
||||
# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
|
||||
# the fit check's printed ledger on each boot is the arbiter, not these
|
||||
# comments.
|
||||
Environment=HALOGEN_MAX_TOK=32768
|
||||
# RAM left for the n-gram table the engine reads through the page cache.
|
||||
# LOWERED 2026-09-12 from 24 (vendor default is 20) to buy the doubled pool at
|
||||
# full prefill speed. The fit check's OWN requirement is the line to honour —
|
||||
# read from a clamped boot, not modelled:
|
||||
# "524288 positions need ~35.0 GiB (plus 1.5 of margin)" -> budget >= 36.5
|
||||
# budget = 121.2 host - 67.7 weights - reserve
|
||||
# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed). At the
|
||||
# old 524,288 pool, 18 -> 35.5 CLAMPED it to 262144, so the ceiling there was
|
||||
# 16; at 401,408 (need ~33.4) it loosens, but a higher reserve still buys
|
||||
# nothing.
|
||||
Environment=HALOGEN_MAX_TOK=24576
|
||||
# RAM the fit check refuses to count on, watched over by the engine's own
|
||||
# n-gram lookup table (47.7 GiB, read through the page cache and never held)
|
||||
# and, since 2026-09-16, by frank-small on the same host.
|
||||
#
|
||||
# AT THE DOUBLED POOL THE CEILING WAS 16, AND 16 BOUGHT NOTHING. Tested live at 524,288: 15 -> 16 changed the fit
|
||||
# margin only. The server held 103.5 GiB and reported "7.1 GiB left for
|
||||
# everything else" at BOTH settings, because this reserve feeds the fit-check
|
||||
# budget and is not an allocation — actual footprint is weights + KV + arena +
|
||||
# vision, none of which it touches. A higher reserve therefore cannot give the
|
||||
# host memory back; it can only take the pool away. 15 keeps the larger margin.
|
||||
# RAISED 15 -> 20 with the 2026-09-16 retune. This knob feeds the fit-check
|
||||
# budget and is NOT an allocation: raising it cannot hand memory back to the
|
||||
# host, because the actual footprint is weights + KV + arena + vision, none of
|
||||
# which it touches — it can only take the pool away. What it does is keep the
|
||||
# fit check honest at the smaller arena. With the arena at 24,576 and no
|
||||
# vision tower, the engine's printed need falls to roughly 25 GiB against a
|
||||
# budget of 121.2 - 67.7 - 20 = 33.5, leaving ~8 GiB of margin, so the pool is
|
||||
# not clamped.
|
||||
#
|
||||
# Measured at 15 before that test: pool reserved at 524288 with no clamp and no
|
||||
# WARNING, and the server held 103.5 GiB. The ledger
|
||||
# reads "host memory left for everything else: 310 contiguous 2 MiB blocks
|
||||
# (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve
|
||||
# baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set).
|
||||
# The 2026-09-13 profile for comparison: 103.5 GiB held and "7.1 GiB left for
|
||||
# everything else" (310 contiguous 2 MiB blocks) at the doubled 524,288 pool,
|
||||
# which is why the pool came down; at 401,408 the same ledger read 9.1 GiB /
|
||||
# 1396 blocks. The 09-16 retune gives back the vision tower (1.97) and ~4.2 of
|
||||
# arena on top of that for the co-tenant, so the expected figure at this boot
|
||||
# is ~15 GiB left. Read the ledger; do not trust this arithmetic.
|
||||
#
|
||||
# The price at 524,288 was that the engine volunteered this note on every boot:
|
||||
# when it runs out, BOTH they and this server can stall for minutes at a
|
||||
# time at full CPU with no disk activity and no output, which is not a
|
||||
# crash and needs no restart.
|
||||
# That is the failure mode to watch, and it is NOT the legible one predicted
|
||||
# above: a minutes-long stall that looks like a hang is harder to diagnose
|
||||
# than a slow prompt. The 09-13 trim to 401,408 was the first application of
|
||||
# the remedy. If it happens again, cut further:
|
||||
# HALOGEN_KV_POOL_POSITIONS=262144 (hands back another 3.6 GiB), or
|
||||
# HALOGEN_MAX_TOK=16384 (hands back 8.4) with reserve then free to rise to 24.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=15
|
||||
# Image input (the sidecar ships inside the weights repo).
|
||||
Environment=HALOGEN_VISION_TOWER=1
|
||||
# To free host memory, change the POOL or the ARENA, and the reserve then
|
||||
# becomes free to rise: HALOGEN_KV_POOL_POSITIONS=262144 hands back 3.6 GiB,
|
||||
# HALOGEN_MAX_TOK=16384 hands back another 4.2. Both are one-line edits the
|
||||
# fit check confirms on the next boot.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=20
|
||||
# Verbose engine logging: the vendor's first-line evidence for load,
|
||||
# allocation and admission questions. Shipping it on means an incident report
|
||||
# already has the full startup account instead of needing a restart-into-debug.
|
||||
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -60,6 +60,7 @@ SERVICE_ORDER = (
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-large",
|
||||
"frank-small",
|
||||
"frank-medium",
|
||||
)
|
||||
|
||||
@@ -90,6 +91,7 @@ DANKCLAW_SERVICES = {
|
||||
"caddy",
|
||||
"homepage",
|
||||
"frank-large",
|
||||
"frank-small",
|
||||
}
|
||||
|
||||
DANKSERVER_SERVICES = {
|
||||
@@ -390,14 +392,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
name = p.name
|
||||
if name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}:
|
||||
_add(services, "homepage")
|
||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
||||
# The ik_llama.cpp runtime (2026-09-15 cutover) keeps its unit,
|
||||
# env and serve files here; each one selects frank-large only.
|
||||
_add(services, "frank-large")
|
||||
if name == "chat-template.jinja":
|
||||
# The shared chat template is read by frank-large's serve script
|
||||
# again; the vllm-radiance runtime on Dankserver ships its own.
|
||||
_add(services, "frank-large")
|
||||
if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
|
||||
# llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-small only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||
# restart a runtime service.
|
||||
return services
|
||||
|
||||
+91
-37
@@ -90,7 +90,7 @@ service_is_allowed() {
|
||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
||||
dankclaw:frank-large|\
|
||||
dankclaw:frank-large|dankclaw:frank-small|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
@@ -965,51 +965,104 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large runs as a ROOT unit again since 2026-09-15: ik_llama.cpp
|
||||
# (qwen4exp support merged upstream) replaced the halogen-flash-server
|
||||
# rootless Quadlet, and the ik server is a native binary — CUDA on the RTX
|
||||
# 3090 for dense/attention/KV, the Strix Halo CPU path for the MoE experts.
|
||||
# There is no container image to pull and no generated unit to avoid:
|
||||
# relink_unit points /etc/systemd/system/frank-large.service at the repo
|
||||
# file, and boot persistence is the unit's [Install] section.
|
||||
# frank-large runs as a rootless Quadlet unit again since 2026-09-16: the
|
||||
# halogen-flash-server runtime is back (it replaced ik_llama.cpp, which had
|
||||
# replaced it), and the RTX 3090 it shared during that interlude now belongs
|
||||
# to frank-small.
|
||||
#
|
||||
# The halogen Quadlet is retired first. Both managers must never know this
|
||||
# unit name at once (they would race for port 7770): stop the user unit,
|
||||
# poll until it is really gone, move the installed .container into the home
|
||||
# trash (never rm), then reload the user manager.
|
||||
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M/Qwen3.8-Flash-Next-Uncensored.i1-Q4_K_M.gguf}"
|
||||
# TWO MANAGERS, ONE UNIT NAME. The ik incarnation ran as a ROOT unit whose
|
||||
# file is a symlink into this checkout; the halogen unit is a USER (quadlet)
|
||||
# unit. Both must never exist at once — they would race for port 7770 — so the
|
||||
# root unit is retired first: stop, disable, move the /etc symlink into the
|
||||
# home trash (never `rm`), reload the root manager. This host's `systemctl
|
||||
# disable` has been observed to delete the unit file itself (2026-08-26
|
||||
# incident), so the move tolerates a file that is already gone.
|
||||
#
|
||||
# FIRST START fetches ~127 GB of weights, and /health cannot answer while it
|
||||
# does. wait_for_llm_large_ready recognises that (checkpoint absent) and waits
|
||||
# for the file instead of burning the health budget on a failure that is not
|
||||
# one.
|
||||
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/halogen/qwen38-flash-next-w4b.hgn}"
|
||||
|
||||
retire_quadlet_llm_large() {
|
||||
local trash="$HOME/.AGENTTRASH/20260915-ik-swap/halogen-quadlet"
|
||||
systemctl --user stop frank-large.service 2>/dev/null || true
|
||||
# A stopping quadlet runs podman stop (SIGTERM, then SIGKILL at the
|
||||
# timeout); llama-family engines never preempt an in-flight generation.
|
||||
# Poll until the user unit is really inactive before removing its file.
|
||||
local i
|
||||
for ((i = 0; i < 30; i++)); do
|
||||
systemctl --user is-active --quiet frank-large.service 2>/dev/null || break
|
||||
sleep 2
|
||||
done
|
||||
if [[ -e "$HOME/.config/containers/systemd/frank-large.container" || -L "$HOME/.config/containers/systemd/frank-large.container" ]]; then
|
||||
retire_root_llm_large() {
|
||||
local trash="$HOME/.AGENTTRASH/20260916-halogen-restore/host-units"
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
run_privileged systemctl disable frank-large.service 2>/dev/null || true
|
||||
if [[ -e /etc/systemd/system/frank-large.service || -L /etc/systemd/system/frank-large.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$HOME/.config/containers/systemd/frank-large.container" \
|
||||
"$trash/frank-large.container"
|
||||
echo "retired halogen quadlet -> $trash"
|
||||
run_privileged mv -- /etc/systemd/system/frank-large.service \
|
||||
"$trash/frank-large.service"
|
||||
echo "retired root unit frank-large.service -> $trash"
|
||||
fi
|
||||
systemctl --user daemon-reload
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_large() {
|
||||
retire_quadlet_llm_large
|
||||
if [[ ! -f "$LLM_LARGE_CHECKPOINT" ]]; then
|
||||
echo "ERROR: checkpoint absent at $LLM_LARGE_CHECKPOINT" >&2
|
||||
echo "frank-large: download the mradermacher i1-Q4_K_M GGUF first; refusing to start a unit that cannot load." >&2
|
||||
retire_root_llm_large
|
||||
# Pull once, then run the localhost/-aliased tag: this host's quadlet path
|
||||
# cannot resolve registry-prefixed refs from local storage at create time
|
||||
# (same workaround as anubis/hermes), and Pull=never keeps boots off the
|
||||
# registry.
|
||||
local image="ghcr.io/peonist-ai/halogen-flash-server:0.5.8"
|
||||
local alias="localhost/peonist-ai/halogen-flash-server:0.5.8"
|
||||
podman pull "$image"
|
||||
podman tag "$image" "$alias"
|
||||
# The weights volume must exist, and the container needs it read-write:
|
||||
# that is where a first start downloads into.
|
||||
mkdir -p /home/dank/models/halogen
|
||||
install_quadlet frank-large dankclaw/frank-large/quadlet frank-large
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence comes from [Install] in the unit file.
|
||||
systemctl --user restart frank-large.service
|
||||
wait_for_llm_large_ready
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
}
|
||||
|
||||
# Bootstrap-aware gate: if the checkpoint is absent the container is fetching
|
||||
# it; wait for the file (the fetch resumes, and the unit restarts it on
|
||||
# failure) rather than gating on a /health that cannot answer yet. Reports
|
||||
# success either way, so a first deploy does not look like a failure while the
|
||||
# engine does exactly what it was told to.
|
||||
wait_for_llm_large_ready() {
|
||||
[[ -f "$LLM_LARGE_CHECKPOINT" ]] && return 0
|
||||
echo "frank-large: checkpoint absent at $LLM_LARGE_CHECKPOINT"
|
||||
echo "frank-large: first start — the container is downloading ~127 GB from"
|
||||
echo "frank-large: peonist-ai/halogen-qwen3.8-flash-next (resumes if interrupted)."
|
||||
local i
|
||||
for ((i = 1; i <= 1200; i++)); do
|
||||
if [[ -f "$LLM_LARGE_CHECKPOINT" ]]; then
|
||||
echo "frank-large: weights landed after approximately $((i * 2))s; gating on /health"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "WARN: frank-large download still in progress after 40m — NOT a failure." >&2
|
||||
echo "WARN: the unit keeps fetching; re-run this deploy (or re-check /health) later." >&2
|
||||
return 0
|
||||
}
|
||||
|
||||
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
|
||||
# ROOT unit: the runtime is a native CUDA binary, so there is no container
|
||||
# image to install and no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
|
||||
# is that unit's [Install] section.
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
|
||||
deploy_llm_small() {
|
||||
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
|
||||
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
|
||||
echo "frank-small: fetch ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (the -mtp build plus the mmproj) first;" >&2
|
||||
echo "frank-small: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
relink_unit frank-large
|
||||
run_privileged systemctl enable frank-large.service
|
||||
run_privileged systemctl restart frank-large.service
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
relink_unit frank-small
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-small.service
|
||||
run_privileged systemctl restart frank-small.service
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium runs as a rootless Quadlet since the
|
||||
@@ -1255,6 +1308,7 @@ deploy_service() {
|
||||
seerr) deploy_seerr ;;
|
||||
marreta) deploy_marreta ;;
|
||||
frank-large) deploy_llm_large ;;
|
||||
frank-small) deploy_llm_small ;;
|
||||
frank-medium) deploy_llm_medium ;;
|
||||
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
||||
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
||||
|
||||
+1
-1
@@ -140,7 +140,7 @@ valid_service() {
|
||||
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
|
||||
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
||||
dankclaw:frank-large|dankclaw:frank-npu|\
|
||||
dankclaw:frank-large|dankclaw:frank-small|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Tunables for frank-small (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the IQ3_S
|
||||
# build with the MTP head, plus the BF16 vision projector) on mainline
|
||||
# llama.cpp, Dankclaw's RTX 3090, since 2026-09-16.
|
||||
#
|
||||
# Runtime: llama.cpp master 37b53fd (2026-09-16), CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is deliberately OFF in that build: it is only needed
|
||||
# for QUANTIZED KV with flash attention, and this service runs f16 KV (below).
|
||||
#
|
||||
# Model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the `-mtp` build of IQ3_S.
|
||||
# GSQ-RCO assigns a separate quantization type to every tensor from a
|
||||
# gradient-based search under a total size budget, rather than applying one
|
||||
# type throughout; IQ3_S at 3.50 bpw is the release's task-lossless operating
|
||||
# point (AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
|
||||
# GPQA-Diamond 0.51 behind, at 11.8 GB against 53.8).
|
||||
#
|
||||
# WHY THAT FILE: the `-mtp` build is ~0.35 GB larger because it carries the
|
||||
# model's Multi-Token Prediction head as GGUF block `blk.64.nextn.*`
|
||||
# (qwen35.nextn_predict_layers = 1). llama.cpp loads it from the SAME file --
|
||||
# `--spec-type draft-mtp` sets load_mtp, and the draft context runs on the main
|
||||
# model's weights (common/common.cpp passes shares_model when there is no
|
||||
# --model-draft; speculative.cpp gives it LLAMA_CONTEXT_TYPE_MTP), so there is
|
||||
# no second 12 GB copy in VRAM. MTP is therefore a property of the FILE, not of
|
||||
# the flags: the plain IQ3_S build has no MTP tensors to load.
|
||||
#
|
||||
# Architecture (GGUF arch qwen35): 64 layers laid out as 16 x (3 x (Gated
|
||||
# DeltaNet -> FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention
|
||||
# layers with CONSTANT-SIZE per-sequence state and 16 full-attention layers,
|
||||
# 4 KV heads x 256 head_dim. Only the attention layers take KV, so a token
|
||||
# costs 2 (K,V) x 4 heads x 256 x 2 B = 64 KiB at f16 (34 KiB at q8_0, 18 KiB
|
||||
# at q4_0), and the DeltaNet layers add ~0.16 GiB per sequence in f32 at any
|
||||
# depth.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB) -- modelled, and the loader's own ledger
|
||||
# ("KV self size", "compute buffer size") outranks it:
|
||||
# weights 11.29 GiB (12,120,016,960 B)
|
||||
# mmproj 0.87 GiB (931,146,528 B, 27 blocks)
|
||||
# MTP draft KV 0.50 GiB (its context is n_ctx long
|
||||
# over one nextn layer: 4 KiB/token)
|
||||
# DeltaNet state 0.16 GiB (one sequence)
|
||||
# CUDA context + prefill/decode bufs ~1.5 GiB
|
||||
# KV at CTX_SIZE 8.00 GiB (131,072 x 64 KiB)
|
||||
# --------
|
||||
# ~22.3 GiB of 24, ~2 GiB of slack
|
||||
# The binding number is the KV: ~9.5 GiB is left after the fixed terms, which
|
||||
# at 64 KiB/token is ~155k tokens across all slots. The deployed shape is one
|
||||
# slot at 131,072, full f16 precision, no KV quantization risk, no rebuild.
|
||||
# Alternatives inside the same budget: 2 x 65536 at f16; 4 x 65536 or
|
||||
# 1 x 262144 at q8_0, both of which need a rebuild with
|
||||
# -DGGML_CUDA_FA_ALL_QUANTS=ON (without it quantized-KV prefill collapses
|
||||
# ~12x on CUDA -- see the CUDA lesson in AGENTS.md).
|
||||
# --parallel N DIVIDES --ctx-size between slots; it does not multiply. One slot
|
||||
# means a second concurrent request queues behind the first.
|
||||
#
|
||||
# Sampling: greedy by default, which is the fleet contract (a request that
|
||||
# sends no temperature decodes greedy; a client that sends one gets its own).
|
||||
# Qwen's own agentic evaluations used temperature 1.0 / top_p 0.95, and the
|
||||
# harness applies exactly that per model in models.yml (extraBody), the same
|
||||
# way it did for the earlier frank-large runtime.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||
# those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
|
||||
# Model package. The -mtp file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp
|
||||
MODEL_FILE=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/mmproj-Qwen3.8-27B-BF16.gguf
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on Dankclaw;
|
||||
# 7770 stays frank-large's. The alias is the id at /v1/models and what clients
|
||||
# ask for -- the harness entry in models.yml names it.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# One slot at 131,072 tokens with f16 KV (see the sizing note above).
|
||||
CTX_SIZE=131072
|
||||
PARALLEL=1
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical). The
|
||||
# prefill wall clock at this model size is set by the card, not by ubatch, and
|
||||
# the small ubatch keeps the DeltaNet chunk buffers small. Raise UBATCH_SIZE
|
||||
# only after reading the compute buffer size the loader reports.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest --
|
||||
# this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention.
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding from the model's own MTP head (see the model note).
|
||||
# Draft depth stays llama.cpp's default of 3 tokens.
|
||||
SPEC_TYPE=draft-mtp
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
@@ -0,0 +1,20 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||
# though the weights themselves are copied into VRAM. Parity with the halogen
|
||||
# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp as frank-small on Dankclaw's
|
||||
# RTX 3090 via mainline llama.cpp.
|
||||
#
|
||||
# Topology: the whole model lives in VRAM (-ngl 999) -- 11.29 GiB of IQ3_S-mtp
|
||||
# weights, 0.87 GiB of BF16 vision projector and ~8 GiB of f16 KV inside the
|
||||
# card's 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
|
||||
# belongs to the halogen engine that serves frank-large.
|
||||
#
|
||||
# MTP: the `-mtp` build carries the NextN block (blk.64.nextn.*), and
|
||||
# `--spec-type draft-mtp` makes llama.cpp load it and run the draft context on
|
||||
# the SAME weights. There is no --model-draft here on purpose: a separate draft
|
||||
# path would load a second 12 GB copy of a 12 GB model. Draft depth is
|
||||
# llama.cpp's default (--spec-draft-n-max 3).
|
||||
#
|
||||
# Vision: --mmproj loads the BF16 projector, so image content parts work. The
|
||||
# fleet's image path moved here from frank-large on 2026-09-16, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
# models.yml. --jinja is pinned even though this build defaults it on: the
|
||||
# model's own template supplies reasoning_effort, preserve_thinking and the
|
||||
# tool-call grammar, and a future build that flips that default must not
|
||||
# silently take tools away.
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
: "${CTX_SIZE:=131072}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${NUM_THREADS:=8}"
|
||||
: "${THREAD_BATCH:=8}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
# is a download that has not finished, and the unit's restart policy would
|
||||
# otherwise turn that into a crash loop that reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
|
||||
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE")
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET"
|
||||
+15
-4
@@ -145,11 +145,12 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# frank-large moved to a rootless quadlet (halogen-flash-server) on
|
||||
# 2026-09-11; its unit carries the fleet's GPU and podman posture, so
|
||||
# it gets its own seam rather than the generic fallback below.
|
||||
# frank-large is a rootless quadlet again since 2026-09-16 (the
|
||||
# halogen-flash-server restore). Its unit carries the fleet's GPU and
|
||||
# podman posture, so it gets its own seam rather than the generic
|
||||
# fallback below.
|
||||
dankclaw/frank-large/*)
|
||||
add_test tests/frank_large_quadlet.bats
|
||||
add_test tests/frank_large_unit.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
@@ -187,6 +188,16 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
# script all live in scripts/, which is why this case has to precede
|
||||
# the generic scripts/*.service and scripts/* fallbacks below.
|
||||
scripts/frank-small.*|scripts/serve-frank-small.sh)
|
||||
add_test tests/frank_small_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/*.service)
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
|
||||
@@ -18,7 +18,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "deployment planner maps large-model changes to the large model only" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-large.env scripts/frank-large.service scripts/serve-frank-large.sh
|
||||
dankclaw/frank-large/quadlet/frank-large.container
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-large" ]
|
||||
}
|
||||
@@ -28,14 +28,18 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
! grep -q 'frank_npu\|frank-npu' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "chat-template changes plan frank-large: the ik runtime reads the tracked template" {
|
||||
# ik_llama.cpp (2026-09-15 cutover) serves the Qwen-Sharp template from
|
||||
# the tracked scripts/chat-template.jinja again. The vllm-radiance
|
||||
# runtime on Dankserver ships its own qwen-fixed-v22.3.jinja out of the
|
||||
# pinned clone, so a template edit deploys nothing there.
|
||||
@test "the retired ik chat template plans nothing, and its archive is still here" {
|
||||
# scripts/chat-template.jinja existed only for the ik_llama.cpp runtime
|
||||
# (retired 2026-09-16 in favour of halogen-flash-server again). The
|
||||
# llama.cpp frank-small runtime serves the template the GGUF itself
|
||||
# carries, which is the one with reasoning_effort, preserve_thinking and
|
||||
# the tool-call grammar. Editing the archive must deploy nothing -- and the
|
||||
# archive must still be present, so this cannot pass by the file having
|
||||
# vanished.
|
||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp" ]
|
||||
run python3 "$PLANNER" --target dankclaw --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-large" ]
|
||||
[ -z "$output" ]
|
||||
|
||||
run python3 "$PLANNER" --target dankserver --paths scripts/chat-template.jinja
|
||||
[ "$status" -eq 0 ]
|
||||
@@ -51,9 +55,19 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
[ "$output" = "frank-medium" ]
|
||||
}
|
||||
|
||||
@test "the retired small endpoint is absent from the deployment inventory" {
|
||||
! grep -q '"frank-small"' "$PLANNER"
|
||||
! grep -qE 'frank-small\) deploy|dankserver:frank-small' "$REMOTE_DEPLOY" "$REPO_ROOT/scripts/deploy.sh"
|
||||
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
|
||||
run python3 "$PLANNER" --target dankclaw --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-small" ]
|
||||
|
||||
# frank-small runs on Dankclaw on the RTX 3090; the Dankserver endpoint of
|
||||
# that name was retired with the FreeToken runtime on 2026-09-15 and must
|
||||
# not come back through a planner mapping.
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "documentation-only changes plan no runtime deployment" {
|
||||
|
||||
+79
-61
@@ -1,86 +1,104 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-large ik_llama.cpp root unit on dankclaw.
|
||||
# ik_llama.cpp replaced the halogen rootless Quadlet on 2026-09-15. This seam
|
||||
# pins the deployment-policy properties that no reviewer can see by reading
|
||||
# the flags alone: a root unit whose files all live in this repo, the hybrid
|
||||
# placement contract (experts + PLE mmap'd on the CPU path — NEVER the
|
||||
# CUDA_Host pinned buffer), no pinned CUDA host memory, the single client
|
||||
# port every consumer already targets, the fleet alias, greedy-by-default
|
||||
# sampling, vision via the mtmd mmproj, and a deploy handler that retires
|
||||
# the halogen quadlet without deleting it.
|
||||
# Seam: the frank-large halogen-flash-server quadlet on dankclaw.
|
||||
# halogen-flash-server replaced ik_llama.cpp again on 2026-09-16, so this
|
||||
# service is once more a rootless Quadlet container on the gfx1151 iGPU. This
|
||||
# seam pins what a flag list cannot show: the boot-persistent podman posture
|
||||
# (both device nodes, memlock, the create-time image contract), the
|
||||
# matched ctx/pool/slots set that IS the two-conversation promise, the
|
||||
# co-tenancy budget the arena and reserve are chosen for, the image handoff to
|
||||
# frank-small, and a deploy handler that retires the previous ROOT unit without
|
||||
# deleting it.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
QUADLET="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "frank-large unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
@test "the frank-large quadlet exists where the deployer installs it from" {
|
||||
[ -f "$QUADLET" ]
|
||||
}
|
||||
|
||||
@test "frank-large unit is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
@test "frank-large is boot-persistent, self-healing and container-shaped" {
|
||||
grep -q '^WantedBy=default.target$' "$QUADLET"
|
||||
grep -q '^Restart=on-failure$' "$QUADLET"
|
||||
grep -q '^RestartSec=' "$QUADLET"
|
||||
grep -q '^Pull=never$' "$QUADLET"
|
||||
# The localhost/-aliased tag is what create-time resolution needs on this
|
||||
# host; the deployer pulls the ghcr.io ref and re-tags it.
|
||||
grep -q '^Image=localhost/peonist-ai/halogen-flash-server:' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-large unit points at the repo serve script and env" {
|
||||
grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh$' "$UNIT"
|
||||
grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
|
||||
@test "frank-large has the gfx1151 devices and the image's documented posture" {
|
||||
grep -q '^AddDevice=/dev/kfd$' "$QUADLET"
|
||||
grep -q '^AddDevice=/dev/dri$' "$QUADLET"
|
||||
grep -q '^PodmanArgs=--ipc=host --security-opt seccomp=unconfined$' "$QUADLET"
|
||||
grep -q '^Ulimit=memlock=-1:-1$' "$QUADLET"
|
||||
# The weights mount must stay read-write: the first start downloads into it.
|
||||
grep -q '^Volume=/home/dank/models/halogen:/models$' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-large places experts and PLE on the mmap'd CPU path, never the pinned buffer" {
|
||||
# --cpu-moe routes CPU-overridden weights through the CUDA_Host PINNED
|
||||
# buffer (108.12 GiB measured on 2026-09-15; it evicted every co-tenant
|
||||
# into swap and starved sshd into unreachability). The -ot ...=CPU plain
|
||||
# CPU backend buffer is the mmap-backed path.
|
||||
grep -q -- '-ot "exps=CPU" -ot "per_layer_token_embd=CPU"' "$SERVE"
|
||||
grep -q '^GGML_CUDA_NO_PINNED_WEIGHTS=1$' "$ENV_FILE"
|
||||
# The serve script refuses to run without the no-pinned policy.
|
||||
grep -q 'GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host' "$SERVE"
|
||||
@test "frank-large keeps the fleet alias and publishes only the API port" {
|
||||
grep -q '^Environment=HALOGEN_MODEL_ID=frank-large$' "$QUADLET"
|
||||
grep -q '^PublishPort=7770:8731$' "$QUADLET"
|
||||
# The engine's own listener stays inside the container: it is unauthenticated.
|
||||
! grep -q 'PublishPort=.*8730' "$QUADLET"
|
||||
}
|
||||
|
||||
@test "frank-large serves the ik runtime from the unified-memory hybrid layout" {
|
||||
grep -q 'ik_llama.cpp/build/bin/llama-server' "$ENV_FILE"
|
||||
grep -q 'mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
@test "the KV pool is exactly two conversations of the per-request ceiling" {
|
||||
local ctx pool slots
|
||||
ctx="$(sed -nE 's/^Environment=HALOGEN_CTX=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
pool="$(sed -nE 's/^Environment=HALOGEN_KV_POOL_POSITIONS=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
slots="$(sed -nE 's/^Environment=HALOGEN_KV_SLOTS=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
[ -n "$ctx" ] && [ -n "$pool" ] && [ -n "$slots" ]
|
||||
# The deployed promise is two resident conversations: the pool is twice the
|
||||
# ceiling, one slot per conversation. Slots still oversubscribe the pool for
|
||||
# any real request, because admission reserves prompt + max_tokens.
|
||||
[ "$pool" -eq $((ctx * 2)) ]
|
||||
[ "$slots" -eq 2 ]
|
||||
}
|
||||
|
||||
@test "frank-large keeps the fleet identity and the client port" {
|
||||
grep -q '^LISTEN_PORT=7770$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-large$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
@test "the prefill arena stays inside the range the vendor tested" {
|
||||
local max_tok reserve
|
||||
max_tok="$(sed -nE 's/^Environment=HALOGEN_MAX_TOK=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
reserve="$(sed -nE 's/^Environment=HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$QUADLET")"
|
||||
[ -n "$max_tok" ] && [ -n "$reserve" ]
|
||||
# 32768 is the vendor default; 65536 cannot boot on this host because the
|
||||
# arena alone outgrows the device budget. The arena is a per-call prefill
|
||||
# chunk, so a smaller value costs prefill speed, not context.
|
||||
[ "$max_tok" -le 32768 ]
|
||||
[ "$max_tok" -ge 16384 ]
|
||||
[ "$reserve" -ge 15 ]
|
||||
}
|
||||
|
||||
@test "frank-large decodes greedy by default (halogen client contract)" {
|
||||
grep -q '^TEMPERATURE=0.0$' "$ENV_FILE"
|
||||
grep -q -- '--temp "$TEMPERATURE"' "$SERVE"
|
||||
@test "frank-large loads no vision tower: the fleet's image path is frank-small" {
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET"
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
}
|
||||
|
||||
@test "frank-large keeps image input via the mtmd mmproj" {
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-f16.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the halogen quadlet without deleting it" {
|
||||
grep -q 'retire_quadlet_llm_large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user stop frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-large.container' "$REMOTE_DEPLOY"
|
||||
@test "the deploy handler retires the previous ROOT unit without deleting it" {
|
||||
grep -q 'retire_root_llm_large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl stop frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl disable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing checkpoint" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'checkpoint absent' "$REMOTE_DEPLOY"
|
||||
! grep -q 'install_quadlet frank-large' "$REMOTE_DEPLOY"
|
||||
@test "the deploy handler installs the quadlet and gates the first start on weights" {
|
||||
grep -q 'install_quadlet frank-large dankclaw/frank-large/quadlet frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_for_llm_large_ready' "$REMOTE_DEPLOY"
|
||||
grep -q 'qwen38-flash-next-w4b.hgn' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the ik_llama.cpp runtime that held this service is archived, not live" {
|
||||
# Its files must not sit in scripts/ where a stale root unit could still be
|
||||
# pointed at them, and must not be gone either: .AGENTTRASH is the rollback.
|
||||
local retired
|
||||
for retired in frank-large.env.ik-llama.cpp frank-large.service.ik-llama.cpp \
|
||||
serve-frank-large.sh.ik-llama.cpp; do
|
||||
[ -f "$REPO_ROOT/scripts/.AGENTTRASH/$retired" ]
|
||||
done
|
||||
[ ! -e "$REPO_ROOT/scripts/frank-large.env" ]
|
||||
[ ! -e "$REPO_ROOT/scripts/frank-large.service" ]
|
||||
[ ! -e "$REPO_ROOT/scripts/serve-frank-large.sh" ]
|
||||
}
|
||||
Executable
+106
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
|
||||
# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
|
||||
# went back to the halogen engine on the iGPU. This seam pins what the flags
|
||||
# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
|
||||
# is involved), the repo-relative paths the unit executes, the separate client
|
||||
# port, the MTP contract (the head lives in the weights file, and no second copy
|
||||
# of the model is loaded for it), the vision handoff from frank-large, and the
|
||||
# KV budget that has to fit a 24 GiB card beside 12 GiB of weights.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-small.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
QUADLET_LARGE="$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container"
|
||||
|
||||
@test "frank-small unit, serve script and env exist in scripts/" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
}
|
||||
|
||||
@test "frank-small is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
|
||||
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-small offloads every layer to the 3090 with flash attention" {
|
||||
grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
|
||||
grep -q '^NGL=999$' "$ENV_FILE"
|
||||
grep -q -- '--gpu-layers "$NGL"' "$SERVE"
|
||||
grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
|
||||
grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint; without --metrics there is no /metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small's MTP head comes from the weights file, not a second copy" {
|
||||
grep -q '^SPEC_TYPE=draft-mtp$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
# The -mtp build is the one carrying blk.64.nextn.*; the plain IQ3_S has no
|
||||
# MTP tensors at all, so MTP is a property of the file, not of the flags.
|
||||
grep -q '^MODEL_FILE=.*-mtp\.gguf$' "$ENV_FILE"
|
||||
# A separate draft model would load a second 12 GiB copy of a 12 GiB model.
|
||||
! grep -q -- '--model-draft' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-small carries the fleet's image path, and only it" {
|
||||
grep -q '^MMPROJ_PATH=.*mmproj-Qwen3\.8-27B-BF16\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
|
||||
! grep -q 'HALOGEN_VISION_TOWER' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "frank-small's KV fits the card beside the weights it loads" {
|
||||
local ctx parallel parallel_slots cache_k cache_v
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads x
|
||||
# 256 dim x 2 B = 64 KiB per token at f16. 11.29 GiB of weights + 0.87 of
|
||||
# mmproj + ~2 GiB of context/compute/MTP leave under 10 GiB of the card's
|
||||
# 24 GiB for KV, and --parallel divides --ctx-size between the slots.
|
||||
[ $((ctx * 64)) -le $((10 * 1024 * 1024)) ]
|
||||
# f16 is the deployed type on purpose: a quantized KV needs the
|
||||
# GGML_CUDA_FA_ALL_QUANTS build option, without which prefill on CUDA
|
||||
# collapses about 12x (see the CUDA lesson in AGENTS.md).
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
}
|
||||
|
||||
@test "frank-small runs on its own port beside frank-large's" {
|
||||
grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
|
||||
grep -q '^ALIAS=frank-small$' "$ENV_FILE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
# The two endpoints cannot share a port, and only the halogen unit publishes
|
||||
# 7770.
|
||||
grep -q '^PublishPort=7770:8731$' "$QUADLET_LARGE"
|
||||
! grep -q 'PublishPort=.*7771' "$QUADLET_LARGE"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing model" {
|
||||
grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp' "$REMOTE_DEPLOY"
|
||||
grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "frank-small's serve script refuses to start on a half-finished download" {
|
||||
grep -q 'frank-small: missing' "$SERVE"
|
||||
}
|
||||
Reference in new issue
Block a user