dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

inference: frank-large returns to halogen, frank-small takes the 3090

Browse Source 
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.

frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.

The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.

Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
This commit is contained in:
Frank
2026-09-16 13:58:00 -07:00
parent cc29875842
commit 43ec77fee6
19 changed files with 648 additions and 198 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy-remote.sh
+91 -37
View File
@@ -90,7 +90,7 @@ service_is_allowed() {
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
dankclaw:frank-large|\
dankclaw:frank-large|dankclaw:frank-small|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:plex|dankserver:tautulli|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
@@ -965,51 +965,104 @@ deploy_anubis() {
wait_container_healthy anubis 60 || return 1
}
# frank-large runs as a ROOT unit again since 2026-09-15: ik_llama.cpp
# (qwen4exp support merged upstream) replaced the halogen-flash-server
# rootless Quadlet, and the ik server is a native binary — CUDA on the RTX
# 3090 for dense/attention/KV, the Strix Halo CPU path for the MoE experts.
# There is no container image to pull and no generated unit to avoid:
# relink_unit points /etc/systemd/system/frank-large.service at the repo
# file, and boot persistence is the unit's [Install] section.
# frank-large runs as a rootless Quadlet unit again since 2026-09-16: the
# halogen-flash-server runtime is back (it replaced ik_llama.cpp, which had
# replaced it), and the RTX 3090 it shared during that interlude now belongs
# to frank-small.
#
# The halogen Quadlet is retired first. Both managers must never know this
# unit name at once (they would race for port 7770): stop the user unit,
# poll until it is really gone, move the installed .container into the home
# trash (never rm), then reload the user manager.
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M/Qwen3.8-Flash-Next-Uncensored.i1-Q4_K_M.gguf}"
# TWO MANAGERS, ONE UNIT NAME. The ik incarnation ran as a ROOT unit whose
# file is a symlink into this checkout; the halogen unit is a USER (quadlet)
# unit. Both must never exist at once — they would race for port 7770 — so the
# root unit is retired first: stop, disable, move the /etc symlink into the
# home trash (never `rm`), reload the root manager. This host's `systemctl
# disable` has been observed to delete the unit file itself (2026-08-26
# incident), so the move tolerates a file that is already gone.
#
# FIRST START fetches ~127 GB of weights, and /health cannot answer while it
# does. wait_for_llm_large_ready recognises that (checkpoint absent) and waits
# for the file instead of burning the health budget on a failure that is not
# one.
LLM_LARGE_CHECKPOINT="${LLM_LARGE_CHECKPOINT:-/home/dank/models/halogen/qwen38-flash-next-w4b.hgn}"
retire_quadlet_llm_large() {
local trash="$HOME/.AGENTTRASH/20260915-ik-swap/halogen-quadlet"
systemctl --user stop frank-large.service 2>/dev/null || true
# A stopping quadlet runs podman stop (SIGTERM, then SIGKILL at the
# timeout); llama-family engines never preempt an in-flight generation.
# Poll until the user unit is really inactive before removing its file.
local i
for ((i = 0; i < 30; i++)); do
systemctl --user is-active --quiet frank-large.service 2>/dev/null || break
sleep 2
done
if [[ -e "$HOME/.config/containers/systemd/frank-large.container" || -L "$HOME/.config/containers/systemd/frank-large.container" ]]; then
retire_root_llm_large() {
local trash="$HOME/.AGENTTRASH/20260916-halogen-restore/host-units"
run_privileged systemctl stop frank-large.service 2>/dev/null || true
run_privileged systemctl disable frank-large.service 2>/dev/null || true
if [[ -e /etc/systemd/system/frank-large.service || -L /etc/systemd/system/frank-large.service ]]; then
mkdir -p "$trash"
mv -- "$HOME/.config/containers/systemd/frank-large.container" \
"$trash/frank-large.container"
echo "retired halogen quadlet -> $trash"
run_privileged mv -- /etc/systemd/system/frank-large.service \
"$trash/frank-large.service"
echo "retired root unit frank-large.service -> $trash"
fi
systemctl --user daemon-reload
run_privileged systemctl daemon-reload
}
deploy_llm_large() {
retire_quadlet_llm_large
if [[ ! -f "$LLM_LARGE_CHECKPOINT" ]]; then
echo "ERROR: checkpoint absent at $LLM_LARGE_CHECKPOINT" >&2
echo "frank-large: download the mradermacher i1-Q4_K_M GGUF first; refusing to start a unit that cannot load." >&2
retire_root_llm_large
# Pull once, then run the localhost/-aliased tag: this host's quadlet path
# cannot resolve registry-prefixed refs from local storage at create time
# (same workaround as anubis/hermes), and Pull=never keeps boots off the
# registry.
local image="ghcr.io/peonist-ai/halogen-flash-server:0.5.8"
local alias="localhost/peonist-ai/halogen-flash-server:0.5.8"
podman pull "$image"
podman tag "$image" "$alias"
# The weights volume must exist, and the container needs it read-write:
# that is where a first start downloads into.
mkdir -p /home/dank/models/halogen
install_quadlet frank-large dankclaw/frank-large/quadlet frank-large
# No enable: generated units cannot be enabled on this host (v1 quadlet
# generator); boot persistence comes from [Install] in the unit file.
systemctl --user restart frank-large.service
wait_for_llm_large_ready
wait_for_health frank-large http://127.0.0.1:7770/health 1200
}
# Bootstrap-aware gate: if the checkpoint is absent the container is fetching
# it; wait for the file (the fetch resumes, and the unit restarts it on
# failure) rather than gating on a /health that cannot answer yet. Reports
# success either way, so a first deploy does not look like a failure while the
# engine does exactly what it was told to.
wait_for_llm_large_ready() {
[[ -f "$LLM_LARGE_CHECKPOINT" ]] && return 0
echo "frank-large: checkpoint absent at $LLM_LARGE_CHECKPOINT"
echo "frank-large: first start — the container is downloading ~127 GB from"
echo "frank-large: peonist-ai/halogen-qwen3.8-flash-next (resumes if interrupted)."
local i
for ((i = 1; i <= 1200; i++)); do
if [[ -f "$LLM_LARGE_CHECKPOINT" ]]; then
echo "frank-large: weights landed after approximately $((i * 2))s; gating on /health"
return 0
fi
sleep 2
done
echo "WARN: frank-large download still in progress after 40m — NOT a failure." >&2
echo "WARN: the unit keeps fetching; re-run this deploy (or re-check /health) later." >&2
return 0
}
# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
# ROOT unit: the runtime is a native CUDA binary, so there is no container
# image to install and no generated unit to avoid. relink_unit points
# /etc/systemd/system/frank-small.service at the repo file and boot persistence
# is that unit's [Install] section.
#
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
# not shared: frank-large's halogen engine runs on the gfx1151 iGPU.
LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
deploy_llm_small() {
if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
echo "frank-small: fetch ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (the -mtp build plus the mmproj) first;" >&2
echo "frank-small: refusing to start a unit that cannot load." >&2
return 1
fi
relink_unit frank-large
run_privileged systemctl enable frank-large.service
run_privileged systemctl restart frank-large.service
wait_for_health frank-large http://127.0.0.1:7770/health 1200
relink_unit frank-small
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-small.service
run_privileged systemctl restart frank-small.service
wait_for_health frank-small http://127.0.0.1:7771/health 900
}
# deploy_llm_medium — frank-medium runs as a rootless Quadlet since the
@@ -1255,6 +1308,7 @@ deploy_service() {
seerr) deploy_seerr ;;
marreta) deploy_marreta ;;
frank-large) deploy_llm_large ;;
frank-small) deploy_llm_small ;;
frank-medium) deploy_llm_medium ;;
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;