retire frank-medium: tests, planning and observability ground on frank-primary
The frank-medium retirement tail the 2026-09-26 lucebox cutover left uncommitted: the radiance quadlet tree, its swap/fetch tooling and its seam test are deleted (archive copies live in dankserver/.AGENTTRASH/ 20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926- frank-medium-radiance/), the planner/allowlist/test-changed mappings drop the service, and frank-large's chat template moves to scripts/ (its old home was the retired frank-medium dir). Finished here: - tests/frank_primary_unit.bats: the lucebox quadlet seam (boot persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved env-name guard, checkout-mounted entrypoint/healthcheck, first-start patience, deployer payload/port gates). - test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move files to trash, and the wrapper refused those additions with "no focused-test mapping". - observability: the vllm: namespace left with the radiance runtime. Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode, queueing, endpoint-down); the dashboard drops the vLLM KV/latency panels; prometheus.yml documents that frank-primary (lucebox) has no /metrics and therefore no scrape job. llm-fleet.json was left corrupted mid-edit (missing opening brace) and is rewritten from HEAD. Suite: 339 green (was 7 red).
This commit is contained in:
Frank
+7
-13
@@ -63,7 +63,6 @@ SERVICE_ORDER = (
|
||||
"readarr",
|
||||
"frank-large",
|
||||
"frank-small",
|
||||
"frank-medium",
|
||||
"frank-primary",
|
||||
)
|
||||
|
||||
@@ -114,7 +113,6 @@ DANKSERVER_SERVICES = {
|
||||
"sonarr",
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-medium",
|
||||
"frank-primary",
|
||||
"node-exporter",
|
||||
}
|
||||
@@ -373,16 +371,6 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
if name == "node-exporter":
|
||||
_add(services, "node-exporter")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-medium":
|
||||
# frank-medium is a CONTAINER again since 2026-09-20: the TP=2
|
||||
# radiance Quadlet, its entrypoint and its healthcheck all live
|
||||
# under this directory, and the container mounts the latter two
|
||||
# straight out of this checkout — so any file here changes deployed
|
||||
# state. (It was a root llama.cpp unit from 2026-09-16, whose trio
|
||||
# lived in scripts/; that shape is archived in
|
||||
# scripts/.AGENTTRASH/20260920-llama-layersplit/ and plans nothing.)
|
||||
_add(services, "frank-medium")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-primary":
|
||||
# frank-primary is the Lucebox DeepSeek V4 Quadlet (2026-09-26
|
||||
# cutover, replacing frank-medium on :7770): its entrypoint and
|
||||
@@ -426,6 +414,13 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# runtime. The download script fetches the payload, which is
|
||||
# host-side state, so it selects nothing.
|
||||
_add(services, "frank-large")
|
||||
if name == "qwen-sharp-v22.4.0.jinja":
|
||||
# The Qwen-Sharp chat template frank-large loads via
|
||||
# --chat-template-file. It lived under dankserver/frank-medium/ until
|
||||
# the 2026-09-26 cutover retired that directory; editing it changes
|
||||
# what the live frank-large endpoint renders, so it plans
|
||||
# frank-large.
|
||||
_add(services, "frank-large")
|
||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||
# restart a runtime service.
|
||||
return services
|
||||
@@ -435,7 +430,6 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
return services
|
||||
return services
|
||||
|
||||
|
||||
def changed_paths(repo: str, base: str, head: str) -> list[str]:
|
||||
result = subprocess.run(
|
||||
["git", "-C", repo, "diff", "--name-only", "--diff-filter=ACDMRTUXB", base, head],
|
||||
|
||||
+7
-162
@@ -95,7 +95,7 @@ service_is_allowed() {
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||
dankserver:frank-medium|dankserver:frank-primary|dankserver:node-exporter|\
|
||||
dankserver:frank-primary|dankserver:node-exporter|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
@@ -1178,159 +1178,6 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance stack as a
|
||||
# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2)
|
||||
# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter.
|
||||
# The entrypoint is quantization-aware (config.json quant_method): the plain
|
||||
# FP8 path, the MXFP4 patch-and-serve path, and the ParoQuant MXFP6 stack
|
||||
# (PR #51's launch6_p kernel). It replaced mainline llama.cpp's layer split,
|
||||
# which held the endpoint since 2026-09-16; the conversation battery measured
|
||||
# radiance at 2.1x decode and a third of the time-to-first-token with 4x the
|
||||
# concurrent throughput (scripts/bench-conversation.py, 2026-09-20).
|
||||
#
|
||||
# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this
|
||||
# unit because TP=2 could not dispatch a collective on the second card (B650
|
||||
# chipset bridges, AtomicOpsCap: Routing-), and the two-replica + routing-proxy
|
||||
# trio was the workaround. The board swap cleared the atomics wall, so one
|
||||
# TP=2 process is both possible and the fastest shape; the replicas and their
|
||||
# affinity proxy stay retired.
|
||||
#
|
||||
# HOST ASSETS. Nothing here installs the image's payloads: the checkpoints, the
|
||||
# pinned radiance patch clone, the patched libr4d build and the Triton compile
|
||||
# cache live outside the checkout (see the unit file's mount comments). The
|
||||
# gates below refuse a deploy that would start a container guaranteed to die.
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-ggz14/vllm-radiance-mxfp4:0.13.0-e060895}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.13.0-e060895}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-PARO-MXFP6}"
|
||||
LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}"
|
||||
LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}"
|
||||
LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq}"
|
||||
|
||||
# The three replica/proxy user units from the 2026-09-16 era are retired HERE,
|
||||
# never left to rot: they must not exist alongside the TP=2 unit (the proxy
|
||||
# published 7770 on the host network). Never `rm` — unit files move into the
|
||||
# home trash; the images stay in local storage. `frank-medium` itself is NOT in
|
||||
# the list: it is the unit this handler installs.
|
||||
retire_vllm_replica_quadlets() {
|
||||
local trash="$HOME/.AGENTTRASH/20260916-vllm-radiance-retire" unit
|
||||
systemctl --user stop frank-medium-a.service frank-medium-b.service \
|
||||
frank-medium-proxy.service 2>/dev/null || true
|
||||
for unit in frank-medium-a frank-medium-b frank-medium-proxy; do
|
||||
if [[ -e "$quadlet_unit_dir/$unit.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/$unit.container" "$trash/$unit.container"
|
||||
echo "retired quadlet $unit.container -> $trash"
|
||||
fi
|
||||
done
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
|
||||
# The llama.cpp root unit is retired HERE, into the home trash, never `rm`:
|
||||
# /etc/systemd/system/frank-medium.service is a relink_unit symlink into this
|
||||
# checkout, and the checkout pull that precedes this handler already moved the
|
||||
# repo file away, leaving the symlink dangling. Stop it while systemd still has
|
||||
# it loaded (an active unit stays loaded without its file), then take both the
|
||||
# symlink and its [Install] alias. Nothing can start it at boot afterwards.
|
||||
retire_llama_medium_unit() {
|
||||
local trash="$HOME/.AGENTTRASH/20260920-llama-layersplit-frank-medium"
|
||||
run_privileged systemctl stop frank-medium.service 2>/dev/null || true
|
||||
if [[ -L /etc/systemd/system/frank-medium.service ||
|
||||
-L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
[[ -L /etc/systemd/system/frank-medium.service ]] &&
|
||||
run_privileged mv -- /etc/systemd/system/frank-medium.service \
|
||||
"$trash/etc-frank-medium.service.symlink"
|
||||
[[ -L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]] &&
|
||||
run_privileged mv -- /etc/systemd/system/multi-user.target.wants/frank-medium.service \
|
||||
"$trash/etc-frank-medium.wants.symlink"
|
||||
echo "retired llama root unit frank-medium.service -> $trash"
|
||||
fi
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_medium() {
|
||||
# Refuse rather than restart-loop. A container whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the endpoint
|
||||
# dark with nothing in the deploy log to say why.
|
||||
local missing=0
|
||||
# Model-aware gates: read the serving shape from the quadlet (the source
|
||||
# of truth), so a model swap is just "edit the quadlet + deploy". The
|
||||
# container /models mount maps to /home/dank/models on the host. The
|
||||
# drafter gate applies only to the dflash path; the patched libr4d gate
|
||||
# applies to the mxfp4/quark/paroquant_mxfp6 path (the plain FP8 path uses
|
||||
# the image's built-in r4d.so and skips the MXFP4 patches).
|
||||
local quadlet_file="dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
local snap_c snap spec_method quant
|
||||
snap_c="$(sed -nE 's/^Environment=SNAP=//p' "$quadlet_file" | head -1)"
|
||||
spec_method="$(sed -nE 's/^Environment=SPEC_METHOD=//p' "$quadlet_file" | head -1)"
|
||||
snap="/home/dank/models${snap_c#/models}"
|
||||
quant="$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get("quantization_config",{}).get("quant_method",""))' "$snap/config.json" 2>/dev/null || true)"
|
||||
[[ -n "$snap_c" ]] || { echo "ERROR: cannot read SNAP from $quadlet_file" >&2; return 1; }
|
||||
if ! podman image exists "$LLM_MEDIUM_IMAGE_LOCAL"; then
|
||||
if podman image exists "$LLM_MEDIUM_IMAGE"; then
|
||||
podman tag "$LLM_MEDIUM_IMAGE" "$LLM_MEDIUM_IMAGE_LOCAL"
|
||||
else
|
||||
echo "ERROR: radiance image absent ($LLM_MEDIUM_IMAGE / $LLM_MEDIUM_IMAGE_LOCAL)" >&2
|
||||
echo "frank-medium: podman pull $LLM_MEDIUM_IMAGE && podman tag $LLM_MEDIUM_IMAGE $LLM_MEDIUM_IMAGE_LOCAL" >&2
|
||||
missing=1
|
||||
fi
|
||||
fi
|
||||
if [[ ! -f "$snap/config.json" ]]; then
|
||||
echo "ERROR: model checkpoint absent at $snap" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ "$spec_method" == "dflash" && ! -f "$LLM_MEDIUM_DRAFTER/config.json" ]]; then
|
||||
echo "ERROR: DFlash2-FP8 drafter absent at $LLM_MEDIUM_DRAFTER" >&2
|
||||
missing=1
|
||||
fi
|
||||
# The patches clone is mounted and its scripts run at EVERY container start
|
||||
# on the MXFP4 path; the chat template is the fleet's reasoning-format
|
||||
# contract on both paths.
|
||||
if [[ ! -f "$LLM_MEDIUM_PATCHES/qwen-fixed-v22.3.jinja" ]]; then
|
||||
echo "ERROR: pinned patch clone / chat template absent at $LLM_MEDIUM_PATCHES" >&2
|
||||
echo "frank-medium: git clone https://github.com/ggz14/radiance-vllm-mxfp4 $LLM_MEDIUM_PATCHES" >&2
|
||||
missing=1
|
||||
fi
|
||||
# The image's shipped r4d.so predates the gated-delta-net overflow fix and
|
||||
# NaNs the MXFP4 model, so the patched build is not optional there. The
|
||||
# plain FP8 path uses the image's built-in r4d.so (no MXFP4 patches).
|
||||
case "$quant" in
|
||||
mxfp4|quark|paroquant_mxfp6)
|
||||
if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then
|
||||
echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2
|
||||
missing=1
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
[[ "$missing" -eq 0 ]] || return 1
|
||||
mkdir -p "$LLM_MEDIUM_CACHE"
|
||||
|
||||
retire_vllm_replica_quadlets
|
||||
retire_llama_medium_unit
|
||||
# 7770 is a host-network bind: nothing new can be created until the old
|
||||
# holder released it (the same gate as the 2026-09-16 cutover, where the
|
||||
# retired proxy held the port and the new engine failed at create). Sweep a
|
||||
# stray manual llama-server too — that is how a bench harness leaves the
|
||||
# port busy. `pkill -x` matches the process name exactly and can never
|
||||
# match the deploying shell; never `pkill -f` here.
|
||||
pkill -x llama-server 2>/dev/null || true
|
||||
pkill -x VLLM::EngineCor 2>/dev/null || true
|
||||
sleep 2
|
||||
wait_port_free 7770 || return 1
|
||||
|
||||
install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence is the unit's [Install] WantedBy=default.target.
|
||||
systemctl --user restart frank-medium.service
|
||||
# Triton/inductor compile on a cold cache runs minutes before the engine
|
||||
# loads ~9.5 GiB per card; HealthStartPeriod in the unit is 1800s to match.
|
||||
wait_container_healthy frank-medium 1800 || {
|
||||
podman logs --tail 40 frank-medium 2>&1 || true
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the Lucebox DeepSeek V4 Quadlet
|
||||
# (the 2026-09-26 cutover that replaces frank-medium on :7770). The model
|
||||
# is a 377 GiB GGUF and the image is a local build, so the gates are:
|
||||
@@ -1356,13 +1203,12 @@ deploy_llm_primary() {
|
||||
fi
|
||||
[[ "$missing" -eq 0 ]] || return 1
|
||||
|
||||
# Cutover: frank-primary takes :7770 from frank-medium. Stop and mask
|
||||
# the radiance unit (masking is mandatory: Restart=on-failure would
|
||||
# restart it and re-take the port). Retiring the radiance unit (moving
|
||||
# it to the home trash) happens after frank-primary is verified — this
|
||||
# handler only releases the port.
|
||||
systemctl --user stop frank-medium 2>/dev/null || true
|
||||
systemctl --user mask frank-medium
|
||||
# Cutover COMPLETED 2026-09-26: frank-primary took :7770 and the radiance
|
||||
# unit is retired. The host-side unit, its mask and the repo files were all
|
||||
# moved to trash by the parent session, so there is nothing left to stop or
|
||||
# mask here — masking a name that no longer exists would only resurrect it
|
||||
# in the unit search path. The port gate stays, because 7770 is a
|
||||
# host-network bind: nothing new can be created until the holder released it.
|
||||
# 7770 is a host-network bind: nothing new can be created until the old
|
||||
# holder released it (the same gate as the 2026-09-16 cutover).
|
||||
sleep 2
|
||||
@@ -1566,7 +1412,6 @@ deploy_service() {
|
||||
marreta) deploy_marreta ;;
|
||||
frank-large) deploy_llm_large ;;
|
||||
frank-small) deploy_llm_small ;;
|
||||
frank-medium) deploy_llm_medium ;;
|
||||
frank-primary) deploy_llm_primary ;;
|
||||
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
||||
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
||||
|
||||
+2
-2
@@ -35,7 +35,7 @@ Options:
|
||||
Examples:
|
||||
./scripts/deploy.sh dankclaw
|
||||
./scripts/deploy.sh dankclaw --service searxng
|
||||
./scripts/deploy.sh dankserver --service frank-medium
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
EOF
|
||||
}
|
||||
@@ -146,7 +146,7 @@ valid_service() {
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
|
||||
dankserver:frank-medium|dankserver:frank-primary|\
|
||||
dankserver:frank-primary|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetch the frank-medium payload: the target GGUF, the f16 vision projector and
|
||||
# the DFlash2 draft model.
|
||||
#
|
||||
# bash scripts/download-frank-medium-gguf.sh
|
||||
# bash scripts/download-frank-medium-gguf.sh --verify-only
|
||||
#
|
||||
# The target repo (orcarouter) is GATED: anonymous requests get a 401, so a
|
||||
# token is REQUIRED for it. The script reads ~/.cache/huggingface/token
|
||||
# (mode 600) or the HF_TOKEN environment variable, and passes it to curl
|
||||
# out-of-band via a mode-600 -K file (a -H header is world-readable in
|
||||
# /proc/<pid>/cmdline -- the 2026-09-16 HF-token leak). The drafter repo
|
||||
# (z-lab) is public; the same token is used for it when present.
|
||||
#
|
||||
# WHY THESE FILES:
|
||||
# * Target: orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 -- the
|
||||
# operator's requested quant (2026-09-19). Arch qwen35, 262144 trained
|
||||
# window, and the file carries the model's MTP layer (blk.64.nextn.*), so
|
||||
# llama.cpp can also draft from the model itself with no second copy in
|
||||
# VRAM.
|
||||
# * Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M -- the block-diffusion
|
||||
# drafter (5 layers, hidden 5120, block_size 8, a 2048-token sliding
|
||||
# window). Q4_K_M is 1.06 GiB against Q8_0's 1.92 GiB and the drafter's
|
||||
# job is throughput, not output quality (the target verifies every drafted
|
||||
# token).
|
||||
#
|
||||
# The script is resumable (curl -C -) and checks every artifact's exact byte
|
||||
# count AND sha256, so a truncated or mis-fetched file fails here instead of
|
||||
# load-crashing the server.
|
||||
set -euo pipefail
|
||||
|
||||
MODEL_REPO="orcarouter/Qwen3.8-27B-Uncensored-GGUF"
|
||||
MODEL_DIR="/home/dank/models/Qwen3.8-27B-Uncensored-GGUF"
|
||||
MODEL_FILE="Qwen3.8-27B-Uncensored-Q8_0.gguf"
|
||||
MODEL_SIZE=29047084416
|
||||
MODEL_SHA256=31756fca94beca71ea4b8706d6fdc896dab2a3c6376ab0c1863b98512a24f8d6
|
||||
# The f16 vision projector from the same repo; the serve script names it via
|
||||
# --mmproj (no autoload in this build), so it must land in MODEL_DIR.
|
||||
MM_PROJ_FILE="mmproj-Qwen3.8-27B-Uncensored-f16.gguf"
|
||||
MM_PROJ_SIZE=931145984
|
||||
MM_PROJ_SHA256=add205b7bfdb3f71f6da36b0a82aa20928dd829a920878c602628cdfbebc5288
|
||||
|
||||
DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF"
|
||||
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF"
|
||||
DRAFT_FILE="Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
DRAFT_SIZE=1143006816
|
||||
DRAFT_SHA256=1a25c56858e1ebe93f2718ac1d49d1151f9323325c1bbfd6209370f4db131ebd
|
||||
|
||||
mode="download"
|
||||
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
||||
|
||||
token="${HF_TOKEN:-}"
|
||||
if [[ -z "$token" && -r "$HOME/.cache/huggingface/token" ]]; then
|
||||
token="$(cat "$HOME/.cache/huggingface/token")"
|
||||
fi
|
||||
|
||||
auth_file=""
|
||||
cleanup() { [[ -n "$auth_file" ]] && rm -f "$auth_file"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
dl() { # dl <repo> <dir> <file>
|
||||
mkdir -p "$2"
|
||||
echo " fetching $3"
|
||||
local args=(-fL -C - -o "$2/$3" --retry 10 --retry-delay 5 --retry-max-time 21600)
|
||||
[[ -n "$auth_file" ]] && args+=(-K "$auth_file")
|
||||
curl "${args[@]}" "https://huggingface.co/$1/resolve/main/$3"
|
||||
}
|
||||
|
||||
if [[ "$mode" == "download" ]]; then
|
||||
if [[ -z "$token" ]]; then
|
||||
echo "ERROR: the target repo is gated and no HF token is available" >&2
|
||||
echo " (set HF_TOKEN or write it to ~/.cache/huggingface/token, mode 600)" >&2
|
||||
exit 1
|
||||
fi
|
||||
auth_file="$(mktemp)"
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$token" > "$auth_file"
|
||||
chmod 600 "$auth_file"
|
||||
|
||||
echo "target: $MODEL_REPO (gated)"
|
||||
dl "$MODEL_REPO" "$MODEL_DIR" "$MODEL_FILE"
|
||||
echo "projector: $MODEL_REPO (gated)"
|
||||
dl "$MODEL_REPO" "$MODEL_DIR" "$MM_PROJ_FILE"
|
||||
echo "draft: $DRAFT_REPO"
|
||||
dl "$DRAFT_REPO" "$DRAFT_DIR" "$DRAFT_FILE"
|
||||
fi
|
||||
|
||||
echo "verifying sizes and sha256"
|
||||
ok=1
|
||||
check() { # check <path> <expected-size> <expected-sha256>
|
||||
local act act_sha
|
||||
act="$(stat -c%s "$1" 2>/dev/null || echo 0)"
|
||||
if [[ "$act" -ne "$2" ]]; then
|
||||
echo " BAD-SIZE $1 ($act bytes, want $2)" >&2
|
||||
ok=0
|
||||
return
|
||||
fi
|
||||
act_sha="$(sha256sum "$1" | cut -d' ' -f1)"
|
||||
if [[ "$act_sha" != "$3" ]]; then
|
||||
echo " BAD-SHA $1 ($act_sha)" >&2
|
||||
ok=0
|
||||
return
|
||||
fi
|
||||
echo " OK $1 ($act bytes)"
|
||||
}
|
||||
check "$MODEL_DIR/$MODEL_FILE" "$MODEL_SIZE" "$MODEL_SHA256"
|
||||
check "$MODEL_DIR/$MM_PROJ_FILE" "$MM_PROJ_SIZE" "$MM_PROJ_SHA256"
|
||||
check "$DRAFT_DIR/$DRAFT_FILE" "$DRAFT_SIZE" "$DRAFT_SHA256"
|
||||
[[ "$ok" -eq 1 ]] || exit 1
|
||||
echo "all artifacts present, sized and hashed correctly"
|
||||
@@ -1,20 +0,0 @@
|
||||
# orcarouter/Qwen3.8-27B-Uncensored-FP8 — FP8 E4M3 block-wise (128x128), dynamic act-quant
|
||||
# 7 safetensors shards + config/tokenizer/processor. Sizes + LFS sha256 from the
|
||||
# HF tree API and X-Linked-ETag (2026-09-20).
|
||||
config.json 51350
|
||||
generation_config.json 202
|
||||
chat_template.jinja 8952
|
||||
merges.txt 3353259
|
||||
preprocessor_config.json 390
|
||||
tokenizer.json 12809320
|
||||
tokenizer_config.json 17928
|
||||
video_preprocessor_config.json 385
|
||||
vocab.json 6722759
|
||||
model.safetensors.index.json 156034
|
||||
model-00001-of-00007.safetensors 3969317448 141e200c1950c59bc42ae28744f50db8f6ea4c66ec3312b55f1ccad2b60af858
|
||||
model-00002-of-00007.safetensors 4939321704 e863e88f53ea5a737581638e41334e9f9871a93053c4251a2cebd4aa6aa1f7c0
|
||||
model-00003-of-00007.safetensors 4939863424 7cbf8db425114b9efcfae64364a60e15e223cc5cadb510b7a3a6d3b48c8617cf
|
||||
model-00004-of-00007.safetensors 4971328696 37338dc30b490663b641567a57565408fee5f1385e7860dff16dca7be07e1519
|
||||
model-00005-of-00007.safetensors 4955596064 fc6fb0dd3675e0cc7e83a454376d12b705a51000f6625cf13f85e384018fc5cb
|
||||
model-00006-of-00007.safetensors 4071440632 5fdcbaf80805a750970e076bcd37e6ebe4c3a3d93ce73c767d2854119947952a
|
||||
model-00007-of-00007.safetensors 3019999168 1bed352a0625d22dc606de47fd3563f24fd9fa73915c4ddffe7947c4b56c28bf
|
||||
@@ -0,0 +1,453 @@
|
||||
{%- set template_version = "qwen3.8-froggeric-v22.4.0" %}
|
||||
{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
|
||||
{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
|
||||
{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
|
||||
{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
|
||||
{%- set _preserve_thinking = preserve_reasoning %}
|
||||
{%- elif preserve_thinking is defined and preserve_thinking is not none %}
|
||||
{%- set _preserve_thinking = preserve_thinking %}
|
||||
{%- else %}
|
||||
{%- set _preserve_thinking = true %}
|
||||
{%- endif %}
|
||||
{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
|
||||
{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
|
||||
{%- set _default_reasoning_effort = 'medium' %}
|
||||
{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
|
||||
{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
|
||||
{%- set _initial_thinking = enable_thinking %}
|
||||
{%- if _effort_raw in ('none', 'off') %}
|
||||
{%- set _initial_thinking = false %}
|
||||
{%- set _initial_effort = 'medium' %}
|
||||
{%- elif _effort_raw in ('minimal', 'low') %}
|
||||
{%- set _initial_effort = 'low' %}
|
||||
{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
|
||||
{%- set _initial_effort = 'xhigh' %}
|
||||
{%- else %}
|
||||
{%- set _initial_effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
|
||||
{%- if auto_disable_thinking_with_tools and _has_tools %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- endif %}
|
||||
{%- for msg in messages %}
|
||||
{%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
|
||||
{%- if msg.content is string %}
|
||||
{%- if '<|think_off|>' in msg.content %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- elif '<|think_on|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'xhigh' %}
|
||||
{%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'low' %}
|
||||
{%- elif '<|think_medium|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- elif msg.content is iterable and msg.content is not mapping %}
|
||||
{%- for item in msg.content %}
|
||||
{%- if item is string %}
|
||||
{%- set _item_text = item %}
|
||||
{%- elif item is mapping and 'text' in item and item.text is string %}
|
||||
{%- set _item_text = item.text %}
|
||||
{%- else %}
|
||||
{%- set _item_text = '' %}
|
||||
{%- endif %}
|
||||
{%- if _item_text %}
|
||||
{%- if '<|think_off|>' in _item_text %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- elif '<|think_on|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'xhigh' %}
|
||||
{%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'low' %}
|
||||
{%- elif '<|think_medium|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if ns_state.thinking %}
|
||||
{%- if ns_state.effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif ns_state.effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if item is mapping %}
|
||||
{%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif item.type == 'video' or 'video' in item %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- item | string }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set head = namespace(count=0, seen_non_system=false) %}
|
||||
{%- for message in messages %}
|
||||
{%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- if _is_sys and not head.seen_non_system %}
|
||||
{%- set head.count = head.count + 1 %}
|
||||
{%- else %}
|
||||
{%- set head.seen_non_system = true %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set sys_state = namespace(content='') %}
|
||||
{%- for message in messages[:head.count] %}
|
||||
{%- set _part = render_content(message.content, false, true) | trim %}
|
||||
{%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
|
||||
{%- if _part %}
|
||||
{%- if sys_state.content %}
|
||||
{%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
|
||||
{%- else %}
|
||||
{%- set sys_state.content = _part %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set _sc = sys_state.content %}
|
||||
{%- set _msgs = messages[head.count:] %}
|
||||
{%- if ns_state.thinking %}
|
||||
{%- set _terse_lead = 'Answer directly, after thinking. Lead with the answer, then only what it needs to be correct and usable.' %}
|
||||
{%- else %}
|
||||
{%- set _terse_lead = 'Answer directly and concisely. Give the answer with only what it needs to be correct and usable.' %}
|
||||
{%- endif %}
|
||||
{%- set _terse_core %}
|
||||
Never: open with preamble or pleasantries; restate the question; add filler transitions; hedge with niceties; or repeat a point you've already made.
|
||||
Always: keep essential steps, caveats, uncertainties, and specifics — never drop correctness or a needed warning for brevity. Keep the final answer lean. Use the least structure that conveys it (plain prose when short; lists or code only when they earn their place). If genuinely uncertain, say so and explain why — never omit uncertainty for the sake of brevity.
|
||||
If a user request is genuinely ambiguous, ask a sharp question, don't guess.
|
||||
{%- endset %}
|
||||
{%- set _terse = _terse_lead ~ '\n' ~ (_terse_core | trim) %}
|
||||
{#- `terse` chat-template kwarg: default true, so behaviour is unchanged unless a caller opts out.
|
||||
Pass {"terse": false} via chat_template_kwargs to serve the model with only its own system
|
||||
prompt -- useful for A/B-ing the terseness effect, or when a downstream prompt conflicts. #}
|
||||
{%- set _terse_on = terse if terse is defined else true %}
|
||||
{%- if _terse_on %}
|
||||
{%- if not _sc %}
|
||||
{%- set _sc = _terse | trim %}
|
||||
{%- else %}
|
||||
{%- set _sc = (_sc | trim) ~ '\n\n' ~ (_terse | trim) %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if _has_tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
|
||||
{%- for tool in tools %}
|
||||
{{- '\n' }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- '\n</tools>' }}
|
||||
{%- if _tool_format == 'json' %}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}{%- if ns_state.thinking %}{{- '<think>\nBrief explanation of tool call\n</think>\n' }}{%- endif %}{{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}{%- if ns_state.thinking %}{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}{%- endif %}{{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY' }}{%- if ns_state.thinking %}{{- ' after thinking' }}{%- endif %}{{- ', with NO conversational text before it.\n- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
|
||||
{%- else %}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}{%- if ns_state.thinking %}{{- '<think>\nBrief explanation of tool call\n</think>\n' }}{%- endif %}{{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}{%- if ns_state.thinking %}{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}{%- endif %}{{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY' }}{%- if ns_state.thinking %}{{- ' after thinking' }}{%- endif %}{{- ', with NO conversational text before it.\n- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
|
||||
{%- endif %}
|
||||
{%- if _sc %}
|
||||
{{- '\n\n' + _sc }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if _sc %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set _last_idx = _msgs | length - 1 %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
|
||||
{%- for message in _msgs[::-1] %}
|
||||
{%- set index = (_msgs | length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == 'user' %}
|
||||
{%- set _rc = render_content(message.content, false) | trim %}
|
||||
{%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if ns.multi_step_tool %}
|
||||
{%- if _last_idx > 50 %}
|
||||
{%- set ns.last_query_index = _last_idx %}
|
||||
{%- else %}
|
||||
{%- set ns.last_query_index = 0 %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
|
||||
{%- for message in _msgs %}
|
||||
{%- set is_system = (message.role == "system" or message.role == "developer") %}
|
||||
{%- set content = render_content(message.content, true, is_system) | trim %}
|
||||
{%- if is_system or message.role == 'user' %}
|
||||
{%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if is_system %}
|
||||
{{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'user' %}
|
||||
{%- set ns2.consecutive_failures = 0 %}
|
||||
{{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'assistant' %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- set _explicit_reasoning = '' %}
|
||||
{%- if message.reasoning_content is defined and message.reasoning_content is not none %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set _explicit_reasoning = message.reasoning_content %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.reasoning_content | string %}
|
||||
{%- endif %}
|
||||
{%- elif message.thinking is defined and message.thinking is not none %}
|
||||
{%- if message.thinking is string %}
|
||||
{%- set _explicit_reasoning = message.thinking %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.thinking | string %}
|
||||
{%- endif %}
|
||||
{%- elif message.reasoning is defined and message.reasoning is not none %}
|
||||
{%- if message.reasoning is string %}
|
||||
{%- set _explicit_reasoning = message.reasoning %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.reasoning | string %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if _explicit_reasoning %}
|
||||
{%- set _lead_end = '' %}
|
||||
{%- if content.startswith('<think>') and '</think>' in content %}
|
||||
{%- set _lead_end = '</think>' %}
|
||||
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
|
||||
{%- set _lead_end = '</thinking>' %}
|
||||
{%- elif content.startswith('</think>') %}
|
||||
{%- set _lead_end = '</think>' %}
|
||||
{%- elif content.startswith('</thinking>') %}
|
||||
{%- set _lead_end = '</thinking>' %}
|
||||
{%- endif %}
|
||||
{%- if _lead_end %}
|
||||
{%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = _explicit_reasoning %}
|
||||
{%- else %}
|
||||
{%- set _think_end = '' %}
|
||||
{%- if content.startswith('</think>') %}
|
||||
{%- set _think_end = '</think>' %}
|
||||
{%- elif content.startswith('</thinking>') %}
|
||||
{%- set _think_end = '</thinking>' %}
|
||||
{%- elif '\n</think>' in content %}
|
||||
{%- set _think_end = '\n</think>' %}
|
||||
{%- elif '\n</thinking>' in content %}
|
||||
{%- set _think_end = '\n</thinking>' %}
|
||||
{%- elif '\n</ think>' in content %}
|
||||
{%- set _think_end = '\n</ think>' %}
|
||||
{%- elif '\n</think >' in content %}
|
||||
{%- set _think_end = '\n</think >' %}
|
||||
{%- elif content.startswith('<think>') and '</think>' in content %}
|
||||
{%- set _think_end = '</think>' %}
|
||||
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
|
||||
{%- set _think_end = '</thinking>' %}
|
||||
{%- endif %}
|
||||
{%- if _think_end %}
|
||||
{%- if 'thinking' in _think_end %}
|
||||
{%- set _think_start = '<thinking>' %}
|
||||
{%- else %}
|
||||
{%- set _think_start = '<think>' %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
|
||||
{%- if _think_start in reasoning_content %}
|
||||
{%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- set content = content.split(_think_end)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content | trim %}
|
||||
{%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
|
||||
{{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>assistant\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined and tool_call.function is not none %}
|
||||
{%- set tc = tool_call.function %}
|
||||
{%- else %}
|
||||
{%- set tc = tool_call %}
|
||||
{%- endif %}
|
||||
{%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
|
||||
{%- if _tool_format == 'json' %}
|
||||
{%- if loop.first %}
|
||||
{%- if content | trim %}
|
||||
{{- '\n\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- set _args = '{}' %}
|
||||
{%- if tc.arguments is defined and tc.arguments is not none %}
|
||||
{%- if tc.arguments is mapping %}
|
||||
{%- set _args = tc.arguments | tojson %}
|
||||
{%- elif tc.arguments is string %}
|
||||
{%- if tc.arguments %}
|
||||
{%- set _args = tc.arguments %}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{%- set _args = tc.arguments | tojson %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
|
||||
{%- else %}
|
||||
{%- if loop.first %}
|
||||
{%- if content | trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tc.arguments is defined and tc.arguments is not none %}
|
||||
{%- if tc.arguments is mapping %}
|
||||
{%- for args_name, args_value in tc.arguments.items() %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- if args_value is string %}
|
||||
{%- set _av = args_value %}
|
||||
{%- else %}
|
||||
{%- set _av = args_value | tojson %}
|
||||
{%- endif %}
|
||||
{%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
|
||||
{{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
|
||||
{%- else %}
|
||||
{{- _av }}
|
||||
{%- endif %}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- else %}
|
||||
{%- if tc.arguments is string %}
|
||||
{%- set _raw_args = tc.arguments %}
|
||||
{%- else %}
|
||||
{%- set _raw_args = tc.arguments | tojson %}
|
||||
{%- endif %}
|
||||
{%- if _raw_args %}
|
||||
{%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
|
||||
{{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
|
||||
{%- else %}
|
||||
{{- _raw_args }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'tool' %}
|
||||
{%- set _content_lower = content | lower %}
|
||||
{%- set _content_head = _content_lower[:120] %}
|
||||
{%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
|
||||
{%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
|
||||
{%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
|
||||
{%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
|
||||
{%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
|
||||
{%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
|
||||
{%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
|
||||
{%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
|
||||
{%- else %}
|
||||
{%- set ns2.consecutive_failures = 0 %}
|
||||
{%- endif %}
|
||||
{%- if ns2.prev_role != 'tool' %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{%- if _tool_format != 'json' and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
|
||||
{%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' + content }}
|
||||
{%- if ns2.consecutive_failures >= 2 %}
|
||||
{{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
|
||||
{%- elif ns2.consecutive_failures == 1 %}
|
||||
{{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
|
||||
{%- endif %}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- set _next_role = _msgs[loop.index0 + 1].role %}
|
||||
{%- if _next_role != 'tool' %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- set ns2.prev_role = message.role %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if not ns_state.thinking %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
@@ -49,7 +49,7 @@ fi
|
||||
: "${PARALLEL_SLOTS:=1}"
|
||||
: "${SPEC_DRAFT_N_MAX:=4}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/../dankserver/frank-medium/qwen-sharp-v22.4.0.jinja}"
|
||||
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}"
|
||||
|
||||
SERVER_BIN="$RUNTIME_DIR/llama-server"
|
||||
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# swap-frank-medium.sh — orchestrate a frank-medium model swap.
|
||||
#
|
||||
# A swap is four steps, and this script drives all of them:
|
||||
# 1. generate a manifest for the new HF repo (hf-manifest.py),
|
||||
# 2. download it to the host, resumable + byte/ sha256-verified (hf-fetch.sh),
|
||||
# 3. detect the checkpoint's weight quantization and point the quadlet at it
|
||||
# (SNAP + the matching SPEC_METHOD/SPEC + KV_MEM),
|
||||
# 4. hand off to deploy.sh (pull, restart, health gate).
|
||||
#
|
||||
# The entrypoint is QUANTIZATION-AWARE: it reads quant_method from the
|
||||
# checkpoint's config.json and serves the plain FP8 path (no MXFP4 patches,
|
||||
# --quantization=fp8, ROCM_AITER unified attention, MTP self-speculation) or
|
||||
# the MXFP4 patch-and-serve path (DFlash2 external drafter). So the ONLY repo
|
||||
# change a swap makes is the quadlet's four env vars — this script edits them.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/swap-frank-medium.sh --repo <owner/repo> --target <host-dir> \
|
||||
# [--generate-manifest] [--manifest <path>] [--spec N] [--kv MB] \
|
||||
# [--apply] [--deploy]
|
||||
#
|
||||
# --repo <owner/repo> the Hugging Face repo to serve
|
||||
# --target <host-dir> the host dir the model lands in
|
||||
# (/home/dank/models/<name>); the quadlet SNAP is
|
||||
# derived as /models/<name>
|
||||
# --generate-manifest write scripts/frank-medium-<name>.manifest from the
|
||||
# live repo tree (needs the host's HF token)
|
||||
# --manifest <path> use an existing manifest instead of generating one
|
||||
# --spec N speculative tokens (default 8 for fp8, 7 for mxfp4)
|
||||
# --kv MB KV pool pin in MiB (default 0 = auto-size)
|
||||
# --apply edit the quadlet (SNAP + SPEC_METHOD + SPEC + KV_MEM)
|
||||
# --deploy run deploy.sh after --apply
|
||||
#
|
||||
# Without --apply the script stops after the download + quantization
|
||||
# detection and prints the quadlet lines to set. The download and the
|
||||
# quantization detection run on the host (the model dir and the token are
|
||||
# there); the quadlet edit runs in the repo.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
HOST="dank@192.168.0.235"
|
||||
QUADLET="dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
MODELS_ROOT="/home/dank/models"
|
||||
|
||||
repo="" target="" manifest="" spec="" kv=0 gen=0 apply=0 deploy=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--repo) repo="$2"; shift 2 ;;
|
||||
--target) target="$2"; shift 2 ;;
|
||||
--manifest) manifest="$2"; shift 2 ;;
|
||||
--spec) spec="$2"; shift 2 ;;
|
||||
--kv) kv="$2"; shift 2 ;;
|
||||
--generate-manifest) gen=1; shift ;;
|
||||
--apply) apply=1; shift ;;
|
||||
--deploy) deploy=1; shift ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[[ -n "$repo" && -n "$target" ]] || {
|
||||
echo "usage: $0 --repo <owner/repo> --target <host-dir> [--generate-manifest|--manifest <path>] [--spec N] [--kv MB] [--apply] [--deploy]" >&2
|
||||
exit 2;
|
||||
}
|
||||
|
||||
name="$(basename "$target")"
|
||||
snap_c="/models/$name"
|
||||
|
||||
# --- 1. manifest -------------------------------------------------------------
|
||||
if [[ "$gen" = 1 ]]; then
|
||||
manifest="scripts/frank-medium-${name}.manifest"
|
||||
echo "==> Generating manifest -> $manifest"
|
||||
scp -q scripts/hf-manifest.py "$HOST:/tmp/hf-manifest.py"
|
||||
ssh "$HOST" "python3 /tmp/hf-manifest.py '$repo' /tmp/swap-manifest"
|
||||
scp -q "$HOST:/tmp/swap-manifest" "$manifest"
|
||||
elif [[ -z "$manifest" ]]; then
|
||||
echo "ERROR: pass --generate-manifest or --manifest <path>" >&2
|
||||
exit 2
|
||||
fi
|
||||
[[ -f "$manifest" ]] || { echo "ERROR: manifest not found: $manifest" >&2; exit 2; }
|
||||
|
||||
# --- 2. download (resumable, verified) ---------------------------------------
|
||||
echo "==> Staging manifest + hf-fetch.sh on $HOST"
|
||||
scp -q "$manifest" "$HOST:/tmp/swap-manifest"
|
||||
scp -q scripts/hf-fetch.sh "$HOST:/tmp/swap-hf-fetch.sh"
|
||||
echo "==> Downloading $repo -> $target"
|
||||
ssh "$HOST" "bash /tmp/swap-hf-fetch.sh '$target' /tmp/swap-manifest '$repo'"
|
||||
|
||||
# --- 3. detect the quantization on the host ----------------------------------
|
||||
quant="$(ssh "$HOST" "python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get(\"quantization_config\",{}).get(\"quant_method\",\"\"))' '$target/config.json' 2>/dev/null || true")"
|
||||
echo "==> Detected quantization: ${quant:-<none>}"
|
||||
|
||||
case "$quant" in
|
||||
fp8) spec_method=dflash; default_spec=7 ;;
|
||||
mxfp4|quark|"") spec_method=dflash; default_spec=7 ;;
|
||||
*) spec_method=dflash; default_spec=7
|
||||
echo "WARNING: unknown quant '$quant'; defaulting to the DFlash2 path" >&2 ;;
|
||||
esac
|
||||
[[ -n "$spec" ]] || spec="$default_spec"
|
||||
|
||||
echo "==> Quadlet env vars for this checkpoint:"
|
||||
echo " Environment=SNAP=$snap_c"
|
||||
echo " Environment=SPEC_METHOD=$spec_method"
|
||||
echo " Environment=SPEC=$spec"
|
||||
echo " Environment=KV_MEM=$kv"
|
||||
|
||||
# --- 4. apply the quadlet edit (in the repo) ---------------------------------
|
||||
if [[ "$apply" = 1 ]]; then
|
||||
echo "==> Editing $QUADLET"
|
||||
sed -i -E "s|^Environment=SNAP=.*|Environment=SNAP=$snap_c|" "$QUADLET"
|
||||
sed -i -E "s|^Environment=SPEC_METHOD=.*|Environment=SPEC_METHOD=$spec_method|" "$QUADLET"
|
||||
sed -i -E "s|^Environment=SPEC=.*|Environment=SPEC=$spec|" "$QUADLET"
|
||||
sed -i -E "s|^Environment=KV_MEM=.*|Environment=KV_MEM=$kv|" "$QUADLET"
|
||||
echo "==> Quadlet updated."
|
||||
fi
|
||||
|
||||
# --- 5. deploy ----------------------------------------------------------------
|
||||
if [[ "$deploy" = 1 ]]; then
|
||||
[[ "$apply" = 1 ]] || { echo "ERROR: --deploy requires --apply" >&2; exit 2; }
|
||||
echo "==> Deploying (pull, restart, health gate)"
|
||||
./scripts/deploy.sh dankserver --service frank-medium --skip-e2e
|
||||
echo "==> Next: run the capability battery"
|
||||
echo " python3 scripts/bench-conversation.py http://192.168.0.235:7770 --model frank-medium --salt ${name} --max-tokens 512 --burst 4"
|
||||
fi
|
||||
+21
-8
@@ -180,17 +180,30 @@ for path in "${CHANGED[@]}"; do
|
||||
scripts/mount-entropy.sh)
|
||||
add_test tests/mount_entropy.bats
|
||||
;;
|
||||
# frank-medium is a rootless quadlet again since the 2026-09-20
|
||||
# tensor-parallel cutover: the TP=2 radiance unit, its entrypoint and
|
||||
# its healthcheck all live under dankserver/frank-medium/, and the
|
||||
# container mounts the latter two straight out of the checkout. The
|
||||
# llama.cpp trio that held this endpoint before that date is archive
|
||||
# material in scripts/.AGENTTRASH/ and maps to nothing.
|
||||
dankserver/frank-medium/*)
|
||||
add_test tests/frank_medium_unit.bats
|
||||
# Archive material: a path under any .AGENTTRASH/ directory is
|
||||
# retired-by-policy (deploy-plan.py plans nothing from it), so it
|
||||
# selects no focused test. It still needs its own case: without one
|
||||
# the wrapper refuses retirement commits, whose trash moves show up
|
||||
# as additions, with "no focused-test mapping".
|
||||
.AGENTTRASH/*|*/.AGENTTRASH/*)
|
||||
;;
|
||||
# frank-primary is the lucebox DS4.1 Quadlet (2026-09-26 cutover on
|
||||
# :7770): its entrypoint and healthcheck live under
|
||||
# dankserver/frank-primary/ and the container mounts them straight out
|
||||
# of the checkout, so any file here changes deployed state. The
|
||||
# radiance quadlet that held this endpoint before that date is archive
|
||||
# material in dankserver/.AGENTTRASH/ and maps to nothing.
|
||||
dankserver/frank-primary/*)
|
||||
add_test tests/frank_primary_unit.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# frank-large's chat template lives in scripts/ (moved there on
|
||||
# 2026-09-26 when its old home was retired): a template edit changes
|
||||
# what the live endpoint renders, so it selects the large unit seam.
|
||||
scripts/qwen-sharp-v22.4.0.jinja)
|
||||
add_test tests/frank_large_unit.bats
|
||||
;;
|
||||
# frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
# script all live in scripts/, which is why this case has to precede
|
||||
|
||||
Reference in New Issue
Block a user