Frank
d7bf9f119a
The frank-medium retirement tail the 2026-09-26 lucebox cutover left uncommitted: the radiance quadlet tree, its swap/fetch tooling and its seam test are deleted (archive copies live in dankserver/.AGENTTRASH/ 20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926- frank-medium-radiance/), the planner/allowlist/test-changed mappings drop the service, and frank-large's chat template moves to scripts/ (its old home was the retired frank-medium dir). Finished here: - tests/frank_primary_unit.bats: the lucebox quadlet seam (boot persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved env-name guard, checkout-mounted entrypoint/healthcheck, first-start patience, deployer payload/port gates). - test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move files to trash, and the wrapper refused those additions with "no focused-test mapping". - observability: the vllm: namespace left with the radiance runtime. Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode, queueing, endpoint-down); the dashboard drops the vLLM KV/latency panels; prometheus.yml documents that frank-primary (lucebox) has no /metrics and therefore no scrape job. llm-fleet.json was left corrupted mid-edit (missing opening brace) and is rewritten from HEAD. Suite: 339 green (was 7 red).
107 lines
3.8 KiB
Bash
Executable File
107 lines
3.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on
|
|
# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp
|
|
# (mainline, the unsloth b11030 prebuilt).
|
|
#
|
|
# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server
|
|
# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5
|
|
# architecture and the DFlash2 drafter (llama_model_qwen35,
|
|
# llama_model_dflash, common_speculative_impl_draft_dflash).
|
|
#
|
|
# Shape: one stream, full 262144 context (the model's native window), all
|
|
# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the
|
|
# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the
|
|
# iGPU's GTT aperture (~114 GiB).
|
|
#
|
|
# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter
|
|
# has a sliding window of 2048 (small draft KV, ~40 MiB).
|
|
#
|
|
# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030
|
|
# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/
|
|
# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint.
|
|
#
|
|
# Text-only (no mmproj) — matching the prior frank-large.
|
|
#
|
|
# The Qwen3.8 family template keeps prior thinking blocks across turns when
|
|
# preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents
|
|
# do not lose the trail. The template gates reasoning_effort to {xhigh,
|
|
# medium, low} — the harness folds its levels onto that set in models.yml
|
|
# (reasoningEffortMap).
|
|
#
|
|
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
|
|
# it; anything not set there falls through to the defaults below.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
|
if [[ -f "$ENV_FILE" ]]; then
|
|
set -a
|
|
source "$ENV_FILE"
|
|
set +a
|
|
fi
|
|
|
|
# --- defaults (used when the env file does not set them) ------------------
|
|
: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
|
|
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
|
: "${HOST:=0.0.0.0}"
|
|
: "${PORT:=7770}"
|
|
: "${CONTEXT_SIZE:=262144}"
|
|
: "${PARALLEL_SLOTS:=1}"
|
|
: "${SPEC_DRAFT_N_MAX:=4}"
|
|
: "${CACHE_RAM:=8192}"
|
|
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}"
|
|
|
|
SERVER_BIN="$RUNTIME_DIR/llama-server"
|
|
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
|
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
|
|
|
if [[ ! -x "$SERVER_BIN" ]]; then
|
|
echo "runtime not found at $SERVER_BIN" >&2
|
|
exit 1
|
|
fi
|
|
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
|
if [[ ! -f "$f" ]]; then
|
|
echo "missing asset $f (run scripts/download-frank-large.sh)" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
# --- env (the unsloth build's ROCm setup) --------------------------------
|
|
export HIP_VISIBLE_DEVICES=0
|
|
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
|
export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}"
|
|
|
|
# --- the serve command ---------------------------------------------------
|
|
ARGS=(
|
|
--host "$HOST"
|
|
--port "$PORT"
|
|
--model "$MODEL_FILE"
|
|
--alias frank-large
|
|
--ctx-size "$CONTEXT_SIZE"
|
|
--parallel "$PARALLEL_SLOTS"
|
|
--n-gpu-layers 999
|
|
--jinja
|
|
--chat-template-file "$TEMPLATE_FILE"
|
|
--chat-template-kwargs '{"preserve_thinking":true}'
|
|
--cache-type-k q8_0
|
|
--cache-type-v q8_0
|
|
--cache-ram "$CACHE_RAM"
|
|
# DFlash2 spec decode (a block-diffusion drafter)
|
|
--spec-type draft-dflash
|
|
--model-draft "$DRAFTER_FILE"
|
|
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
|
|
--spec-draft-n-min 0
|
|
--spec-draft-device ROCm0
|
|
# Greedy sampling (the DFlash2 drafter needs greedy verification)
|
|
--temp 0
|
|
--top-p 1.0
|
|
--top-k 0
|
|
--min-p 0.0
|
|
# Reasoning (thoughts land in reasoning_content)
|
|
--reasoning auto
|
|
--reasoning-format deepseek
|
|
--reasoning-budget -1
|
|
)
|
|
|
|
exec "$SERVER_BIN" "${ARGS[@]}"
|