inference: frank-large returns to halogen, frank-small takes the 3090
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server quadlet again -- the 2026-09-13 unit file was never touched by the two-day ik_llama.cpp interlude, so the runtime was still in the repo and its image is still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192 with a 400,384-position pool and two slots, exactly two 200k conversations. frank-small is new: mainline llama.cpp on the RTX 3090 serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main model's own weights instead of a second 12 GiB copy) plus the BF16 vision projector, as one 131,072-token slot with f16 KV on port 7771. The host is shared now, and that is what the halogen retune pays for: HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to 20 to keep the fit check honest. Vision moved wholesale to frank-small. Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the Qwen-Sharp template only that runtime read, and four benchmark harnesses bound to retired runtimes. Because retiring a file makes any test that greps it pass by absence, the seams assert the archive exists AND the live path does not.
This commit is contained in:
Frank committed
1 parent
cc29875842
commit
43ec77fee6
19 files changed
+648
-198
No files matched your search
@@ -1,123 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-large
|
||||
# via ik_llama.cpp on Dankclaw.
|
||||
#
|
||||
# Topology (ikawrakow's hybrid recipe; upstream issue #2402 for the PLE):
|
||||
# the RTX 3090 (CUDA, sm_86) takes the dense/attention/output weights and
|
||||
# the KV cache (-ngl 999); ALL MoE expert tensors and the ~32 GB PLE
|
||||
# n-gram table stay in the Strix Halo's LPDDR5X via the plain CPU backend
|
||||
# buffer (-ot ...=CPU — mmap-backed page cache; only a tiny slice of PLE is
|
||||
# touched per token). NEVER use --cpu-moe on this host: it routes the
|
||||
# CPU-overridden weights through the CUDA_Host PINNED buffer (measured
|
||||
# 108.12 GiB page-locked on 2026-09-15, which evicted every co-tenant into
|
||||
# swap and starved sshd into unreachability until a power cycle). NEVER add
|
||||
# --no-mmap — the STRIX-era lesson transfers directly (the cgroup killer
|
||||
# takes the process with nothing in the server log).
|
||||
#
|
||||
# Runtime: ik_llama.cpp master d85a1d7 built with CUDA (sm_86) + Zen5
|
||||
# AVX-512 IQK kernels; the startup banner must show HAVE_FANCY_SIMD
|
||||
# is defined. Flag set verified against this binary's --help before this
|
||||
# script was written. ik has no --cache-prompt/--cache-idle-slots (prompt
|
||||
# caching is its default behavior) and no --alias: it is a single-model
|
||||
# server and reports the GGUF's general.name at /v1/models.
|
||||
#
|
||||
# Sampling: greedy-by-default (halogen client contract) — clients that
|
||||
# send their own temperature get their own values.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/ik_llama.cpp/build/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M}"
|
||||
: "${MODEL_FILE:=Qwen3.8-Flash-Next-Uncensored.i1-Q4_K_M.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=262144}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${NUM_THREADS:=16}"
|
||||
: "${THREAD_BATCH:=16}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${NGL:=999}"
|
||||
: "${CPU_MOE:=on}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING:=on}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
: "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}"
|
||||
# Hard guard: on this host CUDA pinned host memory strands the co-tenants
|
||||
# (measured 2026-09-15: a default-policy start tried to page-lock 108.12 GiB,
|
||||
# evicted every co-tenant into swap and starved sshd into unreachability
|
||||
# until a power cycle). GGML_CUDA_NO_PINNED_WEIGHTS=1 keeps the
|
||||
# CPU-overridden weights mmapped (src/llama-load-tensors.cpp); the env file
|
||||
# sets it. Refuse to run without it.
|
||||
if [[ "${GGML_CUDA_NO_PINNED_WEIGHTS:-}" != "1" ]]; then
|
||||
echo "frank-large: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$SLOT_SAVE_PATH"
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first
|
||||
# CUDA open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
CPU_MOE_ARGS=()
|
||||
if [[ "$CPU_MOE" == "on" ]]; then
|
||||
# -ot ...=CPU (plain CPU backend buffer) is the mmap-backed path. NEVER
|
||||
# swap this for --cpu-moe: that routes the weights through the CUDA_Host
|
||||
# PINNED buffer (108.12 GiB measured) and strands the co-tenants.
|
||||
CPU_MOE_ARGS=(-ot "exps=CPU" -ot "per_layer_token_embd=CPU")
|
||||
fi
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
|
||||
TEMPLATE_ARGS=()
|
||||
if [[ -n "${CHAT_TEMPLATE_FILE:-}" && -f "$CHAT_TEMPLATE_FILE" ]]; then
|
||||
TEMPLATE_ARGS=(--chat-template-file "$CHAT_TEMPLATE_FILE")
|
||||
else
|
||||
TEMPLATE_ARGS=(--jinja)
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${TEMPLATE_ARGS[@]}" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
"${CPU_MOE_ARGS[@]}" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
--parallel "$PARALLEL" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--cache-ram "$CACHE_RAM" \
|
||||
--slot-save-path "$SLOT_SAVE_PATH" \
|
||||
--ctx-checkpoints 16 \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning "$REASONING" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET" \
|
||||
"${MMPROJ_ARGS[@]}"
|
||||
Reference in new issue
Block a user