Frank
acd14ab7df
Replaces frank-medium (vllm-radiance TP=2) on :7770. The 377.2 GiB 43-layer MoE (256 routed experts, top-6) is mmapped: dense work and the MLA KV cache on the R9700, routed experts computed on CPU from the page cache (246 GiB RAM). The unit bypasses the image entrypoint (Qwen/DFlash-only flag set) and execs luce_server with the DS4 launch shape; --ds4-fused-decode's monolithic-load fallback IS the serving mode here. deploy_llm_primary gates on image + model byte count, stops/masks the radiance unit, waits for :7770, installs and starts. Runbook: frank-primary-lucebox-ds4.md.
56 lines
2.4 KiB
Bash
Executable File
56 lines
2.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# frank-primary entrypoint: exec the native luce_server with the DeepSeek V4
|
|
# launch shape for this host (2x R9700 gfx1201, 246 GiB RAM).
|
|
#
|
|
# The image's own entrypoint (server/scripts/entrypoint.sh) only forwards a
|
|
# fixed Qwen/DFlash flag set and cannot express the DS4 flags, so this unit
|
|
# bypasses it (PodmanArgs=--entrypoint) and execs the binary directly. The
|
|
# C++ server reads LUCE_* environment variables straight from the process
|
|
# environment, so no env plumbing is needed here.
|
|
#
|
|
# LAUNCH SHAPE (server/docs/DS4.md, server/docs/RECOMMENDED_SETUPS.md):
|
|
# --ds4-fused-decode requests a monolithic GPU load; on a 377 GiB model
|
|
# that allocation fails and the backend falls back
|
|
# to hybrid expert placement + layered decode (dense
|
|
# work on the R9700, routed experts computed on CPU
|
|
# from the mmapped page cache). That fallback IS the
|
|
# intended serving mode on this host.
|
|
# --ds4-prefill sparse the batched prefill; use for prompts beyond a few
|
|
# hundred tokens (exact is token-wise decode cost).
|
|
# --ds4-expert-top-k 6 the model default. Do NOT reduce on adaptive
|
|
# (qtype-105/106) artifacts: measured degenerate
|
|
# output at top-4 (DS4.md, 2026-08-05).
|
|
# --chunk 8192 the qualified Strix Halo profile's chunk.
|
|
# --max-ctx 131072 the qualified Strix Halo window; DS4's MLA KV is
|
|
# compressed, so the pool is small.
|
|
# --cache-type-k/v q4_0 the qualified launch's KV dtype.
|
|
#
|
|
# No DSpark draft is published for DeepSeek-V4.1-Flash, so this is plain
|
|
# autoregressive decode (no --draft, no LUCE_DS4_SPEC).
|
|
set -euo pipefail
|
|
|
|
MODEL="${FRANK_PRIMARY_MODEL:-/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf}"
|
|
BIN="${FRANK_PRIMARY_BIN:-/opt/lucebox-hub/server/build/luce_server}"
|
|
|
|
if [[ ! -f "$MODEL" ]]; then
|
|
echo "frank-primary: model missing at $MODEL" >&2
|
|
exit 1
|
|
fi
|
|
if [[ ! -x "$BIN" ]]; then
|
|
echo "frank-primary: binary missing at $BIN (image build failed?)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
exec "$BIN" "$MODEL" \
|
|
--host 0.0.0.0 \
|
|
--port 7770 \
|
|
--model-name frank-primary \
|
|
--target-device hip:0 \
|
|
--max-ctx 131072 \
|
|
--chunk 8192 \
|
|
--cache-type-k q4_0 \
|
|
--cache-type-v q4_0 \
|
|
--ds4-fused-decode \
|
|
--ds4-prefill sparse \
|
|
--ds4-expert-top-k 6
|