Frank
bf14ac7a7b
The launcher does not change directory, so process cwd becomes sys.path[0] for the 'python -m ...registry' model-inspection subprocess. From the CIRU package root, the top-level vllm/ repo dir shadows the editable install as a namespace package, breaking lazy imports (SamplingParams) and failing model architecture inspection. cd to /home/dank/ling3-ciru before exec.
35 lines
1.3 KiB
Bash
Executable File
35 lines
1.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# serve-ling3-vllm.sh — Launch Ling-3.0-flash CIRU INT4 vLLM (gfx1151) on Dankclaw
|
|
#
|
|
# This wrapper sources the jcbtc ling3-runtime.env (sets VENV and MODEL_PATH),
|
|
# applies our production settings (host, port, served-model-name), and hands
|
|
# off to the validated run-256k.sh launcher (TRITON_MLA, MTP K1, 256k, 6 slots,
|
|
# ROCr idle-fix LD_PRELOAD). Do not hand-roll the vllm serve command; the
|
|
# launcher carries required env vars and the ROCr override.
|
|
set -Eeuo pipefail
|
|
|
|
PKG="/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native"
|
|
|
|
if [ ! -f "$PKG/ling3-runtime.env" ]; then
|
|
echo "CIRU package runtime environment not found: $PKG/ling3-runtime.env" >&2
|
|
exit 1
|
|
fi
|
|
|
|
set -a
|
|
# shellcheck disable=SC1091
|
|
source "$PKG/ling3-runtime.env"
|
|
set +a
|
|
|
|
export HOST="${HOST:-0.0.0.0}"
|
|
export PORT="${PORT:-7770}"
|
|
export SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-frank-default-large}"
|
|
|
|
# Run from a directory with no top-level 'vllm/' dir.
|
|
# Neither launcher changes directory, so the process cwd becomes sys.path[0]
|
|
# for the 'python -m ...registry' inspection subprocess. From the package root,
|
|
# the top-level vllm/ repo dir shadows the editable install as a namespace
|
|
# package and breaks lazy imports (for example, SamplingParams).
|
|
cd /home/dank/ling3-ciru
|
|
|
|
exec bash "$PKG/scripts/run-256k.sh"
|