dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

harness: llama-server lifecycle in pre-clean and health guard

Browse Source
This commit is contained in:
Frank
2026-10-01 21:58:51 -07:00
parent cde91c1a00
commit 01ba41cab4
2 changed files with 13 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-large.env
+10 -4
View File
@@ -88,11 +88,17 @@ FRANK_LARGE_UTIL=0.45
FRANK_LARGE_SPEC=
FRANK_LARGE_SPEC_DIR=/home/dank/models/Qwen3.5-9B
FRANK_LARGE_SPEC_TOKENS=2
# AUTORESEARCH: torchao int8 weight-only (quark int8 + all fp8 kernels
# rejected on gfx1151; torchao is a separate Triton dequant stack)
FRANK_LARGE_QUANT=torchao
FRANK_LARGE_QUANT_CONFIG='{"torchao_config":"int8_weight_only"}'
# --- AUTORESEARCH arm: llama.cpp runtime -------------------------------------
# master build (qwen35 GDN kernels, HIP gfx1151) on Q6_K GGUF: 7.79 GiB
# weights = 0.51x the BF16 bytes, so the LPDDR roof doubles. Cross-model
# DFlash draft (qwen3.5-9b, Q8_0). --reasoning-format deepseek emits
# reasoning_content, NOT vLLM's reasoning.
FRANK_LARGE_RUNTIME=llama
FRANK_LARGE_LLAMA_BIN=/home/dank/GitPackages/llama-master/build-hip/bin/llama-server
FRANK_LARGE_GGUF=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf
FRANK_LARGE_DRAFT_GGUF=/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf
FRANK_LARGE_LLAMA_CTX=1048576
# --- client contract ----------------------------------------------------------
# vLLM emits the thinking text in `reasoning` (NOT llama.cpp/gufo's
# `reasoning_content`) — the harness models.yml compat block for this