frank-medium: orcarouter Q8_0 + DFlash2 Q4_K_M, 2x160k, balanced 32:33
Operator call 2026-09-19: switch the target to orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 (gated repo; the fetch script now requires an HF token and verifies sha256, not just size), take the drafter to the Q4_K_M rung of z-lab/Qwen3.8-27B-DFlash2-GGUF, cut the pool to two 163840-token conversations (327680, entirely inside the trained 262144 window), and balance the tensors across the cards (32:33) in place of the 35:30 x16-lean. The in-file MTP head stays one SPEC_TYPE change away.
This commit is contained in:
1 parent
374041cd7a
commit
d374ab963f
4 files changed
+210
-236
No files matched your search
@@ -1175,9 +1175,9 @@ deploy_llm_small() {
|
||||
# PCIe, once per forward pass. That is the "pipeline parallelism" the endpoint
|
||||
# wanted, with no collective anywhere in it.
|
||||
LLM_MEDIUM_BIN="${LLM_MEDIUM_BIN:-/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}"
|
||||
LLM_MEDIUM_MODEL="${LLM_MEDIUM_MODEL:-/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
|
||||
LLM_MEDIUM_DRAFT="${LLM_MEDIUM_DRAFT:-/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf}"
|
||||
LLM_MEDIUM_MMPROJ="${LLM_MEDIUM_MMPROJ:-/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF/mmproj-BF16.gguf}"
|
||||
LLM_MEDIUM_MODEL="${LLM_MEDIUM_MODEL:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/Qwen3.8-27B-Uncensored-Q8_0.gguf}"
|
||||
LLM_MEDIUM_DRAFT="${LLM_MEDIUM_DRAFT:-/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
LLM_MEDIUM_MMPROJ="${LLM_MEDIUM_MMPROJ:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/mmproj-Qwen3.8-27B-Uncensored-f16.gguf}"
|
||||
|
||||
# The three vLLM user units are retired HERE, not left to rot: the proxy
|
||||
# published 7770 on the host network, so the llama-server below cannot bind
|
||||
|
||||
Reference in new issue
Block a user