frank-large: hand vLLM the CONTAINER model path; link the new runbook
The first crash-looping boot (51 restarts): serve received the host path /home/dank/models/... but the models tree is mounted at /models inside the container, so transformers failed 'Repo id must be in the form namespace/repo_name' before the engine even built. Same conversion the spec draft already used. Pinned by bats. Root docs index links the new runbook (seam required it) and the work log records the crash-loop plus the 52 G payload reclaim.
This commit is contained in:
@@ -213,14 +213,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1,
|
||||
the Strix Halo inference engine, built from source on Dankclaw and
|
||||
serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the
|
||||
z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token
|
||||
conversation, adaptive block policy, greedy, text-only, the two
|
||||
from-source build workarounds, what changed in the client contract, the
|
||||
measured prefill/decode trade against llama.cpp
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — the ACTIVE frank-large runbook
|
||||
since 2026-10-01 (root systemd unit, port 7770): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
|
||||
the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
|
||||
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback via tag
|
||||
frank-large-gufo-q4kxl-20261001): gufo v0.1.1 built from source serving
|
||||
Qwen3.8-27B GGUF + the z-lab DFlash2 drafter on the gfx1151 iGPU — the
|
||||
two from-source build workarounds, the client-contract delta, the
|
||||
measured prefill/decode trade
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
|
||||
@@ -912,3 +912,16 @@ any repo service.
|
||||
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
|
||||
rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
|
||||
Q4_K_XL payload and DFlash2 drafter all remain on the host.
|
||||
- First deploy attempt #2 crash-looped the container 51 times: the serve
|
||||
script handed vLLM the HOST model path while the tree is mounted at
|
||||
`/models` inside the container — transformers rejected it as a malformed
|
||||
repo id. Fixed by converting to the container path (the spec draft was
|
||||
already converted; the target was not), pinned by bats. Also fixed the
|
||||
deployer's own draft gate (same single-file bug as the serve script's).
|
||||
- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
|
||||
`scripts/download-frank-large-swift.sh`) and the orphaned flux-2-klein
|
||||
tree under `models/checkpoints` (23 G; ComfyUI mounts
|
||||
`comfyui-data/models`, not this) purged via
|
||||
`~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
|
||||
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
|
||||
drafter, not a remnant).
|
||||
|
||||
Reference in New Issue
Block a user