frank-small: Q5_K_M -> Q6_K rung at the held 2x200704 f16-KV shape

The freed third slot funded exact V; the remaining ~3.5 GiB of honest
nvidia-smi free (3,571, not the 4,046 total-minus-used I quoted - the
driver holds ~450 MiB of reserve) funds the +875 MiB of Q6_K weights.
Projected landing ~2.7 GiB free, inside the 2-3 GiB floor; measure on
boot, step the pool one 12,288-token increment if it lands under 2 GiB.

- payload dir MiMo-Q6K/ made self-contained (mmproj hardlinked from
  MiMo-Q5KM/, link-count-2 verified)
- env/serve/deployer/bats repointed; download script now fetches and
  verifies Q6_K (7,793,710,624 B, ef96d05a...) with Q5_K_M documented as
  the rollback; payload sha verified live on the host before deploy
- seam green 14/14 against the stub engine
This commit is contained in:
Frank committed 2026-10-03 11:00:19 -07:00
1 parent b80886e318
commit 6ec0b1a85d
5 files changed
+35 -25

No files matched your search

+4 -4
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q6_K) as frank-small on
# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
#
# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
@@ -12,7 +12,7 @@
# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
# separate projector (the Q5_K_M GGUF carries no `v.*` tensors at all), so
# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
# text-only opt-out.
#
@@ -33,7 +33,7 @@ fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
: "${MODEL_FILE:=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf}"
: "${ALIAS:=frank-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
@@ -54,7 +54,7 @@ fi
: "${NGL:=99}"
# Vision projector. No colon on purpose: an explicitly empty value means
# "text-only, on purpose" and must NOT be replaced by the default path.
: "${MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
: "${MMPROJ_FILE=/home/dank/models/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
# Fail loudly rather than half-loading: a missing file is a download that
# has not finished, and Restart=always would turn it into a crash loop that