frank-small: Q5_K_M -> Q6_K rung at the held 2x200704 f16-KV shape

The freed third slot funded exact V; the remaining ~3.5 GiB of honest
nvidia-smi free (3,571, not the 4,046 total-minus-used I quoted - the
driver holds ~450 MiB of reserve) funds the +875 MiB of Q6_K weights.
Projected landing ~2.7 GiB free, inside the 2-3 GiB floor; measure on
boot, step the pool one 12,288-token increment if it lands under 2 GiB.

- payload dir MiMo-Q6K/ made self-contained (mmproj hardlinked from
  MiMo-Q5KM/, link-count-2 verified)
- env/serve/deployer/bats repointed; download script now fetches and
  verifies Q6_K (7,793,710,624 B, ef96d05a...) with Q5_K_M documented as
  the rollback; payload sha verified live on the host before deploy
- seam green 14/14 against the stub engine
This commit is contained in:
Frank committed 2026-10-03 11:00:19 -07:00
1 parent b80886e318
commit 6ec0b1a85d
5 files changed
+35 -25

No files matched your search

+15 -6
View File
@@ -1,5 +1,12 @@
# frank-small — llama.cpp (master, CUDA sm_86) serving
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q6_K on the RTX 3090, :7771.
#
# 2026-10-03 (Frank, second move): the rung went Q5_K_M -> Q6_K with the pool
# HELD at 401,408. The freed third slot funded exact f16 V first; the ~3.5 GiB
# still free funds the +875 MiB of weights (file delta 7,793,710,624 -
# 6,876,124,704). Projected landing ~2.7 GiB free — inside the 2-3 GiB floor,
# so MEASURE after the boot; if it lands under 2 GiB, step the pool down one
# 12,288-token (384 MiB) increment.
#
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
# tokens per conversation, above the 200,700 the harness compacts at — and the
@@ -38,12 +45,14 @@
# and the third slot is gone. The flags stay explicit so the choice is visible
# in the argv either way.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf
# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
# Vision: the Q6_K GGUF ships no `v.*` tensors either, so the projector file
# IS the vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
# scripts/download-frank-small.sh; hardlinked into this dir from MiMo-Q5KM/,
# same bytes — the Q6_K dir is self-contained). Empty string = text-only on
# purpose.
MMPROJ_FILE=/home/dank/models/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
# Fleet endpoint: same port and alias as every frank-small before it, so the
# harness models.yml id and the homepage probe do not move.