docs+scripts: reconcile payload inventory after both swaps

Staged 59 GiB of retired LLM payloads on Dankclaw to
~dank/.AGENTTRASH/20261002-retired-llm-payloads/ (vLLM MiMo BF16, Qwen3.5-9B
MTP source, both DFlash dirs, Gemma4-12B-QAT, SGLang MiMo INT8) and corrected
every doc claim about what is on disk: gufo's rollback pair (Q4_K_XL weights +
the Q8_K_XL DFlash2 drafter) and frank-small's Q6_K stay in place, so the
rollback paths in the tags' env files still resolve. The vLLM-era frank-large
fetch trio moves to old/ with its runtime; runbook links updated.
This commit is contained in:
Frank
2026-10-02 02:31:39 -07:00
parent 72bd4c967e
commit 8f129180d8
9 changed files with 44 additions and 26 deletions
+2 -1
View File
@@ -399,7 +399,8 @@ SGLang run used).
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
(venv + INT8 payload still on the host).
(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
@@ -191,9 +191,13 @@ git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
and redeploy — nothing here deletes anything.
Payloads retired alongside this swap (MiMo-V2.6 BF16, Qwen3.5-9B MTP source,
DFlash draft dirs) are still on disk under `/home/dank/models/` and are the
subject of the [[deployment]] disk-pressure note, not of a rollback plan.
The rollback payloads stay exactly where the tag's env file expects them:
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2
drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`
(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
recoverable until the trash is purged, and never part of a rollback plan.
## When it misbehaves
@@ -108,15 +108,20 @@ chase text drift between spec and no-spec arms as a regression.
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
byte-count + LFS sha256-verified by `scripts/download-frank-large-mimo.sh`
(`--verify-only` re-checks; no token on the curl command line).
- MTP source: `scripts/download-frank-large-qwen35.sh` →
byte-count + LFS sha256-verified by
`old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
the curl command line).
- MTP source: `old/download-frank-large-qwen35.sh` →
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
`model.safetensors.index.json`; the shards carry the odd HF asset names
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
the engine boots from exactly this tree).
- DFlash drafter: `scripts/download-frank-large-dflash.sh` →
- DFlash drafter: `old/download-frank-large-dflash.sh` →
`/home/dank/models/Qwen3.5-9B-DFlash/`.
- **Payload status 2026-10-02:** all three trees moved to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
the other retired frank payloads) and the fetch trio moved to `old/` — the
endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
- The serve script gates the image by REPO DIGEST (rootless podman resolves
tags from the network; the pin exists to boot exactly these bytes
offline), then config + index + exact shard count, then the spec draft
@@ -99,7 +99,7 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|Target|How|
|---|---|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
## When it misbehaves
@@ -20,7 +20,11 @@ timestamp: 2026-10-01
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
> The venv and the INT8 payload stay on disk as the rollback.
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
> payload was staged to
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
> is purged.
## Identity
@@ -159,9 +163,12 @@ serve script refuses to start with a missing shard.
## Rollback
[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
remain intact on disk; its unit/env/serve trio is in `old/` after the
cutover.
[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
Gemma4-12B QAT payload, was staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
2026-10-02; its unit/env/serve trio remains in `old/`.
# Citations
+14 -13
View File
@@ -177,8 +177,8 @@ frank-small, port 7771]
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-gufo-qwen38-27b]] — **the ACTIVE frank-large runbook
again since 2026-10-01 night** (root systemd unit, port 7770): gufo
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
target)** (root systemd unit, port 7770): gufo
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
@@ -228,23 +228,24 @@ frank-small, port 7771]
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract,
troubleshooting (2026-09-16 → 09-19)
- [[frank-small-llama-mimo-q6k]] — **the ACTIVE frank-small runbook
since 2026-10-01 night** (root systemd unit, port 7771): llama.cpp
(master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill-
Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token
sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces
flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no
vision), reasoning_content, no spec decode by default
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
target)**: llama.cpp (master, CUDA sm_86 build) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
GGUF has no vision), reasoning_content, no spec decode; payload still at
`/home/dank/models/MiMo-Q6K/`
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
express; venv + INT8 payload on disk as rollback): SGLang in a
express; venv still on disk, INT8 payload staged to
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
83.9 tok/s decode
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
[[frank-small-sglang-mimo-v26]] → [[frank-small-llama-mimo-q6k]];
rollback target, payload and binary intact on disk): mainline llama.cpp serving
intact on disk): mainline llama.cpp serving
[[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.