docs+scripts: reconcile payload inventory after both swaps
Staged 59 GiB of retired LLM payloads on Dankclaw to ~dank/.AGENTTRASH/20261002-retired-llm-payloads/ (vLLM MiMo BF16, Qwen3.5-9B MTP source, both DFlash dirs, Gemma4-12B-QAT, SGLang MiMo INT8) and corrected every doc claim about what is on disk: gufo's rollback pair (Q4_K_XL weights + the Q8_K_XL DFlash2 drafter) and frank-small's Q6_K stay in place, so the rollback paths in the tags' env files still resolve. The vLLM-era frank-large fetch trio moves to old/ with its runtime; runbook links updated.
This commit is contained in:
@@ -399,7 +399,8 @@ SGLang run used).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
|
||||
Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
|
||||
(venv + INT8 payload still on the host).
|
||||
(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
|
||||
staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
|
||||
|
||||
|
||||
@@ -191,9 +191,13 @@ git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
|
||||
DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
|
||||
and redeploy — nothing here deletes anything.
|
||||
|
||||
Payloads retired alongside this swap (MiMo-V2.6 BF16, Qwen3.5-9B MTP source,
|
||||
DFlash draft dirs) are still on disk under `/home/dank/models/` and are the
|
||||
subject of the [[deployment]] disk-pressure note, not of a rollback plan.
|
||||
The rollback payloads stay exactly where the tag's env file expects them:
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2
|
||||
drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`
|
||||
(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
|
||||
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
|
||||
recoverable until the trash is purged, and never part of a rollback plan.
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
|
||||
@@ -108,15 +108,20 @@ chase text drift between spec and no-spec arms as a regression.
|
||||
|
||||
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
|
||||
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
|
||||
byte-count + LFS sha256-verified by `scripts/download-frank-large-mimo.sh`
|
||||
(`--verify-only` re-checks; no token on the curl command line).
|
||||
- MTP source: `scripts/download-frank-large-qwen35.sh` →
|
||||
byte-count + LFS sha256-verified by
|
||||
`old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
|
||||
the curl command line).
|
||||
- MTP source: `old/download-frank-large-qwen35.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
|
||||
`model.safetensors.index.json`; the shards carry the odd HF asset names
|
||||
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
|
||||
the engine boots from exactly this tree).
|
||||
- DFlash drafter: `scripts/download-frank-large-dflash.sh` →
|
||||
- DFlash drafter: `old/download-frank-large-dflash.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B-DFlash/`.
|
||||
- **Payload status 2026-10-02:** all three trees moved to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
|
||||
the other retired frank payloads) and the fetch trio moved to `old/` — the
|
||||
endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
|
||||
- The serve script gates the image by REPO DIGEST (rootless podman resolves
|
||||
tags from the network; the pin exists to boot exactly these bytes
|
||||
offline), then config + index + exact shard count, then the spec draft
|
||||
|
||||
@@ -99,7 +99,7 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
|
||||
@@ -20,7 +20,11 @@ timestamp: 2026-10-01
|
||||
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
|
||||
> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
|
||||
> The venv and the INT8 payload stay on disk as the rollback.
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
|
||||
> payload was staged to
|
||||
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
|
||||
> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
|
||||
> is purged.
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -159,9 +163,12 @@ serve script refuses to start with a missing shard.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
|
||||
remain intact on disk; its unit/env/serve trio is in `old/` after the
|
||||
cutover.
|
||||
[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
|
||||
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
|
||||
Gemma4-12B QAT payload, was staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
|
||||
2026-10-02; its unit/env/serve trio remains in `old/`.
|
||||
|
||||
# Citations
|
||||
|
||||
|
||||
@@ -177,8 +177,8 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-gufo-qwen38-27b]] — **the ACTIVE frank-large runbook
|
||||
again since 2026-10-01 night** (root systemd unit, port 7770): gufo
|
||||
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
|
||||
target)** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
@@ -228,23 +228,24 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-llama-mimo-q6k]] — **the ACTIVE frank-small runbook
|
||||
since 2026-10-01 night** (root systemd unit, port 7771): llama.cpp
|
||||
(master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill-
|
||||
Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token
|
||||
sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces
|
||||
flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no
|
||||
vision), reasoning_content, no spec decode by default
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
|
||||
target)**: llama.cpp (master, CUDA sm_86 build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||||
`/home/dank/models/MiMo-Q6K/`
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv + INT8 payload on disk as rollback): SGLang in a
|
||||
express; venv still on disk, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]] → [[frank-small-llama-mimo-q6k]];
|
||||
rollback target, payload and binary intact on disk): mainline llama.cpp serving
|
||||
intact on disk): mainline llama.cpp serving
|
||||
[[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
|
||||
Reference in New Issue
Block a user