From 8f129180d87a942a8d633889b51317ff642ea9a3 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 2 Oct 2026 02:31:39 -0700 Subject: [PATCH] docs+scripts: reconcile payload inventory after both swaps Staged 59 GiB of retired LLM payloads on Dankclaw to ~dank/.AGENTTRASH/20261002-retired-llm-payloads/ (vLLM MiMo BF16, Qwen3.5-9B MTP source, both DFlash dirs, Gemma4-12B-QAT, SGLang MiMo INT8) and corrected every doc claim about what is on disk: gufo's rollback pair (Q4_K_XL weights + the Q8_K_XL DFlash2 drafter) and frank-small's Q6_K stay in place, so the rollback paths in the tags' env files still resolve. The vLLM-era frank-large fetch trio moves to old/ with its runtime; runbook links updated. --- AGENTS.md | 3 ++- .../download-frank-large-dflash.sh | 0 {scripts => old}/download-frank-large-mimo.sh | 0 .../download-frank-large-qwen35.sh | 0 .../LLM/frank-large-rocmfpx-q8agent-mtp.md | 10 ++++--- .../LLM/frank-large-vllm-mimo-v26-bf16.md | 13 ++++++--- .../LLM/frank-small-llama-mimo-q5km.md | 2 +- .../LLM/frank-small-sglang-mimo-v26.md | 15 ++++++++--- serverAppConfigs-docs/LLM/index.md | 27 ++++++++++--------- 9 files changed, 44 insertions(+), 26 deletions(-) rename {scripts => old}/download-frank-large-dflash.sh (100%) rename {scripts => old}/download-frank-large-mimo.sh (100%) rename {scripts => old}/download-frank-large-qwen35.sh (100%) diff --git a/AGENTS.md b/AGENTS.md index bcc538b..f344a80 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -399,7 +399,8 @@ SGLang run used). Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`. Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at `/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]] -(venv + INT8 payload still on the host). +(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was +staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02). ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE) diff --git a/scripts/download-frank-large-dflash.sh b/old/download-frank-large-dflash.sh similarity index 100% rename from scripts/download-frank-large-dflash.sh rename to old/download-frank-large-dflash.sh diff --git a/scripts/download-frank-large-mimo.sh b/old/download-frank-large-mimo.sh similarity index 100% rename from scripts/download-frank-large-mimo.sh rename to old/download-frank-large-mimo.sh diff --git a/scripts/download-frank-large-qwen35.sh b/old/download-frank-large-qwen35.sh similarity index 100% rename from scripts/download-frank-large-qwen35.sh rename to old/download-frank-large-qwen35.sh diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md index 8d65f3c..22d269b 100644 --- a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md @@ -191,9 +191,13 @@ git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1 DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio and redeploy — nothing here deletes anything. -Payloads retired alongside this swap (MiMo-V2.6 BF16, Qwen3.5-9B MTP source, -DFlash draft dirs) are still on disk under `/home/dank/models/` and are the -subject of the [[deployment]] disk-pressure note, not of a rollback plan. +The rollback payloads stay exactly where the tag's env file expects them: +`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2 +drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf` +(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB, +Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to +`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 — +recoverable until the trash is purged, and never part of a rollback plan. ## When it misbehaves diff --git a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md index 6000864..55428a3 100644 --- a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md +++ b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md @@ -108,15 +108,20 @@ chase text drift between spec and no-spec arms as a regression. - Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard - byte-count + LFS sha256-verified by `scripts/download-frank-large-mimo.sh` - (`--verify-only` re-checks; no token on the curl command line). -- MTP source: `scripts/download-frank-large-qwen35.sh` → + byte-count + LFS sha256-verified by + `old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on + the curl command line). +- MTP source: `old/download-frank-large-qwen35.sh` → `/home/dank/models/Qwen3.5-9B/` (4 shards behind `model.safetensors.index.json`; the shards carry the odd HF asset names `model.safetensors-0000N-of-00004.safetensors` — the index resolves them, the engine boots from exactly this tree). -- DFlash drafter: `scripts/download-frank-large-dflash.sh` → +- DFlash drafter: `old/download-frank-large-dflash.sh` → `/home/dank/models/Qwen3.5-9B-DFlash/`. +- **Payload status 2026-10-02:** all three trees moved to + `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with + the other retired frank payloads) and the fetch trio moved to `old/` — the + endpoint is two runtimes past this one, and re-fetch is cheaper than disk. - The serve script gates the image by REPO DIGEST (rootless podman resolves tags from the network; the pin exists to boot exactly these bytes offline), then config + index + exact shard count, then the spec draft diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md index c18a35b..71268c8 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md @@ -99,7 +99,7 @@ traffic first. The bats seam fails if a draft flag ships unannounced. |Target|How| |---|---| |Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default| -|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`| +|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`| ## When it misbehaves diff --git a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md index e16a22b..3bb9edc 100644 --- a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md +++ b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md @@ -20,7 +20,11 @@ timestamp: 2026-10-01 > **RETIRED 2026-10-01, hours after cutover.** Replaced by > [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF, > chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. -> The venv and the INT8 payload stay on disk as the rollback. +> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8 +> payload was staged to +> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when +> the llama.cpp endpoint was two runtimes deep — recoverable until the trash +> is purged. ## Identity @@ -159,9 +163,12 @@ serve script refuses to start with a missing shard. ## Rollback -[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary -remain intact on disk; its unit/env/serve trio is in `old/` after the -cutover. +[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the +practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at +`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp +Gemma4-12B QAT payload, was staged to +`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on +2026-10-02; its unit/env/serve trio remains in `old/`. # Citations diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 151bf73..8821399 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -177,8 +177,8 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-large-gufo-qwen38-27b]] — **the ACTIVE frank-large runbook - again since 2026-10-01 night** (root systemd unit, port 7770): gufo +- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback + target)** (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL** with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native @@ -228,23 +228,24 @@ frank-small, port 7771] .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting (2026-09-16 → 09-19) -- [[frank-small-llama-mimo-q6k]] — **the ACTIVE frank-small runbook - since 2026-10-01 night** (root systemd unit, port 7771): llama.cpp - (master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill- - Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token - sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces - flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no - vision), reasoning_content, no spec decode by default +- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback + target)**: llama.cpp (master, CUDA sm_86 build) serving + bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO + native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V + q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the + GGUF has no vision), reasoning_content, no spec decode; payload still at + `/home/dank/models/MiMo-Q6K/` - [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot - express; venv + INT8 payload on disk as rollback): SGLang in a + express; venv still on disk, INT8 payload staged to + `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8 W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision, 83.9 tok/s decode - [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by - [[frank-small-sglang-mimo-v26]] → [[frank-small-llama-mimo-q6k]]; - rollback target, payload and binary intact on disk): mainline llama.cpp serving - intact on disk): mainline llama.cpp serving + [[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged + to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`): + mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX 3090, with the separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.