fix(frank-large): load the model as its native 3-way split, not a concatenated file

The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.

- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
  natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
  canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
  ~/.AGENTTRASH/20260921-frank-large-assembled/
This commit is contained in:
Frank
2026-09-21 13:58:54 -07:00
parent 384747a72f
commit d6f9b8539a
11 changed files with 72 additions and 36 deletions
@@ -73,18 +73,24 @@ One root unit, one process, one published port:
`:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
`00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
(43,836,407,744 B) = **93,682,584,224 B** concatenated in numeric order.
- **Assembled at** `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
- `Qwen3.8-Flash-Next-UD-IQ4_XS.gguf` — main model (87.3 GiB)
(43,836,407,744 B) = **93,682,584,224 B**. The parts ARE the canonical
payload — the serve script points `-m` at part 1 and llama.cpp loads the
3-way split natively (parts 2–3 auto-discovered in the same directory).
Do **not** concatenate them into a single file to serve: part 1's header
carries `ggml.file_count=3`, so a byte-correct concatenation still
advertises itself as a split and the loader rejects the non-split
filename with `invalid split file name` (caught on 2026-09-21 — the
assembled file crash-looped the unit ~110 times before the diagnosis).
- **On disk** at `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
- `UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-0000{1,2,3}-of-00003.gguf` — the
main model, 3-way split (87.3 GiB)
- `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
- `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
`ENABLE_VISION=0`)
- `UD-IQ4_XS/` + `SHA256SUMS` — transport parts and the assembled digest
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
(`curl -fL -C -`), each part verified against its **LFS sha256 fetched
live from the Hub tree API** (not a file that could drift), then the
assembled GGUF against `SHA256SUMS`. `--verify-only` re-checks without
downloading.
live from the Hub tree API** (not a file that could drift).
`--verify-only` re-checks without downloading.
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
there is no external PLE payload and no pager, unlike the CIRU Orca
packaging (79.4 GiB main + 52.4 GiB paged PLE).
@@ -160,6 +166,14 @@ swap file is the backstop.
holds the port. The deploy handler sweeps `pkill -f
'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
is also a `llama-server`) and gates on `wait_port_free 7770`.
- **`invalid split file name` at model load** (fails in ~50 ms, before any
weights are read): `-m` points at a file whose GGUF header advertises a
split (`ggml.file_count > 1`) but whose name is not a
`-NNNNN-of-NNNNN` part. This is what a concatenated split file does —
the concatenation is byte-correct but part 1's header still says
"3-way split". Fix: point `-m` at part 1
(`UD-IQ4_XS/...-00001-of-00003.gguf`) and let the loader pick up parts
2–3 from the same directory.
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
the box. If the co-tenants start thrashing, the levers are KV quant
(`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
+14 -1
View File
@@ -52,7 +52,7 @@ timestamp: 2026-09-14
Qwen3.8-Flash-CIRU-STRIX-Orca, 262144 ctx) to the unsloth llama.cpp
prebuilt (b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm 7.16)
serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB, three HF
split parts concatenated), one conversation at 200,000 ctx, MTP depth 2.
split parts), one conversation at 200,000 ctx, MTP depth 2.
The fleet contract is unchanged: port 7770, alias `frank-large`,
text-only.
- **Why the unsloth prebuilt.** The qwen4exp MTP graph exists only in the
@@ -80,6 +80,19 @@ timestamp: 2026-09-14
- **Files.** `scripts/serve-frank-large.sh`, `scripts/frank-large.env`,
`scripts/frank-large.service`, `scripts/download-frank-large-unsloth.sh`,
`scripts/deploy-remote.sh`, `tests/frank_large_unit.bats`.
- **The split-filename trap (caught at cutover).** The first cutover
crash-looped the unit ~110 times with `invalid split file name` at model
load (~50 ms, before any weights were read): the fetch script
concatenated the three parts into a single GGUF, but part 1's header
carries `ggml.file_count=3`, so the byte-correct concatenation still
advertises itself as a 3-way split and the loader demands a
`-NNNNN-of-NNNNN` filename. Fix: the parts ARE the canonical payload —
the serve script points `-m` at part 1 and llama.cpp loads the split
natively (parts 2–3 auto-discovered in the same directory). A manual
part-1 load confirmed the fix (model loaded in ~34 s from a warm page
cache, health 200) before the redeploy. The redundant 87 GiB assembled
file is staged at `~/.AGENTTRASH/20260921-frank-large-assembled/`
(dankclaw) awaiting purge approval.
## 2026-09-21 — node-dankclaw Prometheus target fixed (netavark host-bind quirk)