diff --git a/AGENTS.md b/AGENTS.md index aef8dbd..b61f546 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -408,12 +408,20 @@ conversation at Frank's 128k cap (Swift's own benchmark protocol ran at class — no true fp8 KV dtype exists; 8.5 effective bits; the order SUPERSEDES the same-day f16/f16 rule written for the Ornith 9B fit), all layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024 (lower -ubatch first, Frank's standing law; 2048 is the first A/B if boot lands -high). Non-KV ≈ 15.5 GiB (weights 11,561 + projector ~1,350 + buffers -~1.6k + CUDA ~500 + DeltaNet ~150/slot + MTP draft ctx ~0.3-0.5k); one q8_0 -slot = 4,352 MiB ⇒ free ≈ 4.5 GiB; a SECOND slot wants +4,352 ⇒ ~0.2 GiB — -far under the 2-3 GiB floor. The pool is capped by the 128k law, not the -card; leftover headroom is honest slack. +ubatch first, Frank's standing law; 2048 is the first A/B if more headroom +is ever wanted). Measured at boot: **19,404 used / 4,721 MiB free** +(`nvidia-smi` free field — the 4.5 GiB prediction held; f16 KV would have +left ~700 MiB, so the free number is also the q8_0 proof). The pool is +capped by the 128k law, not the card; a SECOND slot wants +4,352 MiB and +would land ~0.2 GiB free — far under the 2-3 GiB floor. + +**Measured decode (2026-10-05 probes, draft_n counts from the live +`timings`)**: temp 0.8 → **66.1 tok/s**, acceptance 182/349 (52.1 %); temp 0 +→ 66.9 tok/s, 166/318; **deployed recipe temp 1.0/top_p 0.95/top_k 20 → +58.9 tok/s, 159/355 (44.8 %)** — the higher temperature costs both acceptance +and rate. Vision: a 512×512 solid-red PNG answered `Red` through an +`image_url` part. `reasoning_effort` probe: `xhigh` 200, `high` → 500 +Jinja raise (the models.yml fold stays mandatory). **Speculative decoding is the file's own MTP head**: the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`), so diff --git a/scripts/frank-small.service b/scripts/frank-small.service index e3811b3..7140cbb 100644 --- a/scripts/frank-small.service +++ b/scripts/frank-small.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small) +Description=llama.cpp Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S-mtp GGUF (qwen35 hybrid, 1x131072, q8_0 K+V, in-file MTP head, mmproj vision) on the RTX 3090 (dankclaw, frank-small) After=network-online.target Wants=network-online.target diff --git a/scripts/stack-qwen38-official-fp8.py b/scripts/stack-qwen38-official-fp8.py index e49429a..132c77a 100755 --- a/scripts/stack-qwen38-official-fp8.py +++ b/scripts/stack-qwen38-official-fp8.py @@ -230,6 +230,7 @@ def fill_ple_shard(job): count=(r1 - r0) * cols, offset=src_off + r0 * cols) q = lut.take(codes) + del codes # release the mmap export before sm.close() q[~np.isfinite(q)] = 0.0 q *= np.float32(scale_f32) packed = f32_to_bf16_rne(q) @@ -525,6 +526,7 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd, codes = np.frombuffer(mm, dtype=np.uint8, count=hd, offset=off + r * hd) q = lut.take(codes) * np.float32(scale_f32) + del codes # release the mmap export before mm.close() want = f32_to_bf16_rne(q).tobytes() got_row = sum(rows[:i]) + r with open(out_path, "rb") as ofh: diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md index 6c2faa6..3a37609 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md @@ -102,6 +102,28 @@ probe undersupplies image tokens and the model invents ("video"); the legacy top-level `images` field is IGNORED, only the OpenAI `image_url` content part reaches the model. Same traps the MiMo eras documented. +## Measured at cutover (2026-10-05, live) + +- **Boot ledger**: `creating MTP draft context against the target model + '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal + model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1, + n_ctx_slot = 131072` — every expected line, no q8→f16 fallback. +- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The + 4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free + number doubles as the q8_0 proof. +- **Decode + MTP acceptance** (draft counts from the live `timings`): + temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, + 166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, + 159/355 (44.8 %)** — temperature buys fluency, costs acceptance and rate. + For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this 27B at + IQ3_S runs ~2/3 the rate with ~3× the capability class. +- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`. +- **Template gate**: `reasoning_effort: xhigh` → 200 with + `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the + models.yml fold is mandatory, exactly as the MiMo eras found. +- Streaming completion: 200. No first-request silence window (the + halogen-era 45–60 s stall has no analogue here). + ## Client contract - Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking @@ -143,5 +165,15 @@ recipes: whole-tree docs gate); the live service was still MiMo Q6_K when Swift cut over. Both Ornith payloads were verified on disk before Swift. - Swift header facts (block_count 65, interval 4, KV geometry, nextn) were - read from the Hub file by HTTP range + struct parse — no download - needed — before the payload was fetched. + read from the Hub file by HTTP range + struct parse — no download needed — + before the payload was fetched. +- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label- + committed both sets at Frank's direction — 1919ad7 (frank-primary radiance) + then baf33d0 (this swap: trio, seam, runbooks, fleet docs). Deploy ran + `deploy.sh dankclaw --service frank-small`; the service dispatch and health + gate went green; the first e2e run failed ONE spec (12-crawl, + `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, unrelated to + frank-small), green on rerun. +- The unit's `Description=` (MiMo era) was stale at cutover time and was + updated in-repo afterward — the relinked unit picks it up at the next + deploy/restart, not retroactively.