From 44e7a8dfafb20d96b26a4b2ea490708e996126b9 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 6 Oct 2026 18:13:05 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20frank-secondary=20at=202x114688/ubatch?= =?UTF-8?q?=201024=20=E2=80=94=20runbook=20renamed=20+=20rewritten,=20flee?= =?UTF-8?q?t=20docs=20updated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - runbook git mv ...-mtp-131072.md -> ...-mtp-114688.md; new Measured section (22,374 used / 1,751 MiB free, decode 58.2/66.9 t/s, confounded vs 2x131072 noted), rollback ladder, trail entry - AGENTS.md: topology row, ACTIVE section, contextWindow 114688, runbook index; mmproj name corrected to the publisher's mmproj-BF16.gguf - LLM/index.md, index.md, mimo runbook link, log.md entry + path fix --- AGENTS.md | 63 ++++---- ...frank-secondary-llama-mimo-q6k-2x200704.md | 2 +- ...ry-llama-swift15-unc-iq3xxs-mtp-114688.md} | 137 ++++++++++++------ serverAppConfigs-docs/LLM/index.md | 18 ++- serverAppConfigs-docs/index.md | 13 +- serverAppConfigs-docs/log.md | 22 ++- 6 files changed, 164 insertions(+), 91 deletions(-) rename serverAppConfigs-docs/LLM/{frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md => frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md} (67%) diff --git a/AGENTS.md b/AGENTS.md index 8362f26..3c825ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 | -| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771| +| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 114,688** (Frank's 112k cap each since the 2026-10-06 late reshape, cut from 131,072 to fund **ubatch 1024** — **1,751 MiB measured free**) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024; **58.2 t/s recipe decode / 66.9 (temp 0)** (MTP acceptance 42.8–54.4 %). Renamed from `frank-small` 2026-10-06 (87721ab). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771| Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -308,11 +308,11 @@ image]`) — `frank-large` returned to vision with the flash-server v2 swap 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; `--mm-max-patches` available), and `frank-secondary` through -`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo +the publisher's own `mmproj-BF16.gguf` passed as `--mmproj` — the Swift GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is -**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the -model's native 262144: the harness must compact before llama.cpp answers a -longer prompt with a 400. Note also +**114688** (cut from 131072 on 2026-10-06 late to fund ubatch 1024), the +per-slot window, not the model's native 262144: the harness must compact +before llama.cpp answers a longer prompt with a 400. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — @@ -423,8 +423,10 @@ the publisher's own `mmproj-BF16.gguf`** (931,145,888 B, sha256 `19acd7fc…`), both `/home/dank/models/Swift15-UncIQ3XXS/`, on port 7771. Cutover 2026-10-05 (evening) on Frank's explicit contract: **128k context cap, KV at FP8, vision on, as many 128k contexts as fit**; -reshaped twice on 2026-10-06 — morning to two 131,072 slots no-spec, -evening to this UNCERSED RCO-IQ3_XXS payload with MTP back on. The +reshaped three times on 2026-10-06 — morning to two 131,072 slots +no-spec, evening to this UNCERSED RCO-IQ3_XXS payload with MTP back +on, late the RENAME to **frank-secondary** (87721ab) at two 114,688 +slots with ubatch 1024. The same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed (blocked at the docs gate; the docs debt is what this section @@ -438,31 +440,36 @@ at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at 48 linear layers × 48 value heads). Header facts were read from the Hub file by HTTP range + struct parse before anything was downloaded. -**The shape (2026-10-06, Frank's tight-fit order, MTP back on in the -evening):** `--ctx-size 262144 --parallel 2` = **TWO 131,072-token -conversations** (Frank's 128k cap each; `n_ctx_slot = CTX_SIZE / -PARALLEL`), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's +**The shape (2026-10-06 late; Frank's tight-fit order, MTP back on in +the evening, per-slot cut + ubatch raised late):** `--ctx-size 229376 +--parallel 2` = **TWO 114,688-token conversations** (Frank's 112k cap +each; `n_ctx_slot = CTX_SIZE / PARALLEL`; cut from 131,072 to fund +ubatch 1024), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective bits; supersedes the same-day f16/f16 rule written for the Ornith 9B fit), all layers on -the GPU, `--mmproj` for vision, batch 4096 / ubatch 512. **Speculative +the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024. **Speculative decoding is ON** (draft-mtp, depth 3): the -mtp file's head is drafted against the main model's own weights; ngram-mod skipped on purpose — on a qwen35 hybrid ngram disables recurrent-state rollback, so a rejected draft corrupts the DeltaNet state. -**Measured at the two-slot MTP boot (2026-10-06 evening): 23,636 used / -489 MiB free** (`nvidia-smi` free field) — the tightest fit in the -endpoint's history, under the old 2-3 GiB floor BY ORDER; a 512×512 -solid-red image still answers `Red` afterwards (489 MiB free after the -probe), so the fit carries the vision transients. The MTP draft ctx -costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561) — ~6× the ~0.4 GiB -the one-slot era assumed, because the drafter's KV is built against the -target's full 262,144 context (the draft window), not the 3-token -proposal depth; the lighter RCO-IQ3_XXS quant (−1.56 GiB vs IQ3_S-mtp) -funds it. **Decode: 63.3 tok/s (recipe temp 1.0) / 72.9 (temp 0)** with -MTP acceptance 49.8–62.4 % (48.0 no-spec — the drafter earned its -2.4 GiB). `reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 -Jinja raise (the models.yml fold stays mandatory). +**Measured at the late boot (2026-10-06, 2 × 114,688 / ubatch 1024): +22,374 used / 1,751 MiB free** (`nvidia-smi` free field) — the per-slot +cut freed ~1.05 GiB of KV, which paid for the ubatch-1024 buffers and +netted ~1.25 GiB back over the evening's tightest-ever fit (23,636 used +/ 489 MiB free at 2 × 131,072 / ubatch 512, under the old 2-3 GiB floor +BY ORDER; a 512×512 solid-red image still answered `Red` there, so the +fit carries the vision transients). The MTP draft ctx costs **~2.4 GiB +measured** (23,636 − 22,728 + 1,561 at the 262,144 draft window) — ~6× +the ~0.4 GiB the one-slot era assumed, because the drafter's KV is +built against the target's FULL context, not the 3-token proposal +depth; the lighter RCO-IQ3_XXS quant (−1.56 GiB vs IQ3_S-mtp) funds it. +**Decode at the late boot: 58.2 tok/s (recipe temp 1.0) / 66.9 (temp +0)**, acceptance 42.8–54.4 % (the 2 × 131,072 boot read 63.3 / 72.9 at +49.8–62.4 % — acceptance is text-dependent and the prompts differ: the +comparison is confounded, not a shape verdict). `reasoning_effort` +probe unchanged: `xhigh` 200, `high` → 500 Jinja raise (the models.yml +fold stays mandatory). **Vision with the publisher's own projector**: ajgazin's repo SHIPS `mmproj-BF16.gguf` in the same repo (the IQ3_S era's unsloth mmproj was @@ -485,7 +492,7 @@ payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's 98/100, KL 0.0884). Runbook: -`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md`. +`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`. Rollback: `/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload live until the 2026-10-06 evening swap — revert da36fa3 for the two-slot no-spec shape) or `/home/dank/models/MiMo-Q6K/` (revert to the @@ -1014,7 +1021,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` | -| frank-secondary (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md` | +| frank-secondary (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` | | frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -1028,7 +1035,7 @@ Retired (rollback targets and history): | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` | | llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` | -| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md` | +| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | diff --git a/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md index d710239..d4281a6 100644 --- a/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md @@ -15,7 +15,7 @@ timestamp: 2026-10-03 # frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV > **RETIRED 2026-10-05 (evening).** Superseded by -> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B +> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B > RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail: > committed earlier the same day (77decb5..380aafa) was **never deployed** — > THIS MiMo shape was the live payload when Swift cut over; (2) the KV diff --git a/serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md similarity index 67% rename from serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md rename to serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md index a0f2c73..bf04f36 100644 --- a/serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV" -description: "The active frank-secondary runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic." +title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV" +description: "The active frank-secondary runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic." tags: - llm - frank-secondary @@ -15,26 +15,25 @@ tags: timestamp: 2026-10-06 --- -# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV +# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV -> **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A -> 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit -> contract for the swap: context capped at 128k per conversation, KV at -> FP8 class, vision on, "as many of those contexts as you can". The -> morning order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 -> contexts at 128k — should be a very tight fit") bought the second slot -> by spending the drafter (~0.4 GiB assumed) and ubatch headroom (~0.3 -> GiB): 1,397 MiB measured free. The evening order — swap to the -> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and turn MTP back on — landed -> **489 MiB free**: the lighter quant (−1.56 GiB of weights) funds the -> second slot, but the MTP draft ctx costs **~2.4 GiB measured, not the -> ~0.4 GiB assumed** — the drafter's KV is built against the target's -> full context (the 262,144 draft window), not the 3-token proposal -> depth. The 2–3 GiB floor is knowingly violated, as ordered. This -> supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K -> shape ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the -> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never -> deployed. +> **ACTIVE since 2026-10-05 (evening); reshaped three times on +> 2026-10-06, renamed from frank-small the same night.** A 27B replaces +> the 9B on the same 24 GiB card: KV at FP8 class, vision on. The +> morning order (drop MTP, ubatch 512, TWO 128k contexts) bought the +> second slot — 1,397 MiB free. The evening order swapped to the +> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and put MTP back ON — landing +> **489 MiB free**: the MTP draft ctx costs **~2.4 GiB, not the ~0.4 +> GiB assumed** (the drafter's KV is built against the target's full +> context, not the 3-token proposal depth), paid for by the lighter +> quant (−1.56 GiB). The late order dropped the per-slot window +> 131,072 → **114,688** (Frank's 112k cap) and raised ubatch 512 → +> 1024 — landing **1,751 MiB free**, back above the morning's number +> with the prefill knob raised. This supersedes BOTH earlier 2026-10-05 +> payloads: the live MiMo-V2.6 Q6_K shape +> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the +> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but +> never deployed. ## Identity @@ -64,11 +63,12 @@ escape hatch is the same as every payload here: revert and redeploy. ## Shape and KV contract -`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap -(the GSQ publisher's own benchmark protocol ran at 131,072 per stream). -`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the -per-slot number (`262144 / 2 = 131,072`) is the product contract — -models.yml stays at 131,072. +`--ctx-size 229376 --parallel 2` — TWO conversations at Frank's 112k +cap (2026-10-06 late: cut from the 128k the morning/evening shapes ran +at, to fund ubatch 1024). `CTX_SIZE` stays divisible by `PARALLEL`; the +seam enforces it, and the per-slot number (`229376 / 2 = 114,688`) is +the product contract — models.yml follows at 114,688 (a longer prompt +is a 400, never a truncation). **KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).** llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem @@ -81,18 +81,25 @@ pinned: quantized V requires it, and the build's `GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a silent f16 fallback. -**Two slots + drafter, by arithmetic and by order (2026-10-06).** The -morning order bought the second slot by spending the drafter and ubatch -headroom (1,397 MiB free). The evening swap put the drafter BACK and -changed the arithmetic: the draft ctx is built against the target's FULL -context (262,144 — the draft window, not the 3-token proposal depth), so -it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB -the one-slot era assumed. The lighter quant pays for it: **23,636 used / -489 MiB free** (nvidia-smi `free` field) — the tightest fit in -frank-secondary's history, the 2–3 GiB floor knowingly violated, and the -512×512 vision transient (34 MiB) still fits and recovers. Compute buffers -allocate at graph capture during BOOT, so a miss is loud and fast — the -failure mode is a crash loop, not silent corruption. +**Two slots + drafter, by arithmetic and by order (2026-10-06 morning / +evening, the 2 × 131,072 era).** The morning order bought the second slot +by spending the drafter and ubatch headroom (1,397 MiB free). The evening +swap put the drafter BACK and changed the arithmetic: the draft ctx is +built against the target's FULL context (262,144 — the draft window, not +the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 − +22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter +quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free` +field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor +knowingly violated, and the 512×512 vision transient (34 MiB) still fits +and recovers. + +**The late reshape (2 × 114,688, ubatch 1024) is the current fit.** The +per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; ubatch +512 → 1024 spends part of it on compute buffers (a prefill knob — decode +never moves with it). Measured boot: **22,374 used / 1,751 MiB free**. +The drafter's own KV shrank with its target (draft window 229,376). +Compute buffers allocate at graph capture during BOOT, so a miss is loud +and fast — the failure mode is a crash loop, not silent corruption. ## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3) @@ -161,7 +168,7 @@ leave room for thinking. - **Vision**: 512×512 solid-red PNG → `Red` again on this shape. - Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet. -## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, live) +## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, historical) - **Boot ledger**: `creating MTP draft context against the target model '…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`, @@ -186,6 +193,29 @@ leave room for thinking. slot prefix cache is live. - Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet. +## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, live) + +- **argv** (from `/proc//cmdline`): `--ctx-size 229376 + --parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size + 1024`, `--spec-type draft-mtp`. Deploys 87721ab (the rename — old + `frank-small.service` stopped + disabled first, by hand: `disable` on + a relinked unit removes only its symlink) then f2575a1 (the shape); + health gate + MainPID check green, e2e 32 passed / 1 flaky-retry. +- **VRAM: 22,374 used / 1,751 MiB free** (nvidia-smi `free` field) — + the per-slot cut (≈1.05 GiB of KV) paid for the ubatch-1024 buffers + and netted back ~1.25 GiB against the evening's 489 MiB; back above + the morning's 1,397 MiB. +- **Decode + MTP acceptance** (live `timings`): recipe (temp 1.0) → + **58.2 t/s** (430/1005, 42.8 %); temp 0 → **66.9 t/s** (475/873, + 54.4 %). The 2 × 131,072 boot measured 63.3 / 72.9 — −8–10 %, but + acceptance is text-dependent and these prompts differ from that + measurement's, so the comparison is CONFOUNDED, not a shape verdict; + a clean A/B needs stop + mask (bench policy). +- **Smoke**: `READY` completion + `reasoning_content` populated; the + harness reached :7771 under the new alias (models.yml provider/id + `frank-secondary`, config.yml modelRoles rewritten; per-slot + `contextWindow: 114688`). + ## Client contract - Reasoning in `reasoning_content` (`--reasoning-format deepseek`); @@ -209,14 +239,17 @@ leave room for thinking. ## Rollback -The 2 × 131,072 no-spec IQ3_S-mtp shape is ONE commit away — revert -da36fa3 and redeploy: the env trio then ships `SPEC_TYPE=none`, -`CTX_SIZE=262144`, `PARALLEL=2`, `UBATCH_SIZE=512` against -`/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload, still on disk, -measured 48.0 tok/s no-spec). The 1 × 131,072 MTP shape is two reverts -back (812d9eb's parent): the same IQ3_S-mtp file with -`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, `PARALLEL=1` -(measured 58.9–66.9 tok/s with the head drafting). +**The immediate predecessor shape (2 × 131,072, ubatch 512, same +payload) is ONE revert away** — revert f2575a1 and redeploy +(`CTX_SIZE=262144`, `UBATCH_SIZE=512`). The 2 × 131,072 no-spec +IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env +trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`, +`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the +IQ3_S-mtp payload, still on disk, measured 48.0 tok/s no-spec). The +1 × 131,072 MTP shape is two reverts further back (812d9eb's parent): +the same IQ3_S-mtp file with `SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, +`CTX_SIZE=131072`, `PARALLEL=1` (measured 58.9–66.9 tok/s with the head +drafting). Three payloads on disk, all fetchable by the same script's git-history recipes: @@ -260,3 +293,13 @@ recipes: target's full 262,144 context, not the 3-token proposal depth) plus the second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget, decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec). +- 2026-10-06 (late, Frank): the endpoint renamed frank-small → + **frank-secondary** (87721ab — scripts, unit, seam, runbooks, + identifiers; historical references to the OLD FreeToken/CUDA + frank-small, and to the Dankserver service that became frank-medium, + stay frank-small) and the per-slot window cut 131,072 → 114,688 to + fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand + before the deploy: `systemctl stop frank-small` + `systemctl disable + frank-small`, then `deploy.sh dankclaw --service frank-secondary`. + 1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml + + config.yml retargeted. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index cb44d34..1a17b99 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -103,18 +103,20 @@ frank-secondary, port 7771] (`slow`/`advisor` → frank-primary) carried forward. Superseded by the MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the engine moved with the model -- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since +- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **ACTIVE since 2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since - 2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master + 2026-10-06 evening; RENAMED from frank-small and reshaped to 2 × + 114,688 / ubatch 1024 late 2026-10-06** (root systemd unit, port + 7771): llama.cpp master (CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP- GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED — draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the - RTX 3090, TWO 131,072-token conversations at Frank's 128k cap (pool - 262144 / parallel 2 — the tightest fit in the endpoint's history: - **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with - q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16), ubatch 512, and - vision through the publisher's own mmproj-BF16.gguf. Decode 63.3 t/s - recipe / 72.9 temp 0 (MTP acceptance 49.8–62.4 %; 48.0 no-spec). The + RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool + 229376 / parallel 2 — cut from 131,072 to fund ubatch 1024; **1,751 + MiB measured free**) with + q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and + vision through the publisher's own mmproj-BF16.gguf. Decode 58.2 t/s + recipe / 66.9 temp 0 (MTP acceptance 42.8–54.4 %). RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's 98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 84ca16a..1b5ec2c 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -372,14 +372,15 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE +- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the ACTIVE frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS - with MTP back on since 2026-10-06 evening**): llama.cpp master (CUDA + with MTP back on since 2026-10-06 evening; renamed from frank-small and + reshaped to TWO 114,688-token conversations at Frank's 112k cap with + ubatch 1024 late 2026-10-06**): llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS - (the MTP head in the file is drafted) on the RTX 3090 — TWO - 131,072-token conversations at Frank's 128k cap, the tightest fit in - the endpoint's history (489 MiB measured free), q8_0 K + q8_0 V, vision - through the publisher's own projector, ~63 t/s recipe decode with MTP + (the MTP head in the file is drafted) on the RTX 3090 — 1,751 MiB + measured free, q8_0 K + q8_0 V, vision + through the publisher's own projector, ~58 t/s recipe decode with MTP - [[frank-secondary-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening, superseded by swift15): llama.cpp master serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 45e6300..bc5eb09 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2,6 +2,26 @@ ## 2026-10-06 +- **frank-secondary (Dankclaw): RENAMED from frank-small, then reshaped + to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090).** The rename (commit + 87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers + `FRANK_SECONDARY_*`/`llm_secondary`/`deploy_llm_secondary`; historical + refs to the OLD FreeToken/CUDA frank-small and to the Dankserver + service that became frank-medium stay frank-small, as does + `freetoken-frank-small.md`). Live unit transition by hand before the + deploy: `systemctl stop frank-small` + `systemctl disable frank-small` + (a relinked unit — disable removes only its stale symlink) → + `deploy.sh dankclaw --service frank-secondary`. The reshape (commit + f2575a1, Frank's order "lower max context to 112k so we can increase + ubatch to at least 1k"): `CTX_SIZE 262144→229376` (per-slot + 131,072→114,688), `UBATCH_SIZE 512→1024` → **22,374 used / 1,751 MiB + free** (the per-slot cut frees ~1.05 GiB of KV; ubatch 1024 spends + part, netting ~1.25 GiB back over the 489 MiB fit), decode 58.2 t/s + recipe / 66.9 temp 0 (acceptance 42.8–54.4 % — the drop vs the prior + boot's 63.3/72.9 is confounded by prompt text, not a shape verdict). + Harness retargeted: models.yml (id/name/compactionModels, + `contextWindow: 114688`) + config.yml modelRoles ×14. Runbook renamed: + `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`. - **frank-secondary (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).** Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb): @@ -19,7 +39,7 @@ uncensored model thinks more — 64 tokens is all `reasoning_content`). ngram-mod still skipped (recurrent-state rollback law on the qwen35 hybrid). Runbook renamed + rewritten: - `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md`. + `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`. - **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server 0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the