diff --git a/AGENTS.md b/AGENTS.md index b61f546..c50021b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (~161 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, **2 × 200,000** (first boot; native 262,144), froggeric v22.5 template, vision + tools + `reasoning_content` auto, `/metrics` scraped. Cutover 2026-10-05 (rollback `frank-primary-freetoken-pr587-20261005`, FreeToken one-card image + payload on disk) | 7770 | | Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770| -| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; **MTP head IN the file** — `--spec-type draft-mtp`, depth 3) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **1 × 131,072** (Frank's 128k cap; a second 131,072 slot misses the 2-3 GiB floor) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024. Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 | +| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; the -mtp file's MTP head present but NOT drafted — **spec OFF** since 2026-10-06, ngram skipped on the recurrent-rollback law) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tight fit he ordered 2026-10-06 — **1,397 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **48.0 t/s** no-spec decode (MTP reference 58.9-66.9). Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -402,34 +402,31 @@ at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at 48 linear layers × 48 value heads). Header facts were read from the Hub file by HTTP range + struct parse before anything was downloaded. -**The shape (2026-10-05):** `--ctx-size 131072 --parallel 1` = ONE -conversation at Frank's 128k cap (Swift's own benchmark protocol ran at -131,072), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's FP8 -class — no true fp8 KV dtype exists; 8.5 effective bits; the order -SUPERSEDES the same-day f16/f16 rule written for the Ornith 9B fit), all -layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024 (lower -ubatch first, Frank's standing law; 2048 is the first A/B if more headroom -is ever wanted). Measured at boot: **19,404 used / 4,721 MiB free** -(`nvidia-smi` free field — the 4.5 GiB prediction held; f16 KV would have -left ~700 MiB, so the free number is also the q8_0 proof). The pool is -capped by the 128k law, not the card; a SECOND slot wants +4,352 MiB and -would land ~0.2 GiB free — far under the 2-3 GiB floor. +**The shape (2026-10-06, Frank's tight-fit order):** `--ctx-size 262144 +--parallel 2` = **TWO 131,072-token conversations** (Frank's 128k cap each; +`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` pinned, **q8_0 K + +q8_0 V** (llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective +bits; supersedes the same-day f16/f16 rule written for the Ornith 9B fit), +all layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 512. +**Speculative decoding is OFF** (same order): the -mtp file's head stays but +is not drafted with; ngram-mod skipped on purpose — on a qwen35 hybrid +ngram disables recurrent-state rollback, so a rejected draft corrupts the +DeltaNet state. -**Measured decode (2026-10-05 probes, draft_n counts from the live -`timings`)**: temp 0.8 → **66.1 tok/s**, acceptance 182/349 (52.1 %); temp 0 -→ 66.9 tok/s, 166/318; **deployed recipe temp 1.0/top_p 0.95/top_k 20 → -58.9 tok/s, 159/355 (44.8 %)** — the higher temperature costs both acceptance -and rate. Vision: a 512×512 solid-red PNG answered `Red` through an -`image_url` part. `reasoning_effort` probe: `xhigh` 200, `high` → 500 -Jinja raise (the models.yml fold stays mandatory). - -**Speculative decoding is the file's own MTP head**: the `-mtp` GGUF -carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`), so -`--spec-type draft-mtp --spec-draft-n-max 3` loads it against the main -model's weights — no `--model-draft`, no second draft KV pool. Flags move -as a set (a no-spec baseline strips BOTH). Acceptance reads from the -response `timings` (`draft_n`/`draft_n_accepted`) on real traffic, never a -synthetic repeat. +**Measured at the two-slot boot (2026-10-06): 22,728 used / 1,397 MiB +free** (`nvidia-smi` free field) — under the old 2-3 GiB floor BY ORDER ("a +very tight fit"); a 512×512 solid-red image still answers `Red` afterwards +(1,363 MiB free after the probe), so the fit carries the vision transients. +The slot-2 pricing came from the measured 1×131,072 MTP boot (2026-10-05: +19,404 used / 4,721 free — also the q8_0 proof, f16 would have left ~700): +the second q8_0 slot costs 4,352 MiB KV + ~150 MiB DeltaNet, funded by +dropping the MTP draft ctx (~0.4 GiB) and ubatch 1024→512 (~0.3 GiB, +inferred from the measured 616 MiB/doubling rung on the 9B). **Decode +without the drafter: 48.0 tok/s (temp 0) / 47.8 (recipe temp 1.0)** — the +MTP references stay in frank-small.env for the day the head returns (44.8 % +acceptance at the recipe temp ⇒ 58.9 tok/s; 52.1 % at temp 0.8 ⇒ 66.1). +`reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 Jinja raise +(the models.yml fold stays mandatory). **Vision without a Swift projector**: no Swift repo ships one (ukisai's README is explicit), so the projector is unsloth's mmproj built from the diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md index 3a37609..ad0748c 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV" -description: "The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum." +title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 2 x 131072, no spec, q8_0 KV" +description: "The active frank-small runbook (2026-10-05 evening; 2 × 131,072 no-spec since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap (the tight fit he ordered — 1,397 MiB measured free), q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the full slot-2 arithmetic." tags: - llm - frank-small @@ -15,16 +15,21 @@ tags: timestamp: 2026-10-05 --- -# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV +# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 2 × 131,072, no spec, q8_0 KV -> **ACTIVE since 2026-10-05 (evening).** A 27B replaces the 9B on the same -> 24 GiB card, bought by Frank's explicit contract for this swap: context -> capped at 128k per conversation, KV at FP8 class, vision on, "as many of -> those contexts as you can" — the answer turned out to be **one**, by -> arithmetic, not by choice (see *Sizing*). This supersedes BOTH earlier -> 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape -> ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the Ornith-1.5-9B -> Q8_0 swap that was committed (77decb5..380aafa) but never deployed. +> **ACTIVE since 2026-10-05 (evening); TWO slots, spec OFF since +> 2026-10-06.** A 27B replaces the 9B on the same 24 GiB card, bought by +> Frank's explicit contract for the swap: context capped at 128k per +> conversation, KV at FP8 class, vision on, "as many of those contexts as +> you can" — the 2026-10-05 arithmetic answered **one**; Frank's 2026-10-06 +> order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 contexts at +> 128k — should be a very tight fit") bought the second slot by spending +> the drafter (~0.4 GiB) and ubatch headroom (~0.3 GiB). It fits: +> **1,397 MiB measured free**, the old 2-3 GiB floor knowingly violated. +> This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K +> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the +> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never +> deployed. ## Identity @@ -38,7 +43,7 @@ timestamp: 2026-10-05 | Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) | | Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) | | Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy | -| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out) | +| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, no-spec pair (absence of both flags), batch pair, projector opt-out) | ## Why a base-model projector is legal here @@ -56,9 +61,11 @@ merge 2, ffn 4304, 16 heads, out 5120). ## Shape and KV contract -`--ctx-size 131072 --parallel 1` — ONE conversation at Frank's 128k cap -(the GSQ publisher's own benchmark protocol ran at 131,072). `CTX_SIZE` -stays divisible by `PARALLEL`; the seam enforces it. +`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap +(the GSQ publisher's own benchmark protocol ran at 131,072 per stream). +`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the +per-slot number (`262144 / 2 = 131,072`) is the product contract — +models.yml stays at 131,072. **KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).** llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem @@ -71,28 +78,46 @@ pinned: quantized V requires it, and the build's `GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a silent f16 fallback. -**One slot is the maximum, by arithmetic.** Card 24,576 MiB; weights -11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card -law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet -state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP -draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒ -free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under -the 2–3 GiB floor. The pool is capped by the 128k law, not by the card — -leftover headroom is honest slack, not an undersized pool. +**Two slots, by arithmetic and by order (2026-10-06).** The 2026-10-05 +arithmetic made ONE 131,072 slot the maximum under the 2-3 GiB floor (a +second slot wanted +4,352 MiB of KV against 4,721 free — ~0.2 GiB left, +far under the floor). Frank then ordered the budget spent differently: +drop the MTP drafter (~0.4 GiB: one nextn layer of draft KV + its compute +buffers), drop ubatch 1024→512 (~0.3 GiB, inferred from the measured 616 +MiB/doubling rung on the 9B), and take the second slot anyway — "should +be a very tight fit", i.e. the 2-3 GiB floor is KNOWINGLY VIOLATED. The +measured boot confirms: **22,728 used / 1,397 MiB free** (nvidia-smi +`free` field), and a 512×512 image still answers `Red` afterwards +(1,363 MiB after the probe), so the fit carries the vision transients. +Compute buffers allocate at graph capture during BOOT, so a miss is loud +and fast — the failure mode is a crash loop, not silent corruption. -## MTP (speculative decoding) +## Speculative decoding — OFF since 2026-10-06 -The MTP head ships **in the file** — the `-mtp` GGUF carries -`blk.64.nextn.{enorm,hnorm,shared_head_norm}` + the head layer's tensors at -Q6_K/F32 (publisher RECIPE: "MTP_OK verified"), -`qwen35.nextn_predict_layers=1`. `--spec-type draft-mtp` loads it against -the main model's own weights — no `--model-draft`, no second draft KV pool -(an MTP head can NEVER load as a separate draft model on this stack; two -context-backed drafters cannot share a process). Depth 3 -(`--spec-draft-n-max 3`) matches the fleet's radiance MTP depth. The flags -move as a set: a no-spec baseline must strip BOTH. Acceptance reads from -the response `timings` object (`draft_n`, `draft_n_accepted`) — measure on -real traffic, never on a synthetic repeat (greedy-text acceptance lies). +**Frank's order: drop MTP, skip ngram-mod too.** The MTP head ships IN the +file — the `-mtp` GGUF carries `blk.64.nextn.{enorm,hnorm,shared_head_norm}` +(`qwen35.nextn_predict_layers=1`, publisher RECIPE "MTP_OK verified") — and +`--spec-type draft-mtp` loads it against the main model's own weights (no +`--model-draft`, no second draft model — two context-backed drafters cannot +share a process). It simply is NOT drafted with now: `SPEC_TYPE=none` +strips BOTH speculative flags in the serve script (flags move as a set). + +**Why ngram-mod was skipped, not swapped in:** on a qwen35 hybrid, ngram +speculation logs *"recurrent state rollback is not compatible with +'ngram-mod' — disabling rollback support"* (fleet lesson from frank-large's +chain measurement, 2026-10-03). A rejected draft on Gated DeltaNet layers +cannot be rewound by KV truncation — without the rollback checkpoints the +DeltaNet state carries phantom transitions and output degrades silently. +MTP measured fine here (below); it was spent for the second slot, not +abandoned for ngram. + +**Measured MTP references (2026-10-05, for the day the head returns)**, +draft counts from the live `timings` on real-ish traffic: temp 0.8 → +66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318; +deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355 +(44.8 %). Acceptance reads from the response `timings` object +(`draft_n`, `draft_n_accepted`) — never on a synthetic repeat (greedy-text +acceptance lies). ## Vision contract @@ -102,7 +127,7 @@ probe undersupplies image tokens and the model invents ("video"); the legacy top-level `images` field is IGNORED, only the OpenAI `image_url` content part reaches the model. Same traps the MiMo eras documented. -## Measured at cutover (2026-10-05, live) +## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical) - **Boot ledger**: `creating MTP draft context against the target model '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal @@ -124,6 +149,22 @@ content part reaches the model. Same traps the MiMo eras documented. - Streaming completion: 200. No first-request silence window (the halogen-era 45–60 s stall has no analogue here). +## Measured: the 2 × 131,072 no-spec boot (2026-10-06, live) + +- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/ + mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation + off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144). +- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the + predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly + violated by Frank's order. After a 512×512 image probe: 1,363 MiB — + the vision transients fit. +- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed + recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP + era's 58.9–66.9 tok/s is the honest price of the second context: + −18–28 %. +- **Vision**: 512×512 solid-red PNG → `Red` again on this shape. +- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet. + ## Client contract - Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking @@ -147,6 +188,11 @@ content part reaches the model. Same traps the MiMo eras documented. ## Rollback +The 1 × 131,072 MTP shape is ONE commit away — revert 812d9eb (the +two-slot order) and redeploy: the same env trio then ships +`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, +`PARALLEL=1` (the measured 58.9–66.9 tok/s with the head drafting). + Two payloads on disk, both fetchable by the same script's git-history recipes: @@ -177,3 +223,7 @@ recipes: - The unit's `Description=` (MiMo era) was stale at cutover time and was updated in-repo afterward — the relinked unit picks it up at the next deploy/restart, not retroactively. +- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts + (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode + 48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch + ~0.3 GiB bought the second 4,352 MiB KV slot. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 2ea61ee..52c60f6 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -16,9 +16,10 @@ ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision lives on frank-small)** and **llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B -GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072 -conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via -the base-model projector, on the RTX 3090)**. +GSQ-RCO IQ3_S-mtp GGUF — 27B (head present, NOT drafted), TWO 131,072 +conversations at Frank's 128k cap (tight fit, spec off since 2026-10-06), +q8_0 K + q8_0 V (FP8 class), vision via the base-model projector, on the +RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -81,14 +82,18 @@ frank-small, port 7771] Rollback: revert the swap commit; the ROCmFPX payload + fork build + `build-rocmfpx.sh` pin all remain on disk - [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05 - (evening)** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) - serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B, the - checkpoint's own MTP head IN the file — `--spec-type draft-mtp`, depth 3) - on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a - second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8 - class, 34 KiB/token vs 64 at f16) and vision via unsloth's - mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base - vision stack was hash-proven byte-identical to Swift's (6/6 tensors). + (evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit, + port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5- + Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B; the -mtp file's head is + present but NOT drafted — speculation OFF by Frank's order, ngram skipped + on the recurrent-rollback law) on the RTX 3090, TWO 131,072-token + conversations at Frank's 128k cap (pool 262144 / parallel 2 — the tight + fit he ordered: **1,397 MiB measured free**, the 2-3 GiB floor knowingly + violated) with q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16), + ubatch 512, and vision via unsloth's mmproj-Qwen3.8-27B-F16.gguf — no + Swift projector ships, and the base vision stack was hash-proven + byte-identical to Swift's (6/6 tensors). Decode 48.0 t/s without the + drafter (MTP measured 58.9–66.9; references kept in frank-small.env). The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types. Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index ed3bd54..4742880 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -370,11 +370,13 @@ The LLM inference fleet on the two hosts. Full index: - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) - [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small - runbook since 2026-10-05 (evening): llama.cpp master (CUDA sm_86) serving - ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on - the RTX 3090 — ONE 131,072-token conversation at Frank's 128k cap, q8_0 - K + q8_0 V, vision through the base-model projector; the arithmetic that - makes one slot the maximum + runbook since 2026-10-05 (evening; **2 × 131,072 no-spec since + 2026-10-06**): llama.cpp master (CUDA sm_86) serving ukisai's + Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT + drafted — speculation off) on the RTX 3090 — TWO 131,072-token + conversations at Frank's 128k cap, the tight fit he ordered (1,397 MiB + measured free), q8_0 K + q8_0 V, vision through the base-model + projector, ~48 t/s without the drafter - [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening, superseded by swift15): llama.cpp master serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at