dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-secondary at ubatch 2048 — Measured section, rollback ladder, fleet docs

Browse Source 
23,120 used / 1,005 free at boot, 763 settled after a vision probe
(encoder buffers resident). Rung cost 746 MiB, not the transferable
616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode
unchanged as the law says. 2048 = ceiling for this shape.
This commit is contained in:
Frank
2026-10-06 18:40:26 -07:00
parent d8da7a9eb7
commit 41d2f56269
5 changed files with 104 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md
+47 -14
View File
@@ -27,9 +27,10 @@ timestamp: 2026-10-06
> GiB assumed** (the drafter's KV is built against the target's full
> context, not the 3-token proposal depth), paid for by the lighter
> quant (−1.56 GiB). The late order dropped the per-slot window
> 131,072 → **114,688** (Frank's 112k cap) and raised ubatch 512 →
> 1024 — landing **1,751 MiB free**, back above the morning's number
> with the prefill knob raised. This supersedes BOTH earlier 2026-10-05
> 131,072 → **114,688** (Frank's 112k cap) and climbed the prefill ladder
> ubatch 512 → 1024 → **2048** — landing **1,005 MiB free** (1,751 at
> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
> measured prediction). This supersedes BOTH earlier 2026-10-05
> payloads: the live MiMo-V2.6 Q6_K shape
> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
@@ -93,13 +94,15 @@ field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor
knowingly violated, and the 512×512 vision transient (34 MiB) still fits
and recovers.
**The late reshape (2 × 114,688, ubatch 1024) is the current fit.** The
per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; ubatch
512 → 1024 spends part of it on compute buffers (a prefill knob — decode
never moves with it). Measured boot: **22,374 used / 1,751 MiB free**.
The drafter's own KV shrank with its target (draft window 229,376).
Compute buffers allocate at graph capture during BOOT, so a miss is loud
and fast — the failure mode is a crash loop, not silent corruption.
**The late reshape (2 × 114,688, ubatch 2048) is the current fit.** The
per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; the
ubatch rungs 512 → 1024 → 2048 spend it back on compute buffers (a
prefill knob — decode never moves with it; the rungs cost ~0.3 and
~0.75 GiB measured). Measured boot at 2048: **23,120 used / 1,005 MiB
free**. The drafter's own KV shrank with its target (draft window
229,376). Compute buffers allocate at graph capture during BOOT, so a
miss is loud and fast — the failure mode is a crash loop, not silent
corruption.
## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
@@ -193,7 +196,7 @@ leave room for thinking.
slot prefix cache is live.
- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, live)
## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, historical)
- **argv** (from `/proc/<MainPID>/cmdline`): `--ctx-size 229376
--parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size
@@ -216,6 +219,29 @@ leave room for thinking.
`frank-secondary`, config.yml modelRoles rewritten; per-slot
`contextWindow: 114688`).
## Measured: ubatch 2048 (2026-10-06 midnight, live)
- **argv**: `--ctx-size 229376 --parallel 2 --batch-size 4096
--ubatch-size 2048 --spec-type draft-mtp` (commit d8da7a9, Frank's
"set ubatch to 2k"; deploy green, e2e gate 33/33).
- **VRAM: 23,120 used / 1,005 MiB free at boot** — the 1024 → 2048 rung
cost **746 MiB**, ~130 over the 616 MiB the rung measured on
2026-10-05 (rung cost is shape-dependent, not a constant). The
512×512 solid-red vision probe (`Red`) then took ~240 MiB **and kept
it**: settled **763 MiB free** — on this boot the vision encoder's
buffers stay resident after the request (the evening boot saw its
34 MiB transient return).
- **Decode (ubatch-invariant knob, same prompt as the 1024 rung)**:
recipe → 52.0 t/s (388/1133, 34.2 %) vs 58.2 t/s / 42.8 % at 1024 —
acceptance variance at temp 1.0, not a ubatch effect.
- **Prefill (THE ubatch knob)**: **1,209 t/s @ 3,760 tokens** — first
recorded number for this shape; future rungs compare against this
(no same-prompt 1024 baseline exists; the law predicts ≈ +4.6 %
1024 → 2048).
- **Headroom left ~0.7–1.0 GiB.** ubatch 4096 (= the logical batch, no
micro-splitting) doubles the buffer growth again — not affordable;
**2048 is the ceiling for this shape.**
## Client contract
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
@@ -239,9 +265,9 @@ leave room for thinking.
## Rollback
**The immediate predecessor shape (2 × 131,072, ubatch 512, same
payload) is ONE revert away** — revert f2575a1 and redeploy
(`CTX_SIZE=262144`, `UBATCH_SIZE=512`). The 2 × 131,072 no-spec
**Predecessor shape (same 2 × 114,688, ubatch 1024) is ONE revert
away** — revert d8da7a9 and redeploy. The 2 × 131,072 / ubatch-512
era is one revert further (f2575a1). The 2 × 131,072 no-spec
IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env
trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`,
`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the
@@ -303,3 +329,10 @@ recipes:
frank-small`, then `deploy.sh dankclaw --service frank-secondary`.
1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
config.yml retargeted.
- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
(d8da7a9). Cost 746 MiB: 23,120 used / 1,005 free at boot, settling
**763 MiB free after a vision probe** (encoder buffers stay resident
on this boot). Decode 52.0 t/s recipe on the shared prompt (58.2 at
the 1024 rung — acceptance noise, decode is ubatch-invariant);
prefill 1,209 t/s @ 3,760 tokens (first recorded baseline). 2048 is
the ceiling for this shape; 4096 would not fit.
serverAppConfigs-docs/LLM/index.md
+7 -5
View File
@@ -106,17 +106,19 @@ frank-secondary, port 7771]
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **ACTIVE since
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
114,688 / ubatch 1024 late 2026-10-06** (root systemd unit, port
114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
systemd unit, port
7771): llama.cpp master
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
229376 / parallel 2 — cut from 131,072 to fund ubatch 1024; **1,751
MiB measured free**) with
229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
**1,005 MiB free at boot, 763 settled after a vision probe**) with
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
vision through the publisher's own mmproj-BF16.gguf. Decode 58.2 t/s
recipe / 66.9 temp 0 (MTP acceptance 42.8–54.4 %).
vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
noise 34–54 %).
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
serverAppConfigs-docs/index.md
+4 -4
View File
@@ -376,11 +376,11 @@ The LLM inference fleet on the two hosts. Full index:
frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
with MTP back on since 2026-10-06 evening; renamed from frank-small and
reshaped to TWO 114,688-token conversations at Frank's 112k cap with
ubatch 1024 late 2026-10-06**): llama.cpp master (CUDA
the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA
sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
(the MTP head in the file is drafted) on the RTX 3090 — 1,751 MiB
measured free, q8_0 K + q8_0 V, vision
through the publisher's own projector, ~58 t/s recipe decode with MTP
(the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free
at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision
through the publisher's own projector, ~52–58 t/s recipe decode with MTP
- [[frank-secondary-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
superseded by swift15): llama.cpp master serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
serverAppConfigs-docs/log.md
+17
View File
@@ -2,6 +2,23 @@
## 2026-10-06
- **frank-secondary (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
RTX 3090).** "set ubatch to 2k" → commit d8da7a9: `UBATCH_SIZE`
1024 → 2048 at the 2×114,688 shape. **Cost 746 MiB**, not the 616 MiB
the same rung measured on the MiMo shape on 2026-10-05 (a rung's price
is shape-dependent; the 616 figure was not transferable). Boot:
**23,120 used / 1,005 MiB free**; a 512×512 vision probe then took
~240 MiB **and kept it** — settled **763 MiB free**, the encoder's
buffers staying resident across requests on this build, unlike the
evening boot where the 34 MiB transient returned. Decode 52.0 t/s on
the shared prompt (58.2 at the 1024 rung) with acceptance 34.2 % vs
42.8 %: ubatch does not move decode, so the gap is temp-1.0
acceptance noise, not a regression. **Prefill 1,209 t/s @ 3,760
tokens** — the first prefill number this endpoint has recorded, and
now the baseline future rungs compare against. `BATCH_SIZE` stays
4096, so the seam's batch ≥ ubatch guard holds; **2048 is the ceiling
for this shape** (4096 = no micro-split doubles the buffer growth
again against 0.7–1.0 GiB of headroom). e2e gate 33/33 green.
- **frank-secondary (Dankclaw): RENAMED from frank-small, then reshaped
to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090).** The rename (commit
87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers