docs: frank-secondary at ubatch 2048 — Measured section, rollback ladder, fleet docs
23,120 used / 1,005 free at boot, 763 settled after a vision probe (encoder buffers resident). Rung cost 746 MiB, not the transferable 616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode unchanged as the law says. 2048 = ceiling for this shape.
This commit is contained in:
Frank
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
|
||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 114,688** (Frank's 112k cap each since the 2026-10-06 late reshape, cut from 131,072 to fund **ubatch 1024** — **1,751 MiB measured free**) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024; **58.2 t/s recipe decode / 66.9 (temp 0)** (MTP acceptance 42.8–54.4 %). Renamed from `frank-small` 2026-10-06 (87721ab). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 114,688** (Frank's 112k cap each since the 2026-10-06 late reshape, cut from 131,072 to fund the **ubatch ladder 512 → 1024 → 2048** — **1,005 MiB free at boot, 763 settled after a vision probe** (encoder buffers resident)) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / **ubatch 2048** (the ceiling for this shape — 4096 would not fit); **prefill 1,209 t/s @ 3,760 tokens; ~52–58 t/s recipe decode** (MTP acceptance 34–54 %, temp-1.0 noise; decode is ubatch-invariant). Renamed from `frank-small` 2026-10-06 (87721ab). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -310,7 +310,7 @@ are the engine core; `--mm-max-patches` available),
|
||||
and `frank-secondary` through
|
||||
the publisher's own `mmproj-BF16.gguf` passed as `--mmproj` — the Swift
|
||||
GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is
|
||||
**114688** (cut from 131072 on 2026-10-06 late to fund ubatch 1024), the
|
||||
**114688** (cut from 131072 on 2026-10-06 late to fund the ubatch ladder to 2048), the
|
||||
per-slot window, not the model's native 262144: the harness must compact
|
||||
before llama.cpp answers a longer prompt with a 400. Note also
|
||||
that
|
||||
@@ -426,7 +426,8 @@ context cap, KV at FP8, vision on, as many 128k contexts as fit**;
|
||||
reshaped three times on 2026-10-06 — morning to two 131,072 slots
|
||||
no-spec, evening to this UNCERSED RCO-IQ3_XXS payload with MTP back
|
||||
on, late the RENAME to **frank-secondary** (87721ab) at two 114,688
|
||||
slots with ubatch 1024. The
|
||||
slots with the ubatch ladder climbing 512 → 1024 → 2048 (f2575a1,
|
||||
d8da7a9). The
|
||||
same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K shape and the
|
||||
Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
||||
deployed (blocked at the docs gate; the docs debt is what this section
|
||||
@@ -440,36 +441,36 @@ at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at
|
||||
48 linear layers × 48 value heads). Header facts were read from the Hub
|
||||
file by HTTP range + struct parse before anything was downloaded.
|
||||
|
||||
**The shape (2026-10-06 late; Frank's tight-fit order, MTP back on in
|
||||
the evening, per-slot cut + ubatch raised late):** `--ctx-size 229376
|
||||
--parallel 2` = **TWO 114,688-token conversations** (Frank's 112k cap
|
||||
each; `n_ctx_slot = CTX_SIZE / PARALLEL`; cut from 131,072 to fund
|
||||
ubatch 1024), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's
|
||||
FP8 class — no true fp8 KV dtype exists; 8.5 effective bits; supersedes
|
||||
the same-day f16/f16 rule written for the Ornith 9B fit), all layers on
|
||||
the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024. **Speculative
|
||||
decoding is ON** (draft-mtp, depth 3): the -mtp file's head is drafted
|
||||
against the main model's own weights; ngram-mod skipped on purpose — on
|
||||
a qwen35 hybrid ngram disables recurrent-state rollback, so a rejected
|
||||
draft corrupts the DeltaNet state.
|
||||
**The shape (2026-10-06 midnight; Frank's tight-fit order, MTP back on
|
||||
in the evening, per-slot cut + ubatch climbed to 2048):** `--ctx-size
|
||||
229376 --parallel 2` = **TWO 114,688-token conversations** (Frank's
|
||||
112k cap each; `n_ctx_slot = CTX_SIZE / PARALLEL`; cut from 131,072 to
|
||||
fund the ubatch rungs), `--flash-attn on` pinned, **q8_0 K + q8_0 V**
|
||||
(llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective
|
||||
bits; supersedes the same-day f16/f16 rule written for the Ornith 9B
|
||||
fit), all layers on the GPU, `--mmproj` for vision, batch 4096 /
|
||||
ubatch 2048. **Speculative decoding is ON** (draft-mtp, depth 3): the
|
||||
-mtp file's head is drafted against the main model's own weights;
|
||||
ngram-mod skipped on purpose — on a qwen35 hybrid ngram disables
|
||||
recurrent-state rollback, so a rejected draft corrupts the DeltaNet
|
||||
state.
|
||||
|
||||
**Measured at the late boot (2026-10-06, 2 × 114,688 / ubatch 1024):
|
||||
22,374 used / 1,751 MiB free** (`nvidia-smi` free field) — the per-slot
|
||||
cut freed ~1.05 GiB of KV, which paid for the ubatch-1024 buffers and
|
||||
netted ~1.25 GiB back over the evening's tightest-ever fit (23,636 used
|
||||
/ 489 MiB free at 2 × 131,072 / ubatch 512, under the old 2-3 GiB floor
|
||||
BY ORDER; a 512×512 solid-red image still answered `Red` there, so the
|
||||
fit carries the vision transients). The MTP draft ctx costs **~2.4 GiB
|
||||
**Measured at the midnight boot (2026-10-06, 2 × 114,688 / ubatch
|
||||
2048): 23,120 used / 1,005 MiB free, settling 763 MiB after a vision
|
||||
probe** (`nvidia-smi` free; the encoder buffers stay resident on this
|
||||
boot) — the per-slot cut freed ~1.05 GiB of KV and both ubatch rungs
|
||||
spent it back (~0.3 + ~0.75 GiB measured; the 1024 rung had landed
|
||||
22,374/1,751, the 2048 rung cost 746 MiB, ~130 over the 616 the rung
|
||||
measured on the MiMo shape). The MTP draft ctx costs **~2.4 GiB
|
||||
measured** (23,636 − 22,728 + 1,561 at the 262,144 draft window) — ~6×
|
||||
the ~0.4 GiB the one-slot era assumed, because the drafter's KV is
|
||||
built against the target's FULL context, not the 3-token proposal
|
||||
depth; the lighter RCO-IQ3_XXS quant (−1.56 GiB vs IQ3_S-mtp) funds it.
|
||||
**Decode at the late boot: 58.2 tok/s (recipe temp 1.0) / 66.9 (temp
|
||||
0)**, acceptance 42.8–54.4 % (the 2 × 131,072 boot read 63.3 / 72.9 at
|
||||
49.8–62.4 % — acceptance is text-dependent and the prompts differ: the
|
||||
comparison is confounded, not a shape verdict). `reasoning_effort`
|
||||
probe unchanged: `xhigh` 200, `high` → 500 Jinja raise (the models.yml
|
||||
fold stays mandatory).
|
||||
**Prefill 1,209 t/s @ 3,760 tokens; decode 52.0–58.2 tok/s (recipe
|
||||
temp 1.0) / 66.9 (temp 0)**, acceptance 34.2–54.4 % — decode is
|
||||
ubatch-invariant, the spread is temp-1.0 acceptance noise.
|
||||
`reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 Jinja
|
||||
raise (the models.yml fold stays mandatory).
|
||||
|
||||
**Vision with the publisher's own projector**: ajgazin's repo SHIPS
|
||||
`mmproj-BF16.gguf` in the same repo (the IQ3_S era's unsloth mmproj was
|
||||
|
||||
@@ -27,9 +27,10 @@ timestamp: 2026-10-06
|
||||
> GiB assumed** (the drafter's KV is built against the target's full
|
||||
> context, not the 3-token proposal depth), paid for by the lighter
|
||||
> quant (−1.56 GiB). The late order dropped the per-slot window
|
||||
> 131,072 → **114,688** (Frank's 112k cap) and raised ubatch 512 →
|
||||
> 1024 — landing **1,751 MiB free**, back above the morning's number
|
||||
> with the prefill knob raised. This supersedes BOTH earlier 2026-10-05
|
||||
> 131,072 → **114,688** (Frank's 112k cap) and climbed the prefill ladder
|
||||
> ubatch 512 → 1024 → **2048** — landing **1,005 MiB free** (1,751 at
|
||||
> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
|
||||
> measured prediction). This supersedes BOTH earlier 2026-10-05
|
||||
> payloads: the live MiMo-V2.6 Q6_K shape
|
||||
> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
@@ -93,13 +94,15 @@ field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor
|
||||
knowingly violated, and the 512×512 vision transient (34 MiB) still fits
|
||||
and recovers.
|
||||
|
||||
**The late reshape (2 × 114,688, ubatch 1024) is the current fit.** The
|
||||
per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; ubatch
|
||||
512 → 1024 spends part of it on compute buffers (a prefill knob — decode
|
||||
never moves with it). Measured boot: **22,374 used / 1,751 MiB free**.
|
||||
The drafter's own KV shrank with its target (draft window 229,376).
|
||||
Compute buffers allocate at graph capture during BOOT, so a miss is loud
|
||||
and fast — the failure mode is a crash loop, not silent corruption.
|
||||
**The late reshape (2 × 114,688, ubatch 2048) is the current fit.** The
|
||||
per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; the
|
||||
ubatch rungs 512 → 1024 → 2048 spend it back on compute buffers (a
|
||||
prefill knob — decode never moves with it; the rungs cost ~0.3 and
|
||||
~0.75 GiB measured). Measured boot at 2048: **23,120 used / 1,005 MiB
|
||||
free**. The drafter's own KV shrank with its target (draft window
|
||||
229,376). Compute buffers allocate at graph capture during BOOT, so a
|
||||
miss is loud and fast — the failure mode is a crash loop, not silent
|
||||
corruption.
|
||||
|
||||
## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
|
||||
|
||||
@@ -193,7 +196,7 @@ leave room for thinking.
|
||||
slot prefix cache is live.
|
||||
- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, live)
|
||||
## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, historical)
|
||||
|
||||
- **argv** (from `/proc/<MainPID>/cmdline`): `--ctx-size 229376
|
||||
--parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size
|
||||
@@ -216,6 +219,29 @@ leave room for thinking.
|
||||
`frank-secondary`, config.yml modelRoles rewritten; per-slot
|
||||
`contextWindow: 114688`).
|
||||
|
||||
## Measured: ubatch 2048 (2026-10-06 midnight, live)
|
||||
|
||||
- **argv**: `--ctx-size 229376 --parallel 2 --batch-size 4096
|
||||
--ubatch-size 2048 --spec-type draft-mtp` (commit d8da7a9, Frank's
|
||||
"set ubatch to 2k"; deploy green, e2e gate 33/33).
|
||||
- **VRAM: 23,120 used / 1,005 MiB free at boot** — the 1024 → 2048 rung
|
||||
cost **746 MiB**, ~130 over the 616 MiB the rung measured on
|
||||
2026-10-05 (rung cost is shape-dependent, not a constant). The
|
||||
512×512 solid-red vision probe (`Red`) then took ~240 MiB **and kept
|
||||
it**: settled **763 MiB free** — on this boot the vision encoder's
|
||||
buffers stay resident after the request (the evening boot saw its
|
||||
34 MiB transient return).
|
||||
- **Decode (ubatch-invariant knob, same prompt as the 1024 rung)**:
|
||||
recipe → 52.0 t/s (388/1133, 34.2 %) vs 58.2 t/s / 42.8 % at 1024 —
|
||||
acceptance variance at temp 1.0, not a ubatch effect.
|
||||
- **Prefill (THE ubatch knob)**: **1,209 t/s @ 3,760 tokens** — first
|
||||
recorded number for this shape; future rungs compare against this
|
||||
(no same-prompt 1024 baseline exists; the law predicts ≈ +4.6 %
|
||||
1024 → 2048).
|
||||
- **Headroom left ~0.7–1.0 GiB.** ubatch 4096 (= the logical batch, no
|
||||
micro-splitting) doubles the buffer growth again — not affordable;
|
||||
**2048 is the ceiling for this shape.**
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
|
||||
@@ -239,9 +265,9 @@ leave room for thinking.
|
||||
|
||||
## Rollback
|
||||
|
||||
**The immediate predecessor shape (2 × 131,072, ubatch 512, same
|
||||
payload) is ONE revert away** — revert f2575a1 and redeploy
|
||||
(`CTX_SIZE=262144`, `UBATCH_SIZE=512`). The 2 × 131,072 no-spec
|
||||
**Predecessor shape (same 2 × 114,688, ubatch 1024) is ONE revert
|
||||
away** — revert d8da7a9 and redeploy. The 2 × 131,072 / ubatch-512
|
||||
era is one revert further (f2575a1). The 2 × 131,072 no-spec
|
||||
IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env
|
||||
trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`,
|
||||
`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the
|
||||
@@ -303,3 +329,10 @@ recipes:
|
||||
frank-small`, then `deploy.sh dankclaw --service frank-secondary`.
|
||||
1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
|
||||
config.yml retargeted.
|
||||
- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
|
||||
(d8da7a9). Cost 746 MiB: 23,120 used / 1,005 free at boot, settling
|
||||
**763 MiB free after a vision probe** (encoder buffers stay resident
|
||||
on this boot). Decode 52.0 t/s recipe on the shared prompt (58.2 at
|
||||
the 1024 rung — acceptance noise, decode is ubatch-invariant);
|
||||
prefill 1,209 t/s @ 3,760 tokens (first recorded baseline). 2048 is
|
||||
the ceiling for this shape; 4096 would not fit.
|
||||
|
||||
@@ -106,17 +106,19 @@ frank-secondary, port 7771]
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **ACTIVE since
|
||||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||||
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
|
||||
114,688 / ubatch 1024 late 2026-10-06** (root systemd unit, port
|
||||
114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
|
||||
systemd unit, port
|
||||
7771): llama.cpp master
|
||||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||||
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
|
||||
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
|
||||
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
|
||||
229376 / parallel 2 — cut from 131,072 to fund ubatch 1024; **1,751
|
||||
MiB measured free**) with
|
||||
229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
|
||||
**1,005 MiB free at boot, 763 settled after a vision probe**) with
|
||||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
|
||||
vision through the publisher's own mmproj-BF16.gguf. Decode 58.2 t/s
|
||||
recipe / 66.9 temp 0 (MTP acceptance 42.8–54.4 %).
|
||||
vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
|
||||
t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
|
||||
noise 34–54 %).
|
||||
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
|
||||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
||||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||||
|
||||
@@ -376,11 +376,11 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
|
||||
with MTP back on since 2026-10-06 evening; renamed from frank-small and
|
||||
reshaped to TWO 114,688-token conversations at Frank's 112k cap with
|
||||
ubatch 1024 late 2026-10-06**): llama.cpp master (CUDA
|
||||
the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA
|
||||
sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
|
||||
(the MTP head in the file is drafted) on the RTX 3090 — 1,751 MiB
|
||||
measured free, q8_0 K + q8_0 V, vision
|
||||
through the publisher's own projector, ~58 t/s recipe decode with MTP
|
||||
(the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free
|
||||
at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision
|
||||
through the publisher's own projector, ~52–58 t/s recipe decode with MTP
|
||||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
superseded by swift15): llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
|
||||
|
||||
@@ -2,6 +2,23 @@
|
||||
|
||||
|
||||
## 2026-10-06
|
||||
- **frank-secondary (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
|
||||
RTX 3090).** "set ubatch to 2k" → commit d8da7a9: `UBATCH_SIZE`
|
||||
1024 → 2048 at the 2×114,688 shape. **Cost 746 MiB**, not the 616 MiB
|
||||
the same rung measured on the MiMo shape on 2026-10-05 (a rung's price
|
||||
is shape-dependent; the 616 figure was not transferable). Boot:
|
||||
**23,120 used / 1,005 MiB free**; a 512×512 vision probe then took
|
||||
~240 MiB **and kept it** — settled **763 MiB free**, the encoder's
|
||||
buffers staying resident across requests on this build, unlike the
|
||||
evening boot where the 34 MiB transient returned. Decode 52.0 t/s on
|
||||
the shared prompt (58.2 at the 1024 rung) with acceptance 34.2 % vs
|
||||
42.8 %: ubatch does not move decode, so the gap is temp-1.0
|
||||
acceptance noise, not a regression. **Prefill 1,209 t/s @ 3,760
|
||||
tokens** — the first prefill number this endpoint has recorded, and
|
||||
now the baseline future rungs compare against. `BATCH_SIZE` stays
|
||||
4096, so the seam's batch ≥ ubatch guard holds; **2048 is the ceiling
|
||||
for this shape** (4096 = no micro-split doubles the buffer growth
|
||||
again against 0.7–1.0 GiB of headroom). e2e gate 33/33 green.
|
||||
- **frank-secondary (Dankclaw): RENAMED from frank-small, then reshaped
|
||||
to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090).** The rename (commit
|
||||
87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers
|
||||
|
||||
Reference in New Issue
Block a user