Merge origin/main: reconcile the frank-zealot reslice with the dedupe pass
main moved under this campaign — PR #25/#26 resliced the RTX 3090 endpoint. The frontmatter conflict is resolved in favour of the live shape, confirmed by re-probe the same day: `--ctx-size 602112 --parallel 3` (3 × 200,704), `--cache-type-k f16 --cache-type-v q8_0`, `--spec-type draft-mtp`, `--jinja --reasoning auto`, `--metrics`. The 262,144 figures that remain on the page are era rows and the checkpoint's trained window, not the live shape. log.md's 2026-10-19 fleet-state bullet now records both probes and the law this taught: a docs pass that reports live shapes must re-probe before writing the number, because main moving mid-campaign makes the first probe stale.
This commit is contained in:
Frank
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 2 × 262144, exact f16 KV"
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 3 × 200704, f16 K + q8_0 V, MTP armed"
|
||||
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
|
||||
tags:
|
||||
- llm
|
||||
@@ -13,10 +13,11 @@ tags:
|
||||
timestamp: 2026-10-18
|
||||
---
|
||||
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP, 2 × 262,144, exact f16 KV
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP, 3 × 200,704, f16 K + q8_0 V, MTP armed
|
||||
|
||||
Dankclaw's RTX 3090 endpoint — `http://192.168.0.143:7771`, wire id `frank-zealot`, the fleet's **compaction hub**
|
||||
(widest window it has). One page per endpoint: eras are rows below, not pages. Siblings: [[frank-archon]],
|
||||
(three conversations at 200,704 each, the window the other two endpoints also cap at). One page per
|
||||
endpoint: eras are rows below, not pages. Siblings: [[frank-archon]],
|
||||
[[frank-dragoon]]. Deploy with `./scripts/deploy.sh dankclaw --service frank-zealot`; swap mechanics in
|
||||
[[Model Swap and Rollback]], sizing math in [[Llama Cpp Serving Sizing]]. `scripts/frank-zealot.env` and
|
||||
`scripts/serve-frank-zealot.sh` are the authority for the shape; where any prose disagrees with them, the prose
|
||||
@@ -32,72 +33,87 @@ is the bug.
|
||||
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload |
|
||||
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
|
||||
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
|
||||
| Flags | `--ctx-size 524288 --parallel 2` ⇒ 262,144 per slot · `--cache-type-k f16 --cache-type-v f16` (exact, **16,384 MiB**) · `--flash-attn on` · `--batch-size 4096 --ubatch-size 2048` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · `--log-verbosity 4` · `--presence-penalty 1.0` · **`SPEC_TYPE=none`** |
|
||||
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
|
||||
|
||||
**Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block),
|
||||
`full_attention_interval 4` ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, `nextn_predict_layers 1`
|
||||
(the head **is in the file**), trained window 262,144, `general.sampling.*` 1.0/0.95/20. Only the 8 full layers bill
|
||||
per token (32,768 B at exact f16 ⇒ 16,384 MiB over the pool) and the DeltaNet layers add ~50 MiB of recurrent state
|
||||
**per slot**: exact KV costs the third conversation and nothing else.
|
||||
per token and the DeltaNet layers add ~50 MiB of recurrent state **per slot**, so the shape is a single budget
|
||||
equation at 25,088 B/token (f16 K 16,384 + q8_0 V 8,704): **14,406.00 MiB** of cache over a 602,112 pool +
|
||||
**2,352.00 MiB** of draft cache (blk.32 at its own f16 rate, billed over the WHOLE pool) + ~7.7 GiB of weights,
|
||||
projector, buffers and states, inside the 24,123 MiB the driver leaves addressable. Exact V (32,768 B/token ⇒
|
||||
18,816 MiB here) buys neither the third conversation nor the drafter; the fully quantized pair (17,408 B/token)
|
||||
buys both and costs the keys their precision.
|
||||
|
||||
**MTP is OFF by measurement, not by absence of a head.** The drafter is *good* — 2026-10-18 measured
|
||||
acceptance 0.798 / 0.617 per position, mean accepted length 2.41, and the publisher's card says depth 2 works
|
||||
best — and it still loses here, because the draft context is built against the *target's* pool: `blk.32` bills
|
||||
4,096 B/position at its default f16 ⇒ 2,048 MiB at 524,288 positions on a trunk that projects 21,234 MiB of the
|
||||
24,123 MiB addressable (that projection predicts 2,889 MiB free; replace it with the measured pair). Exact KV and a
|
||||
drafter cannot share this card. `--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's
|
||||
`--help`) would cut the draft to 1,088 MiB — legal on b10777, **never booted beside exact KV**. There is no
|
||||
`--spec-draft-model`: the MTP draft rides the main model's weights. The env carries the recipe
|
||||
(`SPEC_DRAFT_N_MAX=2`, `SPEC_DRAFT_N_MIN=0`, `SPEC_DRAFT_P_MIN=0`); `SPEC_TYPE=none` never sends it.
|
||||
**MTP is ON (Frank, 2026-10-19).** The drafter was always good — 2026-10-18 measured acceptance 0.798 / 0.617 per
|
||||
position, mean accepted length 2.41, and the publisher's card says depth 2 works best — and arming it is a
|
||||
budget decision, not a flag: the draft context is built against the *target's* pool, so `blk.32` bills 4,096
|
||||
B/position (2,352.00 MiB at 602,112) plus 480.97 MiB of its own CUDA0 buffers, and it triples the recurrent
|
||||
state (452.25 MiB for three cells against 150.75 with the drafter off). `CACHE_TYPE_V=q8_0` is what pays that:
|
||||
7,680 B/token cheaper than exact V = **4,410.00 MiB** at this pool. `UBATCH_SIZE=1024` — the rung every measured
|
||||
MTP arm here ran on — contributes only ~406 MiB at THIS shape, not the ~1,306 the 786,432-pool census showed,
|
||||
and `UBATCH_SIZE=2048` beside an armed drafter is still the shape that halved decode (167.2 → 109.2 t/s).
|
||||
`--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's `--help`) would cut the draft's bill again and stay at their default; there is no
|
||||
`--spec-draft-model` — the draft rides the main model's weights, so a draft file would load the trunk twice.
|
||||
`SPEC_TYPE=none` is the rollback and strips the whole set.
|
||||
|
||||
**Harness contract** (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw-3090)`, id `frank-zealot`):
|
||||
`contextWindow: 262144` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a
|
||||
`contextWindow: 200704` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a
|
||||
longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]`,
|
||||
`compat.reasoningContentField: reasoning_content`, `compat.reasoningEffortMap` (**load-bearing**: the template takes
|
||||
only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the
|
||||
`compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here.
|
||||
|
||||
**The exact-KV boot, measured** (2026-10-18 evening, under the unit, verbosity 4). KV **16,384.00 MiB** =
|
||||
32,768 B/token × 524,288 positions, to the MiB — rate derivation and ledger agree with no slop, which is what
|
||||
buries the doc's old "f16 needs 26,112 MiB and cannot boot" figure for good: that was the 27B's 34,816 B/token
|
||||
applied to the 9B. RS **100.50 MiB** (2 cells, one 50.25 MiB cell lighter than the q8_0 era), compute
|
||||
1,408.06 + 1,088.07 CUDA_Host, mmproj worst case **868.32 MiB** charged *after* the engine's own fit check,
|
||||
settled `nvidia-smi` **21,176 used / 2,949 free** of 24,123, `n_slots = 2`, `n_ctx_slot = 262144`,
|
||||
`DropInPaths` empty, no clamp line, no `cudaMalloc` error — within 24 MiB of the pre-flight projection. That is
|
||||
a desktop margin: the ~2 GiB of slack is what an image storm has to live inside.
|
||||
**No rates, by law.** Frank ordered the reslice, not a number: the evidence a swap needs is
|
||||
the seam's fit gate, the deployer's health gate, and the boot ledger below. The previous era's
|
||||
exact-KV boot and its rate floors live in that era's row; those floors were measured while
|
||||
another consumer drove the other slot (56 prompts in three minutes, 96 % GPU-util), the exact
|
||||
condition a bench window exists to exclude.
|
||||
|
||||
**Rates on this shape are FLOORS, not the shape's rates** — measured the same evening greedy with `ignore_eos`
|
||||
at 512 tokens, first request discarded, while another consumer drove the other slot (56 new prompts in three
|
||||
minutes, 96 % GPU-util): the exact condition the bench window exists to exclude, and one contended deep arm
|
||||
even came back truncated (a 512 KB body accepted as `prompt_n = 4`).
|
||||
**The 2026-10-19 boot, measured** — verbosity-4 ledger of `ba69ea5`, started 03:21:56 PDT.
|
||||
Recorded here because projections drift, and nowhere else:
|
||||
|
||||
| Arm | q8_0 3-slot record | f16 2-slot, contended |
|
||||
|---|---|---|
|
||||
| single-stream decode | 168.7 | **134.1 – 141.5** (145.8 and 163.2 in a lighter minute) |
|
||||
| concurrent aggregate | ≈300 across 3 (102.8 each) | **256.6 / 262.4 / 265.1** across 2 (≈128–133 each) |
|
||||
| prefill at ~106k | 2,877 at 106,418 | **3,159.5 at 106,282** |
|
||||
| decode at 106k depth | never measured | **71.5** |
|
||||
| Term | Ledger |
|
||||
|---|---|
|
||||
| Target KV over the 602,112 pool | **14,406.00 MiB** — K (f16) 9,408.00 + V (q8_0) 4,998.00, `200704 cells, 8 layers, 3/3 seqs` |
|
||||
| Draft KV (`blk.32`, its own f16 context) | **2,352.00 MiB** — 1,176.00 + 1,176.00, `200704 cells, 1 layers, 3/3 seqs`: the whole pool, again |
|
||||
| Recurrent state | **452.25 MiB** — `3 cells, 32 layers, 3 seqs, 2 rs_seq`, R (f32) 20.25 + S (f32) 432.00 |
|
||||
| Compute buffers | target 1,665.00 CUDA0 + 424.88 CUDA_Host; draft context 480.97 + 424.86 |
|
||||
| Weights | 2,283.29 CUDA0 + 257.66 CPU-mapped |
|
||||
| Projector | 595.29 model + 248.10 compute + 24.93 CPU (its 868.32 worst case is still charged AFTER the engine's fit check) |
|
||||
| Engine's own verdict | projected 21,639 vs 23,404 free, then **"cannot meet free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"** |
|
||||
| Settled | **22,870 used / 1,255 free**, `n_slots = 3`, `n_ctx_slot = 200704`, `kv_unified = false` |
|
||||
|
||||
What the floors already support: per-conversation concurrency **improved** (two streams hold ~130 t/s each where
|
||||
three held 102.8), endpoint aggregate sits ~12 % under the 3-slot record, prefill beat the record while
|
||||
contended, and single-stream decode is between parity and −20 %. The 1.882× KV byte bill is real; its true size
|
||||
on this card is still unmeasured, so quoting 134 or 163 t/s as "the" number is a guess in either direction —
|
||||
re-run the same arms with the box quiet before a figure is carried anywhere else.
|
||||
This is the first boot on this card that does **not** meet llama.cpp's own free-memory target,
|
||||
and 1,255 MiB settled sits under the ~2 GiB band every previous era held. It boots, `/health`
|
||||
answers, and the seam's gate still passes — it priced 22,428 MiB, conservative by 442 against
|
||||
this ledger. The cheapest rungs back into the band are `CTX_SIZE=401408` (two conversations) or
|
||||
`SPEC_TYPE=none`; both are one line and neither changes the payload.
|
||||
|
||||
**The drafter is engaged, measured on the same boot** (three ordinary requests, not a bench):
|
||||
acceptance 0.706 and 0.763 accepted-per-generated, mean accepted length 2.41 / 2.53,
|
||||
per-position rates (0.882, 0.765) and (0.871, 0.699) — depth 2 earning its keep at 0.3–1.5 MiB
|
||||
of draft state per saved prompt. The three probes that prove a swap: chat → `"zealot alive"`
|
||||
with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY content with
|
||||
`finish_reason: length` — the thinking-budget trap, not a fault); tools →
|
||||
`finish_reason: tool_calls` with `{"a":24,"b":7}`; vision → a 512 px solid-red PNG answers
|
||||
"Red" at `prompt_tokens` 280.
|
||||
|
||||
**Fallback rungs**, cheapest first — one line of `frank-zealot.env` each, and each must keep the seam's fit gate green:
|
||||
|
||||
| Knob | Effect |
|
||||
| Knob | Effect from the shipped shape |
|
||||
|---|---|
|
||||
| `UBATCH_SIZE=1024` | hands back 1,306 MiB of compute buffers; prefill −17 %; it is also the arm that ran the drafter at three windows (213.5 t/s) |
|
||||
| `CACHE_TYPE_V=q8_0` | exact K + quantized V: KV 16,384 → **12,544 MiB**, freeing 3,840 (llama.cpp has no true fp8 KV dtype — q8_0 is the 8-bit class) |
|
||||
| `CTX_SIZE=401408` | two 200,704 conversations, exact KV at 12,544 MiB (same 3,840 freed) — `contextWindow` MUST follow the per-slot number |
|
||||
| `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` | the era below this one: 13,056 MiB of cache, the third conversation back |
|
||||
| `CACHE_TYPE_K/V=q4_0` | 10,240 B/token ⇒ 5,120 MiB (the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704) |
|
||||
| `SPEC_TYPE=none` | the drafter off: hands back its 2,352.00 MiB of cache, 480.97 MiB of CUDA0 buffers and the 301.50 MiB of rollback planes it adds to the recurrent state (~3.1 GiB), at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room |
|
||||
| `UBATCH_SIZE=2048` | +4.6 % prefill and only ~406 MiB of compute buffers at THIS shape (the ~1,306 MiB the 786,432-pool census showed does not transfer) — and the rung that HALVED decode beside an armed drafter (167.2 → 109.2 t/s) |
|
||||
| `CTX_SIZE=401408` | two 200,704 conversations: frees 4,802 MiB of cache — `contextWindow` MUST follow the per-slot number |
|
||||
| `CACHE_TYPE_V=f16` | exact V: +4,410 MiB at this pool with the drafter armed, which is a `cudaMalloc` death the seam refuses — legal only once the drafter or a slot is gone |
|
||||
| `CACHE_TYPE_K=q8_0` | the fully quantized pair (17,408 B/token): frees 4,410 MiB and moves the quantization error into the keys |
|
||||
|
||||
## Eras
|
||||
|
||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||
|---|---|---|---|---|---|
|
||||
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | 2 × 262,144 (pool 524,288), exact f16 K+V = 16,384.00 MiB, `SPEC_TYPE=none`, batch 4096 / ubatch 2048; boot ledger KV 16,384.00 / RS 100.50 / compute 1,408.06 + 1,088.07 CUDA_Host / mmproj worst case 868.32 charged after the fit check, settled **21,176 used / 2,949 free**, `n_ctx_slot = 262144`, within 24 MiB of the projection; floors 134.1–141.5 t/s single-stream, 256.6–265.1 aggregate over two, 3,159.5 t/s prefill at 106,282, 71.5 t/s decode at 106k depth (contended — floors, not the shape's rates) | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
|
||||
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
|
||||
| 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
|
||||
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
|
||||
@@ -139,6 +155,10 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
|
||||
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
|
||||
projections drift.
|
||||
The same trap one knob down: the ubatch 2048 → 1024 step handed back **~1,306 MiB** at the 786,432 / q8_0-pair
|
||||
arm and **406 MiB** at the 602,112 / f16-K + q8_0-V boot (target compute 2,496.13 → 2,089.88, with CUDA0 going
|
||||
UP 257 while CUDA_Host went down 663). Changing the dtype or the pool moves the compute buffers too, so a
|
||||
credit measured at one shape is not a budget line at the next.
|
||||
The two boots that priced non-KV: 2 × 262,144 **without** the projector — 20,190 used, KV 12,800 ⇒ non-KV
|
||||
**7,390 MiB**; 3 × 196,608 **with** it — 22,972 used, KV 14,113 ⇒ non-KV **8,859 MiB**. The difference prices the
|
||||
vision tower at **~1.4 GiB for a 918 MB file**, because the encoder's activation and compute buffers ride with it.
|
||||
@@ -163,7 +183,7 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
|
||||
## Contract
|
||||
|
||||
- `/health` → `{"status":"ok"}`; `/props` reports `n_ctx` and `presence_penalty`; `/slots` shows occupancy; a third
|
||||
- `/health` → `{"status":"ok"}`; `/props` reports `n_ctx` and `presence_penalty`; `/slots` shows occupancy; a fourth
|
||||
concurrent conversation **waits** for a slot — the slot count is the concurrency contract.
|
||||
- `/metrics` answers 200 with 15 `llamacpp:*` families **only with `--metrics`**; without it llama.cpp answers **501**, which Prometheus reads as DOWN while `/health` stays 200. The scrape target is `host.containers.internal:7771/metrics` and the oldest retained sample of `up{job="frank-zealot"}` (2026-10-11) is already 0 — this target was never successfully scraped, across every payload era. `METRICS=0` is the documented opt-out and re-breaks it; the counters cost nothing on the request path.
|
||||
- Reasoning arrives in **`reasoning_content`** (`--reasoning-format deepseek`) and **counts against `max_tokens`**: a short budget yields empty `content` with `finish_reason: "length"`, which reads as a broken model, not a mid-thought one. The slot prefix cache is live (`cached_tokens` in `usage`).
|
||||
@@ -171,9 +191,9 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
(1.0 / 0.95 / 20). `presence_penalty` is the one recipe value with no GGUF metadata key, so it must reach argv;
|
||||
the serve script uses `${PRESENCE_PENALTY=1.0}`, deliberately **not** `:=`, because `""` is the documented opt-out.
|
||||
- Vision: only the OpenAI **`image_url`** content part reaches the model — the legacy top-level `images` field bumps `usage.prompt_tokens` while the model invents an answer. Probe with a **≥ 256 px** solid color (a postage stamp answers "video") and budget **≥ 512** tokens; a 512 px image bills ~280 prompt tokens.
|
||||
- **Q8_0 mmproj, not BF16**: the publisher warns the BF16 tower "can crash llama-server when several image requests run at once", which is exactly the `--parallel 2` shape. `MMPROJ_FILE=` (explicitly empty) is the text-only opt-out and returns ~1.4 GiB; a path at a missing file is a boot **refusal**, never silent blindness.
|
||||
- **Q8_0 mmproj, not BF16**: the publisher warns the BF16 tower "can crash llama-server when several image requests run at once", which is exactly the `--parallel 3` shape. `MMPROJ_FILE=` (explicitly empty) is the text-only opt-out and returns ~1.4 GiB; a path at a missing file is a boot **refusal**, never silent blindness.
|
||||
- Tools: native via `--jinja`; `finish_reason: tool_calls` with valid JSON arguments.
|
||||
- Two non-faults: a `400` naming 262,144 is one conversation outgrowing its **slot**, not pool exhaustion; and
|
||||
- Two non-faults: a `400` naming 200,704 is one conversation outgrowing its **slot**, not pool exhaustion; and
|
||||
the raw `/completion` path is a **dead instrument on this fork build** — a greedy `ignore_eos` run answers
|
||||
**500 "The model produced output that does not match the expected Content-only format"**, because build 10777
|
||||
guards the raw path the moment reasoning markup leaks (older builds only logged `failed to parse reasoning
|
||||
|
||||
@@ -43,9 +43,15 @@
|
||||
holds `StillDeadcode`, `rad`, `tpl.jinja` — so the MXFP4 checkpoint that two
|
||||
pages called the "on-disk rollback" is a re-fetch, now stated once, on the era
|
||||
page that owns that payload.
|
||||
- **Fleet state confirmed live, not from files**: frank-archon 3 × 200,000 fp8 KV,
|
||||
frank-dragoon 2 slots × 200,704, frank-zealot 2 × 262,144; all three answer
|
||||
`/health` 200 and every env value matches its page.
|
||||
- **Fleet state must be probed, and probed again at the end.** The first pass read
|
||||
frank-archon 3 × 200,000 fp8 KV, frank-dragoon 2 × 200,704, frank-zealot
|
||||
2 × 262,144 and every env value matched its page. `origin/main` then moved under
|
||||
the campaign — PR #25/#26 resliced frank-zealot — and the live re-probe on the
|
||||
same day says **`--ctx-size 602112 --parallel 3` ⇒ 3 × 200,704, f16 K + q8_0 V,
|
||||
`--spec-type draft-mtp`, `--jinja --reasoning`, `--metrics`**, matching
|
||||
[[frank-zealot]] exactly. **Law: a documentation pass that reports live shapes is
|
||||
stale the moment main moves; re-probe `/slots` and the running cmdline before
|
||||
writing the number, not after.**
|
||||
|
||||
## 2026-10-18
|
||||
- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).**
|
||||
|
||||
Reference in New Issue
Block a user