frank-primary: record measured battery numbers and the drafter-placement law
Battery 6/6: alias, reasoning_content, tool_calls, vision (probe image described exactly), reasoning_effort high -> 200. Decode 11.8 t/s at depth-0 prose (draft 109/accepted 72), 0.864 mean len 2.73 on tool traffic. VRAM 29%/36% (output-head card carries the drafter). Also records the ROCm1 drafter law in the runbook body (borrowed output.weight + n_copies==1 abort) and the two cutover traps: foreign-process health false green, and llama-server SIGTERM cleanup hang.
This commit is contained in:
@@ -72,7 +72,14 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|
||||
draft load with `invalid vector subscript`). `ngram-mod` is draftless,
|
||||
so it chains legally behind the drafter. Depth 2: each longer
|
||||
verification round re-batches the CPU expert reads, so depth buys less
|
||||
than bandwidth here.
|
||||
than bandwidth here. **The drafter must sit on ROCm1** — the head also
|
||||
borrows the target's `output.weight`, which a `-ts 1,1` split places on
|
||||
the last-layers card; with tensor overrides set the draft scheduler runs
|
||||
`n_copies==1`, and a draft context elsewhere aborts at load
|
||||
(`ggml-backend.cpp:941`, "pre-allocated tensor (output.weight) in a
|
||||
buffer (ROCm1) that cannot run the operation"). The 2026-09-26 cutover
|
||||
crash-looped 4× on `ROCm0` before the probe confirmed `ROCm1` loads —
|
||||
the frank-medium precedent (drafter on the last-layers card) was right.
|
||||
- **Vision, explicitly.** This build never autoloads projectors — without
|
||||
`--mmproj` the server comes up healthy and text-only and answers image
|
||||
requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector
|
||||
@@ -118,7 +125,23 @@ truly free, relinks + enables + restarts the root unit, and waits up to
|
||||
- VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long
|
||||
completion.
|
||||
|
||||
_Measured numbers land here after the deploy battery._
|
||||
_**Measured (2026-09-26 battery, 6/6 pass).** Load: ~7 s warm (mmap from
|
||||
page cache; the fetch's own verify pass pre-warms it — a cold first load
|
||||
pages in 175 GiB). Decode: **11.8 t/s** `predicted_per_second` at
|
||||
depth-0 prose, 128-token generation, draft_n 109 / accepted 72 (0.66 on
|
||||
reasoning-heavy prose; **0.864 mean len 2.73** on the tool-call task).
|
||||
Vision: the probe image (green square centered on a red field) described
|
||||
exactly. VRAM: GPU0 29% / GPU1 36% of 32 GiB (~9.3 / ~11.5 GiB — the
|
||||
output-head card carries the drafter, as designed; ~22 GiB free per
|
||||
card). RAM: ~11 GiB used, ~226 GiB page cache (the payload itself),
|
||||
available ~235 of 246 GiB. `reasoning_effort: high` → 200. Two cutover
|
||||
traps for the record: a LAST-ACK tail from a LAN client held
|
||||
`wait_port_free` until it cleared (a foreign process on the port gates
|
||||
the deploy correctly — but the deployer's `/health` wait cannot tell WHO
|
||||
answers; after any aborted deploy, confirm the responder is the unit's
|
||||
own MainPID), and llama-server's SIGTERM cleanup hung after
|
||||
spec-decode service (logged `cleaning up before exit` and never exited —
|
||||
`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
|
||||
|
||||
## Rollback
|
||||
|
||||
|
||||
Reference in New Issue
Block a user