frank-secondary: record the deployed boot, keep the unit description honest
Deploy verified: nvidia-smi 21,436 MiB used / 2,689 free at ubatch 2048 - the MiMo-priced projection (~21,430/~2,700) landed within 11 MiB, and the verbosity-4 ledger confirms the KV law to the MiB (CUDA0 KV buffer 12,544.00 MiB = K(f16) 6,272 + V(f16) 6,272 = 32 KiB/token over the 401,408 pool), compute buffers 1,136 MiB CUDA0 + 848 MiB host, mmproj 879 + 248 MiB. Playwright gate: 33 passed. Two lessons recorded: the 203/EXEC boot loop my shebang-less rewrite caused (journal signature + the head -1 check), and that llama.cpp's "n_ctx_seq < n_ctx_train" line is informational when the context is BELOW the trained window - the Gemma4 drafter warning was the opposite case. The unit's Description named the Swift-1.5 era (and said 2x131072/q8_0/ draft-mtp); it now names the Q6_K shape. Cosmetic, so it is NOT deployed as its own restart: the next frank-secondary deploy relinks it. Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
@@ -451,13 +451,16 @@ decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
|
||||
property of the file, not of the flags. The publisher card advertises "MTP
|
||||
support" for the base architecture; the quantized export dropped the head.
|
||||
|
||||
**Measured boots** (nvidia-smi, the referee): the pricing boots are the
|
||||
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry — 2 × 200,704 f16 K +
|
||||
f16 V: **21,276 used / 2,849 free** (ubatch 1024) and **21,846 / 2,279**
|
||||
(ubatch 2048). This file is 415 MiB lighter in weights ⇒ projected ~21,430 /
|
||||
~2,700. The deployed boot ledger: see the runbook. Decode is ubatch-invariant
|
||||
(Swift-era 3090 law). The Q6_K rung costs ~10 % decode vs Q5_K_M (weight
|
||||
bandwidth; MiMo-measured 94.4 vs 104.8 t/s).
|
||||
**Measured boots** (nvidia-smi, the referee): the deployed boot is
|
||||
**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
|
||||
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
|
||||
ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
|
||||
bought back what the projection predicted. The engine ledger confirms the KV
|
||||
law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
|
||||
`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
|
||||
mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
|
||||
Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
|
||||
vs 104.8 t/s).
|
||||
|
||||
**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
|
||||
natively multimodal; the Q6_K file carries no vision tensors). Probe traps
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Swift-1.5-Qwen3.8-27B Uncensored RCO IQ3_XXS GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, draft-mtp depth 3, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
|
||||
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -76,9 +76,30 @@ Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
|
||||
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
|
||||
fallback rung if a future shape lands under the 2 GiB floor.
|
||||
|
||||
## Measured boot (2026-10-07)
|
||||
## Measured boot (2026-10-07, deploy `ec6184d`)
|
||||
|
||||
TO BE FILLED BY THE DEPLOY VERIFICATION.
|
||||
`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
|
||||
projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
|
||||
(`--log-verbosity 4`, `journalctl -u frank-secondary`):
|
||||
|
||||
|Line|Value|
|
||||
|---|---|
|
||||
|`llama_context`|`n_ctx = 401408`, `n_ctx_seq = 200704`, `kv_unified = false`|
|
||||
|`llama_kv_cache`|`CUDA0 KV buffer size = 12544.00 MiB` — `size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB`|
|
||||
|`sched_reserve`|`CUDA0 compute buffer size = 1136.06 MiB`, `CUDA_Host compute buffer size = 848.07 MiB`|
|
||||
|`clip_model_loader` (mmproj)|`model size: 878.99 MiB` + `reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB`|
|
||||
|`srv load_model`|`n_slots = 2, n_ctx_slot = 200704`|
|
||||
|`print_info`|`n_ctx_train = 262144`, `n_head_kv = 4`|
|
||||
|
||||
The `32 KiB/token` law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB,
|
||||
which is what the engine printed. The `n_ctx_seq (200704) < n_ctx_train
|
||||
(262144)` line is informational (a context **below** the trained window is
|
||||
safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
|
||||
**past** its window). Weights load in ~6 s from page cache; `/health` answers
|
||||
`{"status":"ok"}` and `/v1/models` serves the alias `frank-secondary`.
|
||||
|
||||
The deploy's Playwright gate passed (33 tests) and the fleet e2e set is
|
||||
unchanged.
|
||||
|
||||
## Client contract
|
||||
|
||||
@@ -111,3 +132,11 @@ one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
|
||||
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
|
||||
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
|
||||
flags — the engine prints the exact figures the shape needs.
|
||||
|
||||
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-secondary.sh`
|
||||
shipped without its `#!/usr/bin/env bash` line, and systemd failed the
|
||||
`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
|
||||
`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
|
||||
`Failed at step EXEC spawning /home/dank/…/serve-frank-secondary.sh`. A
|
||||
shebang is not decoration on this host: every LLM serve script is a direct
|
||||
`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).
|
||||
|
||||
Reference in New Issue
Block a user