dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: record the deployed boot, keep the unit description honest

Browse Source 
Deploy verified: nvidia-smi 21,436 MiB used / 2,689 free at ubatch 2048 -
the MiMo-priced projection (~21,430/~2,700) landed within 11 MiB, and the
verbosity-4 ledger confirms the KV law to the MiB (CUDA0 KV buffer
12,544.00 MiB = K(f16) 6,272 + V(f16) 6,272 = 32 KiB/token over the
401,408 pool), compute buffers 1,136 MiB CUDA0 + 848 MiB host, mmproj
879 + 248 MiB. Playwright gate: 33 passed.

Two lessons recorded: the 203/EXEC boot loop my shebang-less rewrite
caused (journal signature + the head -1 check), and that llama.cpp's
"n_ctx_seq < n_ctx_train" line is informational when the context is BELOW
the trained window - the Gemma4 drafter warning was the opposite case.

The unit's Description named the Swift-1.5 era (and said 2x131072/q8_0/
draft-mtp); it now names the Q6_K shape. Cosmetic, so it is NOT deployed
as its own restart: the next frank-secondary deploy relinks it.

Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frank
2026-10-07 17:33:21 -07:00
co-authored by omp
parent ec6184de9a
commit 5278f5be26
3 changed files with 42 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md
+31 -2
View File
@@ -76,9 +76,30 @@ Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
fallback rung if a future shape lands under the 2 GiB floor.
## Measured boot (2026-10-07)
## Measured boot (2026-10-07, deploy `ec6184d`)
TO BE FILLED BY THE DEPLOY VERIFICATION.
`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
(`--log-verbosity 4`, `journalctl -u frank-secondary`):
|Line|Value|
|---|---|
|`llama_context`|`n_ctx = 401408`, `n_ctx_seq = 200704`, `kv_unified = false`|
|`llama_kv_cache`|`CUDA0 KV buffer size = 12544.00 MiB` — `size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB`|
|`sched_reserve`|`CUDA0 compute buffer size = 1136.06 MiB`, `CUDA_Host compute buffer size = 848.07 MiB`|
|`clip_model_loader` (mmproj)|`model size: 878.99 MiB` + `reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB`|
|`srv load_model`|`n_slots = 2, n_ctx_slot = 200704`|
|`print_info`|`n_ctx_train = 262144`, `n_head_kv = 4`|
The `32 KiB/token` law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB,
which is what the engine printed. The `n_ctx_seq (200704) < n_ctx_train
(262144)` line is informational (a context **below** the trained window is
safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
**past** its window). Weights load in ~6 s from page cache; `/health` answers
`{"status":"ok"}` and `/v1/models` serves the alias `frank-secondary`.
The deploy's Playwright gate passed (33 tests) and the fleet e2e set is
unchanged.
## Client contract
@@ -111,3 +132,11 @@ one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
flags — the engine prints the exact figures the shape needs.
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-secondary.sh`
shipped without its `#!/usr/bin/env bash` line, and systemd failed the
`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
`Failed at step EXEC spawning /home/dank/…/serve-frank-secondary.sh`. A
shebang is not decoration on this host: every LLM serve script is a direct
`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).