diff --git a/AGENTS.md b/AGENTS.md index e61dcbc..336e1e1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -451,13 +451,16 @@ decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`, property of the file, not of the flags. The publisher card advertises "MTP support" for the base architecture; the quantized export dropped the head. -**Measured boots** (nvidia-smi, the referee): the pricing boots are the -MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry — 2 × 200,704 f16 K + -f16 V: **21,276 used / 2,849 free** (ubatch 1024) and **21,846 / 2,279** -(ubatch 2048). This file is 415 MiB lighter in weights ⇒ projected ~21,430 / -~2,700. The deployed boot ledger: see the runbook. Decode is ubatch-invariant -(Swift-era 3090 law). The Q6_K rung costs ~10 % decode vs Q5_K_M (weight -bandwidth; MiMo-measured 94.4 vs 104.8 t/s). +**Measured boots** (nvidia-smi, the referee): the deployed boot is +**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the +MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at +ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights +bought back what the projection predicted. The engine ledger confirms the KV +law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`, +`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the +mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The +Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4 +vs 104.8 t/s). **Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is natively multimodal; the Q6_K file carries no vision tensors). Probe traps diff --git a/scripts/frank-secondary.service b/scripts/frank-secondary.service index 2c3af17..0f08f06 100644 --- a/scripts/frank-secondary.service +++ b/scripts/frank-secondary.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp Swift-1.5-Qwen3.8-27B Uncensored RCO IQ3_XXS GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, draft-mtp depth 3, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary) +Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary) After=network-online.target Wants=network-online.target diff --git a/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md index a377d93..20a5678 100644 --- a/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md @@ -76,9 +76,30 @@ Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 % prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented fallback rung if a future shape lands under the 2 GiB floor. -## Measured boot (2026-10-07) +## Measured boot (2026-10-07, deploy `ec6184d`) -TO BE FILLED BY THE DEPLOY VERIFICATION. +`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced +projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger +(`--log-verbosity 4`, `journalctl -u frank-secondary`): + +|Line|Value| +|---|---| +|`llama_context`|`n_ctx = 401408`, `n_ctx_seq = 200704`, `kv_unified = false`| +|`llama_kv_cache`|`CUDA0 KV buffer size = 12544.00 MiB` — `size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB`| +|`sched_reserve`|`CUDA0 compute buffer size = 1136.06 MiB`, `CUDA_Host compute buffer size = 848.07 MiB`| +|`clip_model_loader` (mmproj)|`model size: 878.99 MiB` + `reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB`| +|`srv load_model`|`n_slots = 2, n_ctx_slot = 200704`| +|`print_info`|`n_ctx_train = 262144`, `n_head_kv = 4`| + +The `32 KiB/token` law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB, +which is what the engine printed. The `n_ctx_seq (200704) < n_ctx_train +(262144)` line is informational (a context **below** the trained window is +safe; the Gemma4 era's warning was the opposite case, a drafter running 53 % +**past** its window). Weights load in ~6 s from page cache; `/health` answers +`{"status":"ok"}` and `/v1/models` serves the alias `frank-secondary`. + +The deploy's Playwright gate passed (33 tests) and the fleet e2e set is +unchanged. ## Client contract @@ -111,3 +132,11 @@ one slot, not the pool; queue behind two long generations ⇒ the slot ceiling is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in `journalctl -u frank-secondary` (KV MiB, compute buffers) before touching flags — the engine prints the exact figures the shape needs. + +**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-secondary.sh` +shipped without its `#!/usr/bin/env bash` line, and systemd failed the +`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and +`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature: +`Failed at step EXEC spawning /home/dank/…/serve-frank-secondary.sh`. A +shebang is not decoration on this host: every LLM serve script is a direct +`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).