Frank
68afc6dbaf
Frank: "shift frank-zealot to 2 parallels of unquantized kv". Shape: CTX_SIZE 786432 -> 524288, PARALLEL 3 -> 2, CACHE_TYPE_K/V q8_0 -> f16. The per-slot window is UNCHANGED at 262,144 (the native trained window, and models.yml contextWindow), so what the exact cache buys is fidelity and what it pays is one concurrent conversation. The POOL is the term that had to move: exact KV bills 32,768 B/token at this geometry (8 full-attention layers x 4 KV heads x 256 dim, 2 B, K and V), so the old 786,432 pool asks 24,576 MiB against the 24,123 MiB the driver leaves addressable. 524,288 positions = 16,384 MiB, projected ~21,200 used / ~2,900 free from the 13:19 boot ledger (only the KV pool and one recurrent-state cell move). Also corrects a repo-wide arithmetic error: "f16 would bill 26,112 MiB" doubled the q8_0 ledger figure INCLUDING its 1/32 scale overhead. q8_0 is 1.0625 B/element, f16 exactly 2.0, so the true ratio is 1.882x. The wall stands; the number was wrong. And it corrects a false claim about the drafter: this build DOES expose --spec-draft-type-k / --spec-draft-type-v (verified against llama-server --help), so the draft cache is quantizable -- the exact-KV-plus-MTP exclusion is a measurement gap, not a law, and the 1,088 MiB quantized-draft arm is recorded as unmeasured. Seam: tests/frank_zealot_unit.bats re-pins the shape contract (2 slots, 524,288 pool, 262,144 per slot) and adds a VRAM fit gate that computes dtype x pool + the measured non-KV ledger terms against 24,123 MiB with >=1 GiB of slack -- and proves the gate REFUSES exact KV at the old pool, so passing means something. 18/18 green.
27 lines
1.1 KiB
Desktop File
27 lines
1.1 KiB
Desktop File
[Unit]
|
|
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 2x262144 slots over one 524288 pool, unquantized f16 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=dank
|
|
Group=dank
|
|
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
|
# though the weights themselves are copied into VRAM.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh
|
|
# llama.cpp can take longer than systemd's 90 s default to drain slots and
|
|
# release the card: a real stop on 2026-10-18 hit `State 'stop-sigterm' timed
|
|
# out. Killing.` and left the unit in `failed` (result 'timeout') even though
|
|
# the process had served health correctly. The engine is SIGKILL-safe (nothing
|
|
# on disk), but a unit that always ends `failed` is a false alarm for the next
|
|
# operator, so give the drain room.
|
|
TimeoutStopSec=180
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|