Frank
466fd477d8
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
27 lines
1.1 KiB
Desktop File
27 lines
1.1 KiB
Desktop File
[Unit]
|
|
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 3x262144 slots over one 786432 pool, q8_0 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=dank
|
|
Group=dank
|
|
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
|
# though the weights themselves are copied into VRAM.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh
|
|
# llama.cpp can take longer than systemd's 90 s default to drain slots and
|
|
# release the card: a real stop on 2026-10-18 hit `State 'stop-sigterm' timed
|
|
# out. Killing.` and left the unit in `failed` (result 'timeout') even though
|
|
# the process had served health correctly. The engine is SIGKILL-safe (nothing
|
|
# on disk), but a unit that always ends `failed` is a false alarm for the next
|
|
# operator, so give the drain room.
|
|
TimeoutStopSec=180
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|