Frank
7fe881de01
Measured after the first 3-slot deploy: 3 x 196,608 booted (n_slots=3, n_ctx_slot=196608, multimodal loaded) but used 22,972/24,576 MiB - 1,153 MiB free, outside the 2-3 GiB band. The two measurements now recorded in frank-small.env price the vision tower at ~1.4 GiB (non-KV 7,390 -> 8,859 MiB), so the pool drops 49,152 tokens (16,384/slot, 1,176 MiB) to land ~2.7 GiB free. The bats budget gate is recalibrated to the measured 9 GiB non-KV cost, which now rejects the 3 x 196,608 shape instead of blessing it.
20 lines
643 B
Desktop File
20 lines
643 B
Desktop File
[Unit]
|
|
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=dank
|
|
Group=dank
|
|
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
|
# though the weights themselves are copied into VRAM.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|