Frank
923e775354
Trial replaces frank-default-large temporarily via frank-large-ling.service (port 7770, same alias). Requires the aetherbird bailingmoe3-support llama.cpp fork at /home/dank/llama.cpp-ling (stock llama.cpp cannot load bailingmoe3). Memory: 8x16GB LPDDR5 = 128GB physical; Vulkan exposes ~111.2GiB heap (UMA 96GiB + GTT) vs ROCm's 96GiB. Q6_K = 98.26GiB; MLA/KDA KV is tiny so 256k ctx x 2 slots fits with ~10GiB headroom. Verified: loads ~45s, concurrent streams both complete (no DFlash-style kill), ~7.7 t/s gen. Restore Qwen: systemctl stop frank-large-ling; systemctl start frank-default-large.
18 lines
427 B
Desktop File
18 lines
427 B
Desktop File
[Unit]
|
|
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-large-ling TRIAL)
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=dank
|
|
Group=dank
|
|
Environment="HOST=0.0.0.0"
|
|
Environment="PORT=7770"
|
|
Environment="FRANK_LING_ENV_FILE=/home/dank/vllmScripts/scripts/frank-default-large.ling.env"
|
|
ExecStart=/home/dank/vllmScripts/scripts/serve-ling-flash.sh
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|