Frank
0c5e054fb8
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
20 lines
603 B
Desktop File
20 lines
603 B
Desktop File
[Unit]
|
|
Description=SGLang MiMo-V2.6-Distill-Qwen-9B-INT8 (qwen3_5 hybrid, MTP+vision) 4x172k fp8 KV (dankclaw, frank-small)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=dank
|
|
Group=dank
|
|
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
|
# though the weights themselves are copied into VRAM.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|