0 Fork 0

Files

T

Frank 4462aa992c Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template

- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
  native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
  explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
  (replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
  ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
  (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
  retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
  --chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
  update index, deployment, and both system specs; models.json names.

Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.

2026-08-24 21:35:11 -07:00

3.1 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

Network

Interface	Type	Speed	IP	Notes
eno1	onboard	2500 Mb/s (2.5 GbE)	192.168.0.235 (DHCP)	RPC worker endpoint (port 50052). Default route via 192.168.0.1.
enxe8ea6ab7ebca	USB-C class adapter	2500 Mb/s	192.168.0.252	UNPLUGGED 2026-08-17. Was idle (unused by the RPC path).
wlp8s0	WiFi	-	-	Unused (carrier down).
docker0	bridge	-	172.17.0.1	Down; internal docker traffic only.

OS and Software

Property	Value
OS	Ubuntu 26.04 LTS (Resolute Raccoon)
Kernel	7.0.0-30-generic
Python	3.14.4

Active Services

The active service on Dankserver is frank-default-small.service: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q8_0) on port 7770, with MoE expert weights held on CPU via --override-tensor. The earlier PP worker (frank-pp-worker.service) was retired with the 2026-08-17 redo. See deployment.

Hardware Relevance

24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at --n-cpu-moe 36 (4 layers' experts on GPU) with ubatch 3072 and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep (frank-default-small-benchmark) guided this state: prefill is the priority, so ubatch 3072 stays (top prefill tier), and ncmoe 36 was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add GGML_CUDA_FORCE_MMQ=1. If decode speed matters more, use ncmoe 32 - ubatch 1024 (gen ~48.7 t/s). The tuning guidance applied to the retired frank-default-small service; the PP worker's allocations are fixed at load.
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

3.1 KiB Raw Blame History