0 Fork 0

Files

T

Frank 968066ff23 docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup

- deployment.md: Boot Policy (2026-08-17) section (disabled units table,
  re-enable recipes, stale frank-restart-small.sh helper, broken
  deploy.sh dankserver target); Update Dankserver -> frank-pp-worker;
  condensed retired 35B history block and 2026-08-13 incident
- AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68),
  deploy.sh dankserver warning, obsolete quirk entries condensed
- README.md: rewrite active services/models tables (PP head + worker)
- index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled)
- qwen3.8-spec-matrix.md: superseded configs merged into compact
  history; stale inline env block removed; bundled-head MTP note
- FastFlowLM, PP Deployment: boot-policy + enablement notes
- System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active
  claims fixed, retired banners, never-shipped plan stubbed

2026-08-17 03:30:00 -07:00

2.7 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

The active service on Dankserver is frank-pp-worker.service: the ggml-rpc worker (RTX 3090) that receives the PP head's tensor share (45 of 65 blocks, ~23.8 GiB of 24 GiB at steady state). The old llama.cpp Qwen3.6-35B-A3B service (frank-default-small) is retired (2026-08-16) and disabled (2026-08-17). See Qwen3.8 PP Deployment.

Hardware Relevance

24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at --n-cpu-moe 36 (4 layers' experts on GPU) with ubatch 3072 and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep (frank-default-small-benchmark) guided this state: prefill is the priority, so ubatch 3072 stays (top prefill tier), and ncmoe 36 was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add GGML_CUDA_FORCE_MMQ=1. If decode speed matters more, use ncmoe 32 - ubatch 1024 (gen ~48.7 t/s). The tuning guidance applied to the retired frank-default-small service; the PP worker's allocations are fixed at load.
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

2.7 KiB Raw Blame History