docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup

- deployment.md: Boot Policy (2026-08-17) section (disabled units table,
  re-enable recipes, stale frank-restart-small.sh helper, broken
  deploy.sh dankserver target); Update Dankserver -> frank-pp-worker;
  condensed retired 35B history block and 2026-08-13 incident
- AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68),
  deploy.sh dankserver warning, obsolete quirk entries condensed
- README.md: rewrite active services/models tables (PP head + worker)
- index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled)
- qwen3.8-spec-matrix.md: superseded configs merged into compact
  history; stale inline env block removed; bundled-head MTP note
- FastFlowLM, PP Deployment: boot-policy + enablement notes
- System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active
  claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
Frank committed 2026-08-17 03:30:00 -07:00
1 parent 3f88b8e1ed
commit 968066ff23
13 files changed
+190 -246

No files matched your search

+2 -2
View File
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
### Active Services
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB.
The active service on Dankserver is `frank-pp-worker.service`: the ggml-rpc worker (RTX 3090) that receives the PP head's tensor share (45 of 65 blocks, ~23.8 GiB of 24 GiB at steady state). The old llama.cpp Qwen3.6-35B-A3B service (`frank-default-small`) is retired (2026-08-16) and disabled (2026-08-17). See [[Qwen3.8 PP Deployment]].
## Hardware Relevance
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). The tuning guidance applied to the retired `frank-default-small` service; the PP worker's allocations are fixed at load.
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).