Files

T

Frank d9af7f1d19 fleet redo 2026-08-17: retire PP+NPU, new topology

- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook

2026-08-17 15:23:46 -07:00

1.5 KiB

Raw Blame History

vLLM Scripts

Deployment scripts for the LLM inference fleet (2026-08-17 topology): llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.

Targets

Host	User	Codename	GPU
dankserver	dank	Dankserver	NVIDIA RTX 3090 24 GB + 62 GB RAM
dankclaw	dank	Dankclaw	AMD Strix Halo iGPU gfx1151 (unified memory)

Active Services

Service	Host	Runtime	Model	Port
frank-default-large	dankserver	llama.cpp (patched FastMTP build)	HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx	7770
frank-default-small	dankclaw	vLLM ROCm (vllm-venv)	Avesed Qwen3.6-35B-A3B INT8 W8A8, full 262144 ctx, no KV quantization, 4 seqs, ngram spec	52625

The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in old/scripts/ for reference.

Structure

docs/ — knowledge base: deployment runbook, model notes, research archive
scripts/ — serve scripts, env files, systemd units, deploy and test tools
old/scripts/ — retired PP/NPU-era scripts (reference only)

Deployment

See docs/deployment.md. Run FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver for the llama.cpp box or FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw for the vLLM box.

Knowledge Base

See docs/index.md for the project knowledge base index.

1.5 KiB Raw Blame History