Frank
3c2db64a78
Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B Q6_K_P, pp16384/tg512, fit-target 0). Key findings: - Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30); ubatch/batch do not affect decode at all. - Prefill + graph VRAM scale with ubatch; logical batch is irrelevant. - VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24. Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024. gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom. Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy; ncmoe32-ub768 (1.3GiB) if headroom matters most. Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.
vLLM Scripts
Scripts for running LLM inference on two local machines.
Targets
| Host | User | Codename | Architecture |
|---|---|---|---|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
Active Services
| Service | Host | Runtime | Model | Port |
|---|---|---|---|---|
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 |
| frank-default-small | dankserver | llama.cpp (ngram-mod) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 |
The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired
vllm-awq-qwen36(AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain inscripts/for reference.
Models
| Model | Target | Format |
|---|---|---|
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) |
| Qwen3.6 35B A3B uncensored Q6_K_P (ngram-mod speculative decoding) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) |
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
Structure
docs/— Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notesscripts/— Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files)
Deployment
See docs/deployment.md for the deploy/update process. Run ./scripts/deploy.sh <dankclaw|dankserver> after pushing committed changes.
Knowledge Base
See docs/index.md for the project knowledge base index.