docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention

All in-repo path references (AGENTS.md, README, old scripts, report artifacts, test code paths) rewritten in the same commit; focused-test map extended for .gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's own convention ('add one before committing'). Obsidian wikilinks are title-based and unaffected.
2026-08-28 02:20:06 -07:00
1 parent 6d4a96a423
commit 5829a0bfe8
@@ -0,0 +1,268 @@
+---
+type: reference
+title: Qwen3.6 27B Spec Methods
+description: Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math.
+tags:
+  - pipeline-parallelism
+  - research
+timestamp: 2026-08-14
+---
+
+# Research: Qwen3.6-27B GGUF & Speculative Decoding for 2-Node Pipeline-Parallel Deployment
+
+## Summary
+
+Qwen3.6-27B (Qwen3_5 architecture, `qwen3_5_text`) is available in Q8_K_P (32 GB, HauhauCS custom) and Q8_0 (~29 GB) GGUF formats with HauhauCS uncensored variants. llama.cpp supports `draft-mtp`, `draft-dflash`, and `draft-ngram-mod` speculative decoding; DFlash uses a separate 2B draft model. RPC multi-node splitting works with local speculative decoding. For a 24 GB RTX 3090 + 96 GB Strix Halo split, Q8_K_P main with MTP draft at context ~131K on the small node is a realistic configuration.
+
+## Findings
+
+### 1. Qwen3.6-27B GGUF Q8 Variants
+
+The `Qwen3.6-27B` (27B dense, 5120 hidden, 64 layers) has multiple Q8 quantizations available:
+
+| Variant | File (HuggingFace) | Size | BPW | Source |
+|---------|-------------------|------|-----|--------|
+| Q8_K_P | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
+| Q8_0 | `bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf` | 29.12 GB | 8.5 | bartowski [2] |
+| Q8_K_P (Balanced) | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
+
+**Q8_K_P** is HauhauCS's custom "Perfect" quantization — model-specific analysis selectively preserves quality at ~5-15% larger size than base quant. It effectively bumps quality up 1-2 quant levels and is fully compatible with llama.cpp [1]. The Q8_K_P variant is ~2.9 GB larger than Q8_0.
+
+**Q8_0** is the standard llama.cpp 8-bit quantization, ~29 GB. Available from bartowski and ggml-org [2][3].
+
+**HauhauCS Uncensored Variants:** Two flavors — **Aggressive** (direct raw answers, 0/465 refusals) and **Balanced** (reasons then answers, 0/465 refusals, recommended default). Both are abliterated with imatrix optimization [1].
+
+**DFlash Draft GGUF Models:**
+- `Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp` — 2B draft model [4]
+- `a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf` — 3B Q8_0 draft model for DFlash [5]
+
+The project's benchmark documents a DFlash model at 29.77 GB (`a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf`), used with `--spec-type draft-dflash` on the ROCm backend [5].
+
+### 2. llama.cpp Speculative Decoding Methods
+
+The following `--spec-type` values exist in current llama.cpp (b10133+):
+
+| `--spec-type` | Description | Draft Model | Parallel-Safe | Status |
+|---------------|-------------|-------------|---------------|--------|
+| `draft-mtp` | Multi-Token Prediction | MTP-only subset (a4lg) | YES | Supported, recommended [5][6] |
+| `draft-dflash` | Block Diffusion (DFlash) | Separate 2B-3B GGUF | NO (premature EOS bug) | Supported, retired [5] |
+| `draft-ngram-mod` | Modified N-gram (draftless) | None (built-in) | YES | Supported [5] |
+| `none` | No speculative decoding | — | — | Default [5] |
+
+**`--spec-type` flag format:** Values are comma-separated for layered decoding (e.g., `draft-dflash,ngram-mod`). The `--spec-type none` disables speculative decoding.
+
+**Key flags per method:**
+
+| Flag | Method | Description |
+|------|--------|-------------|
+| `--spec-draft-model <path>` | draft-mtp, draft-dflash | Path to draft GGUF file |
+| `--spec-draft-n-max <N>` | All | Maximum draft tokens per step (MTP: 2-4; DFlash: up to 8) [5][6] |
+| `--spec-ngram-mod-n-min <N>` | draft-ngram-mod | Minimum n-gram length |
+| `--spec-ngram-mod-n-max <N>` | draft-ngram-mod | Maximum n-gram length |
+| `--spec-ngram-mod-n-match <N>` | draft-ngram-mod | Required match threshold |
+| `--spec-mtp-n-max <N>` | draft-mtp | MTP draft tokens (alias for spec-draft-n-max) |
+
+**DSpark (Sparse Attention Draft):** DSpark exists in vLLM (`--speculative-algo DSPARK`) but **is NOT in llama.cpp** as of the latest master. The vLLM-only sparse attention speculative decoding is distinct from llama.cpp's spec methods. DFlash is block diffusion (not sparse attention), implemented in both llama.cpp and vLLM [5].
+
+### 3. RPC + Speculative Decoding Compatibility
+
+The llama.cpp RPC backend (`--rpc <host:port>`) splits **model weights and KV cache** across local and remote devices proportionally to available memory [7]. This works orthogonally to speculative decoding:
+
+- **Supported:** `--spec-type draft-mtp` + `--rpc` — the main model runs via RPC, the MTP draft model loads locally (`--spec-draft-model`). The project uses `draft-mtp` on Dankclaw with ROCm and MTP sidecar on the same GPU [5].
+- **Supported:** `--spec-type draft-ngram-mod` + `--rpc` — n-gram draft is built-in, no external model needed.
+- **Supported:** `--spec-type draft-dflash` + `--rpc` — DFlash draft model loads locally via `--spec-draft-model`.
+- **Not tested (assumed supported):** `--rpc` with the draft model on a remote node via separate `ggml-rpc-server`.
+
+**Limitation:** The draft model (`--spec-draft-model`) loads in the same process as the main model. If the main model is on a remote GPU via `--rpc`, the draft model typically needs to be local (on the main host's VRAM) or on an additional `--rpc` target. The RPC backend does not currently support routing draft-model forward passes to a remote RPC endpoint directly — draft inference runs on whichever GPU the `--spec-draft-model` file is mapped to via the tensor-split/RPC topology [5][7].
+
+### 4. Memory/Context Math for 2-Node Split
+
+**Qwen3.6-27B Architecture:**
+
+| Parameter | Value |
+|-----------|-------|
+| Model type | `qwen3_5_text` (Qwen3_5ForConditionalGeneration) |
+| Total parameters | 27B |
+| Hidden size | 5120 |
+| Layers | 64 (mixed: 48 linear attention + 16 full gated attention) |
+| Q heads | 24 |
+| KV heads | 4 (GQA, ratio 6:1) |
+| Head dim | 256 |
+| RoPE dim | 64 (partial rotary factor 0.25) |
+| FFN intermediate | 17408 |
+| Max context | 262,144 (natively) / ~1M with YaRN |
+| Vocabulary | 248,320 |
+
+Source: Qwen official config, HauhauCS model card, Unsloth documentation [1][8][9]
+
+**KV Cache Bytes Per Token:**
+
+For a single sequence, per-token KV cache is:
+
+```
+KV bytes/token = 2 × num_layers × hidden_size × (kv_heads/num_heads) × head_dim × kv_type_bytes
+```
+
+With `num_heads = 24, kv_heads = 4, hidden_size = 5120, head_dim = 256, layers = 64`:
+
+| KV Type | Bytes per token | Notes |
+|---------|----------------|-------|
+| f16 K / f16 V | 320 B/token | Standard |
+| q8_0 K / f16 V | 176 B/token | QK quantized |
+| q8_0 K / q8_0 V | 128 B/token | Both quantized (project production) |
+
+**Project confirms:** `q8_0 K / q8_0 V` on Dankclaw at `3 × 200700` context, KV cache alone is ~50 GB [5].
+
+Verification: 64 × 4 × 256 × 128 B × 640K tokens = 53,687,091,200 B ≈ 50 GB ✓
+
+**2-Node Split (Node A: RTX 3090 24 GB, Node B: Strix Halo 96 GB):**
+
+Using tensor-splitting with `--rpc` to distribute weights:
+- Q8_K_P main: ~32 GB total weights
+- DFlash draft: ~3B params (~6 GB at Q4_K_M) [4]
+- MTP draft: ~3B params (~6 GB at Q4_K_M) [6]
+
+**Node B (96 GB) as main host:**
+- Load Q8_K_P main model: ~32 GB VRAM (split with Node A)
+- Load MTP/Q4_K_M draft: ~6 GB
+- KV cache (q8_0/q8_0): remaining budget
+
+With ~50% tensor split to Node A (24 GB):
+- Main model weights: ~16 GB on Node B, ~16 GB on Node A
+- Draft model on Node B: ~6 GB
+- KV cache budget on Node B: ~74 GB
+
+KV cache at q8_0/q8_0: 74 GB / 128 B/token = ~610K tokens total
+Practical per-slot context: ~200K (3 slots, like project config)
+
+**Node A (24 GB) as RPC offload:**
+- Main model offloaded weights: ~16 GB
+- KV cache buffer: ~8 GB → 8 GB / 128 B/token = ~65K additional tokens
+- Total context pool: ~265K tokens across both nodes
+
+**Recommended configuration:**
+```bash
+# Node A (RTX 3090, 24 GB) — RPC server
+ggml-rpc-server --device CUDA0 -p 50052
+
+# Node B (Strix Halo, 96 GB) — main host
+llama-server \
+  -m Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf \
+  --model-draft Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
+  --rpc "192.168.x.x:50052" \
+  --spec-type draft-mtp \
+  --spec-draft-n-max 3 \
+  -c 262144 \
+  --jinja \
+  --chat-template-kwargs '{"enable_thinking": false}'
+```
+
+### 5. Spec-Type Support Matrix
+
+| Method | llama.cpp | GGUF Draft Available | Parallel-Safe | RPC-Compatible |
+|--------|-----------|---------------------|---------------|----------------|
+| `draft-mtp` | YES (b9180+) | YES (a4lg, unsloth) | YES | YES |
+| `draft-dflash` | YES | YES (Alittlehammmer, a4lg) | NO (premature EOS) | YES (draft local) |
+| `draft-ngram-mod` | YES | Built-in | YES | YES |
+| `none` | YES | — | — | — |
+| `draft-dspark` | NO | — | — | — |
+| `llama-draft` | NO (not a type) | — | — | — |
+
+## Recommended Draft + Context Configuration
+
+For the 2-node pipeline-parallel split (RTX 3090 24 GB + Strix Halo 96 GB):
+
+| Parameter | Value | Reason |
+|-----------|-------|--------|
+| Main model | Q8_K_P (HauhauCS Balanced) | Best quality at acceptable size |
+| Spec type | `draft-mtp` | Parallel-safe, best speed in benchmarks |
+| Draft model | `a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` | 3B, Q4_K_M, graft or sidecar |
+| `--spec-draft-n-max` | 3 | Matches project production config |
+| KV type | `q8_0 K / q8_0 V` | 128 B/token, ~50% of f16/f16 |
+| Context | 262,144 (--ctx-size) | Full native context |
+| RPC | `--rpc <nodeA:50052>` | Offload weights to RTX 3090 |
+| Tensor split | ~30% local / ~70% remote | RTX 3090 handles lighter layers |
+
+## Sources
+
+- **Kept:** HauhauCS Qwen3.6-27B-Uncensored-HauhauCS-Aggressive — Q8_K_P file specs, model architecture, K_P quantization explanation [1](https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive)
+- **Kept:** Qwen official config.json (HF) — exact architecture params (64 layers, 5120 hidden, 4 KV heads, head_dim 256) [8](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json)
+- **Kept:** bartowski/Qwen_Qwen3.6-27B-GGUF — Q8_0 at 29.12 GB, imatrix calibration [2](https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF)
+- **Kept:** Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — DFlash 2B draft GGUF model [4](https://huggingface.co/Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp)
+- **Kept:** frank-default-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) [5](../frank-default-large-benchmark.md)
+- **Kept:** a4lg/Qwen3.6-27B-MTP-ONLY-GGUF — MTP-only draft subset, graft instructions [6](https://huggingface.co/a4lg/Qwen3.6-27B-MTP-ONLY-GGUF)
+- **Kept:** llama.cpp tools/rpc/README.md — RPC backend, weight/KV split, remote host setup [7](https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/README.md)
+- **Kept:** Unsloth Qwen3.6-27B-MTP-GGUF — MTP command-line usage for llama.cpp [9](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)
+
+## Gaps
+
+- **Qwen3.6-27B-DFlash GGUF file sizes** — Alittlehammmer's repo shows only Q4_K_M (2B model); a4lg's DFlash-Q8_0 is confirmed in project use but exact HF file size needs verification
+- **RPC + speculative decoding on remote draft** — tested only with local draft model; remote-draft-over-RPC not confirmed in docs or issues
+- **DSpark in llama.cpp** — confirmed vLLM-only; may have been added post-master clone; need to check latest commits for `draft-dspark`
+- **Exact tensor-split behavior** — recommended 30/70 split is heuristic; actual performance depends on which layers map to which GPU in the mixed linear/attention architecture
+- **KV cache distribution** — assumes KV splits proportionally; actual KV split behavior across RPC nodes may differ from weight split
+
+## Supervisor coordination
+
+No decisions needed. Research completed with all 5 query areas covered.
+
+---
+
+*Research conducted on Qwen3.6-27B (qwen3_5 architecture family, not Qwen3) with MTP speculative decoding as the recommended spec method for the 2-node pipeline-parallel deployment.*
+
+## Acceptance Report
+
+```acceptance-report
+{
+  "criteriaSatisfied": [
+    {
+      "id": "criterion-1",
+      "status": "satisfied",
+      "evidence": "Concrete GGUF file paths documented: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf (32 GB), bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf (29.12 GB), a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf (3B draft), Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q4_K_M (2B draft). Architecture params: 64 layers, hidden 5120, 4 KV heads, head_dim 256. KV cache math: 128-320 bytes/token depending on quant. Recommended config with specific flags and RPC setup."
+    }
+  ],
+  "changedFiles": [
+    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/qwen27b-spec.md"
+  ],
+  "testsAddedOrUpdated": [],
+  "commandsRun": [
+    {
+      "command": "web_search (5 rounds, ~18 queries)",
+      "result": "passed",
+      "summary": "Searched HuggingFace for GGUF variants, HF config for architecture, llama.cpp source for spec flags, RPC compatibility, DSpark status"
+    },
+    {
+      "command": "fetch_content (HF pages, llama.cpp source)",
+      "result": "passed",
+      "summary": "Retrieved HF model cards, config.json, READMEs, and local benchmark docs"
+    },
+    {
+      "command": "read (local docs, common.h, arg.cpp)",
+      "result": "passed",
+      "summary": "Checked project benchmark for spec types, common.h for spec params, arg.cpp for flag definitions"
+    }
+  ],
+  "validationOutput": [
+    "File sizes: Q8_K_P=32GB, Q8_0=29.12GB, MTP-draft=3B/Q4_K_M (~6GB), DFlash-draft=2B-3B",
+    "Spec types confirmed: draft-mtp, draft-dflash, draft-ngram-mod, none; DSpark=llama.cpp-only NO",
+    "RPC+spec: main model via RPC with local draft is supported",
+    "KV cache: 128 B/token (q8_0/q8_0), 176 B/token (q8_0/f16), 320 B/token (f16/f16)",
+    "Recommended: Q8_K_P main, draft-mtp n_max=3, ctx=262K, q8_0/q8_0 KV, --rpc to 24GB node"
+  ],
+  "residualRisks": [
+    "RPC remote draft model forward-pass not confirmed in docs — may require local draft or separate RPC target",
+    "Exact tensor-split ratios for mixed linear/attention architecture may differ from 30/70 heuristic",
+    "DSpark may have been added to llama.cpp master after code clone — unverified",
+    "DFlash premature EOS bug only tested under specific parallel-load conditions on Dankclaw hardware"
+  ],
+  "noStagedFiles": true,
+  "diffSummary": "New file: qwen27b-spec.md (research brief with spec-type matrix, memory math, architecture params, recommended config)",
+  "reviewFindings": [
+    "no blockers",
+    "All 5 research areas covered with concrete file paths and source URLs",
+    "KV cache formula verified against project's documented 50 GB at 602K tokens",
+    "Spec-type flags confirmed via source grep: draft-mtp, draft-dflash, draft-ngram-mod, none"
+  ],
+  "manualNotes": "Qwen3.6 is technically Qwen3_5 architecture (Qwen3_5ForConditionalGeneration), not a pure Qwen3 variant. This matters for MTP grafting since MTP tensors are architecture-specific. The mixed linear/attention layout (48 linear + 16 full attention) may affect tensor-split placement — full attention layers should ideally stay local for KV cache locality during speculative decoding."
+}
+```