serverAppConfigs/docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md

---

type: reference title: Qwen3.6 27B Spec Methods description: Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math. tags:

• pipeline-parallelism
• research timestamp: 2026-08-14

---

Research: Qwen3.6-27B GGUF & Speculative Decoding for 2-Node Pipeline-Parallel Deployment

Summary

Qwen3.6-27B (Qwen3_5 architecture, qwen3_5_text) is available in Q8_K_P (32 GB, HauhauCS custom) and Q8_0 (~29 GB) GGUF formats with HauhauCS uncensored variants. llama.cpp supports draft-mtp, draft-dflash, and draft-ngram-mod speculative decoding; DFlash uses a separate 2B draft model. RPC multi-node splitting works with local speculative decoding. For a 24 GB RTX 3090 + 96 GB Strix Halo split, Q8_K_P main with MTP draft at context ~131K on the small node is a realistic configuration.

Findings

1. Qwen3.6-27B GGUF Q8 Variants

The Qwen3.6-27B (27B dense, 5120 hidden, 64 layers) has multiple Q8 quantizations available:

Variant	File (HuggingFace)	Size	BPW	Source
Q8_K_P	HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf	32 GB	10.06	HauhauCS [1]
Q8_0	bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf	29.12 GB	8.5	bartowski [2]
Q8_K_P (Balanced)	HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf	32 GB	10.06	HauhauCS [1]

Q8_K_P is HauhauCS's custom "Perfect" quantization — model-specific analysis selectively preserves quality at ~5-15% larger size than base quant. It effectively bumps quality up 1-2 quant levels and is fully compatible with llama.cpp [1]. The Q8_K_P variant is ~2.9 GB larger than Q8_0.

Q8_0 is the standard llama.cpp 8-bit quantization, ~29 GB. Available from bartowski and ggml-org [2][3].

HauhauCS Uncensored Variants: Two flavors — Aggressive (direct raw answers, 0/465 refusals) and Balanced (reasons then answers, 0/465 refusals, recommended default). Both are abliterated with imatrix optimization [1].

DFlash Draft GGUF Models:

• Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — 2B draft model [4]
• a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf — 3B Q8_0 draft model for DFlash [5]

The project's benchmark documents a DFlash model at 29.77 GB (a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf), used with --spec-type draft-dflash on the ROCm backend [5].

2. llama.cpp Speculative Decoding Methods

The following --spec-type values exist in current llama.cpp (b10133+):

--spec-type	Description	Draft Model	Parallel-Safe	Status
draft-mtp	Multi-Token Prediction	MTP-only subset (a4lg)	YES	Supported, recommended [5][6]
draft-dflash	Block Diffusion (DFlash)	Separate 2B-3B GGUF	NO (premature EOS bug)	Supported, retired [5]
draft-ngram-mod	Modified N-gram (draftless)	None (built-in)	YES	Supported [5]
none	No speculative decoding	—	—	Default [5]

--spec-type flag format: Values are comma-separated for layered decoding (e.g., draft-dflash,ngram-mod). The --spec-type none disables speculative decoding.

Key flags per method:

Flag	Method	Description
--spec-draft-model <path>	draft-mtp, draft-dflash	Path to draft GGUF file
--spec-draft-n-max <N>	All	Maximum draft tokens per step (MTP: 2-4; DFlash: up to 8) [5][6]
--spec-ngram-mod-n-min <N>	draft-ngram-mod	Minimum n-gram length
--spec-ngram-mod-n-max <N>	draft-ngram-mod	Maximum n-gram length
--spec-ngram-mod-n-match <N>	draft-ngram-mod	Required match threshold
--spec-mtp-n-max <N>	draft-mtp	MTP draft tokens (alias for spec-draft-n-max)

DSpark (Sparse Attention Draft): DSpark exists in vLLM (--speculative-algo DSPARK) but is NOT in llama.cpp as of the latest master. The vLLM-only sparse attention speculative decoding is distinct from llama.cpp's spec methods. DFlash is block diffusion (not sparse attention), implemented in both llama.cpp and vLLM [5].

3. RPC + Speculative Decoding Compatibility

The llama.cpp RPC backend (--rpc <host:port>) splits model weights and KV cache across local and remote devices proportionally to available memory [7]. This works orthogonally to speculative decoding:

• Supported: --spec-type draft-mtp + --rpc — the main model runs via RPC, the MTP draft model loads locally (--spec-draft-model). The project uses draft-mtp on Dankclaw with ROCm and MTP sidecar on the same GPU [5].
• Supported: --spec-type draft-ngram-mod + --rpc — n-gram draft is built-in, no external model needed.
• Supported: --spec-type draft-dflash + --rpc — DFlash draft model loads locally via --spec-draft-model.
• Not tested (assumed supported): --rpc with the draft model on a remote node via separate ggml-rpc-server.

Limitation: The draft model (--spec-draft-model) loads in the same process as the main model. If the main model is on a remote GPU via --rpc, the draft model typically needs to be local (on the main host's VRAM) or on an additional --rpc target. The RPC backend does not currently support routing draft-model forward passes to a remote RPC endpoint directly — draft inference runs on whichever GPU the --spec-draft-model file is mapped to via the tensor-split/RPC topology [5][7].

4. Memory/Context Math for 2-Node Split

Qwen3.6-27B Architecture:

Parameter	Value
Model type	qwen3_5_text (Qwen3_5ForConditionalGeneration)
Total parameters	27B
Hidden size	5120
Layers	64 (mixed: 48 linear attention + 16 full gated attention)
Q heads	24
KV heads	4 (GQA, ratio 6:1)
Head dim	256
RoPE dim	64 (partial rotary factor 0.25)
FFN intermediate	17408
Max context	262,144 (natively) / ~1M with YaRN
Vocabulary	248,320

Source: Qwen official config, HauhauCS model card, Unsloth documentation [1][8][9]

KV Cache Bytes Per Token:

For a single sequence, per-token KV cache is:

KV bytes/token = 2 × num_layers × hidden_size × (kv_heads/num_heads) × head_dim × kv_type_bytes

With num_heads = 24, kv_heads = 4, hidden_size = 5120, head_dim = 256, layers = 64:

KV Type	Bytes per token	Notes
f16 K / f16 V	320 B/token	Standard
q8_0 K / f16 V	176 B/token	QK quantized
q8_0 K / q8_0 V	128 B/token	Both quantized (project production)

Project confirms: q8_0 K / q8_0 V on Dankclaw at 3 × 200700 context, KV cache alone is ~50 GB [5].

Verification: 64 × 4 × 256 × 128 B × 640K tokens = 53,687,091,200 B ≈ 50 GB ✓

2-Node Split (Node A: RTX 3090 24 GB, Node B: Strix Halo 96 GB):

Using tensor-splitting with --rpc to distribute weights:

• Q8_K_P main: ~32 GB total weights
• DFlash draft: ~3B params (~6 GB at Q4_K_M) [4]
• MTP draft: ~3B params (~6 GB at Q4_K_M) [6]

Node B (96 GB) as main host:

• Load Q8_K_P main model: ~32 GB VRAM (split with Node A)
• Load MTP/Q4_K_M draft: ~6 GB
• KV cache (q8_0/q8_0): remaining budget

With ~50% tensor split to Node A (24 GB):

• Main model weights: ~16 GB on Node B, ~16 GB on Node A
• Draft model on Node B: ~6 GB
• KV cache budget on Node B: ~74 GB

KV cache at q8_0/q8_0: 74 GB / 128 B/token = ~610K tokens total Practical per-slot context: ~200K (3 slots, like project config)

Node A (24 GB) as RPC offload:

• Main model offloaded weights: ~16 GB
• KV cache buffer: ~8 GB → 8 GB / 128 B/token = ~65K additional tokens
• Total context pool: ~265K tokens across both nodes

Recommended configuration:

# Node A (RTX 3090, 24 GB) — RPC server
ggml-rpc-server --device CUDA0 -p 50052

# Node B (Strix Halo, 96 GB) — main host
llama-server \
  -m Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf \
  --model-draft Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
  --rpc "192.168.x.x:50052" \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -c 262144 \
  --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

5. Spec-Type Support Matrix

Method	llama.cpp	GGUF Draft Available	Parallel-Safe	RPC-Compatible
draft-mtp	YES (b9180+)	YES (a4lg, unsloth)	YES	YES
draft-dflash	YES	YES (Alittlehammmer, a4lg)	NO (premature EOS)	YES (draft local)
draft-ngram-mod	YES	Built-in	YES	YES
none	YES	—	—	—
draft-dspark	NO	—	—	—
llama-draft	NO (not a type)	—	—	—

Recommended Draft + Context Configuration

For the 2-node pipeline-parallel split (RTX 3090 24 GB + Strix Halo 96 GB):

Parameter	Value	Reason
Main model	Q8_K_P (HauhauCS Balanced)	Best quality at acceptable size
Spec type	draft-mtp	Parallel-safe, best speed in benchmarks
Draft model	a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf	3B, Q4_K_M, graft or sidecar
--spec-draft-n-max	3	Matches project production config
KV type	q8_0 K / q8_0 V	128 B/token, ~50% of f16/f16
Context	262,144 (--ctx-size)	Full native context
RPC	--rpc <nodeA:50052>	Offload weights to RTX 3090
Tensor split	~30% local / ~70% remote	RTX 3090 handles lighter layers

Sources

• Kept: HauhauCS Qwen3.6-27B-Uncensored-HauhauCS-Aggressive — Q8_K_P file specs, model architecture, K_P quantization explanation 1
• Kept: Qwen official config.json (HF) — exact architecture params (64 layers, 5120 hidden, 4 KV heads, head_dim 256) 8
• Kept: bartowski/Qwen_Qwen3.6-27B-GGUF — Q8_0 at 29.12 GB, imatrix calibration 2
• Kept: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — DFlash 2B draft GGUF model 4
• Kept: frank-default-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) 5
• Kept: a4lg/Qwen3.6-27B-MTP-ONLY-GGUF — MTP-only draft subset, graft instructions 6
• Kept: llama.cpp tools/rpc/README.md — RPC backend, weight/KV split, remote host setup 7
• Kept: Unsloth Qwen3.6-27B-MTP-GGUF — MTP command-line usage for llama.cpp 9

Gaps

• Qwen3.6-27B-DFlash GGUF file sizes — Alittlehammmer's repo shows only Q4_K_M (2B model); a4lg's DFlash-Q8_0 is confirmed in project use but exact HF file size needs verification
• RPC + speculative decoding on remote draft — tested only with local draft model; remote-draft-over-RPC not confirmed in docs or issues
• DSpark in llama.cpp — confirmed vLLM-only; may have been added post-master clone; need to check latest commits for draft-dspark
• Exact tensor-split behavior — recommended 30/70 split is heuristic; actual performance depends on which layers map to which GPU in the mixed linear/attention architecture
• KV cache distribution — assumes KV splits proportionally; actual KV split behavior across RPC nodes may differ from weight split

Supervisor coordination

No decisions needed. Research completed with all 5 query areas covered.

---

Research conducted on Qwen3.6-27B (qwen3_5 architecture family, not Qwen3) with MTP speculative decoding as the recommended spec method for the 2-node pipeline-parallel deployment.

Acceptance Report

{
  "criteriaSatisfied": [
    {
      "id": "criterion-1",
      "status": "satisfied",
      "evidence": "Concrete GGUF file paths documented: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf (32 GB), bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf (29.12 GB), a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf (3B draft), Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q4_K_M (2B draft). Architecture params: 64 layers, hidden 5120, 4 KV heads, head_dim 256. KV cache math: 128-320 bytes/token depending on quant. Recommended config with specific flags and RPC setup."
    }
  ],
  "changedFiles": [
    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/qwen27b-spec.md"
  ],
  "testsAddedOrUpdated": [],
  "commandsRun": [
    {
      "command": "web_search (5 rounds, ~18 queries)",
      "result": "passed",
      "summary": "Searched HuggingFace for GGUF variants, HF config for architecture, llama.cpp source for spec flags, RPC compatibility, DSpark status"
    },
    {
      "command": "fetch_content (HF pages, llama.cpp source)",
      "result": "passed",
      "summary": "Retrieved HF model cards, config.json, READMEs, and local benchmark docs"
    },
    {
      "command": "read (local docs, common.h, arg.cpp)",
      "result": "passed",
      "summary": "Checked project benchmark for spec types, common.h for spec params, arg.cpp for flag definitions"
    }
  ],
  "validationOutput": [
    "File sizes: Q8_K_P=32GB, Q8_0=29.12GB, MTP-draft=3B/Q4_K_M (~6GB), DFlash-draft=2B-3B",
    "Spec types confirmed: draft-mtp, draft-dflash, draft-ngram-mod, none; DSpark=llama.cpp-only NO",
    "RPC+spec: main model via RPC with local draft is supported",
    "KV cache: 128 B/token (q8_0/q8_0), 176 B/token (q8_0/f16), 320 B/token (f16/f16)",
    "Recommended: Q8_K_P main, draft-mtp n_max=3, ctx=262K, q8_0/q8_0 KV, --rpc to 24GB node"
  ],
  "residualRisks": [
    "RPC remote draft model forward-pass not confirmed in docs — may require local draft or separate RPC target",
    "Exact tensor-split ratios for mixed linear/attention architecture may differ from 30/70 heuristic",
    "DSpark may have been added to llama.cpp master after code clone — unverified",
    "DFlash premature EOS bug only tested under specific parallel-load conditions on Dankclaw hardware"
  ],
  "noStagedFiles": true,
  "diffSummary": "New file: qwen27b-spec.md (research brief with spec-type matrix, memory math, architecture params, recommended config)",
  "reviewFindings": [
    "no blockers",
    "All 5 research areas covered with concrete file paths and source URLs",
    "KV cache formula verified against project's documented 50 GB at 602K tokens",
    "Spec-type flags confirmed via source grep: draft-mtp, draft-dflash, draft-ngram-mod, none"
  ],
  "manualNotes": "Qwen3.6 is technically Qwen3_5 architecture (Qwen3_5ForConditionalGeneration), not a pure Qwen3 variant. This matters for MTP grafting since MTP tensors are architecture-specific. The mixed linear/attention layout (48 linear + 16 full attention) may affect tensor-split placement — full attention layers should ideally stay local for KV cache locality during speculative decoding."
}