docs: vault link integrity restored after rogue-session doc WIP landed

The parallel session's stash pop left GGUF Compatibility, Multi-Device
Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving,
vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9
unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets
that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link,
and two multi-line-wrapped link tokens). All repointed to real pages;
six rows added to the vault index; seam green.

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

@@ -95,6 +95,51 @@ llama-server --hf-repo <target-GGUF> \
 
 The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).
 
+### Chaining laws (measured on this fleet, 2026-09 → 10)
+
+`--spec-type` takes a comma list, and the gate is `ctx_dft` in
+`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
+types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
+unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
+`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
+
+- A chain may hold ONE context-backed drafter plus any number of ngram types.
+  `draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on
+  prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64
+  window.
+- Two context-backed types cannot share one process: `draft-mtp,draft-dflash`
+  dies at load with `context type MTP requested but model doesn't contain MTP
+  layers` — MTP builds its context against whatever the draft context holds,
+  so it met the DRAFTER's file, not a missing head. Pin that in a seam so
+  nobody "fixes the flag" later.
+- Internal order is fixed (ngram types first, then drafters); CLI order is
+  irrelevant.
+- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
+  on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
+  block-copy (2026-10-03): adding the ngram layer logs `recurrent state
+  rollback is not compatible with 'ngram-mod' - disabling rollback support`,
+  and MTP verification depends on that rollback, so acceptance drops on ALL
+  traffic even where the matcher fires. Measure chains per architecture,
+  never by analogy.
+- ngram-mod prints NO startup banner — the load log shows only the drafter's
+  `adding speculative implementation` line. Never conclude "chain not
+  engaged" from the load log; identify behaviourally (`draft_n` > 0 only
+  where text repeats) or by the per-task `draft acceptance = ... mean len`
+  line. It costs ~64 MiB of host-RAM hash table.
+
+### Drafter KV: read the window policy from the header
+
+The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)`
+(`common/speculative.cpp`) — there is no flag to cap it — so a drafter that
+declares NO sliding window caches the whole target window on whatever device
+`--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB
+of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B
+= 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first
+draft graph capture with `ROCm error: out of memory` AFTER the model loaded
+and `/health` answered (measured 2026-09-16). Read `sliding_window` out of
+the drafter's GGUF header before sizing any spec-decode deployment, and
+prefer a skewed `-ts` over a context cut when the full window matters.
+
 DFlash on llama.cpp:
 
 ```
@@ -209,6 +254,151 @@ docker run --rm --name vllm-qwen36-dflash \
 
 History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
 
+## A/B protocol on a live llama.cpp endpoint
+
+Verdicts are **traffic-shaped**: a proposer can only draft text it has seen,
+so a single prompt shape answers the wrong question. Use fixed shapes:
+
+| Shape | Prompt | Who it can favour |
+|---|---|---|
+| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
+| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
+| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
+| identity | short deterministic question | controls for lossless verification |
+
+Design the echo passage so no long span repeats INSIDE it — otherwise you
+measure the matcher against itself. For `structured`, give each line a unique
+id: repeated lines make it a negative control, not a repetition test.
+
+Protocol:
+
+1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait
+   for the port (a retired predecessor's published-port proxy leaves the
+   endpoint port in TIME_WAIT — the bind fails with address-in-use long after
+   the unit stopped).
+2. Per config: write an env file with the overrides, start the serve script
+   with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness:
+   `scripts/bench-spec-ab.py <env-file> <serve-script>`.
+3. **Restore the unit in a `finally`** and shout if it does not come back
+   healthy. A crashed harness that leaves production down is worse than no
+   harness.
+4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff
+   the greedy text. Speculation only buys speed; differing texts are a
+   correctness bug. Empty answers (`content: null` when reasoning ate the
+   budget) make the comparison vacuous — bench clients must fall back to
+   `reasoning_content`, and echo/structured prompts should disable thinking
+   per request: `chat_template_kwargs: {"enable_thinking": false}`.
+5. **Sweep the depth, not just the mode** — draft depth is model-specific and
+   single-valued; deeper often loses (measured: at n=7 a third of proposals
+   survived, at n=3 more than half).
+6. **Sweep the ngram window in the same run as the configs**: too short a
+   `--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments —
+   rejected, verification still paid (measured 43.9 → 36.0 tok/s).
+
+Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
+`predicted_per_second`; cumulative `/metrics`
+`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
+per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
+while tokens flow = the proposer is NOT engaged, whatever the flags say. A
+zero-draft config is a useful negative control: it proves the other config's
+drafts came from the proposer under test.
+
+Harness hygiene: absolute script paths (a relative path plus `cd` to its own
+dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches
+your own command line); nested f-strings with backslash escapes are a
+SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with
+concatenation. Report `decode tok/s` per shape plus acceptance and mean
+accepted length; a config that wins on echo and loses on prose is a traffic
+decision, not a winner. Wire the winner into the deployed env file with the
+measured table as its justification comment; record losing numbers with "do
+not switch without re-measuring".
+
+The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
+slot split, quant rung — not just spec flags) is the transient systemd
+drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
+
+## Is it worth it? (arithmetic before deployment)
+
+Answer against the INCUMBENT, not against zero.
+
+**Step 0 — check whether the target already speculates.** Many engines ship a
+drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`.
+halogen-flash-server: MTP head ON by default — read its own accounting, e.g.
+`serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
+llama.cpp: `/metrics` counters and per-request `draft acceptance` lines.
+
+**Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 /
+(client-visible tok/s ÷ commit-per-round); ask for the rate the operator
+actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic).
+Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a
+plain model on the same box.
+
+**Step 2 — arithmetic:**
+
+```
+round_time = K * d_s + 1.1 * t_s        # K drafts proposed, K+1 verified in ONE target pass
+A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
+rate       = A / round_time             # compare to the incumbent
+break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
+```
+
+Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67
+commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even
+`p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6`
+gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot
+pay for K sequential forwards.
+
+**Step 3 — classify the drafter by per-round cost:**
+
+| drafter | cost per round | needs |
+|---|---|---|
+| plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target |
+| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
+| trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
+
+Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
+states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
+
+**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
+at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
+tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
+for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
+continuation agreement on identical prompts, two API calls, no weights moved.
+llama.cpp can also synthesize the projection without any draft artifact:
+`--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
+PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
+18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
+usually sparse MoE, the worst case for drafting.
+
+**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
+engines refuse external drafters by design: halogen-flash-server's
+`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
+name is a deliberate 400 (a silent downgrade would make comparisons lie);
+there is no device routing in its loopback token protocol, so a drafter can
+never sit on another card. Pure llama.cpp DOES route the draft:
+`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
+`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
+One process sees only the devices its backends were built with — CUDA+Vulkan
+is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
+coexist); a CUDA-only build has no `Vulkan0` to draft with.
+
+**Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation
+compares token IDs, so draft and target must map id → same string. Verify
+cheaply from the GGUF headers (first ~8 MB over HTTP, no full download):
+parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256
+over the first 2000 tokens —
+
+```
+vocab=248320 merges=247587 model=gpt2 pre=qwen35
+sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable
+```
+
+Identity is necessary, NOT sufficient — the drafter still has to be cheap and
+accepted. Report in this order: does the target already draft; break-even
+`p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate;
+whether the drafter fits (VRAM) and the attachment surface exists; the one
+measurement that would settle it.
+
 ## Notes
 
 - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
@@ -225,4 +415,8 @@ History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 20
 - [[Dankclaw System Specs]]
 - [[Dankserver System Specs]]
 - [[VLLM Overview]]
-- [[deployment]]
+- [[deployment]]
+- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
+- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
+- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle