dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: chain DFlash2 with ngram-mod -- measured free where it cannot match

Browse Source 
ngram-mod is draftless (added to the spec config vector unconditionally, while
every model-backed drafter needs ctx_dft), so unlike MTP it can share the
process with DFlash2. Chained at the fleet's 48/64 window, over three shapes:

  prose 1k/512      dflash 43.77 | ngram 17.72 (0 drafts) | chain 43.65
  echo 1.5k/400     dflash 76.82 | ngram 102.50          | chain 200.58
  structured/400    dflash 91.08 | ngram 17.80 (0 drafts) | chain 91.22
  identity          dflash 70.37 | ngram 17.77           | chain 70.40

On prose the chain is byte-identical to the drafter alone (same 119 drafts/49
accepts) -- that is what makes it a safe default -- and on prompt-echo traffic
it is 2.6x. VRAM is unchanged: the matcher is a 64 MiB host-RAM hash table.

The window sweep says keep the long one: 48/64 beats 16/32 (161.0) and 8/16
(113.8) on echo, and shortening it costs prose (43.9 -> 36.0) because the
matcher then proposes on ordinary fragments and the rejections still have to be
verified. That is the 'ngram hurts' result this fleet hit twice before.

Two documentation traps recorded with it: ngram-mod prints NO startup banner in
this build (its drafts are only visible behaviourally), and the harness's
'structured' shape is a negative control because its lines differ.
This commit is contained in:
Frank
2026-09-16 16:44:45 -07:00
parent 95936bbc81
commit a4f4429a99
3 changed files with 111 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_medium_unit.bats
+21 -2
View File
@@ -61,11 +61,30 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
grep -q '^export HIP_VISIBLE_DEVICES$' "$SERVE"
}
@test "the DFlash2 drafter is a draft model at a depth its block allows" {
grep -q '^SPEC_TYPE=draft-dflash$' "$ENV_FILE"
@test "the drafter runs in a chain with the draftless n-gram matcher" {
# The chain is the measured winner (see the runbook's A/B): free where the
# matcher cannot match, ~2.6x the drafter alone on prompt-echo traffic. A
# bare `draft-dflash` here would silently give that up.
grep -qE '^SPEC_TYPE=draft-dflash,ngram-mod$' "$ENV_FILE"
grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DFlash2-Q8_0\.gguf$' "$ENV_FILE"
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
grep -q -- '--spec-draft-model "$DRAFT_MODEL"' "$SERVE"
# ngram-mod's own window -- and only when the type asks for it, so a pure
# DFlash or MTP configuration is not quietly handed n-gram parameters.
grep -q '^SPEC_NGRAM_N_MATCH=24$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_N_MIN=48$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_N_MAX=64$' "$ENV_FILE"
grep -q -- '--spec-ngram-mod-n-match "$SPEC_NGRAM_N_MATCH"' "$SERVE"
grep -q -- '--spec-ngram-mod-n-min "$SPEC_NGRAM_N_MIN"' "$SERVE"
grep -q -- '--spec-ngram-mod-n-max "$SPEC_NGRAM_N_MAX"' "$SERVE"
python3 -c "
lines = open('$SERVE').read().splitlines()
guard = next(i for i, l in enumerate(lines) if '*ngram*' in l)
indent = len(lines[guard]) - len(lines[guard].lstrip())
end = next(i for i in range(guard + 1, len(lines))
if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent)
assert '--spec-ngram-mod-n-match' in chr(10).join(lines[guard:end]), 'ngram flags must sit in the ngram branch'
"
# DFlash2 drafts into a block of 8, so at most 7 draft tokens per
# verification step -- llama.cpp asserts n_max <= block_size - 1.
local n_max