diff --git a/AGENTS.md b/AGENTS.md index c0c08c1..ff07438 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -298,15 +298,14 @@ len ≈ 3` (MTP engaged); an image request is answered from the projector while the halogen side refuses images with a named reason; 22,362 MiB of the card's 24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`. -Runbook: `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`. - -### frank-medium — llama.cpp layer split + DFlash2 (Dankserver, ACTIVE) +### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE) Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root systemd unit `scripts/frank-medium.service`) serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** -(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter, in **one process across -both R9700s**, `--split-mode layer`, 262144 ctx / 1 slot, text only. +(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the draftless +**ngram-mod** matcher, in **one process across both R9700s**, `--split-mode +layer`, 262144 ctx / 1 slot, text only. **Layer split is why both cards now serve ONE model.** Tensor parallelism is impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism — @@ -469,6 +468,27 @@ linked runbooks. the MAIN model's weights (`shares_model`), so there is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP tensors at all, and pointing `--model-draft` at the file would load it twice. +- **What can be chained, and what cannot.** `--spec-type a,b` builds every + enabled implementation and chains drafting through them, in a fixed internal + order (ngram types first, then the drafters — CLI order is irrelevant). The + gate is `ctx_dft`: **draftless** types (the ngram family) are added + unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and + `draft-eagle3` all require a draft CONTEXT. So a chain may hold one + context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod` + is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte- + identical to it on prose at the fleet's 48/64 window) — but + `draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model + doesn't contain MTP layers"*, because MTP builds its context against whatever + model the draft context holds. Two context-backed types cannot share one + process, and MTP's head only loads when MTP **is** the draft model. +- **ngram-mod prints no startup banner.** The load log shows the drafter's + `adding speculative implementation 'draft-dflash'` and nothing at all for the + matcher, so a chain can look unengaged while it is working. Identify it + behaviourally (`draft_n` > 0 only where text repeats) or by the per-task + `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB + host-RAM hash table) and its window decides which traffic it can see: too + short and it proposes on ordinary prose fragments, which costs verification + without acceptance. - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that then served frank-large: every tool-using agent on that endpoint was broken