chore: frank-default-small — ngram-mod, drop DFlash draft

- spec-type → ngram-mod (no separate draft model)
- drop --model-draft, --spec-draft-n-max; add --spec-ngram-mod-n-min/max/match
- DFlash batch default fallback SPEC_TYPE → ngram-mod
- env: SPEC_TYPE=ngram-mod, NGRAM_MOD_N_MIN=48, N_MAX=64, N_MATCH=24
- DRAFT_FILE env var emptied (not used under ngram-mod)
This commit is contained in:
Frank committed 2026-08-11 02:29:32 -07:00
1 parent 5c9e812221
commit fd528ab364
2 files changed
+18 -11

No files matched your search

+8 -3
View File
@@ -1,4 +1,4 @@
# Tunables for frank-default-small (HauhauCS Q6_K_P + DFlash draft, dankserver)
# Tunables for frank-default-small (HauhauCS Q6_K_P, ngram-mod spec, dankserver)
# Runtime: llama-server b10108; RTX 3090 24GB; Ryzen 7 7700X; 62GB RAM.
MODEL_DIR=/home/dank/models
ALIAS=frank-default-small
@@ -8,8 +8,13 @@ PORT=7770
# (Reduced from 262144/stream per operator: allows the larger batch to fit VRAM.)
CTX_SIZE=600000
PARALLEL=3
SPEC_TYPE=draft-dflash
SPEC_N_MAX=7
SPEC_TYPE=ngram-mod
# Ngram-mod speculative decoding: look back this many tokens to guess the
# next tokens. Defaults (48/64/24) are a reasonable starting point; tune if
# you want more or fewer draft lookahead tokens.
NGRAM_MOD_N_MIN=48
NGRAM_MOD_N_MAX=64
NGRAM_MOD_N_MATCH=24
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16
# All 40 layers keep their MoE expert weights on CPU (targeted expert offload).