Commit Graph
8 Commits
Author SHA1 Message Date
Frank cd14d7bf68 Use ngram-mod and prioritize attention experts 2026-08-25 01:40:06 -07:00
Frank f13124f977 Select hybrid expert placement from benchmark 2026-08-25 01:08:00 -07:00
Frank 339325719e Reduce small model context and rebalance GPU layers 2026-08-25 00:59:41 -07:00
Frank 7a17984dbc Tune small model expert placement 2026-08-25 00:31:05 -07:00
Frank af6ffeeb64 Switch frank-default-small to Ornith Q6_K 2026-08-25 00:29:14 -07:00
Frank 7ba1acdbc5 frank-default-small: lower context by 32k per stream for cache headroom
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the
.env, models.json (contextWindow/maxTokens 229376 per slot), and the
index/deployment/ornith runbook references.
2026-08-24 21:50:54 -07:00
Frank 4462aa992c Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
  native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
  explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
  (replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
  ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
  (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
  retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
  --chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
  update index, deployment, and both system specs; models.json names.

Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.
2026-08-24 21:35:11 -07:00
Frank aa410f4439 Reorganize: llm/scripts -> scripts (LLM serve scripts, units, env tunables)
The LLM fleet serve scripts, systemd units, env tunables, and bench
harnesses join the deploy scripts. The vllmScripts deploy.sh is
removed here; its functionality is folded into the unified
scripts/deploy.sh in a follow-up commit.

bench-spec-stack.sh gains the executable bit (required by the
scripts_syntax test seam).
2026-08-22 15:07:31 -07:00