Commit Graph
5 Commits
Author SHA1 Message Date
Frank 1cebbf47f6 frank-default-small: swap to nameistoken Quark W8A8 INT8, batch 8192
- Model: nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark W8A8,
  vLLM-native expert layout). The Avesed quant emitted malformed Qwen3
  XML tool calls; the Quark model follows the format cleanly.
- max-num-batched-tokens lowered 16384 -> 8192 (user request).
- Chat template: neutral path qwen36-chat_template-tools.jinja (same
  base template; adds a concrete worked tool-call example).
2026-08-17 17:49:27 -07:00
Frank 20108e5b7a frank-default-small: use the repo chat_template.jinja so tools render in Qwen3 XML
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
2026-08-17 17:13:35 -07:00
Frank 5108af11aa frank-default-small: enable tool calling (--enable-auto-tool-choice --tool-call-parser qwen3_xml)
Clients send tool_choice=auto; without these flags vLLM rejects with
HTTP 400. The Avesed chat template renders the Qwen3 XML tool format,
so qwen3_xml is the matching parser.
2026-08-17 16:50:23 -07:00
Frank 27788f6f2c fix: make serve scripts + deploy.sh executable 2026-08-17 15:24:40 -07:00
Frank d9af7f1d19 fleet redo 2026-08-17: retire PP+NPU, new topology
- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook
2026-08-17 15:23:46 -07:00