docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192)

This commit is contained in:
Frank committed 2026-08-17 18:01:02 -07:00
1 parent 1cebbf47f6
commit 115b0af8dc
5 files changed
+76 -15

No files matched your search

+10 -1
View File
@@ -8,7 +8,7 @@ local machines. Topology (2026-08-17 redo):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
@@ -119,6 +119,15 @@ flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
**Tool calling (frank-default-small):** the server runs
`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
The template adds a concrete worked tool-call example and defaults tool
requests to non-thinking (Qwen3 tool calls only work reliably with
thinking off; normal requests keep thinking). The Avesed INT8 quant
emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
format cleanly.
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,