docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192)
This commit is contained in:
1 parent
1cebbf47f6
commit
115b0af8dc
5 files changed
+76
-15
No files matched your search
@@ -8,7 +8,7 @@ local machines. Topology (2026-08-17 redo):
|
||||
| Target | Host | Runtime | Service | Model | Port |
|
||||
|--------|------|---------|---------|-------|------|
|
||||
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
|
||||
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 |
|
||||
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 |
|
||||
|
||||
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
|
||||
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
|
||||
@@ -119,6 +119,15 @@ flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
|
||||
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
|
||||
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
|
||||
|
||||
**Tool calling (frank-default-small):** the server runs
|
||||
`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
|
||||
chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
|
||||
The template adds a concrete worked tool-call example and defaults tool
|
||||
requests to non-thinking (Qwen3 tool calls only work reliably with
|
||||
thinking off; normal requests keep thinking). The Avesed INT8 quant
|
||||
emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
|
||||
format cleanly.
|
||||
|
||||
### Env var naming rule — HIP runtime collision
|
||||
|
||||
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
|
||||
|
||||
Reference in new issue
Block a user