docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192)

This commit is contained in:
Frank committed 2026-08-17 18:01:02 -07:00
1 parent 1cebbf47f6
commit 115b0af8dc
5 files changed
+76 -15

No files matched your search

+7 -5
View File
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 |
| Dankclaw | dank@dankclaw | vLLM ROCm (nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec + tool calling) | frank-default-small | 52625 |
## Workflow
@@ -117,11 +117,13 @@ batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
### Dankclaw — frank-default-small
The active service on Dankclaw is vLLM (ROCm) serving
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram
speculative decoding, `--reasoning-parser qwen3`. The unit file is
`scripts/frank-default-small.service`. It launches through
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
speculative decoding, `--reasoning-parser qwen3`, and tool calling
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The
unit file is `scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.