docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192)
This commit is contained in:
1 parent
1cebbf47f6
commit
115b0af8dc
5 files changed
+76
-15
No files matched your search
+7
-5
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 |
|
||||
| Dankclaw | dank@dankclaw | vLLM ROCm (nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec + tool calling) | frank-default-small | 52625 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -117,11 +117,13 @@ batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
|
||||
### Dankclaw — frank-default-small
|
||||
|
||||
The active service on Dankclaw is vLLM (ROCm) serving
|
||||
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name
|
||||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
|
||||
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
|
||||
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram
|
||||
speculative decoding, `--reasoning-parser qwen3`. The unit file is
|
||||
`scripts/frank-default-small.service`. It launches through
|
||||
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
|
||||
speculative decoding, `--reasoning-parser qwen3`, and tool calling
|
||||
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
|
||||
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The
|
||||
unit file is `scripts/frank-default-small.service`. It launches through
|
||||
`scripts/serve-frank-small.sh`, which reads tunables from
|
||||
`scripts/frank-default-small.env`.
|
||||
|
||||
|
||||
Reference in new issue
Block a user