dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192)

Browse Source
This commit is contained in:
Frank
2026-08-17 18:01:02 -07:00
parent 1cebbf47f6
commit 115b0af8dc
5 changed files with 76 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+10 -1
View File
@@ -8,7 +8,7 @@ local machines. Topology (2026-08-17 redo):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
@@ -119,6 +119,15 @@ flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
**Tool calling (frank-default-small):** the server runs
`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
The template adds a concrete worked tool-call example and defaults tool
requests to non-thinking (Qwen3 tool calls only work reliably with
thinking off; normal requests keep thinking). The Avesed INT8 quant
emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
format cleanly.
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
README.md
+1 -1
View File
@@ -15,7 +15,7 @@ llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | dankserver | llama.cpp (patched FastMTP build) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx | 7770 |
| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | Avesed Qwen3.6-35B-A3B INT8 W8A8, full 262144 ctx, no KV quantization, 4 seqs, ngram spec | 52625 |
| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, full 262144 ctx, no KV quantization, 4 seqs, batch 8192, ngram spec + tool calling | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC
worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in
docs/deployment.md
+7 -5
View File
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 |
| Dankclaw | dank@dankclaw | vLLM ROCm (nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec + tool calling) | frank-default-small | 52625 |
## Workflow
@@ -117,11 +117,13 @@ batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
### Dankclaw — frank-default-small
The active service on Dankclaw is vLLM (ROCm) serving
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram
speculative decoding, `--reasoning-parser qwen3`. The unit file is
`scripts/frank-default-small.service`. It launches through
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
speculative decoding, `--reasoning-parser qwen3`, and tool calling
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The
unit file is `scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.
docs/frank-redo-2026-08-17.md
+56 -6
View File
@@ -9,7 +9,7 @@ The old scripts moved to `old/scripts/` in this repository.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-default-small | Dankclaw | vLLM ROCm | Avesed Qwen3.6-35B-A3B INT8 W8A8 | 52625 |
| frank-default-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
tensor split, no NPU.
@@ -31,14 +31,20 @@ tensor split, no NPU.
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
(48/64/24).
### frank-default-small — Avesed/Qwen3.6-35B-A3B-INT8-W8A8
### frank-default-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
- Quant: compressed-tensors INT8 W8A8 (symmetric per-channel weights,
dynamic per-token activations). Expert FFNs quantized; attention, router,
DeltaNet, lm_head kept bf16. Weights ~38 GB.
- Served with vLLM ROCm, ngram speculative decoding.
- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token
dynamic activations). MoE experts stored as per-expert nn.Linear
triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB;
the vision tower, router, lm_head stay BF16.
- Served with vLLM ROCm, ngram speculative decoding, and tool calling.
- Swap history: the initial deployment used
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` (compressed-tensors INT8). Its tool
calls were malformed (the model emitted `<function=get_weather(city=...)>`
instead of the Qwen3 XML structure). The Quark quant follows the format
cleanly; swapped in the evening of 2026-08-17.
## Context Math (frank-default-large, RTX 3090)
@@ -139,3 +145,47 @@ on gfx1151; newer vLLM releases may already include it.
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
Dankserver endpoint.
## Evening Session (2026-08-17): tool calling, model swap, batch 8192
Three changes landed after the initial deployment:
### 1. Tool calling on frank-default-small (HTTP 400 fix)
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
unless the server enables tool calling. Fixes applied:
- `--enable-auto-tool-choice --tool-call-parser qwen3_xml` (the Avesed /
Quark chat template renders the Qwen3 XML tool format, and qwen3_xml
is the matching parser).
- Custom chat template `/home/dank/models/qwen36-chat_template-tools.jinja`:
derived from the repo's `chat_template.jinja` (the tokenizer_config.json
ships no template, so vLLM would use its built-in one, which does not
render the XML tool instructions). The custom template adds:
- a concrete WORKED EXAMPLE built from the actual tool schema
(name + parameter keys) — models follow concrete examples much
better than the generic placeholder;
- tool requests default to non-thinking (`enable_thinking` unset + tools
present -> empty think block). Qwen3 tool calls only work reliably
with thinking off; normal requests keep thinking.
Known pitfall: the earlier Avesed model still emitted malformed XML even
with the strong template (`<function=get_weather(city=...)>`); that was
model behavior, which motivated the swap.
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
`frank-default-small` now serves
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
vLLM-native expert layout). The Avesed compressed-tensors quant's
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
tool requests (no chat_template_kwargs) return clean
`tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]`;
non-tool requests keep reasoning. The Avesed checkpoint stays in the HF
cache for rollback.
### 3. Batch 8192
`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request).
vLLM has no separate ubatch knob; the scheduler issues one batch per
step up to this token cap.
docs/index.md
+2 -2
View File
@@ -21,8 +21,8 @@ FastMTP 32K draft + ngram-mod, 128K ctx]
D[gfx1151 iGPU unified memory]
E[vLLM ROCm
frank-default-small, port 52625]
F[Avesed Qwen3.6-35B-A3B INT8 W8A8
full 262144 ctx, ngram spec, 4 seqs]
F[nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8
full 262144 ctx, ngram spec + tools, 4 seqs]
D --> E --> F
end