docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192)

AGENTS.md

@@ -8,7 +8,7 @@ local machines. Topology (2026-08-17 redo):
 | Target | Host | Runtime | Service | Model | Port |
 |--------|------|---------|---------|-------|------|
 | Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
-| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 |
+| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 |
 
 The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
 NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
@@ -119,6 +119,15 @@ flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
 `flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
 falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
 
+**Tool calling (frank-default-small):** the server runs
+`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
+chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
+The template adds a concrete worked tool-call example and defaults tool
+requests to non-thinking (Qwen3 tool calls only work reliably with
+thinking off; normal requests keep thinking). The Avesed INT8 quant
+emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
+format cleanly.
+
 ### Env var naming rule — HIP runtime collision
 
 The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,

README.md

@@ -15,7 +15,7 @@ llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.
 | Service | Host | Runtime | Model | Port |
 |---------|------|---------|-------|------|
 | frank-default-large | dankserver | llama.cpp (patched FastMTP build) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx | 7770 |
-| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | Avesed Qwen3.6-35B-A3B INT8 W8A8, full 262144 ctx, no KV quantization, 4 seqs, ngram spec | 52625 |
+| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, full 262144 ctx, no KV quantization, 4 seqs, batch 8192, ngram spec + tool calling | 52625 |
 
 The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC
 worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in

docs/deployment.md

@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
 | Target | Host | Runtime | Service | Port |
 |--------|------|---------|---------|------|
 | Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
-| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 |
+| Dankclaw | dank@dankclaw | vLLM ROCm (nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec + tool calling) | frank-default-small | 52625 |
 
 ## Workflow
 
@@ -117,11 +117,13 @@ batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
 ### Dankclaw — frank-default-small
 
 The active service on Dankclaw is vLLM (ROCm) serving
-`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name
+`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
 `frank-default-small` on port 52625. Full 262144 context, unquantized KV
-cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram
+cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
-speculative decoding, `--reasoning-parser qwen3`. The unit file is
+speculative decoding, `--reasoning-parser qwen3`, and tool calling
-`scripts/frank-default-small.service`. It launches through
+(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
+chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The
+unit file is `scripts/frank-default-small.service`. It launches through
 `scripts/serve-frank-small.sh`, which reads tunables from
 `scripts/frank-default-small.env`.
 

docs/frank-redo-2026-08-17.md

@@ -9,7 +9,7 @@ The old scripts moved to `old/scripts/` in this repository.
 | Service | Host | Runtime | Model | Port |
 |---------|------|---------|-------|------|
 | frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
-| frank-default-small | Dankclaw | vLLM ROCm | Avesed Qwen3.6-35B-A3B INT8 W8A8 | 52625 |
+| frank-default-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
 
 The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
 tensor split, no NPU.
@@ -31,14 +31,20 @@ tensor split, no NPU.
   `--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
   (48/64/24).
 
-### frank-default-small — Avesed/Qwen3.6-35B-A3B-INT8-W8A8
+### frank-default-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
 
 - Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
   layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
-- Quant: compressed-tensors INT8 W8A8 (symmetric per-channel weights,
+- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token
-  dynamic per-token activations). Expert FFNs quantized; attention, router,
+  dynamic activations). MoE experts stored as per-expert nn.Linear
-  DeltaNet, lm_head kept bf16. Weights ~38 GB.
+  triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB;
-- Served with vLLM ROCm, ngram speculative decoding.
+  the vision tower, router, lm_head stay BF16.
+- Served with vLLM ROCm, ngram speculative decoding, and tool calling.
+- Swap history: the initial deployment used
+  `Avesed/Qwen3.6-35B-A3B-INT8-W8A8` (compressed-tensors INT8). Its tool
+  calls were malformed (the model emitted `<function=get_weather(city=...)>`
+  instead of the Qwen3 XML structure). The Quark quant follows the format
+  cleanly; swapped in the evening of 2026-08-17.
 
 ## Context Math (frank-default-large, RTX 3090)
 
@@ -139,3 +145,47 @@ on gfx1151; newer vLLM releases may already include it.
 - HF token: `~/.cache/huggingface/token` on both machines (mode 600).
 - OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
   Dankserver endpoint.
+
+## Evening Session (2026-08-17): tool calling, model swap, batch 8192
+
+Three changes landed after the initial deployment:
+
+### 1. Tool calling on frank-default-small (HTTP 400 fix)
+
+Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
+unless the server enables tool calling. Fixes applied:
+
+- `--enable-auto-tool-choice --tool-call-parser qwen3_xml` (the Avesed /
+  Quark chat template renders the Qwen3 XML tool format, and qwen3_xml
+  is the matching parser).
+- Custom chat template `/home/dank/models/qwen36-chat_template-tools.jinja`:
+  derived from the repo's `chat_template.jinja` (the tokenizer_config.json
+  ships no template, so vLLM would use its built-in one, which does not
+  render the XML tool instructions). The custom template adds:
+  - a concrete WORKED EXAMPLE built from the actual tool schema
+    (name + parameter keys) — models follow concrete examples much
+    better than the generic placeholder;
+  - tool requests default to non-thinking (`enable_thinking` unset + tools
+    present -> empty think block). Qwen3 tool calls only work reliably
+    with thinking off; normal requests keep thinking.
+
+Known pitfall: the earlier Avesed model still emitted malformed XML even
+with the strong template (`<function=get_weather(city=...)>`); that was
+model behavior, which motivated the swap.
+
+### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
+
+`frank-default-small` now serves
+`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
+vLLM-native expert layout). The Avesed compressed-tensors quant's
+Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
+tool requests (no chat_template_kwargs) return clean
+`tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]`;
+non-tool requests keep reasoning. The Avesed checkpoint stays in the HF
+cache for rollback.
+
+### 3. Batch 8192
+
+`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request).
+vLLM has no separate ubatch knob; the scheduler issues one batch per
+step up to this token cap.

docs/index.md

@@ -21,8 +21,8 @@ FastMTP 32K draft + ngram-mod, 128K ctx]
         D[gfx1151 iGPU unified memory]
         E[vLLM ROCm
 frank-default-small, port 52625]
-        F[Avesed Qwen3.6-35B-A3B INT8 W8A8
+        F[nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8
-full 262144 ctx, ngram spec, 4 seqs]
+full 262144 ctx, ngram spec + tools, 4 seqs]
         D --> E --> F
     end