diff --git a/AGENTS.md b/AGENTS.md index ace6c12..d4bfb50 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ local machines. Topology (2026-08-17 redo): | Target | Host | Runtime | Service | Model | Port | |--------|------|---------|---------|-------|------| | Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 | -| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 | +| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 | The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The RTX 3090 now runs the whole 27B model standalone @@ -119,6 +119,15 @@ flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in `flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder falls back to torch SDPA and engine init OOMs (256 GiB score matrix). +**Tool calling (frank-default-small):** the server runs +`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom +chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`). +The template adds a concrete worked tool-call example and defaults tool +requests to non-thinking (Qwen3 tool calls only work reliably with +thinking off; normal requests keep thinking). The Avesed INT8 quant +emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the +format cleanly. + ### Env var naming rule — HIP runtime collision The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, diff --git a/README.md b/README.md index 698e6a1..ad74da6 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| | frank-default-large | dankserver | llama.cpp (patched FastMTP build) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx | 7770 | -| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | Avesed Qwen3.6-35B-A3B INT8 W8A8, full 262144 ctx, no KV quantization, 4 seqs, ngram spec | 52625 | +| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, full 262144 ctx, no KV quantization, 4 seqs, batch 8192, ngram spec + tool calling | 52625 | The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in diff --git a/docs/deployment.md b/docs/deployment.md index a4fdebd..bd286a5 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 | -| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 | +| Dankclaw | dank@dankclaw | vLLM ROCm (nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec + tool calling) | frank-default-small | 52625 | ## Workflow @@ -117,11 +117,13 @@ batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is ### Dankclaw — frank-default-small The active service on Dankclaw is vLLM (ROCm) serving -`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name +`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name `frank-default-small` on port 52625. Full 262144 context, unquantized KV -cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram -speculative decoding, `--reasoning-parser qwen3`. The unit file is -`scripts/frank-default-small.service`. It launches through +cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram +speculative decoding, `--reasoning-parser qwen3`, and tool calling +(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom +chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The +unit file is `scripts/frank-default-small.service`. It launches through `scripts/serve-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. diff --git a/docs/frank-redo-2026-08-17.md b/docs/frank-redo-2026-08-17.md index be16daf..cd748b7 100644 --- a/docs/frank-redo-2026-08-17.md +++ b/docs/frank-redo-2026-08-17.md @@ -9,7 +9,7 @@ The old scripts moved to `old/scripts/` in this repository. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| | frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 | -| frank-default-small | Dankclaw | vLLM ROCm | Avesed Qwen3.6-35B-A3B INT8 W8A8 | 52625 | +| frank-default-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 | The RTX 3090 runs the whole 27B model standalone. No RPC worker, no tensor split, no NPU. @@ -31,14 +31,20 @@ tensor split, no NPU. `--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback (48/64/24). -### frank-default-small — Avesed/Qwen3.6-35B-A3B-INT8-W8A8 +### frank-default-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 - Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40 layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx). -- Quant: compressed-tensors INT8 W8A8 (symmetric per-channel weights, - dynamic per-token activations). Expert FFNs quantized; attention, router, - DeltaNet, lm_head kept bf16. Weights ~38 GB. -- Served with vLLM ROCm, ngram speculative decoding. +- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token + dynamic activations). MoE experts stored as per-expert nn.Linear + triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB; + the vision tower, router, lm_head stay BF16. +- Served with vLLM ROCm, ngram speculative decoding, and tool calling. +- Swap history: the initial deployment used + `Avesed/Qwen3.6-35B-A3B-INT8-W8A8` (compressed-tensors INT8). Its tool + calls were malformed (the model emitted `` + instead of the Qwen3 XML structure). The Quark quant follows the format + cleanly; swapped in the evening of 2026-08-17. ## Context Math (frank-default-large, RTX 3090) @@ -139,3 +145,47 @@ on gfx1151; newer vLLM releases may already include it. - HF token: `~/.cache/huggingface/token` on both machines (mode 600). - OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the Dankserver endpoint. + +## Evening Session (2026-08-17): tool calling, model swap, batch 8192 + +Three changes landed after the initial deployment: + +### 1. Tool calling on frank-default-small (HTTP 400 fix) + +Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400 +unless the server enables tool calling. Fixes applied: + +- `--enable-auto-tool-choice --tool-call-parser qwen3_xml` (the Avesed / + Quark chat template renders the Qwen3 XML tool format, and qwen3_xml + is the matching parser). +- Custom chat template `/home/dank/models/qwen36-chat_template-tools.jinja`: + derived from the repo's `chat_template.jinja` (the tokenizer_config.json + ships no template, so vLLM would use its built-in one, which does not + render the XML tool instructions). The custom template adds: + - a concrete WORKED EXAMPLE built from the actual tool schema + (name + parameter keys) — models follow concrete examples much + better than the generic placeholder; + - tool requests default to non-thinking (`enable_thinking` unset + tools + present -> empty think block). Qwen3 tool calls only work reliably + with thinking off; normal requests keep thinking. + +Known pitfall: the earlier Avesed model still emitted malformed XML even +with the strong template (``); that was +model behavior, which motivated the swap. + +### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8 + +`frank-default-small` now serves +`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB, +vLLM-native expert layout). The Avesed compressed-tensors quant's +Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain +tool requests (no chat_template_kwargs) return clean +`tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]`; +non-tool requests keep reasoning. The Avesed checkpoint stays in the HF +cache for rollback. + +### 3. Batch 8192 + +`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request). +vLLM has no separate ubatch knob; the scheduler issues one batch per +step up to this token cap. diff --git a/docs/index.md b/docs/index.md index dc1544c..93cead5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,8 +21,8 @@ FastMTP 32K draft + ngram-mod, 128K ctx] D[gfx1151 iGPU unified memory] E[vLLM ROCm frank-default-small, port 52625] - F[Avesed Qwen3.6-35B-A3B INT8 W8A8 -full 262144 ctx, ngram spec, 4 seqs] + F[nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 +full 262144 ctx, ngram spec + tools, 4 seqs] D --> E --> F end