From 4462aa992ca1ddba9ddb656f4c3d6f6be921fe3b Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 24 Aug 2026 21:35:11 -0700 Subject: [PATCH] Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template - frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU (replaces --n-cpu-moe), Qwen-Sharp chat template. - frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0 ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the retired PromptForge .pfs sidecar lanes. - Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via --chat-template-file on both services. - Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages; update index, deployment, and both system specs; models.json names. Smoke-tested both models on the real hardware on scratch ports before commit: model loads, MTP draft attaches, Sharp template applies, chat completion returns. --- docs/LLM/Dankclaw System Specs.md | 2 +- docs/LLM/Dankserver System Specs.md | 2 +- docs/LLM/dealign-ornith-crack.md | 96 ++++++ docs/LLM/deployment.md | 43 +-- docs/LLM/index.md | 32 +- docs/LLM/kingjones-strix-mtp.md | 91 ++++++ docs/index.md | 4 +- scripts/chat-template.jinja | 453 ++++++++++++++++++++++++++++ scripts/frank-default-large.env | 75 ++--- scripts/frank-default-large.service | 2 +- scripts/frank-default-small.env | 59 ++-- scripts/frank-default-small.service | 2 +- scripts/serve-frank-large.sh | 66 ++-- scripts/serve-frank-small.sh | 56 ++-- 14 files changed, 827 insertions(+), 156 deletions(-) create mode 100644 docs/LLM/dealign-ornith-crack.md create mode 100644 docs/LLM/kingjones-strix-mtp.md create mode 100644 scripts/chat-template.jinja diff --git a/docs/LLM/Dankclaw System Specs.md b/docs/LLM/Dankclaw System Specs.md index fca2210..918fd9d 100644 --- a/docs/LLM/Dankclaw System Specs.md +++ b/docs/LLM/Dankclaw System Specs.md @@ -93,7 +93,7 @@ cable between claw and server is required for 10G. - Unified memory eliminates separate VRAM limits. The GPU uses system RAM. - The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget. -- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]]. +- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]]. - The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`. - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18. - 1.1 TB free disk space is ample for model storage. diff --git a/docs/LLM/Dankserver System Specs.md b/docs/LLM/Dankserver System Specs.md index fd3ca2a..459890d 100644 --- a/docs/LLM/Dankserver System Specs.md +++ b/docs/LLM/Dankserver System Specs.md @@ -60,7 +60,7 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0 ### Active Services -The active service on Dankserver is `frank-default-small.service`: the llama.cpp Qwen3.6-35B-A3B (Q6_K_P + ngram-mod) on port 7770. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. +The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q8_0) on port 7770, with MoE expert weights held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. ## Hardware Relevance diff --git a/docs/LLM/dealign-ornith-crack.md b/docs/LLM/dealign-ornith-crack.md new file mode 100644 index 0000000..2576a82 --- /dev/null +++ b/docs/LLM/dealign-ornith-crack.md @@ -0,0 +1,96 @@ +--- +type: reference +title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver +description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small. +tags: + - llamacpp + - cuda + - dankserver + - moe + - ornith + - mtp +timestamp: 2026-08-24 +--- + +# dealign Ornith-1.5-35B-A3B CRACK on Dankserver + +> **Active 2026-08-24.** `frank-default-small` serves this model. It +> replaced the HauhauCS Qwen3.6-35B-A3B Q6_K_P service on the same upstream +> llama.cpp binary (b10121). + +`frank-default-small` (port 7770) serves +`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` from `dealignai` on Hugging Face. It is +a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + +attention architecture, abliterated (uncensored). The MTP speculative head +is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target +model. + +## Model Files + +| File | Size | +|------|------| +| `Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` | 37.2 GB | +| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.90 GB | + +Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`. + +## Expert Offload — `--override-tensor`, not `--n-cpu-moe` + +The service pins the MoE expert weights to CPU with an **explicit tensor +override** rather than the layer-counter `--n-cpu-moe`. The override uses +the same internal regex that `--cpu-moe` / `--n-cpu-moe` expand: + +``` +OVERRIDE_EXPERT=\.ffn_(up|down|gate|gate_up)_(ch|)exps +OVERRIDE_BUFT=CPU +``` + +This is passed as `--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT"`. +Only the FFN expert tensors move to CPU; every other tensor stays on the +24 GB card up to `--gpu-layers`. To list the buffer type names on the host, +run a bogus override and read the error: + +``` +llama-server --model -c 64 -ot foo=BADTYPE +``` + +On Dankserver the available buffer types are `CPU` and `CUDA0`. + +## Speculative Decoding + +MoE model with a bundled MTP head -> native `draft-mtp`. The head is +auto-detected from the target (b10121 creates the MTP draft context against +the model file). ngram under MTP hurts on this family; MTP alone is the +proven best. `--spec-draft-n-max 4` is the knee. + +``` +--spec-type draft-mtp +--spec-draft-n-max 4 +``` + +## Launch Excerpt + +``` +llama-server \ + -m Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf \ + --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \ + --chat-template-file chat-template.jinja \ + --spec-type draft-mtp \ + --override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ + -ngl all -c 786432 -np 3 \ + --cache-type-k f16 --cache-type-v f16 --no-mmap \ + --temp 1.0 --top-p 0.95 --top-k 20 +``` + +## Known Issues + +1. The 37.2 GB Q8_0 file is large for the 62 GB RAM box; lower `PARALLEL` + or switch quant if VRAM or RAM is tight. +2. `--n-cpu-moe` remains supported; this service deliberately prefers the + explicit tensor override so only the expert tensors reach CPU. + +## See Also + +- [[deployment]] — deploy and update process +- [[Dankserver System Specs]] — the box +- [[frank-redo-2026-08-17]] — fleet topology history \ No newline at end of file diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index 6925581..d628b47 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -17,8 +17,8 @@ repository to a running service. | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 | +| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q8_0, native MTP draft, MoE experts offloaded via --override-tensor) | frank-default-small | 7770 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot) | frank-default-large | 7770 | ## Workflow @@ -122,32 +122,35 @@ curl -sf http://127.0.0.1:7770/health ### Dankserver — frank-default-small The active service on Dankserver is upstream llama.cpp (b10121) serving -`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the -alias `frank-default-small` on port 7770. It runs the stock -`/home/dank/llama.cpp/build/bin/llama-server` binary with the 38 MoE -experts on CPU (`--n-cpu-moe 38`) and 7 CPU threads, ngram-mod -speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both -K and V, context 786432 (3 slots of ~262K each), batch 4096 / ubatch 2048, -`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It -launches through `scripts/serve-frank-small.sh`, which reads tunables from -`scripts/frank-default-small.env`. +`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` under the alias `frank-default-small` +on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server` +binary with the MoE expert weights pinned to CPU by an explicit +`--override-tensor` regex (the expert `\.ffn_(up|down|gate|gate_up)_(ch|)exps` +pattern, buffer type `CPU`) and 7 CPU threads, native `draft-mtp` +speculative decoding (bundled MoE MTP head, n_max 4), f16 KV, context +786432 (3 slots of ~262K each), batch 4096 / ubatch 2048, `--no-mmap`. The +unit is `scripts/frank-default-small.service`. It launches through +`scripts/serve-frank-small.sh`, which reads tunables from +`scripts/frank-default-small.env`. See [[dealign-ornith-crack]]. ### Dankclaw — frank-default-large The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving -`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF -+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias -`frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs +`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` (Q6_0 AGENT, dense) +under the alias `frank-default-large` on port 7770. It runs `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm -7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP -(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context -(1 slot. Concurrent requests queue. The server never preempts a running -generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB -prompt cache, and reasoning in deepseek format. The unit file is +7.15 SDK in the retired ling3-ciru venv) with a standalone MTP Q4_0 draft +head (`--model-draft` and `--spec-type draft-mtp`), f16 KV, full 262144 +context (1 slot. Concurrent requests queue; the server never preempts a +running generation), batch 4096 / ubatch 4096, 16 context checkpoints, +8 GiB prompt cache, and reasoning in deepseek format. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-frank-large.sh`, which reads tunables from -`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]]. +`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template. +See [[kingjones-strix-mtp]]. +> The kingjones Q6 AGENT model has served the slot since 2026-08-24; the +> v3 PromptForge payload is retired on this slot. > Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was > retired the same day. See [[dflash2-large-vulkan]] for the archived runbook. diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 4a1c9e6..2270c0d 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -1,9 +1,10 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and -CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw -(frank-default-large). +(2026-08-24): upstream llama.cpp on Dankserver (frank-default-small, +Ornith-1.5-35B-A3B CRACK) and CIRU ROCmFPX llama.cpp on Dankclaw +(frank-default-large, kingjones Qwen3.8-27B ROCmFP4). Both services +run the Qwen-Sharp chat template. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -18,8 +19,8 @@ graph LR A[RTX 3090 24GB + 62GB RAM] B[upstream llama.cpp b10121 frank-default-small, port 7770] - C[Qwen3.6-35B-A3B-Uncensored Q6_K_P -ngram-mod, 38 MoE experts on CPU, 786432 ctx] + C[Ornith-1.5-35B-A3B-CRACK Q8_0 +native MTP, MoE experts offloaded via override, 786432 ctx] A --> B --> C end @@ -27,8 +28,8 @@ ngram-mod, 38 MoE experts on CPU, 786432 ctx] D[gfx1151 iGPU 64 GiB unified memory] E[CIRU ROCmFPX llama.cpp PromptForge v3 frank-default-large, port 7770] - F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 -262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision] + F[Qwen3.8-27B-Uncensored-ROCmFP4 Q6_0 AGENT +MTP draft + Sharp template, 262K ctx / 1 slot] D --> E --> F end @@ -47,8 +48,12 @@ frank-default-large, port 7770] - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on - Dankclaw (frank-default-large, active. Restored 2026-08-22) +- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model + on Dankclaw (frank-default-large, active 2026-08-24) +- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on + Dankserver (frank-default-small, active 2026-08-24) +- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime + on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small) @@ -56,8 +61,9 @@ frank-default-large, port 7770] ## CIRU Reference (Dankclaw) -- [[ciru-activefpx-promptforge]] — the v3 model, ROCm 7.15 toolchain, - and build history (active deployment) +- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain, + and build history (on Dankclaw; the slot now serves the kingjones + ROCmFP4 model, see [[kingjones-strix-mtp]]) - [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary identity, and experiment summary (retired 2026-08-22) @@ -71,8 +77,8 @@ frank-default-large, port 7770] - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware - notes and sizing (the model family now served by frank-default-small as - Q6_K_P) + notes and sizing (the historical family; frank-default-small now serves + Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]]) ## Memory Management diff --git a/docs/LLM/kingjones-strix-mtp.md b/docs/LLM/kingjones-strix-mtp.md new file mode 100644 index 0000000..7e66399 --- /dev/null +++ b/docs/LLM/kingjones-strix-mtp.md @@ -0,0 +1,91 @@ +--- +type: reference +title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw +description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-default-large. +tags: + - llama.cpp + - rocm + - dankclaw + - gfx1151 + - kingjones + - rocmfp4 + - mtp +timestamp: 2026-08-24 +--- + +# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw + +> **Active 2026-08-24.** `frank-default-large` serves this model. It +> replaced the jcbtc v3 CIRU ActiveFPX PromptForge payload on the same +> ROCmFPX runtime and the same CIRU llama.cpp binary. See +> [[ciru-activefpx-promptforge]] for the retired v3 page. + +`frank-default-large` (port 7770) serves +`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging +Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to +the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as +a **separate** Q4_0 draft GGUF, so a bundled-head auto-detect does not apply +here; the head is loaded with `--model-draft`. + +## Model Files + +The card ships six text files plus the vision projector. The service uses +the Q6 AGENT file (requested for the fleet), the separate MTP draft head +and the Q8 vision projector. + +| File | Size | +|------|------| +| `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` | 23.15 GiB | +| `mtp-Qwen3.8-27B-Q4_0.gguf` (draft head) | 1.56 GiB | +| `mmproj-Qwen3.8-27B-Q8_0.gguf` (vision) | 0.59 GiB | + +Store the files under `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/`. + +## Speculative Decoding + +Dense model, so the finding measured on the dense Qwen3.8 family holds: +**MTP alone at `--spec-draft-n-max 4` is the knee**. ngram under MTP hurts +(measured -4 to -5 t/s on the v3; see [[ciru-activefpx-promptforge]]). The +draft head is a separate file loaded via: + +``` +--spec-type draft-mtp +--model-draft mtp-Qwen3.8-27B-Q4_0.gguf +--spec-draft-n-max 4 +--spec-draft-n-min 0 +--spec-draft-p-min 0.0 +``` + +The draft head loads on `ROCm1` with full offload (`--spec-draft-ngl 999`) +and f16 KV. + +## Launch Excerpt + +The key flags on the CIRU ROCmFPX runtime: + +``` +llama-server \ + -m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \ + --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \ + --chat-template-file chat-template.jinja \ + -ngl 999 -dev ROCm0 -c 262144 -np 1 \ + --spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \ + --spec-draft-ngl 999 --spec-draft-n-max 4 \ + -fa on --no-mmap --cache-type-k f16 --cache-type-v f16 +``` + +## Known Issues + +1. **GPU only**: Vulkan, CUDA, and CPU cannot load these files. +2. `--spec-draft-n-max` defaults to 16; set 4. +3. `reasoning_effort: "none"` throws in the Qwen API. Use + `enable_thinking: false` instead. +4. Small `max_tokens` with thinking on returns empty `content`. The Sharp + template drives thinking on by default (effort medium). + +## See Also + +- [[deployment]] — deploy and update process +- [[ciru-activefpx-promptforge]] — the retired v3 page +- [[Dankclaw System Specs]] — the box +- [[frank-redo-2026-08-17]] — fleet topology history \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index e239567..fc60609 100644 --- a/docs/index.md +++ b/docs/index.md @@ -92,7 +92,9 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on Dankclaw (active) +- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) +- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) +- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21) - [[Dankserver System Specs]] — the upstream llama.cpp box hardware diff --git a/scripts/chat-template.jinja b/scripts/chat-template.jinja new file mode 100644 index 0000000..0a2996e --- /dev/null +++ b/scripts/chat-template.jinja @@ -0,0 +1,453 @@ +{%- set template_version = "qwen3.8-froggeric-v22.4.0" %} +{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %} +{%- set image_count = namespace(value=0) %} +{%- set video_count = namespace(value=0) %} +{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %} +{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %} +{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %} +{%- if preserve_reasoning is defined and preserve_reasoning is not none %} + {%- set _preserve_thinking = preserve_reasoning %} +{%- elif preserve_thinking is defined and preserve_thinking is not none %} + {%- set _preserve_thinking = preserve_thinking %} +{%- else %} + {%- set _preserve_thinking = true %} +{%- endif %} +{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %} +{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %} +{%- set _default_reasoning_effort = 'medium' %} +{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %} +{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %} +{%- set _initial_thinking = enable_thinking %} +{%- if _effort_raw in ('none', 'off') %} + {%- set _initial_thinking = false %} + {%- set _initial_effort = 'medium' %} +{%- elif _effort_raw in ('minimal', 'low') %} + {%- set _initial_effort = 'low' %} +{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %} + {%- set _initial_effort = 'xhigh' %} +{%- else %} + {%- set _initial_effort = 'medium' %} +{%- endif %} +{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %} +{%- if auto_disable_thinking_with_tools and _has_tools %} + {%- set ns_state.thinking = false %} +{%- endif %} +{%- for msg in messages %} + {%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %} + {%- if msg.content is string %} + {%- if '<|think_off|>' in msg.content %} + {%- set ns_state.thinking = false %} + {%- elif '<|think_on|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'xhigh' %} + {%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'low' %} + {%- elif '<|think_medium|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'medium' %} + {%- endif %} + {%- elif msg.content is iterable and msg.content is not mapping %} + {%- for item in msg.content %} + {%- if item is string %} + {%- set _item_text = item %} + {%- elif item is mapping and 'text' in item and item.text is string %} + {%- set _item_text = item.text %} + {%- else %} + {%- set _item_text = '' %} + {%- endif %} + {%- if _item_text %} + {%- if '<|think_off|>' in _item_text %} + {%- set ns_state.thinking = false %} + {%- elif '<|think_on|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'xhigh' %} + {%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'low' %} + {%- elif '<|think_medium|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'medium' %} + {%- endif %} + {%- endif %} + {%- endfor %} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- set reasoning_instructions = '' %} +{%- if ns_state.thinking %} + {%- if ns_state.effort == 'xhigh' %} + {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %} + {%- elif ns_state.effort == 'low' %} + {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %} + {%- endif %} +{%- endif %} +{%- macro render_content(content, do_vision_count, is_system_content=false) %} + {%- if content is string %} + {{- content }} + {%- elif content is iterable and content is not mapping %} + {%- for item in content %} + {%- if item is mapping %} + {%- if item.type == 'image' or 'image' in item or 'image_url' in item %} + {%- if is_system_content %} + {{- raise_exception('System message cannot contain images.') }} + {%- endif %} + {%- if do_vision_count %} + {%- set image_count.value = image_count.value + 1 %} + {%- endif %} + {%- if add_vision_id %} + {{- 'Picture ' ~ image_count.value ~ ': ' }} + {%- endif %} + {{- '<|vision_start|><|image_pad|><|vision_end|>' }} + {%- elif item.type == 'video' or 'video' in item %} + {%- if is_system_content %} + {{- raise_exception('System message cannot contain videos.') }} + {%- endif %} + {%- if do_vision_count %} + {%- set video_count.value = video_count.value + 1 %} + {%- endif %} + {%- if add_vision_id %} + {{- 'Video ' ~ video_count.value ~ ': ' }} + {%- endif %} + {{- '<|vision_start|><|video_pad|><|vision_end|>' }} + {%- elif 'text' in item %} + {{- item.text }} + {%- else %} + {{- raise_exception('Unexpected item type in content.') }} + {%- endif %} + {%- else %} + {{- item | string }} + {%- endif %} + {%- endfor %} + {%- elif content is none or content is undefined %} + {{- '' }} + {%- else %} + {{- raise_exception('Unexpected content type.') }} + {%- endif %} +{%- endmacro %} +{%- if not messages %} + {{- raise_exception('No messages provided.') }} +{%- endif %} +{%- set head = namespace(count=0, seen_non_system=false) %} +{%- for message in messages %} + {%- set _is_sys = (message.role == 'system' or message.role == 'developer') %} + {%- if _is_sys and not head.seen_non_system %} + {%- set head.count = head.count + 1 %} + {%- else %} + {%- set head.seen_non_system = true %} + {%- endif %} +{%- endfor %} +{%- set sys_state = namespace(content='') %} +{%- for message in messages[:head.count] %} + {%- set _part = render_content(message.content, false, true) | trim %} + {%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %} + {%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %} + {%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %} + {%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %} + {%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %} + {%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %} + {%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %} + {%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %} + {%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %} + {%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %} + {%- if _part %} + {%- if sys_state.content %} + {%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %} + {%- else %} + {%- set sys_state.content = _part %} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- set _sc = sys_state.content %} +{%- set _msgs = messages[head.count:] %} +{%- if ns_state.thinking %} + {%- set _terse_lead = 'Answer directly, after thinking. Lead with the answer, then only what it needs to be correct and usable.' %} +{%- else %} + {%- set _terse_lead = 'Answer directly and concisely. Give the answer with only what it needs to be correct and usable.' %} +{%- endif %} +{%- set _terse_core %} +Never: open with preamble or pleasantries; restate the question; add filler transitions; hedge with niceties; or repeat a point you've already made. +Always: keep essential steps, caveats, uncertainties, and specifics — never drop correctness or a needed warning for brevity. Keep the final answer lean. Use the least structure that conveys it (plain prose when short; lists or code only when they earn their place). If genuinely uncertain, say so and explain why — never omit uncertainty for the sake of brevity. +If a user request is genuinely ambiguous, ask a sharp question, don't guess. +{%- endset %} +{%- set _terse = _terse_lead ~ '\n' ~ (_terse_core | trim) %} +{#- `terse` chat-template kwarg: default true, so behaviour is unchanged unless a caller opts out. + Pass {"terse": false} via chat_template_kwargs to serve the model with only its own system + prompt -- useful for A/B-ing the terseness effect, or when a downstream prompt conflicts. #} +{%- set _terse_on = terse if terse is defined else true %} +{%- if _terse_on %} + {%- if not _sc %} + {%- set _sc = _terse | trim %} + {%- else %} + {%- set _sc = (_sc | trim) ~ '\n\n' ~ (_terse | trim) %} + {%- endif %} +{%- endif %} +{%- if _has_tools %} + {{- '<|im_start|>system\n' }} + {%- if reasoning_instructions %} + {{- reasoning_instructions + '\n\n' }} + {%- endif %} + {{- '# Tools\n\nYou have access to the following functions:\n\n' }} + {%- for tool in tools %} + {{- '\n' }} + {{- tool | tojson }} + {%- endfor %} + {{- '\n' }} + {%- if _tool_format == 'json' %} + {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}{%- if ns_state.thinking %}{{- '\nBrief explanation of tool call\n\n' }}{%- endif %}{{- '\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n\n\n\nReminder:\n' }}{%- if ns_state.thinking %}{{- '- You can use the block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the block.\n' }}{%- endif %}{{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside XML tags.\n- If you choose to call a tool, you MUST output the block IMMEDIATELY' }}{%- if ns_state.thinking %}{{- ' after thinking' }}{%- endif %}{{- ', with NO conversational text before it.\n- The tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed block for EACH function. Do NOT nest blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n' }} + {%- else %} + {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}{%- if ns_state.thinking %}{{- '\nBrief explanation of tool call\n\n' }}{%- endif %}{{- '\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n' }}{%- if ns_state.thinking %}{{- '- You can use the block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the block.\n' }}{%- endif %}{{- '- Function calls MUST follow the specified format: an inner block must be nested within XML tags.\n- If you choose to call a tool, you MUST output the block IMMEDIATELY' }}{%- if ns_state.thinking %}{{- ' after thinking' }}{%- endif %}{{- ', with NO conversational text before it.\n- The and tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed block for EACH function. Do NOT nest blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n' }} + {%- endif %} + {%- if _sc %} + {{- '\n\n' + _sc }} + {%- endif %} + {{- '<|im_end|>\n' }} +{%- else %} + {%- if _sc %} + {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }} + {%- elif reasoning_instructions %} + {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set _last_idx = _msgs | length - 1 %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %} +{%- for message in _msgs[::-1] %} + {%- set index = (_msgs | length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == 'user' %} + {%- set _rc = render_content(message.content, false) | trim %} + {%- if not (_rc.startswith('') and _rc.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if ns.multi_step_tool %} + {%- if _last_idx > 50 %} + {%- set ns.last_query_index = _last_idx %} + {%- else %} + {%- set ns.last_query_index = 0 %} + {%- endif %} +{%- endif %} +{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %} +{%- for message in _msgs %} + {%- set is_system = (message.role == "system" or message.role == "developer") %} + {%- set content = render_content(message.content, true, is_system) | trim %} + {%- if is_system or message.role == 'user' %} + {%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %} + {%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %} + {%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %} + {%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %} + {%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %} + {%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %} + {%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %} + {%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %} + {%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %} + {%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %} + {%- endif %} + {%- if is_system %} + {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }} + {%- elif message.role == 'user' %} + {%- set ns2.consecutive_failures = 0 %} + {{- '<|im_start|>user\n' + content + '<|im_end|>\n' }} + {%- elif message.role == 'assistant' %} + {%- set reasoning_content = '' %} + {%- set _explicit_reasoning = '' %} + {%- if message.reasoning_content is defined and message.reasoning_content is not none %} + {%- if message.reasoning_content is string %} + {%- set _explicit_reasoning = message.reasoning_content %} + {%- else %} + {%- set _explicit_reasoning = message.reasoning_content | string %} + {%- endif %} + {%- elif message.thinking is defined and message.thinking is not none %} + {%- if message.thinking is string %} + {%- set _explicit_reasoning = message.thinking %} + {%- else %} + {%- set _explicit_reasoning = message.thinking | string %} + {%- endif %} + {%- elif message.reasoning is defined and message.reasoning is not none %} + {%- if message.reasoning is string %} + {%- set _explicit_reasoning = message.reasoning %} + {%- else %} + {%- set _explicit_reasoning = message.reasoning | string %} + {%- endif %} + {%- endif %} + {%- if _explicit_reasoning %} + {%- set _lead_end = '' %} + {%- if content.startswith('') and '' in content %} + {%- set _lead_end = '' %} + {%- elif content.startswith('') and '' in content %} + {%- set _lead_end = '' %} + {%- elif content.startswith('') %} + {%- set _lead_end = '' %} + {%- elif content.startswith('') %} + {%- set _lead_end = '' %} + {%- endif %} + {%- if _lead_end %} + {%- set content = content.split(_lead_end)[-1].lstrip('\n') %} + {%- endif %} + {%- set reasoning_content = _explicit_reasoning %} + {%- else %} + {%- set _think_end = '' %} + {%- if content.startswith('') %} + {%- set _think_end = '' %} + {%- elif content.startswith('') %} + {%- set _think_end = '' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif content.startswith('') and '' in content %} + {%- set _think_end = '' %} + {%- elif content.startswith('') and '' in content %} + {%- set _think_end = '' %} + {%- endif %} + {%- if _think_end %} + {%- if 'thinking' in _think_end %} + {%- set _think_start = '' %} + {%- else %} + {%- set _think_start = '' %} + {%- endif %} + {%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %} + {%- if _think_start in reasoning_content %} + {%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %} + {%- endif %} + {%- set content = content.split(_think_end)[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- set reasoning_content = reasoning_content | trim %} + {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %} + {{- '<|im_start|>assistant\n\n' + reasoning_content + '\n\n\n' + content }} + {%- else %} + {{- '<|im_start|>assistant\n' + content }} + {%- endif %} + {%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %} + {%- for tool_call in message.tool_calls %} + {%- if tool_call.function is defined and tool_call.function is not none %} + {%- set tc = tool_call.function %} + {%- else %} + {%- set tc = tool_call %} + {%- endif %} + {%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %} + {%- if _tool_format == 'json' %} + {%- if loop.first %} + {%- if content | trim %} + {{- '\n\n' }} + {%- endif %} + {%- else %} + {{- '\n' }} + {%- endif %} + {%- set _args = '{}' %} + {%- if tc.arguments is defined and tc.arguments is not none %} + {%- if tc.arguments is mapping %} + {%- set _args = tc.arguments | tojson %} + {%- elif tc.arguments is string %} + {%- if tc.arguments %} + {%- set _args = tc.arguments %} + {%- endif %} + {%- else %} + {%- set _args = tc.arguments | tojson %} + {%- endif %} + {%- endif %} + {{- '\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n' }} + {%- else %} + {%- if loop.first %} + {%- if content | trim %} + {{- '\n\n\n\n' }} + {%- else %} + {{- '\n\n' }} + {%- endif %} + {%- else %} + {{- '\n\n\n' }} + {%- endif %} + {%- if tc.arguments is defined and tc.arguments is not none %} + {%- if tc.arguments is mapping %} + {%- for args_name, args_value in tc.arguments.items() %} + {{- '\n' }} + {%- if args_value is string %} + {%- set _av = args_value %} + {%- else %} + {%- set _av = args_value | tojson %} + {%- endif %} + {%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %} + {{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }} + {%- else %} + {{- _av }} + {%- endif %} + {{- '\n\n' }} + {%- endfor %} + {%- else %} + {%- if tc.arguments is string %} + {%- set _raw_args = tc.arguments %} + {%- else %} + {%- set _raw_args = tc.arguments | tojson %} + {%- endif %} + {%- if _raw_args %} + {%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %} + {{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }} + {%- else %} + {{- _raw_args }} + {%- endif %} + {%- endif %} + {%- endif %} + {%- endif %} + {{- '\n' }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == 'tool' %} + {%- set _content_lower = content | lower %} + {%- set _content_head = _content_lower[:120] %} + {%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %} + {%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %} + {%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %} + {%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %} + {%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %} + {%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %} + {%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %} + {%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %} + {%- else %} + {%- set ns2.consecutive_failures = 0 %} + {%- endif %} + {%- if ns2.prev_role != 'tool' %} + {{- '<|im_start|>user' }} + {%- endif %} + {%- if _tool_format != 'json' and max_tool_response_chars > 0 and content | length > max_tool_response_chars %} + {%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %} + {%- endif %} + {{- '\n\n' + content }} + {%- if ns2.consecutive_failures >= 2 %} + {{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }} + {%- elif ns2.consecutive_failures == 1 %} + {{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }} + {%- endif %} + {{- '\n' }} + {%- if loop.last %} + {{- '<|im_end|>\n' }} + {%- else %} + {%- set _next_role = _msgs[loop.index0 + 1].role %} + {%- if _next_role != 'tool' %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} + {%- else %} + {{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }} + {%- endif %} + {%- set ns2.prev_role = message.role %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if not ns_state.thinking %} + {{- '\n\n\n\n' }} + {%- else %} + {{- '\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 4d2cb2d..763c112 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -3,40 +3,36 @@ # # Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag # qwen3.8-activefpx-promptforge-v2.3 plus the v3 output-K8 runtime patch -# serving jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3). v3 tensor -# allocation (FFN-equalized, late-Q6 writers, full-Q8 output) with the -# full-W8 PromptForge companions, the Output-K8 draft proxy, and the BF16 -# vision projector. Alias frank-default-large, port 7770 (moved from 52625 -# on 2026-08-22; the primary inference server sits on the standard -# llama.cpp port). -# -# This runtime replaced the retired vLLM service that previously occupied -# this slot. See docs/ciru-activefpx-promptforge.md. +# serving kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4 (dense, Q6 AGENT +# GGUF) with the standalone MTP Q4_0 draft head. Alias frank-default-large, +# port 7770. # +# This model replaced jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) on +# 2026-08-24. The new model is a plain ROCmFP4 GGUF (no PromptForge .pfs +# sidecars, no Output-K8 proxy); the requested file is Q6_. + # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm # HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident). # llama.cpp reads --host/--port/--temp as CLI flags, never as env vars. -# CIRU llama-server binary (ROCmFPX v2.3 + v3 output-K8 patch build, see -# docs/ciru runtime page). Built against the private TheRock ROCm 7.15 SDK -# kept in the retired ling3-ciru venv. ROCm 7.15 is required for the -# advertised Strix Halo performance; the system ROCm 7.2.4 selects -# slower/fallback kernels. +# CIRU llama-server binary (ROCmFPX v2.3 + v3 output-K8 patch build, built +# against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru +# venv). ROCm 7.15 is required for the advertised Strix Halo performance. LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server -# Model files (v3). The five-file payload (GGUF + FFN/GDN .pfs + Output-K8 -# proxy + BF16 vision projector) is the full v3 configuration. The v3 files -# live in a separate directory so the v2 artifacts stay for rollback. -MODEL_DIR=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3 -MODEL_FILE=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf -FFN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs -GDN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs -OUTPUT_K8_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs -MMPROJ_FILE=Qwen3.8-27B-mmproj-BF16.mmproj +# Model files. Plain dense Qwen3.8-27B ROCmFP4 Q6 AGENT GGUF plus the standalone +# MTP draft head and the vision projector. Kingjones 2026-08-24. +MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP +MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf +MTP_DRAFT_FILE=mtp-Qwen3.8-27B-Q4_0.gguf +MMPROJ_FILE=mmproj-Qwen3.8-27B-Q8_0.gguf -# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). The serve script -# exports ROCM_PATH and LD_LIBRARY_PATH from these so the 7.15 libraries -# precede any system ROCm 7.2.4. +# Qwen-Sharp chat template (tracked in the repo, staged on the host). +CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja +# enable_thinking on by default in the Sharp template; medium effort default. +CHAT_TEMPLATE_KWARGS='{"enable_thinking":true}' + +# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel ROCM_LIB=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core @@ -45,13 +41,9 @@ ALIAS=frank-default-large BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# Serving: full native 262144 context (n_ctx_train), 1 slot (user request -# 2026-08-19: parallel slots degraded performance). -c is the TOTAL -# context; with PARALLEL=1 it equals the per-slot context. Concurrent -# requests queue on the single slot; the server never preempts a running -# generation. f16 KV, context checkpoints, prompt cache, native MTP -# depth-4. The Strix Halo is unified memory; rocm-smi reports the 64 GiB -# budget. KV at 262144 x 64 KiB/cell is ~16 GiB. +# Serving: full native 262144 context, 1 slot. f16 KV, context checkpoints, +# prompt cache, draft-mtp with standalone Q4_0 head, n_max 4 (the knee for +# the dense model). Unified memory; rocm-smi reports the 64 GiB budget. CTX_SIZE=262144 PARALLEL=1 BATCH_SIZE=4096 @@ -61,22 +53,21 @@ CACHE_TYPE_V=f16 CTX_CHECKPOINTS=16 CACHE_RAM=8192 -# Native MTP draft (depth-4, p_min 0, backend sampling). The qualified v3 -# profile drives the draft via the Output-K8 proxy; the CPU argmax fast -# path and IU4/ngram lanes are disabled in the serve script. +# Dense-model MTP draft (standalone Q4_0 head, n_max 4). No PromptForge .pfs +# sidecars are loaded; the LLAMA_CPU_ARGMAX fast path stays off. SPEC_DRAFT_N_MAX=4 SPEC_DRAFT_N_MIN=0 SPEC_DRAFT_P_MIN=0.0 SPEC_DRAFT_P_SPLIT=0.10 -# Sampling (model card defaults). -TEMPERATURE=0.7 +# Sampling (kingjones model card / Sharp default). +TEMPERATURE=1.0 +TOP_K=20 TOP_P=0.95 -# Thinking enabled (user request, 2026-08-18). deepseek format moves the -# reasoning into message.reasoning_content (OpenAI-compatible); -# none would leave the thought tags unparsed in content. +# Thinking on (Qwen-Sharp template drives thinking on; reasoning format +# projects the tar into reasoning_content). REASONING=on REASONING_FORMAT=deepseek -VERBOSITY=4 +VERBOSITY=4 \ No newline at end of file diff --git a/scripts/frank-default-large.service b/scripts/frank-default-large.service index c7834dd..c7a323e 100644 --- a/scripts/frank-default-large.service +++ b/scripts/frank-default-large.service @@ -1,5 +1,5 @@ [Unit] -Description=CIRU llama.cpp Qwen3.8-27B ActiveFPX PromptForge MTP (dankclaw, frank-default-large) +Description=CIRU llama.cpp Qwen3.8-27B ROCmFP4 STRIX MTP + Sharp template (dankclaw, frank-default-large) After=network-online.target Wants=network-online.target diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 5c25a67..4ba0843 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -1,38 +1,49 @@ -# Tunables for frank-default-small (HauhauCS Qwen3.6-35B-A3B Q6_K_P, -# ngram-mod spec, Dankserver, RTX 3090). +# Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q8_0, MTP MoE, +# Dankserver, RTX 3090). # -# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; -# Ryzen 7 7700X; 62 GB RAM. MoE experts offloaded to CPU (--n-cpu-moe 36). -# Restored 2026-08-18 (was the pre-2026-08-17-redo frank-default-small). +# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen +# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit +# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS +# Qwen3.6-35B-A3B model on 2026-08-24. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP # incident). This box is NVIDIA/CUDA, but the rule keeps the env portable. -# Model files (already on disk; Q6_K_P GGUF + f16 mmproj vision encoder). -MODEL_DIR=/home/dank/models -MODEL_FILE=Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf -MMPROJ_FILE=mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf +# Model files (Q8_0 CRACK GGUF + f16 mmproj vision encoder + MTP head bundled +# in the GGUF). Dealign2026-08-24. +MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK +MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf +MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf # Upstream llama.cpp binary (b10121). LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server +# Qwen-Sharp chat template (tracked in the repo, staged on the host). +CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja + ALIAS=frank-default-small BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# 3 parallel slots x 262144 ctx each = ~786432 total context pool (max per-slot). -# Dual-f16 KV at 786432 cells leaves ~15 GiB free on the 3090. +# 3 parallel slots x 262144 total context pool (max per-slot), unchanged +# from the prior Qwen3.6 service. Lower PARALLEL if VRAM is tight with the +# larger Q8_0 payload. CTX_SIZE=786432 PARALLEL=3 -# Spec: ngram-mod (n_min=48, n_max=64, n_match=24). -SPEC_TYPE=ngram-mod -NGRAM_MOD_N_MIN=48 -NGRAM_MOD_N_MAX=64 -NGRAM_MOD_N_MATCH=24 +# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4 +# (the knee). ngram under MTP hurts; MTP alone is the proven best for MoE. +SPEC_TYPE=draft-mtp +SPEC_DRAFT_N_MAX=4 +SPEC_DRAFT_N_MIN=0 +SPEC_DRAFT_P_MIN=0.0 -# Expert offload: 38 of 40 layers keep MoE weights on CPU; 2 on GPU. -N_CPU_MOE=38 +# Expert offload via explicit tensor override (replaces --n-cpu-moe). Only +# the MoE FFN exps tensors move to CPU; everything else up to --gpu-layers +# stays on the 24 GB card. The regex is the same one the built-in --cpu-moe +# expands. Buffer type "CPU" is the CPU buft name on this host. +OVERRIDE_EXPERT='\.ffn_(up|down|gate|gate_up)_(ch|)exps' +OVERRIDE_BUFT=CPU N_GPU_LAYERS=all # KV cache on GPU, both K and V at f16. @@ -40,22 +51,22 @@ CACHE_TYPE_K=f16 CACHE_TYPE_V=f16 VERBOSITY=4 -# Store SWA/context checkpoints in RAM to hold peak GPU memory down. +# Store context checkpoints in RAM to hold peak GPU memory down. CTX_CHECKPOINTS=16 -# Big prefill batches (ubatch 2048, batch 4096). +# Big prefill batches. BATCH_SIZE=4096 UBATCH_SIZE=2048 -# Eager RAM load (MoE experts on CPU; ~31 GB RAM). +# Eager RAM load (expert weights on CPU; ~31 GB RAM). NO_MMAP=1 -# Sampling (HauhauCS Qwen3.6-35B-A3B model card, thinking-general). +# Sampling (Dealign Ornith model card, thinking-general). TEMPERATURE=1.0 TOP_K=20 TOP_P=0.95 MIN_P=0.0 -# Compute threads lowered to 7 so other processes keep a core. +# Compute threads lowered to 8 so other processes keep a core. THREADS=7 -THREADS_HTTP=8 +THREADS_HTTP=8 \ No newline at end of file diff --git a/scripts/frank-default-small.service b/scripts/frank-default-small.service index 33d8415..ab74e48 100644 --- a/scripts/frank-default-small.service +++ b/scripts/frank-default-small.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp Qwen3.6-35B-A3B Q6_K_P + ngram-mod (dankserver, frank-default-small) +Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q8_0 MTP + Sharp template (dankserver, frank-default-small) After=network-online.target Wants=network-online.target diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 72db12b..c72c05f 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,19 +1,22 @@ #!/usr/bin/env bash -# Serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) as +# Serve kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (dense) as # frank-default-large via the CIRU ROCmFPX llama.cpp runtime on Dankclaw # (Strix Halo gfx1151). # # Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag # qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) plus the v3 output-K8 -# runtime patch (runtime/qwen38-v3-output-k8-runtime.patch, commit af0eac6 on -# Dankclaw) against the private TheRock ROCm 7.15 SDK (kept in the retired -# ling3-ciru venv). The model is the v3 GGUF with the full-W8 PromptForge -# companion views (FFN/GDN .pfs), the Output-K8 draft proxy, and the BF16 -# vision projector. Native depth-4 MTP speculative decoding. +# runtime patch (commit af0eac6), against the private TheRock ROCm 7.15 SDK +# (kept in the retired ling3-ciru venv). The model is the plain ROCmFP4 +# Q6 AGENT GGUF (no PromptForge .pfs sidecars) with the standalone MTP draft +# head mtp-Qwen3.8-27B-Q4_0.gguf. The Qwen-Sharp chat template is applied at +# runtime. +# +# Spec: dense model -> native MTP draft at n_max 4 (the knee, per the model +# card). The draft head is a separate Q4_0 GGUF loaded via --model-draft. # # The model card requires ROCm 7.15 (not the system ROCm 7.2.4) for the # advertised Strix Halo performance. LD_LIBRARY_PATH therefore points at the -# 7.15 SDK libraries first. An older system ROCm must not precede them. +# 7.15 SDK libraries first. # # Tunables come from frank-default-large.env in this directory unless # FRANK_LARGE_ENV_FILE overrides that path. @@ -30,20 +33,19 @@ fi # --- defaults (used when the env file does not set them) ------------------- : "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3}" -: "${MODEL_FILE:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf}" -: "${FFN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs}" -: "${GDN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs}" -: "${OUTPUT_K8_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs}" -: "${MMPROJ_FILE:=Qwen3.8-27B-mmproj-BF16.mmproj}" +: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP}" +: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}" +: "${MTP_DRAFT_FILE:=mtp-Qwen3.8-27B-Q4_0.gguf}" +: "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-Q8_0.gguf}" +: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" : "${ROCM_DEVEL:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel}" : "${ROCM_LIB:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries}" : "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}" : "${ALIAS:=frank-default-large}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}" -: "${CTX_SIZE:=524288}" -: "${PARALLEL:=2}" +: "${CTX_SIZE:=262144}" +: "${PARALLEL:=1}" : "${BATCH_SIZE:=4096}" : "${UBATCH_SIZE:=4096}" : "${CACHE_TYPE_K:=f16}" @@ -54,30 +56,27 @@ fi : "${SPEC_DRAFT_N_MIN:=0}" : "${SPEC_DRAFT_P_MIN:=0.0}" : "${SPEC_DRAFT_P_SPLIT:=0.10}" -: "${TEMPERATURE:=0.7}" +: "${TEMPERATURE:=1.0}" +: "${TOP_K:=20}" : "${TOP_P:=0.95}" : "${REASONING:=on}" : "${REASONING_FORMAT:=deepseek}" +: "${CHAT_TEMPLATE_KWARGS:={}}" : "${VERBOSITY:=4}" MODEL="$MODEL_DIR/$MODEL_FILE" -export PROMPTFORGE_SIDECAR="$MODEL_DIR/$FFN_PFS" -export PROMPTFORGE_GDN_SIDECAR="$MODEL_DIR/$GDN_PFS" -# v3 qualified runtime: Output-K8 draft proxy + small-row W8 prompt route. -export PROMPTFORGE_MODE="m2048_fused_tail1476" -export GGML_CUDA_GRAPH_OPT=0 -export PROMPTFORGE_ENABLE_SMALLM_W8=1 -export PROMPTFORGE_MTP_OUTPUT_K8=1 -export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0 -export PROMPTFORGE_MTP_OUTPUT_K8_PROXY="$MODEL_DIR/$OUTPUT_K8_PFS" -# The qualified v3 profile disables the CPU argmax draft fast path; the -# Output-K8 proxy + backend sampling replace it. Also drop any IU4/ngram -# companions from the non-IU4 development lane. -unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY || true +MTP_DRAFT="$MODEL_DIR/$MTP_DRAFT_FILE" + +# Plain dense model: no PromptForge .pfs sidecars, no Output-K8 proxy, no IU4 +# lane. The v3 MTP is driven by the standalone Q4_0 draft head through the +# standard draft-mtp path. Unset any leftover PromptForge env lanes. +unset PROMPTFORGE_SIDECAR PROMPTFORGE_GDN_SIDECAR PROMPTFORGE_MTP_OUTPUT_K8_PROXY || true +unset PROMPTFORGE_MODE PROMPTFORGE_ENABLE_SMALLM_W8 || true +unset PROMPTFORGE_MTP_OUTPUT_K8 PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE || true unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR || true unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4 || true -for f in "$MODEL" "$PROMPTFORGE_SIDECAR" "$PROMPTFORGE_GDN_SIDECAR" "$PROMPTFORGE_MTP_OUTPUT_K8_PROXY" "$MODEL_DIR/$MMPROJ_FILE"; do +for f in "$MODEL" "$MTP_DRAFT" "$MODEL_DIR/$MMPROJ_FILE" "$CHAT_TEMPLATE_FILE"; do if [[ ! -f "$f" ]]; then echo "missing required file: $f" >&2 exit 1 @@ -95,6 +94,8 @@ export LD_LIBRARY_PATH="${ROCM_LIB}/lib:${ROCM_CORE}/lib:${ROCM_DEVEL}/lib${LD_L exec "$LLAMA_BIN" \ --model "$MODEL" \ --mmproj "$MODEL_DIR/$MMPROJ_FILE" \ + --chat-template-file "$CHAT_TEMPLATE_FILE" \ + --chat-template-kwargs "$CHAT_TEMPLATE_KWARGS" \ --alias "$ALIAS" \ --host "$BIND_ADDR" \ --port "$LISTEN_PORT" \ @@ -113,6 +114,7 @@ exec "$LLAMA_BIN" \ --metrics \ --no-mmap \ --spec-type draft-mtp \ + --model-draft "$MTP_DRAFT" \ --spec-draft-device ROCm0 \ --spec-draft-ngl 999 \ --spec-draft-type-k f16 \ @@ -121,10 +123,10 @@ exec "$LLAMA_BIN" \ --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \ --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \ --spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \ - --spec-draft-backend-sampling \ --temp "$TEMPERATURE" \ + --top-k "$TOP_K" \ --top-p "$TOP_P" \ --reasoning "$REASONING" \ --reasoning-format "$REASONING_FORMAT" \ --reasoning-budget -1 \ - -lv "$VERBOSITY" + -lv "$VERBOSITY" \ No newline at end of file diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index cd69621..66b7b8d 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,13 +1,24 @@ #!/usr/bin/env bash -# Serve HauhauCS Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6_K_P with -# ngram-mod speculative decoding on Dankserver (RTX 3090, MoE experts on CPU). +# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q8_0) as +# frank-default-small via upstream llama.cpp on Dankserver (RTX 3090, MoE +# experts on CPU). # # Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total / -# 3B-active MoE vision-language model; --n-cpu-moe keeps most expert weights on -# CPU so the non-expert tensors + KV cache fit the 24 GB 3090. +# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF. # -# Tunables come from frank-default-small.env in this directory unless the -# FRANK_SMALL_ENV_FILE environment variable overrides that path. +# Expert offload: instead of the layer-counter --n-cpu-moe, the expert (FFN +# exps) tensors are pinned to CPU explicitly with --override-tensor. The +# pattern is the internal MoE exps regex (the same one --n-cpu-moe expands), +# so only the Mixture-of-Experts weight tensors move to CPU; every other +# tensor stays on the GPU up to --gpu-layers. +# +# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is +# auto-detected from the target model; b10121 creates the MTP draft context +# against the model file). ngram under MTP hurts (measured on the Qwen3.8 +# family), so MTP alone is kept. n_max 4 is the knee. +# +# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE +# overrides that path. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -21,22 +32,27 @@ fi # --- defaults (used when the env file does not set them) ----------------- : "${MODEL_DIR:=/home/dank/models}" -: "${MODEL_FILE:=Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf}" -: "${MMPROJ_FILE:=mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf}" +: "${MODEL_FILE:=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf}" +: "${MMPROJ_FILE:=mmproj-Ornith-1.5-35B-A3B-f16.gguf}" : "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}" +: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" : "${ALIAS:=frank-default-small}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=786432}" : "${PARALLEL:=3}" -: "${SPEC_TYPE:=ngram-mod}" -: "${NGRAM_MOD_N_MIN:=48}" -: "${NGRAM_MOD_N_MAX:=64}" -: "${NGRAM_MOD_N_MATCH:=24}" +: "${SPEC_TYPE:=draft-mtp}" +: "${SPEC_DRAFT_N_MAX:=4}" +: "${SPEC_DRAFT_N_MIN:=0}" +: "${SPEC_DRAFT_P_MIN:=0.0}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" : "${VERBOSITY:=4}" -: "${N_CPU_MOE:=38}" +# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern +# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the +# MoE expert weights and leave every other tensor on the GPU. +: "${OVERRIDE_EXPERT:=\.ffn_(up|down|gate|gate_up)_(ch|)exps}" +: "${OVERRIDE_BUFT:=CPU}" : "${N_GPU_LAYERS:=all}" : "${BATCH_SIZE:=4096}" : "${UBATCH_SIZE:=2048}" @@ -52,7 +68,7 @@ fi MODEL="$MODEL_DIR/$MODEL_FILE" MMPROJ="$MODEL_DIR/$MMPROJ_FILE" -for f in "$MODEL" "$MMPROJ"; do +for f in "$MODEL" "$MMPROJ" "$CHAT_TEMPLATE_FILE"; do if [[ ! -f "$f" ]]; then echo "missing required file: $f" >&2 exit 1 @@ -66,17 +82,17 @@ fi exec "$LLAMA_SERVER" \ --model "$MODEL" \ --mmproj "$MMPROJ" \ - --jinja \ + --chat-template-file "$CHAT_TEMPLATE_FILE" \ --spec-type "$SPEC_TYPE" \ - --spec-ngram-mod-n-min "$NGRAM_MOD_N_MIN" \ - --spec-ngram-mod-n-max "$NGRAM_MOD_N_MAX" \ - --spec-ngram-mod-n-match "$NGRAM_MOD_N_MATCH" \ + --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \ + --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \ + --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \ + --override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \ --cache-type-k "$CACHE_TYPE_K" \ --cache-type-v "$CACHE_TYPE_V" \ --verbosity "$VERBOSITY" \ --ctx-size "$CTX_SIZE" \ --parallel "$PARALLEL" \ - --n-cpu-moe "$N_CPU_MOE" \ --gpu-layers "$N_GPU_LAYERS" \ --batch-size "$BATCH_SIZE" \ --ubatch-size "$UBATCH_SIZE" \ @@ -90,4 +106,4 @@ exec "$LLAMA_SERVER" \ --threads-http "$THREADS_HTTP" \ --alias "$ALIAS" \ --host "$BIND_ADDR" \ - --port "$LISTEN_PORT" + --port "$LISTEN_PORT" \ No newline at end of file