Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU (replaces --n-cpu-moe), Qwen-Sharp chat template. - frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0 ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the retired PromptForge .pfs sidecar lanes. - Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via --chat-template-file on both services. - Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages; update index, deployment, and both system specs; models.json names. Smoke-tested both models on the real hardware on scratch ports before commit: model loads, MTP draft attaches, Sharp template applies, chat completion returns.
This commit is contained in:
Frank
@@ -93,7 +93,7 @@ cable between claw and server is required for 10G.
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (262144 ctx, native MTP, PromptForge DualView). See [[ciru-activefpx-promptforge]].
|
||||
- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]].
|
||||
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
|
||||
@@ -60,7 +60,7 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
|
||||
|
||||
### Active Services
|
||||
|
||||
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Qwen3.6-35B-A3B (Q6_K_P + ngram-mod) on port 7770. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
|
||||
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q8_0) on port 7770, with MoE expert weights held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
type: reference
|
||||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
|
||||
tags:
|
||||
- llamacpp
|
||||
- cuda
|
||||
- dankserver
|
||||
- moe
|
||||
- ornith
|
||||
- mtp
|
||||
timestamp: 2026-08-24
|
||||
---
|
||||
|
||||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
|
||||
> **Active 2026-08-24.** `frank-default-small` serves this model. It
|
||||
> replaced the HauhauCS Qwen3.6-35B-A3B Q6_K_P service on the same upstream
|
||||
> llama.cpp binary (b10121).
|
||||
|
||||
`frank-default-small` (port 7770) serves
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` from `dealignai` on Hugging Face. It is
|
||||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||||
attention architecture, abliterated (uncensored). The MTP speculative head
|
||||
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
|
||||
model.
|
||||
|
||||
## Model Files
|
||||
|
||||
| File | Size |
|
||||
|------|------|
|
||||
| `Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` | 37.2 GB |
|
||||
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.90 GB |
|
||||
|
||||
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
||||
|
||||
## Expert Offload — `--override-tensor`, not `--n-cpu-moe`
|
||||
|
||||
The service pins the MoE expert weights to CPU with an **explicit tensor
|
||||
override** rather than the layer-counter `--n-cpu-moe`. The override uses
|
||||
the same internal regex that `--cpu-moe` / `--n-cpu-moe` expand:
|
||||
|
||||
```
|
||||
OVERRIDE_EXPERT=\.ffn_(up|down|gate|gate_up)_(ch|)exps
|
||||
OVERRIDE_BUFT=CPU
|
||||
```
|
||||
|
||||
This is passed as `--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT"`.
|
||||
Only the FFN expert tensors move to CPU; every other tensor stays on the
|
||||
24 GB card up to `--gpu-layers`. To list the buffer type names on the host,
|
||||
run a bogus override and read the error:
|
||||
|
||||
```
|
||||
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
||||
```
|
||||
|
||||
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
|
||||
auto-detected from the target (b10121 creates the MTP draft context against
|
||||
the model file). ngram under MTP hurts on this family; MTP alone is the
|
||||
proven best. `--spec-draft-n-max 4` is the knee.
|
||||
|
||||
```
|
||||
--spec-type draft-mtp
|
||||
--spec-draft-n-max 4
|
||||
```
|
||||
|
||||
## Launch Excerpt
|
||||
|
||||
```
|
||||
llama-server \
|
||||
-m Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf \
|
||||
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--spec-type draft-mtp \
|
||||
--override-tensor "\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||||
-ngl all -c 786432 -np 3 \
|
||||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||||
--temp 1.0 --top-p 0.95 --top-k 20
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. The 37.2 GB Q8_0 file is large for the 62 GB RAM box; lower `PARALLEL`
|
||||
or switch quant if VRAM or RAM is tight.
|
||||
2. `--n-cpu-moe` remains supported; this service deliberately prefers the
|
||||
explicit tensor override so only the expert tensors reach CPU.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[Dankserver System Specs]] — the box
|
||||
- [[frank-redo-2026-08-17]] — fleet topology history
|
||||
+23
-20
@@ -17,8 +17,8 @@ repository to a running service.
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q8_0, native MTP draft, MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -122,32 +122,35 @@ curl -sf http://127.0.0.1:7770/health
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver is upstream llama.cpp (b10121) serving
|
||||
`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the
|
||||
alias `frank-default-small` on port 7770. It runs the stock
|
||||
`/home/dank/llama.cpp/build/bin/llama-server` binary with the 38 MoE
|
||||
experts on CPU (`--n-cpu-moe 38`) and 7 CPU threads, ngram-mod
|
||||
speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both
|
||||
K and V, context 786432 (3 slots of ~262K each), batch 4096 / ubatch 2048,
|
||||
`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It
|
||||
launches through `scripts/serve-frank-small.sh`, which reads tunables from
|
||||
`scripts/frank-default-small.env`.
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf` under the alias `frank-default-small`
|
||||
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
|
||||
binary with the MoE expert weights pinned to CPU by an explicit
|
||||
`--override-tensor` regex (the expert `\.ffn_(up|down|gate|gate_up)_(ch|)exps`
|
||||
pattern, buffer type `CPU`) and 7 CPU threads, native `draft-mtp`
|
||||
speculative decoding (bundled MoE MTP head, n_max 4), f16 KV, context
|
||||
786432 (3 slots of ~262K each), batch 4096 / ubatch 2048, `--no-mmap`. The
|
||||
unit is `scripts/frank-default-small.service`. It launches through
|
||||
`scripts/serve-frank-small.sh`, which reads tunables from
|
||||
`scripts/frank-default-small.env`. See [[dealign-ornith-crack]].
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
|
||||
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias
|
||||
`frank-default-large` on port 7770 (moved from 52625 on 2026-08-22). It runs
|
||||
`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` (Q6_0 AGENT, dense)
|
||||
under the alias `frank-default-large` on port 7770. It runs
|
||||
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
||||
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
||||
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
|
||||
(1 slot. Concurrent requests queue. The server never preempts a running
|
||||
generation), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
|
||||
prompt cache, and reasoning in deepseek format. The unit file is
|
||||
7.15 SDK in the retired ling3-ciru venv) with a standalone MTP Q4_0 draft
|
||||
head (`--model-draft` and `--spec-type draft-mtp`), f16 KV, full 262144
|
||||
context (1 slot. Concurrent requests queue; the server never preempts a
|
||||
running generation), batch 4096 / ubatch 4096, 16 context checkpoints,
|
||||
8 GiB prompt cache, and reasoning in deepseek format. The unit file is
|
||||
`scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
|
||||
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
|
||||
See [[kingjones-strix-mtp]].
|
||||
|
||||
> The kingjones Q6 AGENT model has served the slot since 2026-08-24; the
|
||||
> v3 PromptForge payload is retired on this slot.
|
||||
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
|
||||
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.
|
||||
|
||||
|
||||
+19
-13
@@ -1,9 +1,10 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and
|
||||
CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw
|
||||
(frank-default-large).
|
||||
(2026-08-24): upstream llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and CIRU ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8-27B ROCmFP4). Both services
|
||||
run the Qwen-Sharp chat template.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -18,8 +19,8 @@ graph LR
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[upstream llama.cpp b10121
|
||||
frank-default-small, port 7770]
|
||||
C[Qwen3.6-35B-A3B-Uncensored Q6_K_P
|
||||
ngram-mod, 38 MoE experts on CPU, 786432 ctx]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q8_0
|
||||
native MTP, MoE experts offloaded via override, 786432 ctx]
|
||||
A --> B --> C
|
||||
end
|
||||
|
||||
@@ -27,8 +28,8 @@ ngram-mod, 38 MoE experts on CPU, 786432 ctx]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[CIRU ROCmFPX llama.cpp PromptForge v3
|
||||
frank-default-large, port 7770]
|
||||
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3
|
||||
262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision]
|
||||
F[Qwen3.8-27B-Uncensored-ROCmFP4 Q6_0 AGENT
|
||||
MTP draft + Sharp template, 262K ctx / 1 slot]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
@@ -47,8 +48,12 @@ frank-default-large, port 7770]
|
||||
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on
|
||||
Dankclaw (frank-default-large, active. Restored 2026-08-22)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model
|
||||
on Dankclaw (frank-default-large, active 2026-08-24)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
|
||||
@@ -56,8 +61,9 @@ frank-default-large, port 7770]
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the v3 model, ROCm 7.15 toolchain,
|
||||
and build history (active deployment)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||||
and build history (on Dankclaw; the slot now serves the kingjones
|
||||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||||
identity, and experiment summary (retired 2026-08-22)
|
||||
|
||||
@@ -71,8 +77,8 @@ frank-default-large, port 7770]
|
||||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||||
notes and sizing (the model family now served by frank-default-small as
|
||||
Q6_K_P)
|
||||
notes and sizing (the historical family; frank-default-small now serves
|
||||
Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]])
|
||||
|
||||
## Memory Management
|
||||
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- kingjones
|
||||
- rocmfp4
|
||||
- mtp
|
||||
timestamp: 2026-08-24
|
||||
---
|
||||
|
||||
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
|
||||
> **Active 2026-08-24.** `frank-default-large` serves this model. It
|
||||
> replaced the jcbtc v3 CIRU ActiveFPX PromptForge payload on the same
|
||||
> ROCmFPX runtime and the same CIRU llama.cpp binary. See
|
||||
> [[ciru-activefpx-promptforge]] for the retired v3 page.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
|
||||
Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
|
||||
the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
|
||||
a **separate** Q4_0 draft GGUF, so a bundled-head auto-detect does not apply
|
||||
here; the head is loaded with `--model-draft`.
|
||||
|
||||
## Model Files
|
||||
|
||||
The card ships six text files plus the vision projector. The service uses
|
||||
the Q6 AGENT file (requested for the fleet), the separate MTP draft head
|
||||
and the Q8 vision projector.
|
||||
|
||||
| File | Size |
|
||||
|------|------|
|
||||
| `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` | 23.15 GiB |
|
||||
| `mtp-Qwen3.8-27B-Q4_0.gguf` (draft head) | 1.56 GiB |
|
||||
| `mmproj-Qwen3.8-27B-Q8_0.gguf` (vision) | 0.59 GiB |
|
||||
|
||||
Store the files under `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/`.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
Dense model, so the finding measured on the dense Qwen3.8 family holds:
|
||||
**MTP alone at `--spec-draft-n-max 4` is the knee**. ngram under MTP hurts
|
||||
(measured -4 to -5 t/s on the v3; see [[ciru-activefpx-promptforge]]). The
|
||||
draft head is a separate file loaded via:
|
||||
|
||||
```
|
||||
--spec-type draft-mtp
|
||||
--model-draft mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
--spec-draft-n-max 4
|
||||
--spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.0
|
||||
```
|
||||
|
||||
The draft head loads on `ROCm1` with full offload (`--spec-draft-ngl 999`)
|
||||
and f16 KV.
|
||||
|
||||
## Launch Excerpt
|
||||
|
||||
The key flags on the CIRU ROCmFPX runtime:
|
||||
|
||||
```
|
||||
llama-server \
|
||||
-m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
|
||||
--mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
|
||||
--chat-template-file chat-template.jinja \
|
||||
-ngl 999 -dev ROCm0 -c 262144 -np 1 \
|
||||
--spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
|
||||
--spec-draft-ngl 999 --spec-draft-n-max 4 \
|
||||
-fa on --no-mmap --cache-type-k f16 --cache-type-v f16
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **GPU only**: Vulkan, CUDA, and CPU cannot load these files.
|
||||
2. `--spec-draft-n-max` defaults to 16; set 4.
|
||||
3. `reasoning_effort: "none"` throws in the Qwen API. Use
|
||||
`enable_thinking: false` instead.
|
||||
4. Small `max_tokens` with thinking on returns empty `content`. The Sharp
|
||||
template drives thinking on by default (effort medium).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[ciru-activefpx-promptforge]] — the retired v3 page
|
||||
- [[Dankclaw System Specs]] — the box
|
||||
- [[frank-redo-2026-08-17]] — fleet topology history
|
||||
+3
-1
@@ -92,7 +92,9 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on Dankclaw (active)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box hardware
|
||||
|
||||
@@ -0,0 +1,453 @@
|
||||
{%- set template_version = "qwen3.8-froggeric-v22.4.0" %}
|
||||
{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
|
||||
{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
|
||||
{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
|
||||
{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
|
||||
{%- set _preserve_thinking = preserve_reasoning %}
|
||||
{%- elif preserve_thinking is defined and preserve_thinking is not none %}
|
||||
{%- set _preserve_thinking = preserve_thinking %}
|
||||
{%- else %}
|
||||
{%- set _preserve_thinking = true %}
|
||||
{%- endif %}
|
||||
{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
|
||||
{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
|
||||
{%- set _default_reasoning_effort = 'medium' %}
|
||||
{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
|
||||
{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
|
||||
{%- set _initial_thinking = enable_thinking %}
|
||||
{%- if _effort_raw in ('none', 'off') %}
|
||||
{%- set _initial_thinking = false %}
|
||||
{%- set _initial_effort = 'medium' %}
|
||||
{%- elif _effort_raw in ('minimal', 'low') %}
|
||||
{%- set _initial_effort = 'low' %}
|
||||
{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
|
||||
{%- set _initial_effort = 'xhigh' %}
|
||||
{%- else %}
|
||||
{%- set _initial_effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
|
||||
{%- if auto_disable_thinking_with_tools and _has_tools %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- endif %}
|
||||
{%- for msg in messages %}
|
||||
{%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
|
||||
{%- if msg.content is string %}
|
||||
{%- if '<|think_off|>' in msg.content %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- elif '<|think_on|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'xhigh' %}
|
||||
{%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'low' %}
|
||||
{%- elif '<|think_medium|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- elif msg.content is iterable and msg.content is not mapping %}
|
||||
{%- for item in msg.content %}
|
||||
{%- if item is string %}
|
||||
{%- set _item_text = item %}
|
||||
{%- elif item is mapping and 'text' in item and item.text is string %}
|
||||
{%- set _item_text = item.text %}
|
||||
{%- else %}
|
||||
{%- set _item_text = '' %}
|
||||
{%- endif %}
|
||||
{%- if _item_text %}
|
||||
{%- if '<|think_off|>' in _item_text %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- elif '<|think_on|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'xhigh' %}
|
||||
{%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'low' %}
|
||||
{%- elif '<|think_medium|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if ns_state.thinking %}
|
||||
{%- if ns_state.effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif ns_state.effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if item is mapping %}
|
||||
{%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif item.type == 'video' or 'video' in item %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- item | string }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set head = namespace(count=0, seen_non_system=false) %}
|
||||
{%- for message in messages %}
|
||||
{%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- if _is_sys and not head.seen_non_system %}
|
||||
{%- set head.count = head.count + 1 %}
|
||||
{%- else %}
|
||||
{%- set head.seen_non_system = true %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set sys_state = namespace(content='') %}
|
||||
{%- for message in messages[:head.count] %}
|
||||
{%- set _part = render_content(message.content, false, true) | trim %}
|
||||
{%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
|
||||
{%- if _part %}
|
||||
{%- if sys_state.content %}
|
||||
{%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
|
||||
{%- else %}
|
||||
{%- set sys_state.content = _part %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set _sc = sys_state.content %}
|
||||
{%- set _msgs = messages[head.count:] %}
|
||||
{%- if ns_state.thinking %}
|
||||
{%- set _terse_lead = 'Answer directly, after thinking. Lead with the answer, then only what it needs to be correct and usable.' %}
|
||||
{%- else %}
|
||||
{%- set _terse_lead = 'Answer directly and concisely. Give the answer with only what it needs to be correct and usable.' %}
|
||||
{%- endif %}
|
||||
{%- set _terse_core %}
|
||||
Never: open with preamble or pleasantries; restate the question; add filler transitions; hedge with niceties; or repeat a point you've already made.
|
||||
Always: keep essential steps, caveats, uncertainties, and specifics — never drop correctness or a needed warning for brevity. Keep the final answer lean. Use the least structure that conveys it (plain prose when short; lists or code only when they earn their place). If genuinely uncertain, say so and explain why — never omit uncertainty for the sake of brevity.
|
||||
If a user request is genuinely ambiguous, ask a sharp question, don't guess.
|
||||
{%- endset %}
|
||||
{%- set _terse = _terse_lead ~ '\n' ~ (_terse_core | trim) %}
|
||||
{#- `terse` chat-template kwarg: default true, so behaviour is unchanged unless a caller opts out.
|
||||
Pass {"terse": false} via chat_template_kwargs to serve the model with only its own system
|
||||
prompt -- useful for A/B-ing the terseness effect, or when a downstream prompt conflicts. #}
|
||||
{%- set _terse_on = terse if terse is defined else true %}
|
||||
{%- if _terse_on %}
|
||||
{%- if not _sc %}
|
||||
{%- set _sc = _terse | trim %}
|
||||
{%- else %}
|
||||
{%- set _sc = (_sc | trim) ~ '\n\n' ~ (_terse | trim) %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if _has_tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
|
||||
{%- for tool in tools %}
|
||||
{{- '\n' }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- '\n</tools>' }}
|
||||
{%- if _tool_format == 'json' %}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}{%- if ns_state.thinking %}{{- '<think>\nBrief explanation of tool call\n</think>\n' }}{%- endif %}{{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}{%- if ns_state.thinking %}{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}{%- endif %}{{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY' }}{%- if ns_state.thinking %}{{- ' after thinking' }}{%- endif %}{{- ', with NO conversational text before it.\n- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
|
||||
{%- else %}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}{%- if ns_state.thinking %}{{- '<think>\nBrief explanation of tool call\n</think>\n' }}{%- endif %}{{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}{%- if ns_state.thinking %}{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}{%- endif %}{{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY' }}{%- if ns_state.thinking %}{{- ' after thinking' }}{%- endif %}{{- ', with NO conversational text before it.\n- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
|
||||
{%- endif %}
|
||||
{%- if _sc %}
|
||||
{{- '\n\n' + _sc }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if _sc %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set _last_idx = _msgs | length - 1 %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
|
||||
{%- for message in _msgs[::-1] %}
|
||||
{%- set index = (_msgs | length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == 'user' %}
|
||||
{%- set _rc = render_content(message.content, false) | trim %}
|
||||
{%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if ns.multi_step_tool %}
|
||||
{%- if _last_idx > 50 %}
|
||||
{%- set ns.last_query_index = _last_idx %}
|
||||
{%- else %}
|
||||
{%- set ns.last_query_index = 0 %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
|
||||
{%- for message in _msgs %}
|
||||
{%- set is_system = (message.role == "system" or message.role == "developer") %}
|
||||
{%- set content = render_content(message.content, true, is_system) | trim %}
|
||||
{%- if is_system or message.role == 'user' %}
|
||||
{%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if is_system %}
|
||||
{{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'user' %}
|
||||
{%- set ns2.consecutive_failures = 0 %}
|
||||
{{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'assistant' %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- set _explicit_reasoning = '' %}
|
||||
{%- if message.reasoning_content is defined and message.reasoning_content is not none %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set _explicit_reasoning = message.reasoning_content %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.reasoning_content | string %}
|
||||
{%- endif %}
|
||||
{%- elif message.thinking is defined and message.thinking is not none %}
|
||||
{%- if message.thinking is string %}
|
||||
{%- set _explicit_reasoning = message.thinking %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.thinking | string %}
|
||||
{%- endif %}
|
||||
{%- elif message.reasoning is defined and message.reasoning is not none %}
|
||||
{%- if message.reasoning is string %}
|
||||
{%- set _explicit_reasoning = message.reasoning %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.reasoning | string %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if _explicit_reasoning %}
|
||||
{%- set _lead_end = '' %}
|
||||
{%- if content.startswith('<think>') and '</think>' in content %}
|
||||
{%- set _lead_end = '</think>' %}
|
||||
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
|
||||
{%- set _lead_end = '</thinking>' %}
|
||||
{%- elif content.startswith('</think>') %}
|
||||
{%- set _lead_end = '</think>' %}
|
||||
{%- elif content.startswith('</thinking>') %}
|
||||
{%- set _lead_end = '</thinking>' %}
|
||||
{%- endif %}
|
||||
{%- if _lead_end %}
|
||||
{%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = _explicit_reasoning %}
|
||||
{%- else %}
|
||||
{%- set _think_end = '' %}
|
||||
{%- if content.startswith('</think>') %}
|
||||
{%- set _think_end = '</think>' %}
|
||||
{%- elif content.startswith('</thinking>') %}
|
||||
{%- set _think_end = '</thinking>' %}
|
||||
{%- elif '\n</think>' in content %}
|
||||
{%- set _think_end = '\n</think>' %}
|
||||
{%- elif '\n</thinking>' in content %}
|
||||
{%- set _think_end = '\n</thinking>' %}
|
||||
{%- elif '\n</ think>' in content %}
|
||||
{%- set _think_end = '\n</ think>' %}
|
||||
{%- elif '\n</think >' in content %}
|
||||
{%- set _think_end = '\n</think >' %}
|
||||
{%- elif content.startswith('<think>') and '</think>' in content %}
|
||||
{%- set _think_end = '</think>' %}
|
||||
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
|
||||
{%- set _think_end = '</thinking>' %}
|
||||
{%- endif %}
|
||||
{%- if _think_end %}
|
||||
{%- if 'thinking' in _think_end %}
|
||||
{%- set _think_start = '<thinking>' %}
|
||||
{%- else %}
|
||||
{%- set _think_start = '<think>' %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
|
||||
{%- if _think_start in reasoning_content %}
|
||||
{%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- set content = content.split(_think_end)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content | trim %}
|
||||
{%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
|
||||
{{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>assistant\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined and tool_call.function is not none %}
|
||||
{%- set tc = tool_call.function %}
|
||||
{%- else %}
|
||||
{%- set tc = tool_call %}
|
||||
{%- endif %}
|
||||
{%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
|
||||
{%- if _tool_format == 'json' %}
|
||||
{%- if loop.first %}
|
||||
{%- if content | trim %}
|
||||
{{- '\n\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- set _args = '{}' %}
|
||||
{%- if tc.arguments is defined and tc.arguments is not none %}
|
||||
{%- if tc.arguments is mapping %}
|
||||
{%- set _args = tc.arguments | tojson %}
|
||||
{%- elif tc.arguments is string %}
|
||||
{%- if tc.arguments %}
|
||||
{%- set _args = tc.arguments %}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{%- set _args = tc.arguments | tojson %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
|
||||
{%- else %}
|
||||
{%- if loop.first %}
|
||||
{%- if content | trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tc.arguments is defined and tc.arguments is not none %}
|
||||
{%- if tc.arguments is mapping %}
|
||||
{%- for args_name, args_value in tc.arguments.items() %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- if args_value is string %}
|
||||
{%- set _av = args_value %}
|
||||
{%- else %}
|
||||
{%- set _av = args_value | tojson %}
|
||||
{%- endif %}
|
||||
{%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
|
||||
{{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
|
||||
{%- else %}
|
||||
{{- _av }}
|
||||
{%- endif %}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- else %}
|
||||
{%- if tc.arguments is string %}
|
||||
{%- set _raw_args = tc.arguments %}
|
||||
{%- else %}
|
||||
{%- set _raw_args = tc.arguments | tojson %}
|
||||
{%- endif %}
|
||||
{%- if _raw_args %}
|
||||
{%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
|
||||
{{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
|
||||
{%- else %}
|
||||
{{- _raw_args }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'tool' %}
|
||||
{%- set _content_lower = content | lower %}
|
||||
{%- set _content_head = _content_lower[:120] %}
|
||||
{%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
|
||||
{%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
|
||||
{%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
|
||||
{%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
|
||||
{%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
|
||||
{%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
|
||||
{%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
|
||||
{%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
|
||||
{%- else %}
|
||||
{%- set ns2.consecutive_failures = 0 %}
|
||||
{%- endif %}
|
||||
{%- if ns2.prev_role != 'tool' %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{%- if _tool_format != 'json' and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
|
||||
{%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' + content }}
|
||||
{%- if ns2.consecutive_failures >= 2 %}
|
||||
{{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
|
||||
{%- elif ns2.consecutive_failures == 1 %}
|
||||
{{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
|
||||
{%- endif %}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- set _next_role = _msgs[loop.index0 + 1].role %}
|
||||
{%- if _next_role != 'tool' %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- set ns2.prev_role = message.role %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if not ns_state.thinking %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
@@ -3,40 +3,36 @@
|
||||
#
|
||||
# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
|
||||
# qwen3.8-activefpx-promptforge-v2.3 plus the v3 output-K8 runtime patch
|
||||
# serving jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3). v3 tensor
|
||||
# allocation (FFN-equalized, late-Q6 writers, full-Q8 output) with the
|
||||
# full-W8 PromptForge companions, the Output-K8 draft proxy, and the BF16
|
||||
# vision projector. Alias frank-default-large, port 7770 (moved from 52625
|
||||
# on 2026-08-22; the primary inference server sits on the standard
|
||||
# llama.cpp port).
|
||||
#
|
||||
# This runtime replaced the retired vLLM service that previously occupied
|
||||
# this slot. See docs/ciru-activefpx-promptforge.md.
|
||||
# serving kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4 (dense, Q6 AGENT
|
||||
# GGUF) with the standalone MTP Q4_0 draft head. Alias frank-default-large,
|
||||
# port 7770.
|
||||
#
|
||||
# This model replaced jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) on
|
||||
# 2026-08-24. The new model is a plain ROCmFP4 GGUF (no PromptForge .pfs
|
||||
# sidecars, no Output-K8 proxy); the requested file is Q6_.
|
||||
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm
|
||||
# HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident).
|
||||
# llama.cpp reads --host/--port/--temp as CLI flags, never as env vars.
|
||||
|
||||
# CIRU llama-server binary (ROCmFPX v2.3 + v3 output-K8 patch build, see
|
||||
# docs/ciru runtime page). Built against the private TheRock ROCm 7.15 SDK
|
||||
# kept in the retired ling3-ciru venv. ROCm 7.15 is required for the
|
||||
# advertised Strix Halo performance; the system ROCm 7.2.4 selects
|
||||
# slower/fallback kernels.
|
||||
# CIRU llama-server binary (ROCmFPX v2.3 + v3 output-K8 patch build, built
|
||||
# against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru
|
||||
# venv). ROCm 7.15 is required for the advertised Strix Halo performance.
|
||||
LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server
|
||||
|
||||
# Model files (v3). The five-file payload (GGUF + FFN/GDN .pfs + Output-K8
|
||||
# proxy + BF16 vision projector) is the full v3 configuration. The v3 files
|
||||
# live in a separate directory so the v2 artifacts stay for rollback.
|
||||
MODEL_DIR=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3
|
||||
MODEL_FILE=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf
|
||||
FFN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs
|
||||
GDN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs
|
||||
OUTPUT_K8_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs
|
||||
MMPROJ_FILE=Qwen3.8-27B-mmproj-BF16.mmproj
|
||||
# Model files. Plain dense Qwen3.8-27B ROCmFP4 Q6 AGENT GGUF plus the standalone
|
||||
# MTP draft head and the vision projector. Kingjones 2026-08-24.
|
||||
MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP
|
||||
MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf
|
||||
MTP_DRAFT_FILE=mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
MMPROJ_FILE=mmproj-Qwen3.8-27B-Q8_0.gguf
|
||||
|
||||
# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). The serve script
|
||||
# exports ROCM_PATH and LD_LIBRARY_PATH from these so the 7.15 libraries
|
||||
# precede any system ROCm 7.2.4.
|
||||
# Qwen-Sharp chat template (tracked in the repo, staged on the host).
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
# enable_thinking on by default in the Sharp template; medium effort default.
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true}'
|
||||
|
||||
# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv).
|
||||
ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
|
||||
ROCM_LIB=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries
|
||||
ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core
|
||||
@@ -45,13 +41,9 @@ ALIAS=frank-default-large
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Serving: full native 262144 context (n_ctx_train), 1 slot (user request
|
||||
# 2026-08-19: parallel slots degraded performance). -c is the TOTAL
|
||||
# context; with PARALLEL=1 it equals the per-slot context. Concurrent
|
||||
# requests queue on the single slot; the server never preempts a running
|
||||
# generation. f16 KV, context checkpoints, prompt cache, native MTP
|
||||
# depth-4. The Strix Halo is unified memory; rocm-smi reports the 64 GiB
|
||||
# budget. KV at 262144 x 64 KiB/cell is ~16 GiB.
|
||||
# Serving: full native 262144 context, 1 slot. f16 KV, context checkpoints,
|
||||
# prompt cache, draft-mtp with standalone Q4_0 head, n_max 4 (the knee for
|
||||
# the dense model). Unified memory; rocm-smi reports the 64 GiB budget.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
BATCH_SIZE=4096
|
||||
@@ -61,22 +53,21 @@ CACHE_TYPE_V=f16
|
||||
CTX_CHECKPOINTS=16
|
||||
CACHE_RAM=8192
|
||||
|
||||
# Native MTP draft (depth-4, p_min 0, backend sampling). The qualified v3
|
||||
# profile drives the draft via the Output-K8 proxy; the CPU argmax fast
|
||||
# path and IU4/ngram lanes are disabled in the serve script.
|
||||
# Dense-model MTP draft (standalone Q4_0 head, n_max 4). No PromptForge .pfs
|
||||
# sidecars are loaded; the LLAMA_CPU_ARGMAX fast path stays off.
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.0
|
||||
SPEC_DRAFT_P_SPLIT=0.10
|
||||
|
||||
# Sampling (model card defaults).
|
||||
TEMPERATURE=0.7
|
||||
# Sampling (kingjones model card / Sharp default).
|
||||
TEMPERATURE=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
|
||||
# Thinking enabled (user request, 2026-08-18). deepseek format moves the
|
||||
# <think> reasoning into message.reasoning_content (OpenAI-compatible);
|
||||
# none would leave the thought tags unparsed in content.
|
||||
# Thinking on (Qwen-Sharp template drives thinking on; reasoning format
|
||||
# projects the tar into reasoning_content).
|
||||
REASONING=on
|
||||
REASONING_FORMAT=deepseek
|
||||
|
||||
VERBOSITY=4
|
||||
VERBOSITY=4
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=CIRU llama.cpp Qwen3.8-27B ActiveFPX PromptForge MTP (dankclaw, frank-default-large)
|
||||
Description=CIRU llama.cpp Qwen3.8-27B ROCmFP4 STRIX MTP + Sharp template (dankclaw, frank-default-large)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -1,38 +1,49 @@
|
||||
# Tunables for frank-default-small (HauhauCS Qwen3.6-35B-A3B Q6_K_P,
|
||||
# ngram-mod spec, Dankserver, RTX 3090).
|
||||
# Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q8_0, MTP MoE,
|
||||
# Dankserver, RTX 3090).
|
||||
#
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB;
|
||||
# Ryzen 7 7700X; 62 GB RAM. MoE experts offloaded to CPU (--n-cpu-moe 36).
|
||||
# Restored 2026-08-18 (was the pre-2026-08-17-redo frank-default-small).
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen
|
||||
# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit
|
||||
# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS
|
||||
# Qwen3.6-35B-A3B model on 2026-08-24.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
|
||||
|
||||
# Model files (already on disk; Q6_K_P GGUF + f16 mmproj vision encoder).
|
||||
MODEL_DIR=/home/dank/models
|
||||
MODEL_FILE=Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf
|
||||
MMPROJ_FILE=mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf
|
||||
# Model files (Q8_0 CRACK GGUF + f16 mmproj vision encoder + MTP head bundled
|
||||
# in the GGUF). Dealign2026-08-24.
|
||||
MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK
|
||||
MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf
|
||||
MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf
|
||||
# Upstream llama.cpp binary (b10121).
|
||||
LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
|
||||
|
||||
# Qwen-Sharp chat template (tracked in the repo, staged on the host).
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
|
||||
ALIAS=frank-default-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# 3 parallel slots x 262144 ctx each = ~786432 total context pool (max per-slot).
|
||||
# Dual-f16 KV at 786432 cells leaves ~15 GiB free on the 3090.
|
||||
# 3 parallel slots x 262144 total context pool (max per-slot), unchanged
|
||||
# from the prior Qwen3.6 service. Lower PARALLEL if VRAM is tight with the
|
||||
# larger Q8_0 payload.
|
||||
CTX_SIZE=786432
|
||||
PARALLEL=3
|
||||
|
||||
# Spec: ngram-mod (n_min=48, n_max=64, n_match=24).
|
||||
SPEC_TYPE=ngram-mod
|
||||
NGRAM_MOD_N_MIN=48
|
||||
NGRAM_MOD_N_MAX=64
|
||||
NGRAM_MOD_N_MATCH=24
|
||||
# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
|
||||
# (the knee). ngram under MTP hurts; MTP alone is the proven best for MoE.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.0
|
||||
|
||||
# Expert offload: 38 of 40 layers keep MoE weights on CPU; 2 on GPU.
|
||||
N_CPU_MOE=38
|
||||
# Expert offload via explicit tensor override (replaces --n-cpu-moe). Only
|
||||
# the MoE FFN exps tensors move to CPU; everything else up to --gpu-layers
|
||||
# stays on the 24 GB card. The regex is the same one the built-in --cpu-moe
|
||||
# expands. Buffer type "CPU" is the CPU buft name on this host.
|
||||
OVERRIDE_EXPERT='\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
OVERRIDE_BUFT=CPU
|
||||
N_GPU_LAYERS=all
|
||||
|
||||
# KV cache on GPU, both K and V at f16.
|
||||
@@ -40,22 +51,22 @@ CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
VERBOSITY=4
|
||||
# Store SWA/context checkpoints in RAM to hold peak GPU memory down.
|
||||
# Store context checkpoints in RAM to hold peak GPU memory down.
|
||||
CTX_CHECKPOINTS=16
|
||||
|
||||
# Big prefill batches (ubatch 2048, batch 4096).
|
||||
# Big prefill batches.
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=2048
|
||||
|
||||
# Eager RAM load (MoE experts on CPU; ~31 GB RAM).
|
||||
# Eager RAM load (expert weights on CPU; ~31 GB RAM).
|
||||
NO_MMAP=1
|
||||
|
||||
# Sampling (HauhauCS Qwen3.6-35B-A3B model card, thinking-general).
|
||||
# Sampling (Dealign Ornith model card, thinking-general).
|
||||
TEMPERATURE=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
MIN_P=0.0
|
||||
|
||||
# Compute threads lowered to 7 so other processes keep a core.
|
||||
# Compute threads lowered to 8 so other processes keep a core.
|
||||
THREADS=7
|
||||
THREADS_HTTP=8
|
||||
THREADS_HTTP=8
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.6-35B-A3B Q6_K_P + ngram-mod (dankserver, frank-default-small)
|
||||
Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q8_0 MTP + Sharp template (dankserver, frank-default-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -1,19 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) as
|
||||
# Serve kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (dense) as
|
||||
# frank-default-large via the CIRU ROCmFPX llama.cpp runtime on Dankclaw
|
||||
# (Strix Halo gfx1151).
|
||||
#
|
||||
# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
|
||||
# qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) plus the v3 output-K8
|
||||
# runtime patch (runtime/qwen38-v3-output-k8-runtime.patch, commit af0eac6 on
|
||||
# Dankclaw) against the private TheRock ROCm 7.15 SDK (kept in the retired
|
||||
# ling3-ciru venv). The model is the v3 GGUF with the full-W8 PromptForge
|
||||
# companion views (FFN/GDN .pfs), the Output-K8 draft proxy, and the BF16
|
||||
# vision projector. Native depth-4 MTP speculative decoding.
|
||||
# runtime patch (commit af0eac6), against the private TheRock ROCm 7.15 SDK
|
||||
# (kept in the retired ling3-ciru venv). The model is the plain ROCmFP4
|
||||
# Q6 AGENT GGUF (no PromptForge .pfs sidecars) with the standalone MTP draft
|
||||
# head mtp-Qwen3.8-27B-Q4_0.gguf. The Qwen-Sharp chat template is applied at
|
||||
# runtime.
|
||||
#
|
||||
# Spec: dense model -> native MTP draft at n_max 4 (the knee, per the model
|
||||
# card). The draft head is a separate Q4_0 GGUF loaded via --model-draft.
|
||||
#
|
||||
# The model card requires ROCm 7.15 (not the system ROCm 7.2.4) for the
|
||||
# advertised Strix Halo performance. LD_LIBRARY_PATH therefore points at the
|
||||
# 7.15 SDK libraries first. An older system ROCm must not precede them.
|
||||
# 7.15 SDK libraries first.
|
||||
#
|
||||
# Tunables come from frank-default-large.env in this directory unless
|
||||
# FRANK_LARGE_ENV_FILE overrides that path.
|
||||
@@ -30,20 +33,19 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf}"
|
||||
: "${FFN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs}"
|
||||
: "${GDN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs}"
|
||||
: "${OUTPUT_K8_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs}"
|
||||
: "${MMPROJ_FILE:=Qwen3.8-27B-mmproj-BF16.mmproj}"
|
||||
: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${MTP_DRAFT_FILE:=mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
: "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-Q8_0.gguf}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${ROCM_DEVEL:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel}"
|
||||
: "${ROCM_LIB:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries}"
|
||||
: "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}"
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=524288}"
|
||||
: "${PARALLEL:=2}"
|
||||
: "${CTX_SIZE:=262144}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
@@ -54,30 +56,27 @@ fi
|
||||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||||
: "${SPEC_DRAFT_P_MIN:=0.0}"
|
||||
: "${SPEC_DRAFT_P_SPLIT:=0.10}"
|
||||
: "${TEMPERATURE:=0.7}"
|
||||
: "${TEMPERATURE:=1.0}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.95}"
|
||||
: "${REASONING:=on}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${CHAT_TEMPLATE_KWARGS:={}}"
|
||||
: "${VERBOSITY:=4}"
|
||||
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
export PROMPTFORGE_SIDECAR="$MODEL_DIR/$FFN_PFS"
|
||||
export PROMPTFORGE_GDN_SIDECAR="$MODEL_DIR/$GDN_PFS"
|
||||
# v3 qualified runtime: Output-K8 draft proxy + small-row W8 prompt route.
|
||||
export PROMPTFORGE_MODE="m2048_fused_tail1476"
|
||||
export GGML_CUDA_GRAPH_OPT=0
|
||||
export PROMPTFORGE_ENABLE_SMALLM_W8=1
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8=1
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY="$MODEL_DIR/$OUTPUT_K8_PFS"
|
||||
# The qualified v3 profile disables the CPU argmax draft fast path; the
|
||||
# Output-K8 proxy + backend sampling replace it. Also drop any IU4/ngram
|
||||
# companions from the non-IU4 development lane.
|
||||
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY || true
|
||||
MTP_DRAFT="$MODEL_DIR/$MTP_DRAFT_FILE"
|
||||
|
||||
# Plain dense model: no PromptForge .pfs sidecars, no Output-K8 proxy, no IU4
|
||||
# lane. The v3 MTP is driven by the standalone Q4_0 draft head through the
|
||||
# standard draft-mtp path. Unset any leftover PromptForge env lanes.
|
||||
unset PROMPTFORGE_SIDECAR PROMPTFORGE_GDN_SIDECAR PROMPTFORGE_MTP_OUTPUT_K8_PROXY || true
|
||||
unset PROMPTFORGE_MODE PROMPTFORGE_ENABLE_SMALLM_W8 || true
|
||||
unset PROMPTFORGE_MTP_OUTPUT_K8 PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE || true
|
||||
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR || true
|
||||
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4 || true
|
||||
|
||||
for f in "$MODEL" "$PROMPTFORGE_SIDECAR" "$PROMPTFORGE_GDN_SIDECAR" "$PROMPTFORGE_MTP_OUTPUT_K8_PROXY" "$MODEL_DIR/$MMPROJ_FILE"; do
|
||||
for f in "$MODEL" "$MTP_DRAFT" "$MODEL_DIR/$MMPROJ_FILE" "$CHAT_TEMPLATE_FILE"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "missing required file: $f" >&2
|
||||
exit 1
|
||||
@@ -95,6 +94,8 @@ export LD_LIBRARY_PATH="${ROCM_LIB}/lib:${ROCM_CORE}/lib:${ROCM_DEVEL}/lib${LD_L
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL" \
|
||||
--mmproj "$MODEL_DIR/$MMPROJ_FILE" \
|
||||
--chat-template-file "$CHAT_TEMPLATE_FILE" \
|
||||
--chat-template-kwargs "$CHAT_TEMPLATE_KWARGS" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
@@ -113,6 +114,7 @@ exec "$LLAMA_BIN" \
|
||||
--metrics \
|
||||
--no-mmap \
|
||||
--spec-type draft-mtp \
|
||||
--model-draft "$MTP_DRAFT" \
|
||||
--spec-draft-device ROCm0 \
|
||||
--spec-draft-ngl 999 \
|
||||
--spec-draft-type-k f16 \
|
||||
@@ -121,10 +123,10 @@ exec "$LLAMA_BIN" \
|
||||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
|
||||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
|
||||
--spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \
|
||||
--spec-draft-backend-sampling \
|
||||
--temp "$TEMPERATURE" \
|
||||
--top-k "$TOP_K" \
|
||||
--top-p "$TOP_P" \
|
||||
--reasoning "$REASONING" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget -1 \
|
||||
-lv "$VERBOSITY"
|
||||
-lv "$VERBOSITY"
|
||||
@@ -1,13 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6_K_P with
|
||||
# ngram-mod speculative decoding on Dankserver (RTX 3090, MoE experts on CPU).
|
||||
# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q8_0) as
|
||||
# frank-default-small via upstream llama.cpp on Dankserver (RTX 3090, MoE
|
||||
# experts on CPU).
|
||||
#
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total /
|
||||
# 3B-active MoE vision-language model; --n-cpu-moe keeps most expert weights on
|
||||
# CPU so the non-expert tensors + KV cache fit the 24 GB 3090.
|
||||
# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF.
|
||||
#
|
||||
# Tunables come from frank-default-small.env in this directory unless the
|
||||
# FRANK_SMALL_ENV_FILE environment variable overrides that path.
|
||||
# Expert offload: instead of the layer-counter --n-cpu-moe, the expert (FFN
|
||||
# exps) tensors are pinned to CPU explicitly with --override-tensor. The
|
||||
# pattern is the internal MoE exps regex (the same one --n-cpu-moe expands),
|
||||
# so only the Mixture-of-Experts weight tensors move to CPU; every other
|
||||
# tensor stays on the GPU up to --gpu-layers.
|
||||
#
|
||||
# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is
|
||||
# auto-detected from the target model; b10121 creates the MTP draft context
|
||||
# against the model file). ngram under MTP hurts (measured on the Qwen3.8
|
||||
# family), so MTP alone is kept. n_max 4 is the knee.
|
||||
#
|
||||
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
|
||||
# overrides that path.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -21,22 +32,27 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -----------------
|
||||
: "${MODEL_DIR:=/home/dank/models}"
|
||||
: "${MODEL_FILE:=Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf}"
|
||||
: "${MMPROJ_FILE:=mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf}"
|
||||
: "${MODEL_FILE:=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf}"
|
||||
: "${MMPROJ_FILE:=mmproj-Ornith-1.5-35B-A3B-f16.gguf}"
|
||||
: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${ALIAS:=frank-default-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=786432}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${SPEC_TYPE:=ngram-mod}"
|
||||
: "${NGRAM_MOD_N_MIN:=48}"
|
||||
: "${NGRAM_MOD_N_MAX:=64}"
|
||||
: "${NGRAM_MOD_N_MATCH:=24}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${SPEC_DRAFT_N_MAX:=4}"
|
||||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||||
: "${SPEC_DRAFT_P_MIN:=0.0}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${VERBOSITY:=4}"
|
||||
: "${N_CPU_MOE:=38}"
|
||||
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
|
||||
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
|
||||
# MoE expert weights and leave every other tensor on the GPU.
|
||||
: "${OVERRIDE_EXPERT:=\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
|
||||
: "${OVERRIDE_BUFT:=CPU}"
|
||||
: "${N_GPU_LAYERS:=all}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
@@ -52,7 +68,7 @@ fi
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
MMPROJ="$MODEL_DIR/$MMPROJ_FILE"
|
||||
|
||||
for f in "$MODEL" "$MMPROJ"; do
|
||||
for f in "$MODEL" "$MMPROJ" "$CHAT_TEMPLATE_FILE"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "missing required file: $f" >&2
|
||||
exit 1
|
||||
@@ -66,17 +82,17 @@ fi
|
||||
exec "$LLAMA_SERVER" \
|
||||
--model "$MODEL" \
|
||||
--mmproj "$MMPROJ" \
|
||||
--jinja \
|
||||
--chat-template-file "$CHAT_TEMPLATE_FILE" \
|
||||
--spec-type "$SPEC_TYPE" \
|
||||
--spec-ngram-mod-n-min "$NGRAM_MOD_N_MIN" \
|
||||
--spec-ngram-mod-n-max "$NGRAM_MOD_N_MAX" \
|
||||
--spec-ngram-mod-n-match "$NGRAM_MOD_N_MATCH" \
|
||||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
|
||||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
|
||||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
|
||||
--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--verbosity "$VERBOSITY" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
--n-cpu-moe "$N_CPU_MOE" \
|
||||
--gpu-layers "$N_GPU_LAYERS" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
@@ -90,4 +106,4 @@ exec "$LLAMA_SERVER" \
|
||||
--threads-http "$THREADS_HTTP" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT"
|
||||
--port "$LISTEN_PORT"
|
||||
Reference in New Issue
Block a user