dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault

Browse Source 
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
This commit is contained in:
Frank committed 2026-09-03 16:17:14 -07:00
1 parent 5a083eee6c
commit 8212d4451a
68 files changed
+1053 -1207
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md
+3 -3
View File
@@ -160,9 +160,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090)
The Dankserver llama.cpp deployment (`frank-default-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
@@ -195,7 +195,7 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
`frank-default-large` on Dankclaw runs the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec is the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. See [[qwen3.8-spec-matrix]] and [[Qwen3.8 PP Deployment]] for the current state and the 2026-08-15 Vulkan0/ROCm0 high-context bench.
`frank-large` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+4 -4
View File
@@ -14,7 +14,7 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
hold on the 8060S. See [[frank-default-large-benchmark]].
hold on the 8060S. See [[frank-large-benchmark]].
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
@@ -40,7 +40,7 @@ lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
different, driver-reported figure that has always been larger than the kernel
cap.
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-default-large-benchmark]].
limits documented in [[frank-large-benchmark]].
## 1. Self-Hosted Agentic Tools
@@ -98,7 +98,7 @@ All require root and a reboot for the modprobe change.
## Sources
- [[frank-default-large-benchmark]] — measured backend/spec numbers
- [[frank-large-benchmark]] — measured backend/spec numbers
- [[deployment]] — deploy and update process
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
@@ -107,7 +107,7 @@ All require root and a reboot for the modprobe change.
## See Also
- [[frank-default-large-benchmark]]
- [[frank-large-benchmark]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
- [[index]]
serverAppConfigs-docs/LLM/Dankclaw System Specs.md
+2 -2
View File
@@ -38,7 +38,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-08-17**) |
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
The NPU is retired; the 52625 port and the `frank-default-small` name now serve the CIRU ROCmFPX llama.cpp runtime (see [[ciru-activefpx-promptforge]]).
The NPU is retired; the 52625 port and the `frank-small` name now serve the CIRU ROCmFPX llama.cpp runtime (see [[ciru-activefpx-promptforge]]).
### ROCm
@@ -93,7 +93,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]].
- The active service is `frank-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]].
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 1.1 TB free disk space is ample for model storage.
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+1 -1
View File
@@ -60,7 +60,7 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
The active service on Dankserver is `frank-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
+11 -11
View File
@@ -1,7 +1,7 @@
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
> The `frank-default-small` name and port 52625 now belong to the vLLM
> The `frank-small` name and port 52625 now belong to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
---
@@ -19,7 +19,7 @@ timestamp: 2026-08-16
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-default-small` (port 52625) serves in production. See Promotion.
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
## Verdict (2026-08-16)
@@ -168,7 +168,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-default-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
- The `llama-server` (frank-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
@@ -186,7 +186,7 @@ Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens)
## Model Bakeoff (measured 2026-08-16, 4 models)
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-default-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
@@ -204,21 +204,21 @@ Notes:
## Promotion (2026-08-16)
`frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
`frank-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
The `frank-default-small` unit is disabled for boot. Only the PP pair (`frank-default-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
The `frank-small` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/frank-small.service
sudo systemctl daemon-reload
sudo systemctl enable frank-default-small
sudo systemctl enable frank-small
```
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`).
The PP split was renamed `frank-pp-head` → `frank-large` (2026-08-16, commit `48142f5`).
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
@@ -277,7 +277,7 @@ graph TD
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-default-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-default-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
@@ -314,7 +314,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16).** `frank-default-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
- **NPU production role (2026-08-16).** `frank-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
## Citations
serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md
+1 -1
View File
@@ -26,7 +26,7 @@ conclusions.
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-default-large.env` |
| ubatch | 256 | `UBATCH_SIZE` in `frank-large.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md
+1 -1
View File
@@ -189,7 +189,7 @@ For the 2-node pipeline-parallel split (RTX 3090 24 GB + Strix Halo 96 GB):
- **Kept:** Qwen official config.json (HF) — exact architecture params (64 layers, 5120 hidden, 4 KV heads, head_dim 256) [8](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json)
- **Kept:** bartowski/Qwen_Qwen3.6-27B-GGUF — Q8_0 at 29.12 GB, imatrix calibration [2](https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF)
- **Kept:** Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — DFlash 2B draft GGUF model [4](https://huggingface.co/Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp)
- **Kept:** frank-default-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) [5](../frank-default-large-benchmark.md)
- **Kept:** frank-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) [5](../frank-large-benchmark.md)
- **Kept:** a4lg/Qwen3.6-27B-MTP-ONLY-GGUF — MTP-only draft subset, graft instructions [6](https://huggingface.co/a4lg/Qwen3.6-27B-MTP-ONLY-GGUF)
- **Kept:** llama.cpp tools/rpc/README.md — RPC backend, weight/KV split, remote host setup [7](https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/README.md)
- **Kept:** Unsloth Qwen3.6-27B-MTP-GGUF — MTP command-line usage for llama.cpp [9](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md
+9 -9
View File
@@ -11,7 +11,7 @@ morning-of runbook. Research background:
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
This split is the production `frank-default-large` service. Current
This split is the production `frank-large` service. Current
state, verified against the live journal and output smoke tests
(2026-08-17, load 02:06):
@@ -32,17 +32,17 @@ state, verified against the live journal and output smoke tests
Vulkan).
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
stable; all allocations are upfront).
- Unit `frank-default-large.service`, launched through
`serve-pp-head.sh` + `frank-default-large.env`.
- Unit `frank-large.service`, launched through
`serve-pp-head.sh` + `frank-large.env`.
## Service enablement (2026-08-17)
The PP pair is the only inference service enabled for boot. Both units
(`frank-default-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-default-small` on Dankclaw) and all retired
(`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-small` on Dankclaw) and all retired
units are disabled. Restart order: worker first, then head. The Dankserver
helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-default-small` unit); restart the worker with
`frank-small` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. `deploy.sh dankserver` is broken until retargeted. See
@@ -164,9 +164,9 @@ llama-server b10433 ggml-rpc-server b10433
|------|---------|---------|
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/frank-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/frank-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
@@ -220,7 +220,7 @@ FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
```
Config variants: edit `scripts/frank-default-large.env` on the claw (git commit +
Config variants: edit `scripts/frank-large.env` on the claw (git commit +
pull if the bench runs from the repo), then re-run `start` + `bench`:
- No spec baseline: `SPEC_NONE=1`
serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md
+1 -1
View File
@@ -76,6 +76,6 @@ graph LR
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[frank-default-small-benchmark]]
- [[frank-small-benchmark]]
- [[Dankserver System Specs]]
- [[Dankclaw System Specs]]
serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md
+1 -1
View File
@@ -47,7 +47,7 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB
## Prerequisites
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-default-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md
+1 -1
View File
@@ -114,7 +114,7 @@ export HF_HOME=/mnt/entropy/huggingface
### Active Service Conflict
The retired `frank-default-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
The retired `frank-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
```
nvidia-smi # verify GPU is free
serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md
+7 -7
View File
@@ -1,7 +1,7 @@
---
type: reference
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-large.
tags:
- llama.cpp
- rocm
@@ -26,7 +26,7 @@ timestamp: 2026-08-29
> `--flash-attn on` work on the charlie build-qwen38 tree — all still
> true for that binary + model pair, which remains the rollback target.
`frank-default-large` (port 7770) served
`frank-large` (port 7770) served
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
with the author's own MTP draft head
@@ -166,7 +166,7 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Config files
- Env: `scripts/frank-default-large.env`
- Env: `scripts/frank-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
@@ -175,10 +175,10 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Swap procedure
```bash
sudo systemctl stop frank-default-large
# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large
sudo systemctl start frank-default-large # or via deploy.sh
sudo systemctl stop frank-large
# edit scripts/frank-large.env + scripts/serve-frank-large.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-large
sudo systemctl start frank-large # or via deploy.sh
curl http://192.168.0.143:7770/health
```
serverAppConfigs-docs/LLM/ciru-activefpx-promptforge.md
+6 -6
View File
@@ -1,7 +1,7 @@
---
type: reference
title: CIRU ActiveFPX PromptForge on Dankclaw
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service.
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-large service.
tags:
- llama.cpp
- rocm
@@ -14,12 +14,12 @@ timestamp: 2026-08-22
# CIRU ActiveFPX PromptForge on Dankclaw
> **Active 2026-08-22.** `frank-default-large` serves this v3 model again
> **Active 2026-08-22.** `frank-large` serves this v3 model again
> after a same-day Kairic Edge IU4 experiment was reverted (operator
> decision). Commit `c60156f` restored the v3 script pair. See
> [[kairic-edge]] for the retired experiment reference.
`frank-default-large` (port 7770, moved from 52625 on 2026-08-22) serves
`frank-large` (port 7770, moved from 52625 on 2026-08-22) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
@@ -135,7 +135,7 @@ graph TD
C --> D[Build llama-server]
D --> E[Verify --list-devices shows gfx1151]
E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
F --> G[Deploy frank-default-large]
F --> G[Deploy frank-large]
G --> H[Health check + chat completion]
```
@@ -150,8 +150,8 @@ against the model card before the service restart.
## Service
- Unit: `frank-default-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
- Unit: `frank-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
the system ROCm. `ROCM_PATH` points at the devel SDK.
serverAppConfigs-docs/LLM/dankserver-vram-growth.md
+4 -4
View File
@@ -5,7 +5,7 @@
---
type: reference
title: Dankserver VRAM and RAM Growth Under Load
description: Investigation of the recurring cuBLAS crash on frank-default-small. What grows, what does not, and what to do next.
description: Investigation of the recurring cuBLAS crash on frank-small. What grows, what does not, and what to do next.
tags:
- dankserver
- llamacpp
@@ -18,7 +18,7 @@ tags:
# Dankserver VRAM and RAM Growth Under Load
This page records the investigation of the recurring cuBLAS crash on
`frank-default-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
`frank-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
with `CUDA error: an unsupported value or parameter was passed to the
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
## Mitigation Options
The documentation for the benchmark ([[frank-default-small-benchmark]])
The documentation for the benchmark ([[frank-small-benchmark]])
prescribes the first option for this crash class. None of these change
the model.
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
## See Also
- [[frank-default-small-benchmark]]
- [[frank-small-benchmark]]
- [[Dankserver System Specs]]
- [[deployment]]
serverAppConfigs-docs/LLM/dealign-ornith-crack.md
+3 -3
View File
@@ -1,7 +1,7 @@
---
type: reference
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-small.
tags:
- llamacpp
- cuda
@@ -14,12 +14,12 @@ timestamp: 2026-08-24
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
> **Active 2026-08-25.** `frank-default-small` serves this model. It
> **Active 2026-08-25.** `frank-small` serves this model. It
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
> The Q6_K file was downloaded from the model repository and validated with
> `llama-gguf` before installation.
`frank-default-small` (port 7770) serves
`frank-small` (port 7770) serves
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
attention architecture, abliterated (uncensored). The GGUF also contains a
serverAppConfigs-docs/LLM/deployment.md
+24 -18
View File
@@ -19,8 +19,9 @@ SearXNG, Caddy, or another container service does **not** restart inference.
| Target | Host | Service | Port |
|---|---|---|---:|
| Dankserver | `dank@dankserver` | `frank-default-small` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-default-large` | 7770 |
| Dankserver | `dank@dankserver` | `frank-small` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-npu` (FLM on the XDNA2 NPU) | 7771 |
## Scoped workflow
@@ -38,34 +39,39 @@ graph LR
Run the unified deployer from the repository root:
```bash
./scripts/deploy.sh dankserver --service frank-default-small
./scripts/deploy.sh dankclaw --service frank-default-large
./scripts/deploy.sh dankserver --service frank-small
./scripts/deploy.sh dankclaw --service frank-large
./scripts/deploy.sh dankclaw --dry-run
```
Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
on the selected host. The following files select the corresponding service:
- `scripts/frank-default-small.env`, `frank-default-small.service`, and
`serve-frank-small.sh` select `frank-default-small` on Dankserver.
- `scripts/frank-default-large.env`, `frank-default-large.service`, and
`serve-frank-large.sh` select `frank-default-large` on Dankclaw.
- `scripts/frank-small.env`, `frank-small.service`, and
`serve-frank-small.sh` select `frank-small` on Dankserver.
- `scripts/frank-large.env`, `frank-large.service`, and
`serve-frank-large.sh` select `frank-large` on Dankclaw.
- `scripts/frank-npu.env`, `frank-npu.service`,
and `serve-frank-npu.sh` select `frank-npu`
on Dankclaw.
- `scripts/chat-template.jinja` selects the LLM on the selected host only.
- Documentation, tests, deployment tooling, and unrelated container changes
do not select either inference service.
The target-side `scripts/deploy-remote.sh` relinks the selected unit into
`/etc/systemd/system/`, reloads systemd, enables it, restarts it, and waits for
`http://127.0.0.1:7770/health`. The wait budget is 20 minutes because cold model
loads can be slow. A failed handler does not advance its per-service state.
that service's health endpoint (`http://127.0.0.1:7770/health` for the two
inference services; `/v1/version` for the NPU sight+sound endpoint — FLM has no
`/health` route). The wait budget is 20 minutes because cold model loads can be
slow. A failed handler does not advance its per-service state.
## Unit and artifact paths
Both units point into the stable target checkout:
```text
/home/dank/serverAppConfigs/scripts/frank-default-small.service
/home/dank/serverAppConfigs/scripts/frank-default-large.service
/home/dank/serverAppConfigs/scripts/frank-small.service
/home/dank/serverAppConfigs/scripts/frank-large.service
```
The unit files and serve scripts validate their model/runtime artifacts before
@@ -74,8 +80,8 @@ the unit absolute paths and the deployment tests in the same change.
## NPU unit
`frank-default-npu` is an inactive, historical Dankclaw endpoint. It is
relinked only when explicitly selected with `--service frank-default-npu`; the
`frank-npu` is an inactive, historical Dankclaw endpoint. It is
relinked only when explicitly selected with `--service frank-npu`; the
large-model handler never restarts it. This is intentional: the NPU is not part
of the active inference path and should not be disturbed by a large-model
change.
@@ -99,16 +105,16 @@ change in that service's inputs.
If the LLM handler fails after the target pull:
1. Inspect `journalctl -u frank-default-small -n 200` or
`journalctl -u frank-default-large -n 200`.
1. Inspect `journalctl -u frank-small -n 200` or
`journalctl -u frank-large -n 200`.
2. Confirm the model files, custom runtime binary, and chat template are
readable by `dank`.
3. Retry only the affected service:
```bash
./scripts/deploy.sh dankserver --service frank-default-small
./scripts/deploy.sh dankserver --service frank-small
# or
./scripts/deploy.sh dankclaw --service frank-default-large
./scripts/deploy.sh dankclaw --service frank-large
```
4. Verify the matching health endpoint before declaring recovery.
serverAppConfigs-docs/LLM/dflash2-large-vulkan.md
+7 -7
View File
@@ -1,6 +1,6 @@
---
type: reference
title: DFlash2 on Dankclaw (frank-default-large)
title: DFlash2 on Dankclaw (frank-large)
description: ARCHIVED - unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Retired 2026-08-21 for speed; kept as a runbook.
tags:
- llama.cpp
@@ -13,14 +13,14 @@ tags:
timestamp: 2026-08-21
---
# DFlash2 on Dankclaw (frank-default-large) — ARCHIVED
# DFlash2 on Dankclaw (frank-large) — ARCHIVED
> **Archived 2026-08-21.** This DFlash2 Vulkan deployment was retired the
> same day it shipped because it was slower than the CIRU runtime it
> replaced. `frank-default-large` is back on CIRU. This page stays as the
> replaced. `frank-large` is back on CIRU. This page stays as the
> runbook and provenance for the DFlash2 attempt.
`frank-default-large` (port 52625) briefly served
`frank-large` (port 52625) briefly served
`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
runtime was upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
@@ -114,14 +114,14 @@ graph TD
C --> D[Build llama-server]
D --> E[Verify Vulkan0 device]
E --> F[Add DFlash2 drafter]
F --> G[Swap frank-default-large]
F --> G[Swap frank-large]
G --> H[Health check + completion]
```
## Service
- Unit: `frank-default-large.service`, port 52625
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
- Unit: `frank-large.service`, port 52625
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
### Launch profile
serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md
+10 -10
View File
@@ -1,7 +1,7 @@
---
type: reference
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-default-large.
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-large.
tags:
- llama.cpp
- rocm
@@ -15,12 +15,12 @@ timestamp: 2026-08-25
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
> **Active 2026-08-25.** `frank-default-large` serves this model. It replaced
> **Active 2026-08-25.** `frank-large` serves this model. It replaced
> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same
> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See
> [[kingjones-strix-mtp]] for the retired page.
`frank-default-large` (port 7770) serves
`frank-large` (port 7770) serves
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
@@ -127,8 +127,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Service
- Unit: `frank-default-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
- Unit: `frank-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
## Tuning MTP
@@ -142,14 +142,14 @@ Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
```
sudo systemctl stop frank-default-large
sudo systemctl stop frank-large
./scripts/bench-ubatch.sh
sudo systemctl start frank-default-large
sudo systemctl start frank-large
```
The script stops `frank-default-large` for a clean GPU, boots one server
The script stops `frank-large` for a clean GPU, boots one server
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
and restarts the service. See [[frank-default-large-benchmark]].
and restarts the service. See [[frank-large-benchmark]].
## Known Issues
@@ -183,4 +183,4 @@ The ubatch sweep above gives the production ubatch choice.
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
- [[Dankclaw System Specs]] — the box
- [[frank-default-large-benchmark]] — backend/spec benchmark history
- [[frank-large-benchmark]] — backend/spec benchmark history
serverAppConfigs-docs/LLM/frank-large-benchmark.md
+2 -2
View File
@@ -2,10 +2,10 @@
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# frank-default-large — Backend and Spec Benchmark
# frank-large — Backend and Spec Benchmark
This page records the benchmarks that chose the runtime for
`frank-default-large` on Dankclaw. Every number came from the live
`frank-large` on Dankclaw. Every number came from the live
llama.cpp build on that machine (upstream, b10133+).
## Summary
serverAppConfigs-docs/LLM/frank-npu.md
+11 -11
View File
@@ -1,6 +1,6 @@
---
type: configuration
title: frank-default-sightsound Runbook
title: frank-npu Runbook
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy.
tags:
- flm
@@ -8,17 +8,17 @@ tags:
- xdna2
- npu
- dankclaw
- frank-default-sightsound
- frank-npu
timestamp: 2026-09-01
---
# frank-default-sightsound — FLM NPU sight+sound Runbook
# frank-npu — FLM NPU sight+sound Runbook
`frank-default-sightsound` is the third inference endpoint on Dankclaw: a
`frank-npu` is the third inference endpoint on Dankclaw: a
FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`)
running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) —
which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]],
`frank-default-large`) does not do. The model-selection tradeoff: the only
`frank-large`) does not do. The model-selection tradeoff: the only
NPU2 pack family on this box with BOTH a vision and an audio tower is the
Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx)
is text-only and fails the hard modality requirement, so this endpoint trades
@@ -29,9 +29,9 @@ is text-only and fails the hard modality requirement, so this endpoint trades
| Field | Value |
|-------|-------|
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
| Service | `frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
| Files | `scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service` |
| Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
| Service | `frank-npu.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
| Files | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
| Deploy | `scripts/deploy.sh dankclaw --service frank-npu` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
| Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) |
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
| Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) |
@@ -57,8 +57,8 @@ clients choose, send requests with the desired `model` field.
**To swap the active model** (changes what the service loads at start):
```
sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-default-sightsound.env
./scripts/deploy.sh dankclaw --service frank-default-sightsound
sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-npu.env
./scripts/deploy.sh dankclaw --service frank-npu
```
**To switch per-request** (service stays loaded, client sends the tag):
@@ -119,7 +119,7 @@ chain, all live-verified on the host:
3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT`
on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the
iGPU default domain identity-mapped (zero DMA-translation overhead for
`frank-default-large`) while enabling IOMMU hardware + PASID, which is what
`frank-large`) while enabling IOMMU hardware + PASID, which is what
the NPU driver's SVA bind requires — PASID domains are paged regardless of
the default-domain type.
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md
+18 -18
View File
@@ -8,15 +8,15 @@ The old scripts moved to `old/scripts/` in this repository.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-default-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
| frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
tensor split, no NPU.
## Models
### frank-default-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
### frank-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
@@ -31,7 +31,7 @@ tensor split, no NPU.
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
(48/64/24).
### frank-default-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
### frank-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
@@ -46,7 +46,7 @@ tensor split, no NPU.
instead of the Qwen3 XML structure). The Quark quant follows the format
cleanly; swapped in the evening of 2026-08-17.
## Context Math (frank-default-large, RTX 3090)
## Context Math (frank-large, RTX 3090)
KV per token: 16 attention layers x 4 KV heads x 256 dim.
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
@@ -63,11 +63,11 @@ Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
Production choice: **131072 (128K)** — the maximum that keeps about
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
`scripts/frank-default-large.env` to 122880 for ~1.8 GiB headroom.
`scripts/frank-large.env` to 122880 for ~1.8 GiB headroom.
## Install Runbook
### Dankserver (frank-default-large)
### Dankserver (frank-large)
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
@@ -87,17 +87,17 @@ Production choice: **131072 (128K)** — the maximum that keeps about
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
4. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-default-large
sudo systemctl restart frank-default-large
sudo ln -sf /home/dank/vLLM/scripts/frank-large.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-large
sudo systemctl restart frank-large
```
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
6. Confirm VRAM headroom with `nvidia-smi`.
### Dankclaw (frank-default-small)
### Dankclaw (frank-small)
1. Stop the old services: `sudo systemctl stop frank-default-large
frank-default-small` (retired PP head and NPU endpoint).
1. Stop the old services: `sudo systemctl stop frank-large
frank-small` (retired PP head and NPU endpoint).
2. Install uv and create the venv:
```
curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -122,9 +122,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about
vision encoder falls back to torch SDPA and engine init OOMs.
5. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
sudo systemctl restart frank-default-small
sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-small
sudo systemctl restart frank-small
```
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
@@ -150,7 +150,7 @@ on gfx1151; newer vLLM releases may already include it.
Three changes landed after the initial deployment:
### 1. Tool calling on frank-default-small (HTTP 400 fix)
### 1. Tool calling on frank-small (HTTP 400 fix)
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
unless the server enables tool calling. Fixes applied:
@@ -175,7 +175,7 @@ model behavior, which motivated the swap.
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
`frank-default-small` now serves
`frank-small` now serves
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
vLLM-native expert layout). The Avesed compressed-tensors quant's
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
serverAppConfigs-docs/LLM/frank-small-benchmark.md
+4 -4
View File
@@ -1,6 +1,6 @@
---
type: reference
title: frank-default-small Benchmark — ncmoe and Batch Tuning
title: frank-small Benchmark — ncmoe and Batch Tuning
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
tags:
- benchmark
@@ -10,11 +10,11 @@ tags:
- dankserver
---
# frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning
# frank-small Benchmark — ncmoe and Batch/Ubatch Tuning
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
`--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`.
`--n-cpu-moe` and `--ubatch-size` values for `frank-small`.
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
@@ -105,7 +105,7 @@ service runs 700416 cells (13,680 MiB).
## Chosen State
> **Live drift (2026-08-22 audit):** the deployed `frank-default-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
> **Live drift (2026-08-22 audit):** the deployed `frank-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
Prefill speed is the priority, so the state keeps the top prefill tier and
moves one expert layer to CPU for stability:
serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md
+2 -2
View File
@@ -6,7 +6,7 @@ tags:
- freetoken
- benchmark
- dankserver
- frank-default-small
- frank-small
- qwen3.6
timestamp: 2026-09-02
---
@@ -90,5 +90,5 @@ Tiel-Coder finetune.
## See Also
- [[freetoken-frank-default-small]] — production runbook
- [[freetoken-frank-small]] — production runbook
- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)
serverAppConfigs-docs/LLM/freetoken-frank-small.md
+8 -8
View File
@@ -1,20 +1,20 @@
---
type: configuration
title: FreeToken frank-default-small Runbook
description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-default-small on Dankserver (RTX 3090).
title: FreeToken frank-small Runbook
description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090).
tags:
- freetoken
- dankserver
- frank-default-small
- frank-small
- qwen3.6
- moe
- fp8
timestamp: 2026-08-30
---
# FreeToken frank-default-small Runbook
# FreeToken frank-small Runbook
The `frank-default-small` service on Dankserver is served by
The `frank-small` service on Dankserver is served by
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
expert weights in host RAM with an LRU GPU cache. Model history on this
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
@@ -130,8 +130,8 @@ modelopt block-fp8 remains the only FP8 path that loads.
| File | Purpose |
|------|---------|
| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
| `scripts/frank-default-small.env` | Tunables (model path, pool sizes) |
| `scripts/frank-default-small.service` | Systemd unit |
| `scripts/frank-small.env` | Tunables (model path, pool sizes) |
| `scripts/frank-small.service` | Systemd unit |
## Limitations
@@ -150,7 +150,7 @@ modelopt block-fp8 remains the only FP8 path that loads.
- **`model` field is required** in `/v1/chat/completions` bodies —
FreeToken rejects requests without it with a 422 (`Field required`),
unlike llama.cpp's server which defaults to the loaded model. Send
`"model": "frank-default-small"`.
`"model": "frank-small"`.
- Load on this swap took well under the old llama.cpp numbers:
weights → experts (parallel build) → CUDA graph capture → Uvicorn
in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md
+5 -5
View File
@@ -1,7 +1,7 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
@@ -16,7 +16,7 @@ timestamp: 2026-08-26
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
> **Active 2026-08-26.** `frank-default-small` switched from upstream
> **Active 2026-08-26.** `frank-small` switched from upstream
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
> RTX 3090. This page documents the build, the mandatory FA build flags, and
> the MoE / CPU-offload runtime arguments.
@@ -101,8 +101,8 @@ a hard compile error.
## Runtime configuration
Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
`scripts/frank-default-small.service` in the repo. The service just calls
Files: `scripts/serve-frank-small.sh`, `scripts/frank-small.env`,
`scripts/frank-small.service` in the repo. The service just calls
the serve script; the env file holds all tunables.
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
@@ -241,7 +241,7 @@ apply.
The previous runtime `upstream llama.cpp b10121` and its binary
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
rollback to the old stack, change `LLAMA_SERVER=` in
`scripts/frank-default-small.env` back to that path and
`scripts/frank-small.env` back to that path and
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
spec / `--spec-type` flags. The model file is not changed either way.
serverAppConfigs-docs/LLM/index.md
+20 -17
View File
@@ -1,9 +1,9 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on two local machines
(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-default-small,
(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-small,
Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX
llama.cpp on Dankclaw (frank-default-large, kingjones777
llama.cpp on Dankclaw (frank-large, kingjones777
Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template.
> 2026-08-22: the former `vllmScripts` repository is merged into the
@@ -18,7 +18,7 @@ graph LR
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
A[RTX 3090 24GB + 60GB RAM]
B2[FreeToken v0.1.2
frank-default-small, port 7770]
frank-small, port 7770]
C[Huihui-Qwen3.6-35B-A3B-abliterated-FP8
block-fp8, 20% MoE residency, 453k KV / 4 streams]
A --> B2 --> C
@@ -27,7 +27,7 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
E2[gfx1151 iGPU 64 GiB unified memory]
E[kingjones30/ROCmFPX build
frank-default-large, port 7770]
frank-large, port 7770]
F[kingjones777 Qwen3.8-Flash-Next
ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
E2 --> E --> F
@@ -46,18 +46,18 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
## Active Deployment
- [[freetoken-frank-default-small]] — the active frank-default-small
- [[freetoken-frank-small]] — the active frank-small
runbook: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-
abliterated-FP8 on Dankserver (swapped 2026-08-30; same format/pools)
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[kingjones-strix-rocmfp4]] — ACTIVE frank-default-large runbook:
- [[kingjones-strix-rocmfp4]] — ACTIVE frank-large runbook:
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
kingjones30/ROCmFPX fork (swapped 2026-08-30)
- [[agentionai-fast-imatrix]] — retired frank-default-large runbook:
- [[agentionai-fast-imatrix]] — retired frank-large runbook:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38
(2026-08-29 → 2026-08-30); rollback binary + payload retained
@@ -66,9 +66,9 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
on Dankclaw (retired)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-default-small, retired 2026-09-02)
Dankserver (frank-small, retired 2026-09-02)
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
frank-default-small (2026-08-26): q8_0 + Hadamard KV cache,
frank-small (2026-08-26): q8_0 + Hadamard KV cache,
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
@@ -76,9 +76,10 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on
- [[Dankserver System Specs]] — the RTX 3090 box (frank-small)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large,
served on the pinned kingjones30/ROCmFPX llama.cpp fork since 2026-08-30)
- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on
Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port
7771, whole-registry serving, per-request model routing), incl. the
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
@@ -101,8 +102,10 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (the historical family; frank-default-small now serves
Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]])
notes and sizing (this architecture family is what frank-small
serves today in the Huihui abliterated FP8 variant — see
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
that preceded it is archived at [[dealign-ornith-crack]])
## Memory Management
@@ -115,7 +118,7 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
@@ -134,7 +137,7 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
## Related
- [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
Gitea instance on Dankclaw hosts this repository.
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
git-based deploy with inference isolation
serverAppConfigs-docs/LLM/kairic-edge.md
+4 -4
View File
@@ -17,10 +17,10 @@ timestamp: 2026-08-22
> **Retired 2026-08-22.** This experiment was deployed, benchmarks were
> collected, and the service reverted to v3 the same day (operator
> decision). `frank-default-large` serves v3 again. This page is kept as
> decision). `frank-large` serves v3 again. This page is kept as
> the retired experiment's build runbook and reference.
`frank-default-large` (port 52625) briefly served
`frank-large` (port 52625) briefly served
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` through the Kairic Edge CIRU ROCmFPX
llama.cpp runtime. The runtime routes qualified prompt and multi-token
verification shapes through the RDNA 3.5 native
@@ -215,7 +215,7 @@ service restart confirms the fresh tree, then it is a deletion candidate
The service runs the pinned runner profile plus the local chat
deviations below. Tunables live in
`scripts/frank-default-large.env`; flags in `scripts/serve-frank-large.sh`.
`scripts/frank-large.env`; flags in `scripts/serve-frank-large.sh`.
See [[deployment]] for the update process.
| Setting | Value | Origin |
@@ -301,7 +301,7 @@ ubatches do not. Production runs the runner's qualified 2048/512.
The v3 payload and binary stay on the machine for rollback.
1. Stop the service: `sudo systemctl stop frank-default-large`
1. Stop the service: `sudo systemctl stop frank-large`
2. Restore the v3 scripts:
`git revert 4c43839 7aee1df 40d50fe` (or checkout the script pair from
commit `132e147`)
serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md
+6 -6
View File
@@ -1,11 +1,11 @@
# frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
# frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
**Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by
the `set_rows` build bug** — the model loads but crashes in
`ggml_compute_forward_set_rows` before it can serve. All config goals are met;
a build fix is required for the model to run.
**Alias:** `frank-default-large`, port **7770**.
**Alias:** `frank-large`, port **7770**.
The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF
from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a
@@ -26,7 +26,7 @@ fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` |
| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on |
| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` |
| Tunables | `scripts/frank-default-large.env` |
| Tunables | `scripts/frank-large.env` |
| Serve script | `scripts/serve-frank-large.sh` |
## Serving status: the `set_rows` build bug (blocks serving)
@@ -107,7 +107,7 @@ dl mtp-Qwen3.8-27B-Q4_0.gguf
```
The serve script checks each artifact is present and readable before launch
(`scripts/frank-default-large.env` -> the four `MODEL_*` files plus the chat
(`scripts/frank-large.env` -> the four `MODEL_*` files plus the chat
template). A missing file fails the artifact check with `missing or
unreadable required artifact`.
@@ -116,7 +116,7 @@ unreadable required artifact`.
`scripts/serve-frank-large.sh` builds the `llama-server` call from the env:
```
/dev/llama-server -m <main> --alias frank-default-large
/dev/llama-server -m <main> --alias frank-large
--host 0.0.0.0 --port 7770 --jinja
-dev ROCm0 -ngl 999 -np 1 -c 262144
-b 512 -ub 512 -fa on
@@ -197,7 +197,7 @@ and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the
## Env tunables
`scripts/frank-default-large.env`. Serving: `CTX_SIZE=262144`,
`scripts/frank-large.env`. Serving: `CTX_SIZE=262144`,
`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`,
`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`,
`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`,
serverAppConfigs-docs/LLM/kingjones-strix-mtp.md
+3 -3
View File
@@ -1,7 +1,7 @@
---
type: reference
title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-default-large.
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-large.
tags:
- llama.cpp
- rocm
@@ -16,13 +16,13 @@ timestamp: 2026-08-24
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
> **Active again 2026-09-01.** The kingjones ROCmFP4 STRIX MTP payload is
> restored as `frank-default-large` after the Q8_0 AGENT experiment. The Q8_0
> restored as `frank-large` after the Q8_0 AGENT experiment. The Q8_0
> model was retired because it crashed before serving. The Q6 AGENT model,
> standalone MTP draft, Q8 projector, and PromptForge runtime are now active.
> This page documents the active model and its runbook.
`frank-default-large` (port 7770) serves
`frank-large` (port 7770) serves
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md
+7 -7
View File
@@ -1,7 +1,7 @@
---
type: reference
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
description: ACTIVE frank-default-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
description: ACTIVE frank-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
tags:
- llama.cpp
- rocm
@@ -23,10 +23,10 @@ timestamp: 2026-08-30
> decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads
> do not load on this loader family — see MTP Draft Head).
`frank-default-large` (port 7770) serves
`frank-large` (port 7770) serves
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
via `scripts/serve-frank-large.sh` + `scripts/frank-default-large.env`
via `scripts/serve-frank-large.sh` + `scripts/frank-large.env`
and the Qwen-Sharp chat template.
## Model Architecture
@@ -178,7 +178,7 @@ cmake --build build --target llama-server llama-quantize -j16
## Serving Profile
Deployed args (`scripts/frank-default-large.env` — our validated
Deployed args (`scripts/frank-large.env` — our validated
profile, unchanged from the FAST deployment except the binary and model):
```
@@ -272,9 +272,9 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
1. Build the runtime (see Runtime build) BEFORE touching the service.
2. Point the model file's directory at the canonical path (a plain `mv`
— never delete a 113 GiB payload to re-fetch it).
3. Edit `scripts/frank-default-large.env` (+ serve script defaults) via
3. Edit `scripts/frank-large.env` (+ serve script defaults) via
git: commit, push, `./scripts/deploy.sh dankclaw --service
frank-default-large`.
frank-large`.
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
`/v1/chat/completions` probe.
@@ -284,7 +284,7 @@ binary and the agentionai payload remain on disk.
### Config files
- Env: `scripts/frank-default-large.env`
- Env: `scripts/frank-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Chat template: `scripts/chat-template.jinja`
serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md
+6 -10
View File
@@ -1,10 +1,6 @@
# Ling-3.0-flash Q6_K Trial on Dankclaw
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. `frank-default-large` now runs the Qwen3.8-27B PP head, not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
This page records the Ling-3.0-flash trial that temporarily replaced
`frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on
port 7770 under the same API alias.
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. At this page's retirement date `frank-large` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
## Status
@@ -73,22 +69,22 @@ point varies (385-1588 tokens). Single streams never freeze.
`SPEC_TYPE=none`, `PARALLEL=2`. All concurrent streams complete (EOS).
Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with
MTP. Fallback chain if spec-off regresses: A) `PARALLEL=1` keep MTP,
B) switch back to Qwen3.6-27B (frank-default-large, MTP n_max 3).
B) switch back to Qwen3.6-27B (frank-large, MTP n_max 3).
## Files
- `scripts/frank-large-ling.service` - systemd unit.
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
draft-mtp).
- `scripts/frank-default-large.ling.env` - tunables.
- `scripts/frank-large.ling.env` - tunables.
Passwordless management is installed:
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
## Switch Back (obsolete)
The switch-back commands below are obsolete. `frank-default-large` now
runs the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
The switch-back commands below are obsolete. At retirement time
`frank-large` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
re-enable `frank-large-ling` from the repo unit
(`scripts/frank-large-ling.service`) and serve script
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
@@ -107,6 +103,6 @@ re-enable `frank-large-ling` from the repo unit
## See Also
- [[frank-default-large-benchmark]]
- [[frank-large-benchmark]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
serverAppConfigs-docs/LLM/ling-3.0-flash-vllm-plan.md
+1 -1
View File
@@ -6,5 +6,5 @@ Dankclaw. The deployment never shipped. The actual Ling work used the
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
The trial service (`frank-large-ling`) is retired and disabled under the
boot policy (2026-08-17). `frank-default-large` is the Qwen3.8-27B PP
boot policy (2026-08-17). `frank-large` is the Qwen3.8-27B PP
head, not the Ling path. See git history for the full decision table.
serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md
+3 -3
View File
@@ -5,7 +5,7 @@
# Qwen3.8-27B Speculative-Decoding Matrix
This page records the speculative-decoding benchmark history and the
production tuning for the Qwen3.8-27B service (`frank-default-large`),
production tuning for the Qwen3.8-27B service (`frank-large`),
now a pipeline-parallel head with a Dankserver worker.
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
@@ -145,7 +145,7 @@ that is the long-context attention penalty, not a spec artifact.
## Production Configuration
The live tunables live in `scripts/frank-default-large.env` (head) and
The live tunables live in `scripts/frank-large.env` (head) and
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
state section at the top of this page records the current values: Q6_K
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
@@ -153,6 +153,6 @@ q8_0/f16 KV.
## See Also
- [[frank-default-large-benchmark]]
- [[frank-large-benchmark]]
- [[ling-3.0-flash-trial]]
- [[deployment]]
serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md
+4 -4
View File
@@ -7,7 +7,7 @@ and the test matrix planned for 2026-08-16 morning.
Related pages:
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
- [Backend and spec benchmark](frank-default-large-benchmark.md)
- [Backend and spec benchmark](frank-large-benchmark.md)
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
@@ -58,7 +58,7 @@ Consequences of the hybrid design:
## Current production state (baseline)
`frank-default-large` (Dankclaw, Vulkan0, 2026-08-15):
`frank-large` (Dankclaw, Vulkan0, 2026-08-15):
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
@@ -225,7 +225,7 @@ tensor cache warm).
| E | A with n_max 3 and 5 | n_max curve on the split |
| F | A, plain Q8_0 model file | same-bit A/B |
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
| Ref | production frank-default-large (Vulkan, 2 slots) | the bar to beat |
| Ref | production frank-large (Vulkan, 2 slots) | the bar to beat |
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
@@ -245,5 +245,5 @@ production @16k) with the same 8-bit model.
`-dio` for RPC).
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
- Local benchmarks: [frank-default-large-benchmark](frank-default-large-benchmark.md),
- Local benchmarks: [frank-large-benchmark](frank-large-benchmark.md),
[qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
serverAppConfigs-docs/LLM/ubatch-benchmark-large.md
+5 -5
View File
@@ -1,6 +1,6 @@
---
type: benchmark plan
title: frank-default-large ubatch sweep
title: frank-large ubatch sweep
description: Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment.
tags:
- llama.cpp
@@ -10,9 +10,9 @@ tags:
timestamp: 2026-09-01
---
# frank-default-large ubatch sweep
# frank-large ubatch sweep
This benchmark uses the active `frank-default-large` model on Dankclaw with
This benchmark uses the active `frank-large` model on Dankclaw with
`llama-bench`. It does not use `llama-server`, HTTP requests, or server-log
scraping. Stop the production service before running it and start the service
again afterward.
@@ -29,9 +29,9 @@ Each point runs three 128-token repetitions by default.
Run on Dankclaw from the repository checkout:
```bash
sudo systemctl stop frank-default-large
sudo systemctl stop frank-large
./scripts/bench-ubatch.sh
sudo systemctl start frank-default-large
sudo systemctl start frank-large
```
The script uses the active Q6 AGENT model, an 8K prompt workload, equal