llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
Frank
2026-10-10 17:03:14 -07:00
parent 5ced474eb9
commit 55f87fcdce
195 changed files with 2085 additions and 2064 deletions
@@ -115,7 +115,7 @@ unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
- Internal order is fixed (ngram types first, then drafters); CLI order is
irrelevant.
- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
on the qwen35 hybrid (frank-dragoon era) measured −15 % prose / −7 %
block-copy (2026-10-03): adding the ngram layer logs `recurrent state
rollback is not compatible with 'ngram-mod' - disabling rollback support`,
and MTP verification depends on that rollback, so acceptance drops on ALL
@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090)
The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
The Dankserver llama.cpp deployment (`frank-zealot`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-zealot` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
@@ -250,7 +250,7 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
`frank-large` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
@@ -315,7 +315,7 @@ not switch without re-measuring".
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
slot split, quant rung — not just spec flags) is the transient systemd
drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
## Is it worth it? (arithmetic before deployment)
@@ -419,4 +419,4 @@ measurement that would settle it.
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
@@ -22,7 +22,7 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
hold on the 8060S. See [[frank-large-benchmark]].
hold on the 8060S. See [[frank-dragoon-benchmark]].
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
@@ -48,7 +48,7 @@ lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
different, driver-reported figure that has always been larger than the kernel
cap.
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-large-benchmark]].
limits documented in [[frank-dragoon-benchmark]].
## 1. Self-Hosted Agentic Tools
@@ -106,7 +106,7 @@ All require root and a reboot for the modprobe change.
## Sources
- [[frank-large-benchmark]] — measured backend/spec numbers
- [[frank-dragoon-benchmark]] — measured backend/spec numbers
- [[deployment]] — deploy and update process
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
@@ -115,7 +115,7 @@ All require root and a reboot for the modprobe change.
## See Also
- [[frank-large-benchmark]]
- [[frank-dragoon-benchmark]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
- [[index]]
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-zealot-gemma4-12b]]).
### ROCm
@@ -101,7 +101,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]].
- The active service is `frank-dragoon` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-dragoon-gufo-qwen38-27b]].
- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 1.1 TB free disk space is ample for model storage.
@@ -17,7 +17,7 @@ timestamp: 2026-09-20
Dankserver is a media/library server and the `frank-medium` LLM host. The
host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and replaced with
two AMD Radeon AI PRO R9700 cards.
## Hardware Specifications
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
The active inference service is `frank-medium.service`: mainline llama-server on
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
its unit is archived in `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`,
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
See [[deployment]].
@@ -13,12 +13,12 @@ timestamp: 2026-09-03
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-secondary` name and port 52625 then belonged to the vLLM
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-zealot` (port 52625) serves in production. See Promotion.
## Verdict (2026-08-16)
@@ -167,7 +167,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
@@ -185,7 +185,7 @@ Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens)
## Model Bakeoff (measured 2026-08-16, 4 models)
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
@@ -203,21 +203,21 @@ Notes:
## Promotion (2026-08-16)
`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-secondary
sudo systemctl enable frank-zealot
```
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
The PP split was renamed `frank-pp-head` → `frank-large` (2026-08-16, commit `48142f5`).
The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
@@ -353,7 +353,7 @@ installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
@@ -390,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
- **NPU production role (2026-08-16).** `frank-zealot` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
## Citations
@@ -23,7 +23,7 @@ read the returned content:
```python
import json, urllib.request
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req, timeout=600)))
```
@@ -119,5 +119,5 @@ re-download.
- [[Llama Cpp Runtimes]] — which fork ships which custom types
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
@@ -13,7 +13,7 @@ Fleet law first: **STOP the production service before any GPU experiment**
GPU and corrupts results; an unisolated GPU experiment has precedent for
wedging a host until power cycle. Reference harnesses:
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
`scripts/autoresearch/frank-primary-arms.sh`.
`scripts/autoresearch/frank-archon-arms.sh`.
## 1. The window structure (one config per arm, a driver owns the window)
@@ -27,7 +27,7 @@ wedging a host until power cycle. Reference harnesses:
with `container state improper`.
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
LLM hosts — one card, one server). The serve script takes its config via
env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
pointing at a staged copy; the checkout is never mutated.
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
the loop — a refused boot dies in one poll instead of 20 min. Capture the
@@ -245,17 +245,17 @@ nearly shipped the wrong setting.
the comment); verify multi-line command construction with `bash -n` AND a
render test.
## 8. Results location and recovery (frank-primary arms)
## 8. Results location and recovery (frank-archon arms)
| copy | path |
|---|---|
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
constants 99.9/88.8 identical across arms by construction; **identical
numbers across arms == stub or bug, never a measurement.** After compaction,
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
@@ -161,8 +161,8 @@ of the same RAM, not extra memory.
## 7. IBS model-access attribution (AMD CPUs)
Real CPU access patterns for mmap'd weights, live, no stop (established on
frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
frank-archon, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
`scripts/frank-archon-expert-activity.py` (self-contained dump + sample +
analyze; `--analyze-only`).
1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
@@ -80,7 +80,7 @@ co-tenants.
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
"move layers to the card" converts reclaimable cache into device memory and
frees nothing for co-tenants. A served model's footprint ≠ weight size
(frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
(frank-zealot: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
- **Context lengths: derive, never remember.** `llama-bench` computes
@@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed
## Related
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
@@ -8,8 +8,8 @@ timestamp: 2026-10-05T00:00:00-07:00
# Model Swap and Rollback
Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
frank-primary). Repo law: all config flows through git; the live models are
Applies to every endpoint swap on the fleet (frank-zealot / frank-dragoon /
frank-archon). Repo law: all config flows through git; the live models are
host state under `/home/dank/models/<name>/`.
## 1. Fit preflight — never deploy a swap without confirming it FITS
@@ -26,7 +26,7 @@ conclusions.
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-large.env` |
| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
@@ -20,7 +20,7 @@ morning-of runbook. Research background:
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
This split is the production `frank-large` service. Current
This split is the production `frank-dragoon` service. Current
state, verified against the live journal and output smoke tests
(2026-08-17, load 02:06):
@@ -41,17 +41,17 @@ state, verified against the live journal and output smoke tests
Vulkan).
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
stable; all allocations are upfront).
- Unit `frank-large.service`, launched through
`serve-pp-head.sh` + `frank-large.env`.
- Unit `frank-dragoon.service`, launched through
`serve-pp-head.sh` + `frank-dragoon.env`.
## Service enablement (2026-08-17)
The PP pair is the only inference service enabled for boot. Both units
(`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-secondary` on Dankclaw) and all retired
(`frank-dragoon` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-zealot` on Dankclaw) and all retired
units are disabled. Restart order: worker first, then head. The Dankserver
helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-secondary` unit); restart the worker with
`frank-zealot` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. `deploy.sh dankserver` is broken until retargeted. See
@@ -173,9 +173,9 @@ llama-server b10433 ggml-rpc-server b10433
|------|---------|---------|
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
| `scripts/frank-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/frank-dragoon.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
| `scripts/frank-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/frank-dragoon.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
@@ -229,7 +229,7 @@ FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
```
Config variants: edit `scripts/frank-large.env` on the claw (git commit +
Config variants: edit `scripts/frank-dragoon.env` on the claw (git commit +
pull if the bench runs from the repo), then re-run `start` + `bench`:
- No spec baseline: `SPEC_NONE=1`
@@ -65,6 +65,6 @@ graph LR
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[frank-secondary-benchmark]]
- [[frank-zealot-benchmark]]
- [[Dankserver System Specs]]
- [[Dankclaw System Specs]]
@@ -78,12 +78,12 @@ Laws that steer everything here:
variants **[verified (tool existence: heretic; rest survey)]**.
- Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm
weights, so it composes with any expert quantization.
- Fleet status: frank-large runs an abliterated base **by Frank's choice**
- Fleet status: frank-dragoon runs an abliterated base **by Frank's choice**
(AGENTS: censored-base refusals were the accepted trade of the 27B era);
frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.
### HauhauCS QX_K_P "Perfect" quants
- Our frank-secondary publisher. Method, from the publisher's own cards
- Our frank-zealot publisher. Method, from the publisher's own cards
**[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced"),
(2) quantize with **imatrix calibration on the ABLITERATED weights**,
(3) K_P = a **per-model, per-tensor mixed-precision profile** —
@@ -106,7 +106,7 @@ Laws that steer everything here:
## Per-card playbooks
### RTX 3090 — the unlocked card (frank-secondary runs llama.cpp BY CHOICE)
### RTX 3090 — the unlocked card (frank-zealot runs llama.cpp BY CHOICE)
The 3090 is the only fleet device with no format lock: radiance needs
`.rad`, halogen needs `.hgn`, and the v2 checkpoint loads in NO other
engine. sm_86 + CUDA is the industry's default target, so the runtime
@@ -179,7 +179,7 @@ Q8_K_P 40.6 GiB + mmproj 0.84) — Q4_K_P IS the protected-tensor blend.
The self-made delta: calibration on OUR corpora, our hot-expert law,
and a KLD gate scored at the serving shape.
### R9700 — frank-primary (radiance)
### R9700 — frank-archon (radiance)
The full self-quant pipeline already exists end to end: fetch BF16 →
record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap,
`RADIANCE_CALIB_DIR/TOKENS`) → derive protected experts (energy-share) →
@@ -219,7 +219,7 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
until proven), and does its MTP cast to the fp8/u2 encodings like
qwen4exp. The MNBT-2048 disk-tier law is MOOT for a resident model.
### Strix Halo — frank-large (halogen) and friends
### Strix Halo — frank-dragoon (halogen) and friends
- `.hgn`: **closed production door.** No public quantizer exists
(verified absence across peonist-ai repos); the format's lock is the
engine's lock. We consume vendor artifacts; the repo-side tooling is
@@ -255,6 +255,6 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
`scripts/build-kairic-edge-gfx1151.sh`, [[agentionai-fast-imatrix]],
[[kingjones-strix-rocmfp4]], [[qwen3.8-strix-optimizations]],
[[frank-primary-radiance-swift15-w4dn8]],
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]],
[[frank-large-flash-server-v2-swift15]], [[GGUF Compatibility]].
[[frank-archon-radiance-swift15-w4dn8]],
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]],
[[frank-dragoon-flash-server-v2-swift15]], [[GGUF Compatibility]].
@@ -247,4 +247,4 @@ see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
measured campaigns
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — window discipline on live hosts
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — window discipline on live hosts
@@ -8,7 +8,7 @@ timestamp: 2026-10-05T00:00:00-07:00
# RDNA4 MoE Expert Offload VRAM Budget
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-primary: Qwen3.8-
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-archon: Qwen3.8-
Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
Tuning history and the deployed shape live in
[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
@@ -198,7 +198,7 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
`17.`):
```
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
```
@@ -206,7 +206,7 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
3. Verify after deploy: plan lines show new budget/graphs/arena;
`GPU KV cache size` matches the ask; graph-estimate vs post-capture;
`rocm-smi` high-water per card = the real margin;
`systemctl show frank-primary -p NRestarts --value` = 0; a real chat
`systemctl show frank-archon -p NRestarts --value` = 0; a real chat
completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
`localhost` inside an ssh session on the host — the latter returns `000`
and looks like an outage).
@@ -55,7 +55,7 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB
## Prerequisites
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-dragoon.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
@@ -122,7 +122,7 @@ export HF_HOME=/mnt/entropy/huggingface
### Active Service Conflict
The retired `frank-secondary` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
The retired `frank-zealot` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
```
nvidia-smi # verify GPU is free
@@ -10,9 +10,9 @@ timestamp: 2026-10-05T00:00:00-07:00
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
first-class expert-offload planner) served `frank-primary` at shape G until
first-class expert-offload planner) served `frank-archon` at shape G until
the 2026-10-05 FreeToken cutover; it remains the rollback target
(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
[[LLM Bench Windows and Measurement]].
@@ -213,7 +213,7 @@ through two image tags to bring an overlay up was entirely wasted work.
## Compile-cache persistence (survives `podman rm -f`)
Serve scripts that recreate the container discard the writable layer, so
every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
cold, of which 173 s was one graph).
1. **Prove it and locate it:** fresh container vs live
@@ -241,7 +241,7 @@ cold, of which 173 s was one graph).
## Plan-line verification checklist (after any fork deploy)
```
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
```
@@ -250,7 +250,7 @@ these columns to `17.`. Verify: new budget/graphs/arena in the plan;
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
real margin; `systemctl show frank-archon -p NRestarts --value` = 0;
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
completion from a PEER — `curl localhost:7770/health` inside an ssh session
on the host can return `000` and looks like an outage that isn't.
+3 -3
View File
@@ -244,7 +244,7 @@ vLLM supports multiple parallelism strategies:
already uses them.** The tcclaviger fork is a vLLM fork, so the standard
scheduler flags are present in the shipped image (verified by reading the
`vllm.engine.arg_utils` and `vllm.config.scheduler` source in the live
container `frank-primary-qwen`, not from a stub):
container `frank-archon-qwen`, not from a stub):
| Flag | Default | What it controls |
|---|---|---|
@@ -268,14 +268,14 @@ ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-secondary
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-zealot
was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
counterpart here, and the two backends should not be reasoned about by analogy.
**MNBT is not a lever on this shape either.** The deployed 2048 is the fork's
default, so a change is a deliberate departure from the shipped value, not a
tune. The one thing MNBT does cost: every change re-enters the compile RANGE
path, so `FRANK_PRIMARY_CACHE_DIR` pays one recompilation.
path, so `FRANK_ARCHON_CACHE_DIR` pays one recompilation.
## See Also
@@ -1,7 +1,7 @@
---
type: reference
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-large.
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-dragoon.
tags:
- llama.cpp
- rocm
@@ -26,7 +26,7 @@ timestamp: 2026-08-29
> `--flash-attn on` work on the charlie build-qwen38 tree — all still
> true for that binary + model pair, which remains the rollback target.
`frank-large` (port 7770) served
`frank-dragoon` (port 7770) served
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
with the author's own MTP draft head
@@ -166,8 +166,8 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Config files
- Env: `scripts/frank-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Env: `scripts/frank-dragoon.env`
- Serve: `scripts/serve-frank-dragoon.sh`
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
any clean fork rebuild — see Serving profile)
@@ -175,10 +175,10 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Swap procedure
```bash
sudo systemctl stop frank-large
# edit scripts/frank-large.env + scripts/serve-frank-large.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-large
sudo systemctl start frank-large # or via deploy.sh
sudo systemctl stop frank-dragoon
# edit scripts/frank-dragoon.env + scripts/serve-frank-dragoon.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-dragoon
sudo systemctl start frank-dragoon # or via deploy.sh
curl http://192.168.0.143:7770/health
```
@@ -1,7 +1,7 @@
---
type: reference
title: CIRU ActiveFPX PromptForge on Dankclaw
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-large service.
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-dragoon service.
tags:
- llama.cpp
- rocm
@@ -14,13 +14,13 @@ timestamp: 2026-08-22
# CIRU ActiveFPX PromptForge on Dankclaw
> **RETIRED 2026-08-24.** `frank-large` served this v3 model from
> **RETIRED 2026-08-24.** `frank-dragoon` served this v3 model from
> 2026-08-22 (restored after a same-day Kairic Edge IU4 experiment was
> reverted — operator decision, commit `c60156f`; see [[kairic-edge]])
> until [[eemin-rocmi4-mtp]] took over on 2026-08-24. Kept as the
> historical record of the v3 deployment.
`frank-large` (port 7770, moved from 52625 on 2026-08-22) serves
`frank-dragoon` (port 7770, moved from 52625 on 2026-08-22) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
@@ -136,7 +136,7 @@ graph TD
C --> D[Build llama-server]
D --> E[Verify --list-devices shows gfx1151]
E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
F --> G[Deploy frank-large]
F --> G[Deploy frank-dragoon]
G --> H[Health check + chat completion]
```
@@ -151,8 +151,8 @@ against the model card before the service restart.
## Service
- Unit: `frank-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
- Unit: `frank-dragoon.service`, port 7770
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
the system ROCm. `ROCM_PATH` points at the devel SDK.
@@ -1,14 +1,14 @@
---
type: research
title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-large), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-dragoon), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
tags:
- dankclaw
- iommu
- kernel
- grub
- prefill
- frank-large
- frank-dragoon
- halogen
- frank-npu
timestamp: 2026-09-11
@@ -54,7 +54,7 @@ takes the NPU with it.
1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and
`/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure
documented in [[frank-npu]]. Audio input left the fleet with it; image input
survives on `frank-large` through halogen's vision tower.
survives on `frank-dragoon` through halogen's vision tower.
2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge
and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the
observability stack. DMA translation is now off for all of them. This was
@@ -80,7 +80,7 @@ so only that one was taken:
cat /proc/cmdline # expect amd_iommu=off, no iommu=pt
ls /sys/kernel/iommu_groups/ | wc -l # expect 0 groups (was 30)
systemctl is-active frank-npu # expect inactive (masked)
curl -s http://127.0.0.1:7770/health # frank-large must still answer 200
curl -s http://127.0.0.1:7770/health # frank-dragoon must still answer 200
```
**To revert:** put `iommu=pt` back in `GRUB_CMDLINE_LINUX_DEFAULT` in place of
@@ -168,7 +168,7 @@ Not advisable.
| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
| **radiance-vLLM** (drwolfen) | yes — served frank-primary 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
| **radiance-vLLM** (drwolfen) | yes — served frank-archon 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
@@ -203,7 +203,7 @@ reproducible **from a bf16 source**, not from our file. Uncensored-family
bf16 source candidates: `windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF`
(ships the bf16 GGUF) and `mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF`
(i1 quants only). 622 G free fits source + container. A trial is:
download → `rad-convert` (CPU-side) → stop+mask frank-primary window to
download → `rad-convert` (CPU-side) → stop+mask frank-archon window to
serve. Caveats on record: `--tp-wire wht6` is a lossy int8 all-reduce
wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
an opt-in f16 that changes generated text; every pool is budgeted
@@ -221,7 +221,7 @@ gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
## See Also
- [[frank-primary-lucebox-ds4]] — the retired Lucebox frank-primary runbook
- [[frank-archon-lucebox-ds4]] — the retired Lucebox frank-archon runbook
- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
- [[frank-archon-r9v-radiance-qwen38-q8]] — the ACTIVE frank-archon
@@ -30,7 +30,7 @@ timestamp: 2026-09-29
This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
image `docker.io/tcclaviger/vllm:latest`, engine
`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-primary` systemd service —
`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-archon` systemd service —
the R9V production unit is untouched, so no stop-and-mask was required for
any of this. All numbers are decode tok/s measured from the server's own
`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
@@ -58,7 +58,7 @@ Published to the host LAN via the serve script's `PORTMAP` hook
(`PORTMAP=7773:7770`): health and completions answer on
`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
decode 36.49/34.55/35.98 through the published path). Host port 7773 was
chosen over 7770 because `frank-primary` (7770) is stopped but ENABLED —
chosen over 7770 because `frank-archon` (7770) is stopped but ENABLED —
any future start would collide with a published 7770.
Device notes carried from the bring-up: `--device /dev/dri` must be the
@@ -249,8 +249,8 @@ re-copies them; a recreate wipes container-local files).
Two same-day cutovers moved this page's findings into production:
1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
`frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
window is documented in [[frank-archon-glm53-flash-w4a16]]; git tag
`frank-archon-fp8-mtp-20261001` is the last commit of that shape). The
production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
measured 37.4-40.6 tok/s at ~54% acceptance.
2. **The overnight redo campaign re-measured EVERY lever at matched
@@ -278,20 +278,20 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
FRANK_ARCHON_MAXLEN=200704 200704-token window
FRANK_ARCHON_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_ARCHON_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_ARCHON_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_ARCHON_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_ARCHON_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
FRANK_ARCHON_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_ARCHON_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit)
FRANK_ARCHON_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_ARCHON_CACHE_DIR=/home/dank/.cache/frank-archon mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
FRANK_PRIMARY_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
FRANK_ARCHON_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
```
### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
@@ -299,7 +299,7 @@ FRANK_PRIMARY_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the Q
**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
~40k tokens into a long conversation; short sessions stayed clean.
**Evidence (live `podman logs frank-primary-qwen`, the 2026-10-04 18:19
**Evidence (live `podman logs frank-archon-qwen`, the 2026-10-04 18:19
boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
running request, and the engine's own 18:46:08 log line reading
"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
@@ -343,7 +343,7 @@ load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
overclock); and a two-head, single-slot NaN state is a state-path fault,
not scattered finite bit corruption.
**Fix.** `FRANK_PRIMARY_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
**Fix.** `FRANK_ARCHON_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
(the fork-named mitigation; the flag renders in both directions from the
knob so a fork default change cannot silently flip the contract). Cost:
no prefix reuse — every turn re-prefills the whole history instead of at
@@ -388,7 +388,7 @@ conversation cannot distinguish fixed from unfixed; (2) a second
long-context mechanism with no guard coverage — fp8 KV degradation at
depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
and would settle it). Prefix caching is **back ON**
(`FRANK_PRIMARY_NO_PREFIX_CACHING` commented out — the cost bought no
(`FRANK_ARCHON_NO_PREFIX_CACHING` commented out — the cost bought no
fix). The knob and the serve branch stay for a future fork build with a
fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
it still breaks at ~40k with zero faults, A/B the KV dtype.
@@ -472,7 +472,7 @@ running** — the bound tracks whatever the co-tenants hold at that moment. The
demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
boot where the bound drops near it silently converts next-hottest experts to
VRAM-only (the `pinned_extra` warning is the only trace).
`FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
`FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
on a host with 246.1 GiB total. Semantics verified in the shipped image
(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
@@ -484,7 +484,7 @@ was itself fitted at nbt 4096, which means 2048 is covered by an activation
figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
SEQS property). The costs are a one-time recompile (the compile **range** is
part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
part of the cache key, so `FRANK_ARCHON_CACHE_DIR` pays once) and a wider
prefill chunk, which the 09-29 sweep found decode-interchangeable across
1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
@@ -656,7 +656,7 @@ read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
only available while the container lives).
Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
`frank-archon-fp8-mtp-20261001` (env trio + seam re-pinned there).
## Parallelism options on this box (2026-10-01 survey)
@@ -696,9 +696,9 @@ more cards.
run in the engine's namespace and the files must be copied in first:
```
podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
podman cp /home/dank/bench_iter.py frank-primary-qwen:/
podman exec frank-primary-qwen python3 /mtp_measure.py
podman cp /home/dank/mtp_measure.py frank-archon-qwen:/
podman cp /home/dank/bench_iter.py frank-archon-qwen:/
podman exec frank-archon-qwen python3 /mtp_measure.py
```
Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
@@ -717,7 +717,7 @@ reproducible across boots.
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
V4.1 scale)
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
fork era; production has been the tcclaviger fork since)
- [[x4-pair-experiment]] — the same residency law measured on a narrower
link: streaming over PCIe loses, computing in RAM does not care
@@ -1,7 +1,7 @@
---
type: reference
title: Dankserver VRAM and RAM Growth Under Load
description: Investigation of the recurring cuBLAS crash on frank-secondary. What grows, what does not, and what to do next.
description: Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next.
tags:
- dankserver
- llamacpp
@@ -18,7 +18,7 @@ timestamp: 2026-09-03
# Dankserver VRAM and RAM Growth Under Load
This page records the investigation of the recurring cuBLAS crash on
`frank-secondary` (llama.cpp, RTX 3090 24 GiB). The service aborts
`frank-zealot` (llama.cpp, RTX 3090 24 GiB). The service aborts
with `CUDA error: an unsupported value or parameter was passed to the
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
## Mitigation Options
The documentation for the benchmark ([[frank-secondary-benchmark]])
The documentation for the benchmark ([[frank-zealot-benchmark]])
prescribes the first option for this crash class. None of these change
the model.
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
## See Also
- [[frank-secondary-benchmark]]
- [[frank-zealot-benchmark]]
- [[Dankserver System Specs]]
- [[deployment]]
@@ -1,7 +1,7 @@
---
type: reference
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-secondary.
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-zealot.
tags:
- llamacpp
- cuda
@@ -14,12 +14,12 @@ timestamp: 2026-08-24
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
> **Active 2026-08-25.** `frank-secondary` serves this model. It
> **Active 2026-08-25.** `frank-zealot` serves this model. It
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
> The Q6_K file was downloaded from the model repository and validated with
> `llama-gguf` before installation.
`frank-secondary` (port 7770) serves
`frank-zealot` (port 7770) serves
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
attention architecture, abliterated (uncensored). The GGUF also contains a
+16 -16
View File
@@ -20,8 +20,8 @@ SearXNG, Caddy, or another container service does **not** restart inference.
| Target | Host | Service | Port |
|---|---|---|---:|
| Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-secondary` | 7771 |
| Dankclaw | `dank@dankclaw` | `frank-dragoon` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-zealot` | 7771 |
## Scoped workflow
@@ -40,7 +40,7 @@ Run the unified deployer from the repository root:
```bash
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankclaw --service frank-large
./scripts/deploy.sh dankclaw --service frank-dragoon
./scripts/deploy.sh dankclaw --dry-run
```
@@ -51,23 +51,23 @@ on the selected host. The following files select the corresponding service:
entrypoint and its healthcheck) selects `frank-medium` on Dankserver —
the container mounts the entrypoint and healthcheck straight out of the
checkout, so any file there changes deployed state.
- `scripts/frank-large.env`, `frank-large.service`, and
`serve-frank-large.sh` select `frank-large` on Dankclaw.
- `scripts/frank-secondary.env`, `frank-secondary.service`, and
`serve-frank-secondary.sh` select `frank-secondary` on Dankclaw.
- `scripts/frank-dragoon.env`, `frank-dragoon.service`, and
`serve-frank-dragoon.sh` select `frank-dragoon` on Dankclaw.
- `scripts/frank-zealot.env`, `frank-zealot.service`, and
`serve-frank-zealot.sh` select `frank-zealot` on Dankclaw.
- Documentation, tests, deployment tooling, and unrelated container changes
do not select any inference service.
The target-side `scripts/deploy-remote.sh` installs the selected service.
`frank-large` and `frank-secondary` are **root units**: the handler relinks the
`frank-dragoon` and `frank-zealot` are **root units**: the handler relinks the
repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
handler retires its predecessors (the replica Quadlets and the retired llama
root unit), stops the service's own unit, waits for the published port to
clear, installs the `.container` and restarts the generated unit. Each
handler then waits on that service's health endpoint —
`http://127.0.0.1:7770/health` for frank-large and frank-medium,
`http://127.0.0.1:7771/health` for frank-secondary. The wait budget is 20 minutes
`http://127.0.0.1:7770/health` for frank-dragoon and frank-medium,
`http://127.0.0.1:7771/health` for frank-zealot. The wait budget is 20 minutes
because cold model loads are slow. A failed handler does not advance its
per-service state.
@@ -75,8 +75,8 @@ per-service state.
The root units point into the stable target checkout:
```text
/home/dank/serverAppConfigs/scripts/frank-large.service
/home/dank/serverAppConfigs/scripts/frank-secondary.service
/home/dank/serverAppConfigs/scripts/frank-dragoon.service
/home/dank/serverAppConfigs/scripts/frank-zealot.service
```
`frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
(`frank-medium.container`) rather than a unit in `scripts/`.
@@ -110,7 +110,7 @@ change in that service's inputs.
If the LLM handler fails after the target pull:
1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-secondary
1. Inspect `journalctl -u frank-dragoon -n 200` or `journalctl -u frank-zealot
-n 200` (root units), or `journalctl --user -u frank-medium -n 200`
(rootless Quadlet).
2. Confirm the model files, the runtime binary and (for frank-medium) the
@@ -119,14 +119,14 @@ If the LLM handler fails after the target pull:
```bash
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankclaw --service frank-large
./scripts/deploy.sh dankclaw --service frank-secondary
./scripts/deploy.sh dankclaw --service frank-dragoon
./scripts/deploy.sh dankclaw --service frank-zealot
```
4. Verify the matching health endpoint before declaring recovery.
Do not stop the other host's inference service during recovery. The two
Dankclaw endpoints are independent processes on different devices: restarting
frank-large does not disturb frank-secondary, and the reverse holds too — with one
frank-dragoon does not disturb frank-zealot, and the reverse holds too — with one
exception worth knowing, that the deployer waits for port 7770 to be released
before it starts the replacement container. For benchmark work, follow the
separate stop/benchmark/restart procedure in [[Testing Policy]] and the
@@ -1,6 +1,6 @@
---
type: reference
title: DFlash2 on Dankclaw (frank-large)
title: DFlash2 on Dankclaw (frank-dragoon)
description: ARCHIVED - unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Retired 2026-08-21 for speed; kept as a runbook.
tags:
- llama.cpp
@@ -13,14 +13,14 @@ tags:
timestamp: 2026-08-21
---
# DFlash2 on Dankclaw (frank-large) — ARCHIVED
# DFlash2 on Dankclaw (frank-dragoon) — ARCHIVED
> **Archived 2026-08-21.** This DFlash2 Vulkan deployment was retired the
> same day it shipped because it was slower than the CIRU runtime it
> replaced. `frank-large` is back on CIRU. This page stays as the
> replaced. `frank-dragoon` is back on CIRU. This page stays as the
> runbook and provenance for the DFlash2 attempt.
`frank-large` (port 52625) briefly served
`frank-dragoon` (port 52625) briefly served
`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
runtime was upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
@@ -114,14 +114,14 @@ graph TD
C --> D[Build llama-server]
D --> E[Verify Vulkan0 device]
E --> F[Add DFlash2 drafter]
F --> G[Swap frank-large]
F --> G[Swap frank-dragoon]
G --> H[Health check + completion]
```
## Service
- Unit: `frank-large.service`, port 52625
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
- Unit: `frank-dragoon.service`, port 52625
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
### Launch profile
+11 -11
View File
@@ -1,7 +1,7 @@
---
type: reference
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-large.
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-dragoon.
tags:
- llama.cpp
- rocm
@@ -15,14 +15,14 @@ timestamp: 2026-08-25
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
> **RETIRED 2026-08-25.** `frank-large` served this model from 2026-08-18
> **RETIRED 2026-08-25.** `frank-dragoon` served this model from 2026-08-18
> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
> on the same port and the same Strix Halo iGPU, running the new
> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
> the page it replaced), and was itself replaced by
> [[kingjones-strix-rocmfp4]].
`frank-large` (port 7770) serves
`frank-dragoon` (port 7770) serves
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
@@ -98,7 +98,7 @@ graph TD
D --> E[Sharp chat template<br/>reasoning deepseek]
```
Key flags (see `scripts/serve-frank-large.sh`):
Key flags (see `scripts/serve-frank-dragoon.sh`):
```
--spec-type draft-mtp --spec-mtp-strict-qwen
@@ -129,8 +129,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
### Service
- Unit: `frank-large.service`, port 7770
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
- Unit: `frank-dragoon.service`, port 7770
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
## Tuning MTP
@@ -144,14 +144,14 @@ Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
```
sudo systemctl stop frank-large
sudo systemctl stop frank-dragoon
./scripts/bench-ubatch.sh
sudo systemctl start frank-large
sudo systemctl start frank-dragoon
```
The script stops `frank-large` for a clean GPU, boots one server
The script stops `frank-dragoon` for a clean GPU, boots one server
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
and restarts the service. See [[frank-large-benchmark]].
and restarts the service. See [[frank-dragoon-benchmark]].
## Known Issues
@@ -185,4 +185,4 @@ The ubatch sweep above gives the production ubatch choice.
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
- [[Dankclaw System Specs]] — the box
- [[frank-large-benchmark]] — backend/spec benchmark history
- [[frank-dragoon-benchmark]] — backend/spec benchmark history
@@ -1,7 +1,7 @@
# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
# frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
cutover — see [[frank-primary-radiance-qwen38-native-fp8]]; kept as the
cutover — see [[frank-archon-radiance-qwen38-native-fp8]]; kept as the
rollback runbook: the image + orcarouter payload stay on disk until the
window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
@@ -9,11 +9,11 @@ window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
on the host into the local image
`localhost/frank-primary-freetoken:pr587-77a7201` by
`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
alias (`frank-primary`), same 131-shard payload
`localhost/frank-archon-freetoken:pr587-77a7201` by
`scripts/docker/frank-archon-freetoken.Dockerfile`. Same port, same served
alias (`frank-archon`), same 131-shard payload
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
git tag `frank-archon-vllm-shapeG-20261003` (the vLLM trio + its env lore),
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
## Why FreeToken
@@ -31,8 +31,8 @@ deterministic pass — no per-boot estimator to fight.
```bash
cd /home/dank/serverAppConfigs
podman build -t frank-primary-freetoken:pr587-77a7201 \
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
podman build -t frank-archon-freetoken:pr587-77a7201 \
-f scripts/docker/frank-archon-freetoken.Dockerfile scripts/docker/
```
Laws baked into the Dockerfile (each one cost a failure):
@@ -129,7 +129,7 @@ native 262,144-token conversation + full budget. `models.yml`
PLE ngram table in host RAM → the unit carries
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
container's ~170 GiB MUST be down first; `systemctl stop frank-archon`
frees it (measured: 207 GB free right after the stop).
## Resize live
@@ -156,11 +156,11 @@ ft ctl stats # pool occupancy, VRAM, throughput
## Rollback
```bash
git checkout frank-primary-vllm-shapeG-20261003 -- \
scripts/frank-primary.env scripts/serve-frank-primary.sh \
scripts/frank-primary.service
./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
&& git push && ./scripts/deploy.sh dankserver --service frank-primary
git checkout frank-archon-vllm-shapeG-20261003 -- \
scripts/frank-archon.env scripts/serve-frank-archon.sh \
scripts/frank-archon.service
./tests/run.sh && git commit -m "frank-archon: rollback to vLLM shape G" \
&& git push && ./scripts/deploy.sh dankserver --service frank-archon
```
The tcclaviger image and the FreeToken image coexist in dank's rootless
@@ -1,7 +1,7 @@
---
type: runbook
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
title: frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-archon cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
tags:
- dankserver
- r9700
@@ -16,7 +16,7 @@ tags:
timestamp: 2026-09-30
---
# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
# frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
@@ -118,7 +118,7 @@ register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
spec sets through the patched grouping/layout/builder functions. A 7-minute
boot per hypothesis is the alternative.
**Window discipline:** stop + mask `frank-primary` FIRST and gate on
**Window discipline:** stop + mask `frank-archon` FIRST and gate on
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
@@ -152,7 +152,7 @@ by this page's kernel-time measurement),
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
| unit | `scripts/frank-archon.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
```mermaid
@@ -698,14 +698,14 @@ newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankserver --service frank-archon
# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only
bash scripts/download-frank-archon.sh --verify-only
# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
podman logs frank-archon-glm 2>&1 | grep "expert_lru.py"
```
**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
@@ -719,11 +719,11 @@ mask symlink **and** the container is owned by the unit and not `Stopping`. The
repair recipe, all four steps needed:
```bash
sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
sudo mv /run/systemd/system/frank-archon.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
sudo systemctl daemon-reload
sudo systemctl reset-failed frank-primary.service # a failed unit refuses to start
podman rm -f frank-primary-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
sudo systemctl start frank-primary.service # then ~7 min to health 200
sudo systemctl reset-failed frank-archon.service # a failed unit refuses to start
podman rm -f frank-archon-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
sudo systemctl start frank-archon.service # then ~7 min to health 200
```
Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
@@ -795,4 +795,4 @@ toggling on the mini itself).
| state | where |
| --- | --- |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-archon-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
@@ -1,9 +1,9 @@
---
type: configuration
title: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
description: ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
title: frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
description: ACTIVE frank-archon runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
tags:
- frank-primary
- frank-archon
- dankserver
- llamacpp
- unsloth
@@ -19,15 +19,15 @@ tags:
timestamp: 2026-09-26
---
# frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
# frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**):
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-archon**):
unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the
Lucebox DS4.1 quadlet that held the port for hours earlier the same day
([[frank-primary-lucebox-ds4]]), which itself replaced
([[frank-archon-lucebox-ds4]]), which itself replaced
[[frank-medium-radiance-mxfp4]].
## Why this shape
@@ -89,20 +89,20 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|Piece|Value|
|---|---|
|Unit|`scripts/frank-primary.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|Serve script|`scripts/serve-frank-primary.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|Env tunables|`scripts/frank-primary.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|Unit|`scripts/frank-archon.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|Serve script|`scripts/serve-frank-archon.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|Env tunables|`scripts/frank-archon.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`|
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-primary.sh` (sha256 + byte-count verified)|
|Port / alias|7770 / `frank-primary` — unchanged from the lucebox and radiance eras; no client moves|
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-archon.sh` (sha256 + byte-count verified)|
|Port / alias|7770 / `frank-archon` — unchanged from the lucebox and radiance eras; no client moves|
|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)|
|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request|
|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)|
|Observability|`frank-primary` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
|Observability|`frank-archon` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
### Cutover mechanics
The deployer (`deploy_llm_primary` in `scripts/deploy-remote.sh`) gates on
The deployer (`deploy_llm_archon` in `scripts/deploy-remote.sh`) gates on
every artifact at its declared byte count (`stat -c %s`, never `wc -c` —
which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash
@@ -138,7 +138,7 @@ between slots — it does not multiply.
## Verification (2026-09-26 deploy)
- `/v1/models` reports `frank-primary`.
- `/v1/models` reports `frank-archon`.
- Chat completion returns `content` + `reasoning_content`.
- Tool call returns well-formed `tool_calls` (`--jinja`).
- Real image request answered from the projector (not a health check —
@@ -218,7 +218,7 @@ vision and the target on the same battery class.
**Which blocks, measured (same day):** asked whether the `-ot` regex
could target the *most active* blocks, I instrumented the live server
instead of guessing — `scripts/frank-primary-expert-activity.py`
instead of guessing — `scripts/frank-archon-expert-activity.py`
samples llama-server with AMD IBS (`perf record -e ibs_op// --data`)
while 4 mixed concurrent generations run, maps each sampled load
address through `/proc/<pid>/maps` back to its backing tensor (extents
@@ -235,7 +235,7 @@ per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of
samples; hottest cell 0.1%), below the reach of a tensor-level
override. Not a benchmark: it instruments production serving in place
at negligible IBS overhead; re-measure any future checkpoint or
traffic mix with `frank-primary-expert-activity.py <pid>` (or
traffic mix with `frank-archon-expert-activity.py <pid>` (or
`--analyze-only` against the saved sample file in `/tmp`).
## Rollback
@@ -243,8 +243,8 @@ traffic mix with `frank-primary-expert-activity.py <pid>` (or
The lucebox DS4.1 material is fully retained on the host: image
`localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era
repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
[[frank-primary-lucebox-ds4]]). Restore = move the `.container` back,
uninstall the root unit (`systemctl disable --now frank-primary` — the
[[frank-archon-lucebox-ds4]]). Restore = move the `.container` back,
uninstall the root unit (`systemctl disable --now frank-archon` — the
link, not the file; never `systemctl disable` a unit whose file must
survive), `daemon-reload`, start the user quadlet. The Qwen payload
(~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a
@@ -252,9 +252,9 @@ refetch.
## Fleet context
- [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
b11030-mix binary family on gfx1151.
- [[frank-secondary-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
- [[frank-zealot-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
3090; the only audio-capable endpoint.
- The harness contract lives in `/home/blank/.omp/agent/models.yml`
(`llamacpp (dankserver)` provider, `contextWindow 262144`,
@@ -1,9 +1,9 @@
---
type: configuration
title: frank-primary Lucebox DeepSeek V4 Runbook
description: RETIRED 2026-09-26 frank-primary runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-primary-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
title: frank-archon Lucebox DeepSeek V4 Runbook
description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
tags:
- frank-primary
- frank-archon
- dankserver
- lucebox
- deepseek-v4
@@ -16,16 +16,16 @@ tags:
timestamp: 2026-09-26
---
# frank-primary Lucebox DeepSeek V4 Runbook
# frank-archon Lucebox DeepSeek V4 Runbook
> [!warning] RETIRED 2026-09-26 (same day it went live)
> Replaced by [[frank-primary-llamacpp-qwen38-flash-next-q8]] — unsloth
> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
> served for hours; the rollback material (image `localhost/lucebox-rocm:
> gfx1201` and the 378 GiB payload) is retained on the host, and the
> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`.
The dankserver LLM endpoint (:7770, alias **frank-primary**) as it stood on
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
2026-09-26: the
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
@@ -62,7 +62,7 @@ mode here, not an error.
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
# Three local modifications (repo-tracked at
# dankserver/frank-primary/lucebox-gfx1201.patch):
# dankserver/frank-archon/lucebox-gfx1201.patch):
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
# base already ships them in /opt/rocm; the apt repo does not serve
# them) and the three runtime math libs.
@@ -94,11 +94,11 @@ podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
400 against HF's xethub CDN for this repo; see the fetch note in
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
`deploy_llm_primary`.
`deploy_llm_archon`.
## Deployed flags (2026-09-26)
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (the unit
From `dankserver/frank-archon/frank-archon-entrypoint.sh` (the unit
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
flag set and cannot express the DS4 flags):
@@ -113,7 +113,7 @@ flag set and cannot express the DS4 flags):
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
| Identity | `--model-name frank-primary`, `--host 0.0.0.0 --port 7770` |
| Identity | `--model-name frank-archon`, `--host 0.0.0.0 --port 7770` |
## API surface
@@ -123,11 +123,11 @@ metadata — the fleet contract is unchanged from frank-medium.
## Cutover procedure
1. Image built and model fetched (gates in `deploy_llm_primary`).
2. `deploy.sh` → `deploy_llm_primary`: stops and **masks** the radiance
1. Image built and model fetched (gates in `deploy_llm_archon`).
2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
unit (`frank-medium`), waits for :7770 to free, installs
`dankserver/frank-primary/quadlet/frank-primary.container`, starts
`frank-primary.service`, waits for health (1800 s start period — first
`dankserver/frank-archon/quadlet/frank-archon.container`, starts
`frank-archon.service`, waits for health (1800 s start period — first
start page-ins the dense weights and builds the graphs).
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
call, and an Open WebUI completion.
@@ -137,7 +137,7 @@ metadata — the fleet contract is unchanged from frank-medium.
## Rollback
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
systemctl --user start frank-medium`) and stop/mask frank-primary. The
systemctl --user start frank-medium`) and stop/mask frank-archon. The
`localhost/radiance-vllm:0.13.0-e060895` image and the
`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
@@ -1,25 +1,25 @@
---
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
service: frank-primary
title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
service: frank-archon
host: dankserver
port: 7770
status: active
deployed: 2026-09-27
supersedes: "[[frank-primary-r9v-radiance-qwen38-uncensored]]"
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, q8]
supersedes: "[[frank-archon-r9v-radiance-qwen38-uncensored]]"
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, q8]
---
# frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)
# frank-archon — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
/ `0000:07:00.0`) as `frank-primary` on **:7770**.
/ `0000:07:00.0`) as `frank-archon` on **:7770**.
Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
direction. Rollback target: [[frank-primary-r9v-radiance-qwen38-uncensored]]
direction. Rollback target: [[frank-archon-r9v-radiance-qwen38-uncensored]]
— the uncensored state dir (`/home/dank/r9v-state/uncensored/`, `ready:
true`), its frozen config (`restore-config-20260927.env`) and payload stay
on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.
@@ -39,10 +39,10 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
**Why the tuned shape lives on-host:** the q8 profile's setup run
qualified the `R9V_*` config (tiered-expert placement, per-rank offload
@@ -55,7 +55,7 @@ would drift from the qualification — don't.
## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
frank-secondary/large). The fork's `speculative_config` is a **single dict**
frank-zealot/large). The fork's `speculative_config` is a **single dict**
(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
mutually exclusive — no chaining seam exists in the fork. What IS active:
@@ -67,7 +67,7 @@ mutually exclusive — no chaining seam exists in the fork. What IS active:
- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram
proposer) — swapping MTP4 for it is an either/or, not a chain.
## Sharp chat template (fleet parity with frank-large)
## Sharp chat template (fleet parity with frank-dragoon)
`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is
@@ -176,7 +176,7 @@ Laws from the matrix:
after a bench teardown; a production boot failed on it the same morning
(unit "Deactivated successfully", no error in journal — the serve script
exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
`frank-primary.env` (forwarded by launch.sh `2de0357`).
`frank-archon.env` (forwarded by launch.sh `2de0357`).
Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
@@ -256,7 +256,7 @@ quoting through ssh+zsh layers.
## Ops
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
script re-gates everything, relaunch through the shim, waits).
- **First start after boot** re-warms ~175 GiB of page cache; expect
minutes to /health; `TimeoutStartSec=1800`.
@@ -266,11 +266,11 @@ quoting through ssh+zsh layers.
produces a false-green deploy (the 2026-09-26 lesson, guarded by the
MainPID check).
- **Rollback to uncensored**: point
`scripts/frank-primary.env` `R9V_STATE_FILE` back at
`scripts/frank-archon.env` `R9V_STATE_FILE` back at
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
(adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
uncensored state dir and payload are intact on disk.
- **Models on the wire**: served alias `frank-primary`; thinking channel
- **Models on the wire**: served alias `frank-archon`; thinking channel
is `reasoning` (vLLM); tool parser `qwen3_coder`.
## Known costs (accepted at cutover)
@@ -1,19 +1,19 @@
---
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
service: frank-primary
title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
service: frank-archon
host: dankserver
port: 7770
status: retired
retired: 2026-09-27
superseded_by: "[[frank-primary-r9v-radiance-qwen38-q8]]"
supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]"
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, retired]
superseded_by: "[[frank-archon-r9v-radiance-qwen38-q8]]"
supersedes: "[[frank-archon-llamacpp-qwen38-flash-next-q8]]"
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, retired]
---
# frank-primary — R9V radiance-vLLM (dankserver, RETIRED)
# frank-archon — R9V radiance-vLLM (dankserver, RETIRED)
> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover,
> [[frank-primary-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook:
> [[frank-archon-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook:
> the uncensored state dir, frozen config and payload are intact on disk;
> the rollback procedure lives in that page's Ops section.
@@ -22,11 +22,11 @@ The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770**.
`0000:03:00.0` / `0000:07:00.0`) as `frank-archon` on **:7770**.
Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
Rollback target: [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the
Rollback target: [[frank-archon-llamacpp-qwen38-flash-next-q8]] — the
llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
wiring; host unit was the relinked one) and the GGUF payload stays on disk,
so a rollback is a redeploy.
@@ -37,10 +37,10 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs |
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs |
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
**Why the tuned shape lives on-host:** the profile's setup run qualified a
104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
@@ -83,7 +83,7 @@ unit. `docker wait` cannot stop the container child, hence `ExecStop`.
- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
`runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
all-reduce verified inside serving workers** on both BDFs.
- Wire: `/v1/models` → `frank-primary`; `/health` + `/metrics` 200
- Wire: `/v1/models` → `frank-archon`; `/health` + `/metrics` 200
(Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
thinking requests populate `reasoning` (vLLM channel).
- Memlock: the full tiered-expert load ran to healthy **under a plain dank
@@ -107,7 +107,7 @@ unit. `docker wait` cannot stop the container child, hence `ExecStop`.
## Ops
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
script re-gates everything, relaunch through the shim, waits).
- **First start after boot** re-warms ~93 GiB of page cache; expect several
minutes to /health; `TimeoutStartSec=1800`.
@@ -119,7 +119,7 @@ unit. `docker wait` cannot stop the container child, hence `ExecStop`.
- **Qualification artifacts** (kept on-host):
`/home/dank/r9v-state/uncensored/qualification-*`,
failure bundles `failure-*` (review before sharing — raw logs).
- **Models on the wire**: served alias `frank-primary`; thinking channel is
- **Models on the wire**: served alias `frank-archon`; thinking channel is
`reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
vllm030 wiring had planned).
@@ -1,4 +1,4 @@
# frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
# frank-archon — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
@@ -6,12 +6,12 @@ The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
libr4d kernels for gfx1201, no Python in the serving path) driving the
digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
(v1.0.7 line, 287 MiB, `libr4d@0.5.0`) as a plain `podman run` through
`scripts/serve-frank-primary.sh` under the **root systemd unit**
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
same served alias (`frank-primary`), same container identity
(`frank-primary` = the unit name). Rollback: git tag
`frank-primary-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
`localhost/frank-primary-freetoken:pr587-77a7201`, payload
`scripts/serve-frank-archon.sh` under the **root systemd unit**
(`frank-archon.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
same served alias (`frank-archon`), same container identity
(`frank-archon` = the unit name). Rollback: git tag
`frank-archon-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
`localhost/frank-archon-freetoken:pr587-77a7201`, payload
`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`.
## Why radiance
@@ -33,14 +33,14 @@ same served alias (`frank-primary`), same container identity
- **Raw checkpoint**: `/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8`
(ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision
configs). Fetched by `scripts/download-frank-primary.sh` (vendored
configs). Fetched by `scripts/download-frank-archon.sh` (vendored
`hfget.sh`, curl+jq, no HF token needed) and **verified byte-for-byte**:
all 131 shards sha256-match the Hub LFS oids (2026-10-05, log
`/home/dank/logs/verify-qwen38-official-fp8.log`).
- **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad`
(177,109 MiB measured = 173.0 GiB; floor
`FRANK_PRIMARY_RAD_MIN_MIB=177000` in the env gates a
truncated convert). Built by `scripts/build-frank-primary-rad.sh`:
`FRANK_ARCHON_RAD_MIN_MIB=177000` in the env gates a
truncated convert). Built by `scripts/build-frank-archon-rad.sh`:
1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with
**`quantization_config` dropped** + relative symlinks into the raw
checkout;
@@ -99,12 +99,12 @@ same served alias (`frank-primary`), same container identity
(declare = CPU-side layout queries); add `--device` only if a plugin
load ever demands it.
## The shape (`scripts/frank-primary.env`, every value commented there)
## The shape (`scripts/frank-archon.env`, every value commented there)
| Knob | Value | Why |
|---|---|---|
| Image | `stilldeadcode/radiance@sha256:4c3f4845…` | digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH |
| Container | `frank-primary` | identity = service name across runtimes; legacy list retires `frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe` |
| Container | `frank-archon` | identity = service name across runtimes; legacy list retires `frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe` |
| `--tp 2 --tp-wire wht6` | both R9700s | fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB) |
| `--max-num-seqs 2` | first boot | Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8` |
| `--max-model-len 200000` | served window | native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated |
@@ -113,7 +113,7 @@ same served alias (`frank-primary`), same container identity
| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-archon --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
@@ -122,7 +122,7 @@ same served alias (`frank-primary`), same container identity
spare ≈ **134 GiB**. `ttm.pages_limit` (this kernel has NO `pages_max` —
the two-knob story was stale) caps GPU-mappable pinned pages (default = half
of RAM): the pre-cutover cap was **123 GiB** — BELOW the ledger — so the
serve script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
serve script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=140000`) refuses to
start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
verified read-back; raising at runtime is harmless and allocates nothing).
@@ -175,16 +175,16 @@ KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom.
```bash
# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh --plan-only' # declare + plan, no bytes
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh' # full convert + rad-info
./scripts/deploy.sh dankserver --service frank-primary
ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh --plan-only' # declare + plan, no bytes
ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh' # full convert + rad-info
./scripts/deploy.sh dankserver --service frank-archon
```
Post-cutover checklist (each is observable, not assumed):
1. Unit: `is-active` active, `is-enabled` enabled, container answers
`/health` with 200 JSON.
2. Boot ledger (engine stdout → `/home/dank/logs/frank_primary.log`):
2. Boot ledger (engine stdout → `/home/dank/logs/frank_archon.log`):
P2P probe per rank pair, TP=2 shard placement, expert pool coverage
line ("backs N % of the worst case … M full-length sessions"), KV pool
bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
@@ -194,7 +194,7 @@ Post-cutover checklist (each is observable, not assumed):
(tool-call + thinking turn both exercise it).
5. Vision: a ≥256 px solid-color PNG answers the exact color through an
`image_url` content part.
6. Prometheus target `frank-primary` green (not never-green).
6. Prometheus target `frank-archon` green (not never-green).
## Cutover measured (2026-10-06, all values observed live)
@@ -225,25 +225,25 @@ Post-cutover checklist (each is observable, not assumed):
converter's scale-name law, the PLE file-shadow and the
symlink-through-raw-index bug are in the payload section above.
- Prometheus: `/metrics` answers 200 with vLLM-compatible families
(verified from Dankclaw); the `frank-primary` job needs a
(verified from Dankclaw); the `frank-archon` job needs a
prometheus-service deploy to go green (config pushed, reload
deferred behind a live parallel writer's dirty tree).
## Tuning levers (in order of likely use)
- **`FRANK_PRIMARY_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
- **`FRANK_ARCHON_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
headroom line and real slot occupancy agree; each slot buys a
recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
- **`FRANK_PRIMARY_HOST_POOL_MIB`**: the ledger's coverage line is the
- **`FRANK_ARCHON_HOST_POOL_MIB`**: the ledger's coverage line is the
oracle — it prints what the pool backs at the worst case and how many
full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of
the host's 246.
- **`FRANK_PRIMARY_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
- **`FRANK_ARCHON_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
point; 0.87 zeroes it. Move only with a measured hit-rate reason.
- **`FRANK_PRIMARY_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
- **`FRANK_ARCHON_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
+5.6 % prefill for bit-exact reductions; the fused one-shot forms are
declared over the seqs × (1+spec) row bound.
- **`FRANK_PRIMARY_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
- **`FRANK_ARCHON_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
unquoted.
- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
the tunable space at this model's shapes on this machine, writes
@@ -252,14 +252,14 @@ Post-cutover checklist (each is observable, not assumed):
--num-speculative-tokens 3`), or you measure a program the serve never
runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
`--rm` scratch container's cache dies with it: the serve script
bind-mounts `$FRANK_PRIMARY_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
bind-mounts `$FRANK_ARCHON_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
`gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
stopped-service window) survives and is read at every production boot.
The two attention requests have no shape recipe (`attn_paged`,
`attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
`RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
- **`FRANK_PRIMARY_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
- **`FRANK_ARCHON_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
stopped-service window (real-text probes: 20,417-token cold prefill,
1,985-token warm prefill, 256-token decode). The prefill ladder is
NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
@@ -274,14 +274,14 @@ Post-cutover checklist (each is observable, not assumed):
publisher's `scripts/` harnesses (`fnpf`, `fnconc`, `cstep2`, `ident`,
`mctools`) — note `mcserve/fnstop`-family scripts `pkill` the engine
binary by name; on this host the service is a systemd unit, so use
`sudo systemctl stop frank-primary` instead.
`sudo systemctl stop frank-archon` instead.
## Rollback
| Layer | Path |
|---|---|
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-primary-freetoken-pr587-20261005 -- scripts/frank-primary.env scripts/frank-primary.service scripts/serve-frank-primary.sh scripts/docker/ tests/frank_primary_unit.bats` + redeploy; image `localhost/frank-primary-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-primary-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-archon-freetoken-pr587-20261005 -- scripts/frank-archon.env scripts/frank-archon.service scripts/serve-frank-archon.sh scripts/docker/ tests/frank_archon_unit.bats` + redeploy; image `localhost/frank-archon-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-archon-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` + `--host-pool-mib 12288` — the documented fallback if the native container ever proves unbootable |
Retire the FreeToken-era orcarouter payload to `~/.AGENTTRASH/` only after
@@ -290,7 +290,7 @@ the rollback window closes (trash policy, never `rm`).
## Links
- [[LLM/index]] — fleet index
- [[frank-primary-freetoken-pr587]] — the retired FreeToken era
- [[frank-archon-freetoken-pr587]] — the retired FreeToken era
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM era
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
- [[RDNA4 Container Serving]] — rootless-podman GPU laws on these hosts
@@ -1,4 +1,4 @@
# frank-primary — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
**Status: LIVE — deployed 2026-10-08 (main `3ca64e1`, e2e 33/33).** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
@@ -10,7 +10,7 @@ finetune) converted with the recipe
**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
Same port, alias, container name, systemd unit. Rollback: revert the
`MODEL_FILE` line in `scripts/frank-primary.env` + redeploy —
`MODEL_FILE` line in `scripts/frank-archon.env` + redeploy —
`/home/dank/models/rad/qwen4exp-fp8-native.rad` stays on disk (trash
policy; 173 GiB, purge when the window closes).
@@ -22,7 +22,7 @@ policy; 173 GiB, purge when the window closes).
host and VRAM budget for KV and slots at equal-or-better measured
decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control).
- **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's
fleet-personality call; frank-large serves the same family).
fleet-personality call; frank-dragoon serves the same family).
- **GPTQ closes the precision gap**: RTN alone on BF16 weights lands
measurably worse; GPTQ with Hessians from a 10M-token corpus of this
model's own distribution recovers most of it, verified end-to-end by a
@@ -30,7 +30,7 @@ policy; 173 GiB, purge when the window closes).
## The payload chain
1. **Fetch**: `scripts/download-frank-primary-swift15.sh` →
1. **Fetch**: `scripts/download-frank-archon-swift15.sh` →
`/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3
GiB, per-shard LFS sha256 verified, no token on the command line).
48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560,
@@ -95,7 +95,7 @@ policy; 173 GiB, purge when the window closes).
| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |
## The shape (`scripts/frank-primary.env`)
## The shape (`scripts/frank-archon.env`)
Unchanged from the FP8-native era: TP=2 + `wht6`, `--max-num-seqs 3`,
`--max-model-len 200000`, MNBT 2048 (the MTP resonance law — 2560
@@ -103,7 +103,7 @@ collapses decode to 2.6–7.7 t/s), fp8 KV, `expert_tiered` host pool
40960 (the load-bearing floor), `--ngram-placement ram`, froggeric
v22.5 template override. `MODEL_FILE` points at
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416
MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
MiB measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
177000 → **110000**.
## Gates (measured at cutover)
@@ -149,8 +149,8 @@ MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
budget and `content` comes back empty (the answer lands in the
reasoning channel).
- **Deployed verification (2026-10-08)**: `deploy.sh dankserver
--service frank-primary` green incl. Playwright e2e 33/33; unit
`active`, `/health` 200, `/v1/models` id `frank-primary` (the harness
--service frank-archon` green incl. Playwright e2e 33/33; unit
`active`, `/health` 200, `/v1/models` id `frank-archon` (the harness
alias is the served-model-name, independent of the file — models.yml
untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2,
seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert
@@ -159,23 +159,23 @@ MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
### Wire rename + the bench drop-in trap (2026-10-09)
`FRANK_PRIMARY_SERVED_NAME` moved `frank-primary` → **frank-archon**; the
`FRANK_ARCHON_SERVED_NAME` moved `frank-archon` → **frank-archon**; the
service, unit, scripts and env file keep the old name. Radiance is the only
fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
**requested** model name in chat responses — so a chat probe that returns 200
proves nothing about the served id. Always read `/v1/models`.
The first deploy after the edit came up green and `/v1/models` still said
`frank-primary`. Cause: a campaign leftover drop-in
`/etc/systemd/system/frank-primary.service.d/bench.conf` (2026-10-08 04:09)
overrode BOTH `Environment=FRANK_PRIMARY_ENV_FILE` and `ExecStart` to the
`frank-archon`. Cause: a campaign leftover drop-in
`/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09)
overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the
staging worktree `/home/dank/serverAppConfigs-swift15`. Every
`deploy.sh dankserver --service frank-primary` since then restarted the
`deploy.sh dankserver --service frank-archon` since then restarted the
staging tree: green health gate, old configuration, silently. Verified before
removing: the staging `serve-frank-primary.sh` and `frank-primary.service`
are byte-identical to the checkout, its `frank-primary.env` differed only by
removing: the staging `serve-frank-archon.sh` and `frank-archon.service`
are byte-identical to the checkout, its `frank-archon.env` differed only by
the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-primary-bench-dropin/`.
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`.
After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
26071/3543).
@@ -190,9 +190,9 @@ run. After an LLM deploy, verify the served configuration behaviourally
## Deploy + verify
```bash
sudo systemctl stop frank-primary # drained window
sudo systemctl stop frank-archon # drained window
bash convert-swift15-w4dn8.sh # if not already run
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankserver --service frank-archon
```
Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger
@@ -206,9 +206,9 @@ probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
| Layer | Path |
|---|---|
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-primary.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
| FreeToken PR #587 | git tag `frank-primary-freetoken-pr587-20261005` |
| tcclaviger vLLM shape G | git tag `frank-primary-vllm-shapeG-20261003` |
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-archon.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
| FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` |
| tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` |
Campaign artifacts on disk: BF16 checkpoint (335 GiB), `kld-ref` (24
GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, logs
@@ -218,7 +218,7 @@ re-convert source if the window ever reopens.
## Links
- [[frank-primary-radiance-qwen38-native-fp8]] — the replaced FP8-native era
- [[frank-archon-radiance-qwen38-native-fp8]] — the replaced FP8-native era
- [[LLM/index]] — fleet index
- [[swift15-campaign-worklog]] — campaign record
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
@@ -1,6 +1,6 @@
---
type: reference
description: "This page records the benchmarks that chose the runtime for `frank-large` on Dankclaw."
description: "This page records the benchmarks that chose the runtime for `frank-dragoon` on Dankclaw."
tags:
- llm
- frank
@@ -11,10 +11,10 @@ timestamp: 2026-09-03
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# frank-large — Backend and Spec Benchmark
# frank-dragoon — Backend and Spec Benchmark
This page records the benchmarks that chose the runtime for
`frank-large` on Dankclaw. Every number came from the live
`frank-dragoon` on Dankclaw. Every number came from the live
llama.cpp build on that machine (upstream, b10133+).
## Summary
@@ -0,0 +1,16 @@
---
type: reference
title: "CIRU v4.4.1 Orca on Dankclaw (frank-dragoon)"
description: "The retired frank-dragoon runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
tags:
- llm
- ciru
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-19
---
@@ -1,14 +1,14 @@
# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
`scripts/frank-large.service`); the retired 27B payload stays on disk at
(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
`scripts/frank-dragoon.service`); the retired 27B payload stays on disk at
`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
(`scripts/download-frank-large-halogen.sh`) stays in the repo.
(`scripts/download-frank-dragoon-halogen.sh`) stays in the repo.
## Why this shape exists
Frank's ask: replace the frank-large model with
Frank's ask: replace the frank-dragoon model with
`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
That payload is an **abliterated Swift1.5 finetune of peonist's
Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
@@ -70,13 +70,13 @@ Payload notes worth keeping:
dangle inside the container mount; both the serve gate and the deployer
refuse them).
Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
Fetch: `scripts/download-frank-dragoon-flashnext.sh` (`--verify-only` to
re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
resume law throughout; no token — the repos are public).
## Shape — the knobs that are not free
`scripts/frank-large.env` carries the full rationale inline; the table:
`scripts/frank-dragoon.env` carries the full rationale inline; the table:
| Env | Value | Why |
|---|---|---|
@@ -88,7 +88,7 @@ resume law throughout; no token — the repos are public).
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-large`); engine port 8730 never published |
| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-dragoon`); engine port 8730 never published |
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
@@ -112,7 +112,7 @@ instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
mandatory.
- Vision: OpenAI `image_url` content parts only (the fleet's standard
contract; the legacy top-level `images` field is the frank-secondary trap).
contract; the legacy top-level `images` field is the frank-zealot trap).
≥256 px probe images or the model answers "video".
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
truncation. `max_tokens` INCLUDES thinking.
@@ -133,12 +133,12 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
## Mechanics
- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
- **Root systemd unit** (`scripts/frank-dragoon.service`, `User=dank`,
`LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
frank-dragoon-halogen`), driven by `scripts/serve-frank-dragoon.sh` — the
fleet's `podman run`-direct pattern. The script runs the container
FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
prompt cache, any clamp) lands in `/home/dank/logs/frank_dragoon.log`.
- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
image digest in podman storage, byte-exact checkpoint + n-gram table +
vision tower, flat tokenizer with all six files as regular files,
@@ -151,7 +151,7 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
when the dying 27B engine freed RAM). The serve script now checks
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_DRAGOON_RAM_WAIT`
(default 600 s) for the RAM to clear, so a transient dip boots once
instead of crash-looping; if it never clears it refuses loudly. The 16
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
@@ -162,7 +162,7 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
## Verification checklist (run after any redeploy)
1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
1. Boot ledger greps in `/home/dank/logs/frank_dragoon.log`: resident
weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
**no clamp named**.
2. `curl http://192.168.0.143:7770/health` → 200.
@@ -183,7 +183,7 @@ BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
`maxTokens: 45056`, `input: [text, image]`,
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
frank-secondary (the `compactionModel` steering is unchanged; the input
frank-zealot (the `compactionModel` steering is unchanged; the input
≈ 170k fits its 200,704 slot).
## Rollback
@@ -1,13 +1,13 @@
---
type: reference
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
tags:
- llm
- gufo
- swift-1.5
- dankclaw
- frank-large
- frank-dragoon
- strix-halo
- gfx1151
- q8
@@ -16,12 +16,12 @@ tags:
timestamp: 2026-09-28
---
# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
# frank-dragoon — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
> **RETIRED 2026-10-02 — the rollback target.** Superseded by
> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
> [[frank-dragoon-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
> MTP4, 17.0 t/s measured). This era was itself a return after one day of
> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
> [[frank-dragoon-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
> ≥6-bit vLLM quant path dead on gfx1151). The shape
> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
@@ -30,10 +30,10 @@ timestamp: 2026-09-28
> Swift-1.5 Q8 history below is retained for the engine/template law,
> which still governs.
> **Originally ACTIVE 2026-09-28.** Replaces
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
> b11030 prebuilt), which had replaced
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
> alias (`frank-large`).
> [[frank-dragoon-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
> alias (`frank-dragoon`).
>
> **The payload also changed on 2026-09-28**, later the same day: the target
> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
@@ -44,16 +44,16 @@ timestamp: 2026-09-28
## Identity
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
inference engine", a vertical engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
**v0.1.1** (commit `b0f8673`), built from source on the host to
`/home/dank/GitPackages/gufo-v0.1.1/`.
- **Service shape:** root systemd unit `scripts/frank-large.service`
- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
(`User=dank`, `LimitMEMLOCK=infinity`), launched by
`scripts/serve-frank-large.sh` with tunables from
`scripts/frank-large.env`.
`scripts/serve-frank-dragoon.sh` with tunables from
`scripts/frank-dragoon.env`.
gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
are empty) and no Nix on this host, so the runtime is a source build. It is
@@ -77,7 +77,7 @@ gufo's own measurement of the two engines on the same Qwen3.8-27B targets
Read that honestly: **prefill, load time and short-context decode improve;
deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
around 32k of cached depth. frank-large runs one 262,144-token conversation,
around 32k of cached depth. frank-dragoon runs one 262,144-token conversation,
so a long agent session spends its life on the losing side of that line
while a short turn wins. The Q4_K_XL target does not show the regression
(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
@@ -116,7 +116,7 @@ architecture, driver and queue budget.
- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
sha256-verified (`9076f99a…`, the Hub LFS oid) by
`scripts/download-frank-large-swift.sh`. It is an **abliterated
`scripts/download-frank-dragoon-swift.sh`. It is an **abliterated
(uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
`mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
@@ -160,7 +160,7 @@ were confirmed against gufo's source rather than assumed:
from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
drafter's filename rather than the target's.
- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
frank-large served. The harness overrides per request via `extraBody`
frank-dragoon served. The harness overrides per request via `extraBody`
(1.0 / 0.95 / top-k 20).
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
@@ -227,8 +227,8 @@ tokens or `inter_token` is first-token noise.
## The template law: Qwen-Sharp is gone
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was
`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no
wired into `frank-archon` as `R9V_CHAT_TEMPLATE_FILE` and it was
`frank-dragoon`'s `--chat-template-file` under llama.cpp. **frank-dragoon can no
longer use it**, and the reason is stronger than "gufo lacks the flag":
> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
@@ -265,7 +265,7 @@ second is the same class of change that produced the 2026-09-21
## Client contract — what actually changed
Everything the fleet depended on still holds: port 7770, alias `frank-large`,
Everything the fleet depended on still holds: port 7770, alias `frank-dragoon`,
`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
auto-discovers a `mmproj` beside the model; this payload has none).
@@ -300,9 +300,9 @@ The differences worth knowing:
## Failure modes worth knowing
- **A missing runtime is a deploy error, not a restart loop.** The handler
gates on `LLM_LARGE_RUNTIME` *before* it stops anything, and the serve
gates on `LLM_DRAGOON_RUNTIME` *before* it stops anything, and the serve
script exits with a `build-gufo.sh` pointer rather than exec'ing.
- **Host memory is the ceiling, and it is shared.** frank-large and ComfyUI
- **Host memory is the ceiling, and it is shared.** frank-dragoon and ComfyUI
both hold gfx1151 compute queues. gufo reads
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
@@ -321,13 +321,13 @@ The differences worth knowing:
Two independent rollbacks, because two things changed on 2026-09-28:
1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-large.env` at
1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-dragoon.env` at
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
and redeploy. **The 29.3 GiB target was reclaimed on 2026-09-28**, so this
path now needs a re-fetch first: `scripts/download-frank-large.sh`, then
path now needs a re-fetch first: `scripts/download-frank-dragoon.sh`, then
the env edit. The DFlash2 drafter was never touched and still lives in
that directory.
2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
2. **Engine** — [[frank-dragoon-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
script's llama.cpp command, and redeploy. This is also the only way to
@@ -343,8 +343,8 @@ Two independent rollbacks, because two things changed on 2026-09-28:
- GGUF header evidence for the payload swap: both files' headers were parsed
and diffed directly (architecture, all 866 tensor names in order, per-tensor
`ggml_type` and dims) before either was loaded
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
tunables: `scripts/frank-large.env`
- Payload: `scripts/download-frank-large-swift.sh` (target),
`scripts/download-frank-large.sh` (DFlash2 drafter)
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-dragoon.sh`;
tunables: `scripts/frank-dragoon.env`
- Payload: `scripts/download-frank-dragoon-swift.sh` (target),
`scripts/download-frank-dragoon.sh` (DFlash2 drafter)
- Predecessor: [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
@@ -1,14 +1,14 @@
# frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
# frank-dragoon — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
frank-large is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
frank-dragoon is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
pinned digest, launched by `scripts/serve-frank-large.sh` under a root
pinned digest, launched by `scripts/serve-frank-dragoon.sh` under a root
systemd unit as a podman container — serving
**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
**DFlash2** drafter at **one slot x 200,700 ctx**.
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-large-rocmfpx-q6agent-uncensored-mtp]]),
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]),
which remains the rollback target with its payload on disk.
## Why swap
@@ -25,21 +25,21 @@ confirmed by Frank before deploy:
1. **Text-only.** halogen loads the language tower only; README "honest
limits": *"The vision encoder is ignored; image inputs are not supported."*
The mmproj-era vision role moves wholly to frank-secondary.
The mmproj-era vision role moves wholly to frank-zealot.
2. **Censored base.** The checkpoint is a finetune of the censored
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
## Shape and why
`scripts/frank-large.env` carries every tunable with its evidence; the load-
`scripts/frank-dragoon.env` carries every tunable with its evidence; the load-
bearing numbers:
- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
a step batches when more than one conversation is running, so the drafter
stays off with more than one slot"). Slots>1 silently loses DFlash2; the
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-large
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-dragoon
was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
prompt is a 400 naming the limit, never a truncation. The harness side is
@@ -83,7 +83,7 @@ bearing numbers:
|thinking field|`reasoning_content` (same as before)|
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|image input|ignored/error — route to frank-secondary|
|image input|ignored/error — route to frank-zealot|
## Deploy mechanics
@@ -91,7 +91,7 @@ bearing numbers:
(published tag 0.1.4). The deployer gates the exact digest exists in podman
storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
was the Flash-Next w4b era, different image, retired.
- **Payload:** `scripts/download-frank-large-halogen.sh` (fetch law: `curl -f`,
- **Payload:** `scripts/download-frank-dragoon-halogen.sh` (fetch law: `curl -f`,
exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
@@ -101,20 +101,20 @@ bearing numbers:
docker-run parity). The deployer and serve script refuse a missing file or
ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
those links dangle inside the container mount.
- **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
- **Unit:** `scripts/frank-dragoon.service` (root, `User=dank`, relinked into
`/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
stop -t 120 frank-large-halogen`, script runs `podman run` FOREGROUND
stop -t 120 frank-dragoon-halogen`, script runs `podman run` FOREGROUND
(2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
only the container ID — the engine ledger existed nowhere on disk and
the post-deploy grep below read a one-line hex file; foreground run
returns the container's exit code so `Restart=always` is real and the
ledger lands in the log). Engine stdout goes to
`/home/dank/logs/frank_large.log` (the boot ledger lives there, not in
`/home/dank/logs/frank_dragoon.log` (the boot ledger lives there, not in
the journal).
- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
- **Cutover:** `deploy_llm_dragoon` in `scripts/deploy-remote.sh` sweeps the
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
the legacy `frank-large` container name by NAME (never `pkill -f
llama-server` — that hits frank-secondary), retires the old quadlet if the 2026-09
the legacy `frank-dragoon` container name by NAME (never `pkill -f
llama-server` — that hits frank-zealot), retires the old quadlet if the 2026-09
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
- **Bench law:** stop+mask the unit before any GPU experiment on this box
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
@@ -122,7 +122,7 @@ bearing numbers:
## Verification (post-deploy checklist)
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_large.log`
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_dragoon.log`
— ledger shows the pool at the asked shape, DFlash2 active, the
prompt cache line naming the pinned budget, NO clamp line. EXPECT the
WARNING "smaller than ONE full-context entry (18.4 GB at
@@ -136,7 +136,7 @@ bearing numbers:
`reasoning_content` populated; decode rate from the timings.
4. Spec decode fires: multi-token step behavior visible in rate; compare
against DRAFTER=0 arm if in doubt (never against gufo — bench law).
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-large`
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-dragoon`
incl. a tool call (V1TOOLS).
6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
@@ -172,10 +172,10 @@ Measured on real agent traffic (serve_api lines) the day after cutover:
one with public evidence, and it moved nothing that matters here.
- **Routing is the applied mitigation (2026-10-06):** the harness's
deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
moved to frank-primary (radiance on Dankserver: fp8 KV at
moved to frank-archon (radiance on Dankserver: fp8 KV at
7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
200k window, prefix-cache host+disk tiers). frank-large keeps OpenWebUI
and shallow bursts, where it runs 30-50 t/s. frank-primary's
200k window, prefix-cache host+disk tiers). frank-dragoon keeps OpenWebUI
and shallow bursts, where it runs 30-50 t/s. frank-archon's
depth-decode rate is designed-for but not yet measured on record.
- **The lever that existed was the prompt cache, and it was OFF** (see
the shape bullet). The pin turns the 85-180 s re-prefill per deep
@@ -192,7 +192,7 @@ Measured on real agent traffic (serve_api lines) the day after cutover:
## Rollback
Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:
Revert the frank-dragoon swap commit and redeploy `dankclaw --service frank-dragoon`:
the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
@@ -1,11 +1,11 @@
---
type: reference
description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance]
description: frank-dragoon on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
tags: [llm, frank-dragoon, ik_llama, dankclaw, rtx3090, strix-halo, performance]
timestamp: 2026-09-16T03:30:00-07:00
---
# frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
# frank-dragoon — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
**RETIRED 2026-09-16, after one day.** Replaced by [[halogen-flash-server]],
which was the runtime before it as well — the fleet went ik → halogen → back.
@@ -23,10 +23,10 @@ hardware, and they are why the fleet went back. It served
`/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
-DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
- Unit: **root systemd unit** `/etc/systemd/system/frank-large.service`
symlinked to `scripts/frank-large.service` in the repo; enabled at boot;
`Restart=always`. Serve script `scripts/serve-frank-large.sh` + env
`scripts/frank-large.env`.
- Unit: **root systemd unit** `/etc/systemd/system/frank-dragoon.service`
symlinked to `scripts/frank-dragoon.service` in the repo; enabled at boot;
`Restart=always`. Serve script `scripts/serve-frank-dragoon.sh` + env
`scripts/frank-dragoon.env`.
- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
@@ -108,9 +108,9 @@ decoding is unavailable for this GGUF — mradermacher stripped the NextN head
- **The systemd unit self-heals through manual tests**: `pkill` the server and
the unit respawns it ~4 s later (Restart=always) — two instances, port
races, phantom measurements. For manual experiments:
`sudo systemctl stop frank-large.service` first (it is a root unit; plain
`sudo systemctl stop frank-dragoon.service` first (it is a root unit; plain
`systemctl --user` lines in older docs are stale), and restore with
`sudo systemctl start frank-large.service`.
`sudo systemctl start frank-dragoon.service`.
- Verify "HTTP server listening" in the CURRENT log and
`pgrep -xc llama-server` == 1 before trusting any measurement.
- FA must stay ON — FA-off crashes the loader on this arch.
@@ -1,13 +1,13 @@
---
type: reference
title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)"
description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- kairic-edge
- rocmfpx
- dankclaw
- frank-large
- frank-dragoon
- strix-halo
- gfx1151
- iu4
@@ -18,11 +18,11 @@ tags:
timestamp: 2026-09-21
---
# frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)
# frank-dragoon — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
> runbook on 2026-09-21 before it was replaced by
> [[frank-large-unsloth-qwen38-27b-q8xl]] later the same day. The payload
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] later the same day. The payload
> and runtime remain on disk as the rollback target.
## What it served
@@ -38,8 +38,8 @@ timestamp: 2026-09-21
## Rollback
Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-large-unsloth-qwen38-flash-next]]
Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-dragoon-unsloth-qwen38-flash-next]]
for the predecessor.
# Citations
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
description: "The active frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
tags:
- llm
- frank-large
- frank-dragoon
- dankclaw
- llama-cpp
- rocmfpx
@@ -15,14 +15,14 @@ tags:
timestamp: 2026-10-03
---
# frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
# frank-dragoon — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
> build script all on disk).** Replaced by [[frank-large-halogen27b-dflash2]]
> build script all on disk).** Replaced by [[frank-dragoon-halogen27b-dflash2]]
> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
> remains the contract record for the rollback. Original banner follows.
> ACTIVE since 2026-10-03. Cut over from
> [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
> [[frank-dragoon-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
> on Frank's request: the *uncensored* variant of the same protected-heads
> family, one quant rung down. The swap turned out to be a strict upgrade on
> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
@@ -34,15 +34,15 @@ timestamp: 2026-10-03
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
| Fetch | [`scripts/download-frank-large-kingjones-uncensored.sh`](../../scripts/download-frank-large-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
| Seam | `tests/frank_large_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
| Fetch | [`scripts/download-frank-dragoon-kingjones-uncensored.sh`](../../scripts/download-frank-dragoon-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
| Seam | `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
## Why this file
@@ -78,7 +78,7 @@ Unchanged from the Q8 era — the swap was payload-only:
| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
| `--fit off` | on | the auto-fitter would silently resize the window |
| `--flash-attn on` | on | required for the long-window attention path |
| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-large-rocmfpx-q8agent-mtp]] |
| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-dragoon-rocmfpx-q8agent-mtp]] |
Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
@@ -138,37 +138,37 @@ models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
## Client contract
Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
`frank-large`, native tools, thinking in `reasoning_content`,
`frank-dragoon`, native tools, thinking in `reasoning_content`,
`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
card-recipe sampling defaults server-side, vision through the BF16 projector
(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
`content: null` when reasoning eats the budget, the self-recovering MTP
boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
in [[frank-dragoon-rocmfpx-q8agent-mtp]] and still applies.
Sweep key for scratch servers is now the payload name:
`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
llama-server` (frank-secondary is also one).
llama-server` (frank-zealot is also one).
## Operations
```bash
./scripts/deploy.sh dankclaw --service frank-large
systemctl status frank-large
journalctl -u frank-large -n 200 --no-pager
scripts/download-frank-large-kingjones-uncensored.sh --verify-only
./scripts/deploy.sh dankclaw --service frank-dragoon
systemctl status frank-dragoon
journalctl -u frank-dragoon -n 200 --no-pager
scripts/download-frank-dragoon-kingjones-uncensored.sh --verify-only
```
## Rollback
| Target | How |
|---|---|
| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_LARGE_MODEL_FILE` + `LLM_LARGE_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-large-kingjones.sh --verify-only` |
| gufo Q4_K_XL era | git tag `frank-large-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-large-gufo-qwen38-27b]] |
| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_DRAGOON_MODEL_FILE` + `LLM_DRAGOON_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-dragoon-kingjones.sh --verify-only` |
| gufo Q4_K_XL era | git tag `frank-dragoon-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-dragoon-gufo-qwen38-27b]] |
## When it misbehaves
Same decision tree as [[frank-large-rocmfpx-q8agent-mtp]]:
Same decision tree as [[frank-dragoon-rocmfpx-q8agent-mtp]]:
`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
description: "The retired frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
description: "The retired frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
tags:
- llm
- frank-large
- frank-dragoon
- dankclaw
- llama-cpp
- rocmfpx
@@ -15,17 +15,17 @@ tags:
timestamp: 2026-10-02
---
# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
# frank-dragoon — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
> **RETIRED 2026-10-03** — superseded by
> [[frank-large-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
> [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
> runbook stays live as the rollback reference and the trap catalogue; the
> payload it documents remains on disk at
> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
>
> Cut over 2026-10-02 from [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
> Cut over 2026-10-02 from [[frank-dragoon-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
> on two requests from Frank: *highest quality that runs*, and *one full
> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
> could load — at 17.0 t/s, so quality was not bought with speed.
@@ -39,15 +39,15 @@ timestamp: 2026-10-02
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
| Fetch | [`scripts/download-frank-dragoon-kingjones.sh`](../../scripts/download-frank-dragoon-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
| Seam | `tests/frank_dragoon_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
## Why this file, and why only this runtime
@@ -117,7 +117,7 @@ there is no second 26 GiB copy in memory. See the fleet law in
`AGENTS.md → llama.cpp serving`.
**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
`cache_prompt: false`, `frank-dragoon.service` stopped for every scratch arm (the
benchmarking law), baseline arm stripped of `--model-draft` **and** every
`--spec-*` flag:
@@ -144,7 +144,7 @@ re-litigates this:
| Behavior | Observation |
|---|---|
| Model id | `/v1/models` → `frank-large` |
| Model id | `/v1/models` → `frank-dragoon` |
| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
@@ -162,7 +162,7 @@ Three traps, all hit for real:
enough for the vision probe).
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
appears in the log under draft resets. Self-recovering; not an outage signal.
3. **Never `pkill -f llama-server` on Dankclaw** — frank-secondary is also a
3. **Never `pkill -f llama-server` on Dankclaw** — frank-zealot is also a
`llama-server`. The sweep key here is the payload name:
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
@@ -170,24 +170,24 @@ Three traps, all hit for real:
```bash
# deploy (planner selects this endpoint only via its own unit/env/serve trio)
./scripts/deploy.sh dankclaw --service frank-large
./scripts/deploy.sh dankclaw --service frank-dragoon
# status / logs / cause of a crash loop
systemctl status frank-large
journalctl -u frank-large -n 200 --no-pager
systemctl status frank-dragoon
journalctl -u frank-dragoon -n 200 --no-pager
# rebuild the pinned fork (only when deliberately moving the pin)
scripts/build-rocmfpx.sh
# re-verify the payload without re-downloading
scripts/download-frank-large-kingjones.sh --verify-only
scripts/download-frank-dragoon-kingjones.sh --verify-only
```
`serve-frank-large.sh` refuses to start — before touching the port — unless the
fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
`serve-frank-dragoon.sh` refuses to start — before touching the port — unless the
fork binary, the weights, the MTP head and (unless `FRANK_DRAGOON_MMPROJ=""`) the
projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
broken payload never stops the running unit. Caller environment **overrides**
`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
`frank-dragoon.env`, and defaults use `:=` so an explicitly empty value surfaces
as the error it reads as.
## Rollback (as of retirement, 2026-10-03)
@@ -196,16 +196,16 @@ as the error it reads as.
payload and remains on disk** at
`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
2026-10-03 swap commit (unit/env/serve trio) or repoint
`FRANK_LARGE_MODEL_FILE`/`LLM_LARGE_MODEL_FILE`, verify with
`scripts/download-frank-large-kingjones.sh --verify-only`, redeploy.
`FRANK_DRAGOON_MODEL_FILE`/`LLM_DRAGOON_MODEL_FILE`, verify with
`scripts/download-frank-dragoon-kingjones.sh --verify-only`, redeploy.
git tag **`frank-large-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
git tag **`frank-dragoon-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
DFlash2) is now a **re-download** rollback: the gufo payloads
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
removed from disk on 2026-10-03 with Frank's explicit approval to free space.
`scripts/build-gufo.sh` remains on disk; re-fetch per
[[frank-large-gufo-qwen38-27b]].
[[frank-dragoon-gufo-qwen38-27b]].
The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
@@ -215,7 +215,7 @@ recoverable until the trash is purged, and never part of a rollback plan.
## When it misbehaves
- **`unknown tensor type` / ftype 115 refused** — something moved off the
ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
ROCmFPX binary. Check `readlink -f $(systemctl show frank-dragoon -p ExecStart)`
and the fork commit; this file cannot run on stock llama.cpp or gufo.
- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
@@ -1,13 +1,13 @@
---
type: reference
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
description: "The RETIRED frank-large runbook (2026-09-21 → 2026-09-28; replaced by [[frank-large-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (2026-09-21 → 2026-09-28; replaced by [[frank-dragoon-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- llama.cpp
- unsloth
- dankclaw
- frank-large
- frank-dragoon
- strix-halo
- gfx1151
- q8
@@ -17,31 +17,31 @@ tags:
timestamp: 2026-09-21
---
# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
# frank-dragoon — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
> **RETIRED 2026-09-28**, replaced by [[frank-large-gufo-qwen38-27b]] (the
> **RETIRED 2026-09-28**, replaced by [[frank-dragoon-gufo-qwen38-27b]] (the
> gufo engine, same weights, same port and alias); the prebuilt is intact on
> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
> [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
> [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
> [[frank-dragoon-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
> preceded).
## Identity
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
(`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
A mainline build, not a fork: it supports the `qwen3_5` architecture and the
DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
`common_speculative_impl_draft_dflash`).
- **Service shape:** root systemd unit `scripts/frank-large.service`
(`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
`scripts/frank-large.env`.
- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
(`User=dank`), launched by `scripts/serve-frank-dragoon.sh` with tunables from
`scripts/frank-dragoon.env`.
## Shape
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-dragoon.sh`).
- **Context:** one stream, full 262,144 tokens (the model's native window),
`PARALLEL_SLOTS=1`.
- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
@@ -75,18 +75,18 @@ harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
## Payload
`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
`scripts/download-frank-dragoon.sh` fetches the main GGUF and the DFlash2
drafter, each verified against its declared byte count + sha256
(`--verify-only` re-checks). The serve script refuses to start if the binary,
the main GGUF, or the drafter is missing.
## Client contract
- **Alias:** `frank-large` (the `id` at `/v1/models`).
- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
`--reasoning-format deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned).
- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
- **Vision:** none — text-only (no mmproj), matching the prior frank-dragoon.
- **Sampling:** greedy by default; a client that sends a temperature gets its
own, and the harness applies per-model overrides from `models.yml`.
@@ -105,8 +105,8 @@ the main GGUF, or the drafter is missing.
## Rollback
Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
Payload and runtime for [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge
IU4 W4A4) remain intact on disk. See [[frank-dragoon-unsloth-qwen38-flash-next]]
for the Flash-Next predecessor it replaced on 2026-09-21.
# Citations
@@ -114,4 +114,4 @@ for the Flash-Next predecessor it replaced on 2026-09-21.
- unsloth prebuilts: https://huggingface.co/unsloth
- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
- DFlash2 drafter: https://huggingface.co/z-lab
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
@@ -1,12 +1,12 @@
---
type: reference
title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)"
description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- unsloth
- dankclaw
- frank-large
- frank-dragoon
- strix-halo
- gfx1151
- qwen4exp
@@ -15,11 +15,11 @@ tags:
timestamp: 2026-09-21
---
# frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
# frank-dragoon — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
> runbook on 2026-09-21 before it was replaced by
> [[frank-large-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
> same day. The payload and runtime remain on disk as the rollback target.
## What it served
@@ -37,8 +37,8 @@ timestamp: 2026-09-21
## Rollback
Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-large-ciru-orca]] for the
Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-dragoon-ciru-orca]] for the
predecessor.
# Citations
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
title: "frank-dragoon on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
description: "The RETIRED frank-dragoon runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-dragoon-gufo-qwen38-27b]] by Frank the same night."
tags:
- llm
- frank-large
- frank-dragoon
- dankclaw
- vllm
- strix-halo
@@ -15,27 +15,27 @@ tags:
timestamp: 2026-10-01
---
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
# frank-dragoon — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
> torchao config unwired). Back to [[frank-dragoon-gufo-qwen38-27b]]
> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
## Identity
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
`podman run` driven by `scripts/serve-frank-large.sh` under a root unit
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
`podman run` driven by `scripts/serve-frank-dragoon.sh` under a root unit
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
(17.53 GiB, 4 shards) — the same checkpoint frank-secondary serves as an INT8
(17.53 GiB, 4 shards) — the same checkpoint frank-zealot serves as an INT8
W8A16 AutoRound conversion on the 3090, here unquantized.
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
are loaded into the draft pass).
@@ -67,7 +67,7 @@ scheduler instead of carved per session.
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
One session, temperature 0, 28,647-token prefill / 512-token decode, same
client for all three arms (2026-10-01 bench window, frank-large stopped per
client for all three arms (2026-10-01 bench window, frank-dragoon stopped per
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
@@ -82,7 +82,7 @@ Qwen3.5-9B that preserves the hidden-state geometry — identical
per-position acceptance is the direct measurement that the distillation
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
(verified in both the BF16 base and the INT8 conversion) — which is exactly
why frank-secondary's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
why frank-zealot's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
reads the head from the TARGET checkpoint, vLLM reads the draft from a
separate path. That difference is the whole trick:
@@ -91,8 +91,8 @@ separate path. That difference is the whole trick:
```
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
[1,5,…,29]) stays fetched on disk; flip `FRANK_DRAGOON_SPEC=dflash` and repoint
`FRANK_DRAGOON_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
## The numerics trap (greedy text is NOT preserved by spec decode)
@@ -109,14 +109,14 @@ chase text drift between spec and no-spec arms as a regression.
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
byte-count + LFS sha256-verified by
`old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
`old/download-frank-dragoon-mimo.sh` (`--verify-only` re-checks; no token on
the curl command line).
- MTP source: `old/download-frank-large-qwen35.sh` →
- MTP source: `old/download-frank-dragoon-qwen35.sh` →
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
`model.safetensors.index.json`; the shards carry the odd HF asset names
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
the engine boots from exactly this tree).
- DFlash drafter: `old/download-frank-large-dflash.sh` →
- DFlash drafter: `old/download-frank-dragoon-dflash.sh` →
`/home/dank/models/Qwen3.5-9B-DFlash/`.
- **Payload status 2026-10-02:** all three trees moved to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
@@ -131,7 +131,7 @@ chase text drift between spec and no-spec arms as a regression.
## Client contract
- **Alias:** `frank-large` (the `id` at `/v1/models`).
- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
`models.yml` compat block for this provider sets
@@ -146,7 +146,7 @@ chase text drift between spec and no-spec arms as a regression.
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
`chat_template_kwargs` form is what vLLM reads; a top-level
`preserve_thinking` is ignored there.
- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
- **Env hygiene:** every tunable is `FRANK_DRAGOON_`-prefixed because HIP
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
the env file with `set -a`.
@@ -154,23 +154,23 @@ chase text drift between spec and no-spec arms as a regression.
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
- The serve script sweeps stale container identities before starting:
`frank-large-mimo` itself and the bench-window container `vllm-mimo`.
`frank-dragoon-mimo` itself and the bench-window container `vllm-mimo`.
- Container death → `podman wait` returns non-zero from ExecStart →
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
frank-primary lesson).
- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
frank-archon lesson).
- Bench windows: `systemctl stop frank-dragoon` is enough (mask fails on the
relinked unit; the deploy restores).
- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
262,144 tokens per request: 3.13x` — four sessions share the pool
(the scheduler, not a carve). Live battery green: alias `frank-large`,
(the scheduler, not a carve). Live battery green: alias `frank-dragoon`,
reasoning split into `reasoning`, tool_calls with correct args, and
`/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
matching the bench window.
## Rollback
Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
Git tag **`frank-dragoon-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
payload + DFlash2 drafter, all still on the host. Restoring the tag's
unit/env/serve trio and redeploying returns the 27B endpoint.
@@ -181,6 +181,6 @@ unit/env/serve trio and redeploying returns the 27B endpoint.
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
[[frank-secondary-sglang-mimo-v26]]
- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
- Predecessor: [[frank-dragoon-gufo-qwen38-27b]]; sibling on the same arch:
[[frank-zealot-sglang-mimo-v26]]
@@ -1,16 +0,0 @@
---
type: reference
title: "CIRU v4.4.1 Orca on Dankclaw (frank-large)"
description: "The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
tags:
- llm
- ciru
- dankclaw
- frank-large
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-19
---
@@ -211,7 +211,7 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
history
- [[frank-secondary-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
- [[frank-zealot-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
difference: with no KV to attend over, per-pass overhead dominates and a
bigger micro-batch wins; at 64k of resident KV it does not.
@@ -312,7 +312,7 @@ came up healthy and *text-only*: image requests answered 500 "image input is
not supported — hint: … provide the mmproj" and the load log had no mmproj
line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the
model directory. The serve script now passes `--mmproj` explicitly (the
frank-secondary pattern), refuses to start without the file when vision is on, and
frank-zealot pattern), refuses to start without the file when vision is on, and
the deploy handler's preflight checks for it. Verified end-to-end after the
fix: a generated 64x32 test image was described from its pixels — with
thinking on the model reasoned about "the three horizontal bars" (its reading
@@ -724,7 +724,7 @@ fleet.
2026-09-15, whose `--split-mode layer` measurements this service reuses
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
collective, and every software escape that is closed
- [[frank-secondary-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
- [[frank-zealot-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
@@ -602,7 +602,7 @@ but not both. That is the wall, and it is capacity, not scheduling.
### Watching it instead of measuring it by hand
Everything above was diagnosed with ad-hoc probes. The same signals are now
scraped — Prometheus has `frank-medium` and `frank-large` jobs
scraped — Prometheus has `frank-medium` and `frank-dragoon` jobs
(`shared/prometheus/prometheus.yml`), so the numbers that matter have history:
`vllm:kv_cache_usage_perc` (per-card pool fill), `vllm:num_requests_waiting`
(queue depth), `rate(vllm:prefix_cache_hits_total[5m]) /
@@ -144,7 +144,7 @@ ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
First deploy (2026-09-15) also ran `retire_root_llm_small()` in
`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
disabled and moved to `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units`
disabled and moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units`
(the host's `systemctl disable` has been observed to delete unit files,
2026-08-26 incident, so both outcomes are tolerated).
@@ -156,7 +156,7 @@ Deployed flags (see `scripts/frank-medium.env` for the full set):
hidden state is produced. Reasoning on, deepseek format, unlimited budget;
sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8,
top-k 20, min-p 0, presence 1.5). Text only — vision stays on
`frank-large` (Dankclaw).
`frank-dragoon` (Dankclaw).
## Verified serving behavior (2026-09-15)
@@ -181,7 +181,7 @@ are projected into `reasoning_content` (deepseek format); requests that send
no sampler fields get the server's CLI defaults, not greedy. Text only.
Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default`
→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small
routes) → `frank-medium`, vision stays `frank-large`.
routes) → `frank-medium`, vision stays `frank-dragoon`.
## Host ROCm posture (post-cleanup)
+7 -7
View File
@@ -48,17 +48,17 @@ upstream project on this exact silicon. That single flag is the whole decision:
Consequences accepted at retirement: **audio input left the fleet with no
replacement** (halogen reads images only, no audio or video), and the surviving
image path is halogen's vision tower on `frank-large`.
image path is halogen's vision tower on `frank-dragoon`.
## Where its routes went
| consumer | setting | now |
|---|---|---|
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) |
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-secondary` |
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` |
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-secondary` |
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` |
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-dragoon` (halogen vision tower) |
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-zealot` |
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-dragoon` |
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-zealot` |
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-dragoon:medium` |
The `flm (dankclaw-npu)` provider block was removed from the harness model
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
@@ -104,7 +104,7 @@ all live-verified on the host:
carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID
state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`.
3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain
identity-mapped (no DMA-translation overhead for `frank-large`) while
identity-mapped (no DMA-translation overhead for `frank-dragoon`) while
enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
zero `SVA bind device failed` entries since boot, `accel0` recreated at
@@ -17,15 +17,15 @@ The old scripts moved to `old/scripts/` in this repository.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-secondary | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
| frank-dragoon | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-zealot | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
tensor split, no NPU.
## Models
### frank-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
### frank-dragoon — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
@@ -40,7 +40,7 @@ tensor split, no NPU.
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
(48/64/24).
### frank-secondary — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
### frank-zealot — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
@@ -55,7 +55,7 @@ tensor split, no NPU.
instead of the Qwen3 XML structure). The Quark quant follows the format
cleanly; swapped in the evening of 2026-08-17.
## Context Math (frank-large, RTX 3090)
## Context Math (frank-dragoon, RTX 3090)
KV per token: 16 attention layers x 4 KV heads x 256 dim.
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
@@ -72,11 +72,11 @@ Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
Production choice: **131072 (128K)** — the maximum that keeps about
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
`scripts/frank-large.env` to 122880 for ~1.8 GiB headroom.
`scripts/frank-dragoon.env` to 122880 for ~1.8 GiB headroom.
## Install Runbook
### Dankserver (frank-large)
### Dankserver (frank-dragoon)
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
@@ -96,17 +96,17 @@ Production choice: **131072 (128K)** — the maximum that keeps about
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
4. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vLLM/scripts/frank-large.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-large
sudo systemctl restart frank-large
sudo ln -sf /home/dank/vLLM/scripts/frank-dragoon.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-dragoon
sudo systemctl restart frank-dragoon
```
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
6. Confirm VRAM headroom with `nvidia-smi`.
### Dankclaw (frank-secondary)
### Dankclaw (frank-zealot)
1. Stop the old services: `sudo systemctl stop frank-large
frank-secondary` (retired PP head and NPU endpoint).
1. Stop the old services: `sudo systemctl stop frank-dragoon
frank-zealot` (retired PP head and NPU endpoint).
2. Install uv and create the venv:
```
curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -131,9 +131,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about
vision encoder falls back to torch SDPA and engine init OOMs.
5. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-secondary
sudo systemctl restart frank-secondary
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-zealot
sudo systemctl restart frank-zealot
```
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
@@ -159,7 +159,7 @@ on gfx1151; newer vLLM releases may already include it.
Three changes landed after the initial deployment:
### 1. Tool calling on frank-secondary (HTTP 400 fix)
### 1. Tool calling on frank-zealot (HTTP 400 fix)
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
unless the server enables tool calling. Fixes applied:
@@ -184,7 +184,7 @@ model behavior, which motivated the swap.
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
`frank-secondary` now serves
`frank-zealot` now serves
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
vLLM-native expert layout). The Avesed compressed-tensors quant's
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
@@ -1,15 +0,0 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
description: "The RETIRED frank-secondary runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-secondary
- dankclaw
- llama.cpp
- rtx3090
- qwen35
- mtp
- vision
timestamp: 2026-09-16
---
@@ -1,7 +1,7 @@
---
timestamp: 2026-09-03
type: reference
title: frank-secondary Benchmark — ncmoe and Batch Tuning
title: frank-zealot Benchmark — ncmoe and Batch Tuning
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
tags:
- benchmark
@@ -11,11 +11,11 @@ tags:
- dankserver
---
# frank-secondary Benchmark — ncmoe and Batch/Ubatch Tuning
# frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
`--n-cpu-moe` and `--ubatch-size` values for `frank-secondary`.
`--n-cpu-moe` and `--ubatch-size` values for `frank-zealot`.
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
@@ -106,7 +106,7 @@ service runs 700416 cells (13,680 MiB).
## Chosen State
> **Live drift (2026-08-22 audit):** the deployed `frank-secondary` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
> **Live drift (2026-08-22 audit):** the deployed `frank-zealot` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
Prefill speed is the priority, so the state keeps the top prefill tier and
moves one expert layer to CPU for stability:
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
description: "The active frank-secondary runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
title: "frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
description: "The active frank-zealot runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- llama.cpp
- rtx3090
@@ -14,25 +14,25 @@ tags:
timestamp: 2026-09-17
---
# frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
# frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
> **ACTIVE since 2026-09-17.** Replaces [[frank-secondary-qwen38-gsq-rco]]
> **ACTIVE since 2026-09-17.** Replaces [[frank-zealot-qwen38-gsq-rco]]
> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
> [[frank-secondary-vllm-gemma4]] (the same-day vLLM experiment, also the
> rollback target). Same port (7771), same alias (`frank-secondary`), same
> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment, also the
> rollback target). Same port (7771), same alias (`frank-zealot`), same
> client contract surface.
## Identity
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
- **Endpoint:** `frank-zealot` at `192.168.0.143:7771` (Dankclaw).
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
collapses prefill ~12x on CUDA.
- **Service shape:** root systemd unit `scripts/frank-secondary.service`
(`User=dank`), launched by `scripts/serve-frank-secondary.sh` with tunables from
`scripts/frank-secondary.env`.
- **Service shape:** root systemd unit `scripts/frank-zealot.service`
(`User=dank`), launched by `scripts/serve-frank-zealot.sh` with tunables from
`scripts/frank-zealot.env`.
## Shape
@@ -51,7 +51,7 @@ timestamp: 2026-09-17
the card. 4 is the measured max.
- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
only tokenizes and samples (its CPU belongs to the halogen engine serving
frank-large).
frank-dragoon).
## Spec method
@@ -90,7 +90,7 @@ fault).
## Client contract
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
- **Alias:** `frank-zealot` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned even though this build defaults
@@ -109,12 +109,12 @@ pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
## Rollback
Payload and runtime for [[frank-secondary-qwen38-gsq-rco]] (ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-secondary-vllm-gemma4]]
Payload and runtime for [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-zealot-vllm-gemma4]]
is the same-day vLLM experiment (also a rollback target).
# Citations
- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
- llama.cpp: https://github.com/ggml-org/llama.cpp
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`
@@ -1,7 +1,7 @@
# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
# frank-zealot — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
**RETIRED 2026-10-07** — same day it went live. Superseded by
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
2 × 200,704 with exact KV); the payload stays at
`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
@@ -108,4 +108,4 @@ free).
Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
payload remains on disk. See
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]].
[[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]].
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
description: "The retired 2026-10-02/03 frank-secondary runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
description: "The retired 2026-10-02/03 frank-zealot runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- llama-cpp
- rtx3090
@@ -14,17 +14,17 @@ tags:
timestamp: 2026-10-03
---
# frank-secondary — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
# frank-zealot — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
> **RETIRED 2026-10-03** — superseded by
> [[frank-secondary-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
> [[frank-zealot-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
> decode is the measured price of the higher rung on this bandwidth-bound
> card. This file remains the rollback reference and the owner of the
> measured boot-pair table.
> Its own history: one quant step down from [[frank-secondary-llama-mimo-q6k]]
> Its own history: one quant step down from [[frank-zealot-llama-mimo-q6k]]
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
> conversations of 200,704 tokens with the KV VALUES unquantized** —
@@ -36,11 +36,11 @@ timestamp: 2026-10-03
|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-secondary.sh`|
|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-zealot.sh`|
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
## Shape and the KV contract
@@ -100,13 +100,13 @@ The per-slot window is the whole price, in both directions:
- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
sets `contextWindow: 200704` for this endpoint — the harness compacts before
it reaches the ceiling instead of discovering it.
- **frank-large's tallest pre-compaction prompt now fits here.** Compaction
for frank-large sessions runs on this endpoint (`compactionModel` in
`models.yml`); frank-large compacts at 70 % of 262,144 = 183,501 tokens,
- **frank-dragoon's tallest pre-compaction prompt now fits here.** Compaction
for frank-dragoon sessions runs on this endpoint (`compactionModel` in
`models.yml`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
tokens to spare. The 2026-10-02 window trim had restored the fit for
frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens
the fit to frank-large itself. Nothing routes compaction automatically —
frank-archon's smaller 140,492-token trigger; the 2026-10-03 reslice widens
the fit to frank-dragoon itself. Nothing routes compaction automatically —
the point is the ceiling no longer forbids it.
- Two simultaneous generations are the slot ceiling; a third request queues
behind a released slot (the mechanism measured on the 3-slot shape).
@@ -128,7 +128,7 @@ The per-slot window is the whole price, in both directions:
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
but refuses a request without `messages` in this build, so chat is the only
vision path.
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-secondary.env` drops
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-zealot.env` drops
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
refusal, never a silent fallback to blindness.
@@ -145,15 +145,15 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|Target|How|
|---|---|
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-secondary.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|Q6_K, same runtime|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-secondary-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-zealot.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SECONDARY_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-secondary-sglang-int8.sh`|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_ZEALOT_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
## When it misbehaves
- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script exits 1
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script exits 1
naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
description: "The RETIRED frank-secondary runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
description: "The RETIRED frank-zealot runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- llama-cpp
- gguf
@@ -12,10 +12,10 @@ tags:
timestamp: 2026-10-03
---
# frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
# frank-zealot — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
> **RETIRED 2026-10-05 (evening).** Superseded by
> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
> [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
@@ -29,14 +29,14 @@ timestamp: 2026-10-03
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-zealot`, `/v1` OpenAI surface |
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
## Shape and KV contract
@@ -62,12 +62,12 @@ MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
under the floor — and the calibrated bandwidth law (it predicted this file's
94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
pool to 2 × 182,272 — which also breaks frank-large's 183,501-token
pool to 2 × 182,272 — which also breaks frank-dragoon's 183,501-token
compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
were declined. Re-open only on evidence the Q6_K residual matters.
**Pool prices on this card come from boots, never formulas** — the measured
boot-pair table lives in [[frank-secondary-llama-mimo-q5km]] (*Shape and KV
boot-pair table lives in [[frank-zealot-llama-mimo-q5km]] (*Shape and KV
contract*) and still prices every rollback shape.
## Compute-buffer geometry (2026-10-05)
@@ -78,7 +78,7 @@ measured rather than assumed, because the predecessor shape recorded **"ubatch
**That failure was a different shape, and it did not reproduce here.** The
service itself is the bench (one card holds one server, so no scratch port):
the old rung was re-booted through a transient `FRANK_SECONDARY_ENV_FILE` drop-in
the old rung was re-booted through a transient `FRANK_ZEALOT_ENV_FILE` drop-in
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
2,849 MiB free — which makes the ubatch the only variable in the comparison.
@@ -99,7 +99,7 @@ Three readings:
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
and the pool's 12,288-token (384 MiB) ladder in
[[frank-secondary-llama-mimo-q5km]] is the other way to buy room back.
[[frank-zealot-llama-mimo-q5km]] is the other way to buy room back.
**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
@@ -125,7 +125,7 @@ that is the pair the curve was measured at.
## Client contract
llama.cpp surface, live-verified on this model family: model id `frank-secondary`,
llama.cpp surface, live-verified on this model family: model id `frank-zealot`,
native tools via `--jinja` (`tool_calls` in the response), thinking in
**`reasoning_content`**, vision via `image_url` content parts. Two vision
traps, both measured here:
@@ -147,13 +147,13 @@ so the harness compacts before llama.cpp answers a longer prompt with a 400.
|Target|How|
|---|---|
|Q5_K_M at the same shape|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|Q5_K_M at the same shape|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
## When it misbehaves
Same decision tree as [[frank-secondary-llama-mimo-q5km]] (*When it misbehaves*):
Same decision tree as [[frank-zealot-llama-mimo-q5km]] (*When it misbehaves*):
`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
description: "The retired 2026-10-01 frank-secondary runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
description: "The retired 2026-10-01 frank-zealot runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- llama-cpp
- rtx3090
@@ -14,10 +14,10 @@ tags:
timestamp: 2026-10-01
---
# frank-secondary — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
# frank-zealot — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
> [[frank-secondary-llama-mimo-q5km]], the same runtime and shape one quant rung
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
> target.
>
@@ -29,11 +29,11 @@ timestamp: 2026-10-01
|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
## Shape and the KV contract
@@ -95,12 +95,12 @@ cmake --build build-cuda -j 16 --target llama-server
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
stay on disk as the rollback: repoint the unit's serve script and env from
git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
git history (`git show frank-dragoon-gufo-q4kxl-20261001` era is BEFORE this;
use the parent of the 2026-10-01 night swap commit).
## When it misbehaves
- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script
exits 1 naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
description: "The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
description: "The ACTIVE frank-zealot runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- llama-cpp
- gguf
@@ -12,10 +12,10 @@ tags:
timestamp: 2026-10-07
---
# frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
# frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
Live since 2026-10-07, replacing the
[[frank-secondary-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
[[frank-zealot-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
conversation slot and unquantized KV back.
@@ -25,14 +25,14 @@ conversation slot and unquantized KV back.
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-secondary` name), `/v1` OpenAI surface |
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine) |
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
## Payload geometry (read from the GGUF header, not the model card)
@@ -68,7 +68,7 @@ that is what makes two 200k slots affordable where the 26B MoE fit only one.
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
2,279** at ubatch 2048 — see [[frank-secondary-llama-mimo-q6k-2x200704]].
2,279** at ubatch 2048 — see [[frank-zealot-llama-mimo-q6k-2x200704]].
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
so the projection is ~21,430 used / ~2,700 free.
@@ -80,7 +80,7 @@ fallback rung if a future shape lands under the 2 GiB floor.
`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
(`--log-verbosity 4`, `journalctl -u frank-secondary`):
(`--log-verbosity 4`, `journalctl -u frank-zealot`):
|Line|Value|
|---|---|
@@ -98,7 +98,7 @@ safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
**past** its window). Weights load in ~6 s from page cache; `/health` answers
`{"status":"ok"}` and `/v1/models` serves the alias `frank-zealot`.
**Wire rename (2026-10-09):** `ALIAS` moved `frank-secondary` → `frank-zealot`,
**Wire rename (2026-10-09):** `ALIAS` moved `frank-zealot` → `frank-zealot`,
so the served id matches the harness model id and the family-facing name.
Consumers re-keyed in the same change: OpenWebUI (`model` row, `access_grant`
row, `openai.api_configs` index-2 `model_ids`) and `models.yml` (already
@@ -111,7 +111,7 @@ unchanged.
## Client contract
llama.cpp surface: model id `frank-secondary`, native tools via `--jinja`,
llama.cpp surface: model id `frank-zealot`, native tools via `--jinja`,
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
@@ -128,7 +128,7 @@ behind a live generation (the Gemma4 era's `--parallel 1` law), and the
|Target|How|
|---|---|
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-secondary-llama-gemma4-q4km-200704]]|
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-zealot-llama-gemma4-q4km-200704]]|
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
@@ -138,13 +138,13 @@ Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
`journalctl -u frank-zealot` (KV MiB, compute buffers) before touching
flags — the engine prints the exact figures the shape needs.
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-secondary.sh`
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-zealot.sh`
shipped without its `#!/usr/bin/env bash` line, and systemd failed the
`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
`Failed at step EXEC spawning /home/dank/…/serve-frank-secondary.sh`. A
`Failed at step EXEC spawning /home/dank/…/serve-frank-zealot.sh`. A
shebang is not decoration on this host: every LLM serve script is a direct
`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
description: "The active frank-secondary runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
title: "frank-zealot on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
description: "The active frank-zealot runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- llama-cpp
- gguf
@@ -15,7 +15,7 @@ tags:
timestamp: 2026-10-06
---
# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
# frank-zealot — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
> **ACTIVE since 2026-10-05 (evening); reshaped three times on
> 2026-10-06, renamed from frank-small the same night.** A 27B replaces
@@ -32,7 +32,7 @@ timestamp: 2026-10-06
> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
> measured prediction). This supersedes BOTH earlier 2026-10-05
> payloads: the live MiMo-V2.6 Q6_K shape
> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
> ([[frank-zealot-llama-mimo-q6k-2x200704]], now retired) and the
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
> never deployed.
@@ -41,14 +41,14 @@ timestamp: 2026-10-06
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-zealot`, `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
## Why this payload (2026-10-06 evening)
@@ -90,7 +90,7 @@ built against the target's FULL context (262,144 — the draft window, not
the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 −
22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter
quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free`
field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor
field) — the tightest fit in frank-zealot's history, the 2–3 GiB floor
knowingly violated, and the 512×512 vision transient (34 MiB) still fits
and recovers.
@@ -114,7 +114,7 @@ drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
ngram speculation logs *"recurrent state rollback is not compatible with
'ngram-mod' — disabling rollback support"* (fleet lesson from
frank-large's chain measurement, 2026-10-03). A rejected draft on Gated
frank-dragoon's chain measurement, 2026-10-03). A rejected draft on Gated
DeltaNet layers cannot be rewound by KV truncation — without the rollback
checkpoints the DeltaNet state carries phantom transitions and output
degrades silently. The flags move as a set: the serve script emits BOTH
@@ -216,7 +216,7 @@ leave room for thinking.
a clean A/B needs stop + mask (bench policy).
- **Smoke**: `READY` completion + `reasoning_content` populated; the
harness reached :7771 under the new alias (models.yml provider/id
`frank-secondary`, config.yml modelRoles rewritten; per-slot
`frank-zealot`, config.yml modelRoles rewritten; per-slot
`contextWindow: 114688`).
## Measured: ubatch 2048 (2026-10-06 midnight, live)
@@ -284,7 +284,7 @@ recipes:
`/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
[[frank-secondary-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
[[frank-zealot-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
380aafa's parent), one `MODEL_FILE` + shape edit.
3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
@@ -300,12 +300,12 @@ recipes:
read from the Hub file by HTTP range + struct parse — no download needed
— before the payload was fetched.
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
label-committed both sets at Frank's direction — 1919ad7 (frank-primary
label-committed both sets at Frank's direction — 1919ad7 (frank-archon
radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
Deploy ran `deploy.sh dankclaw --service frank-secondary`; the service
Deploy ran `deploy.sh dankclaw --service frank-zealot`; the service
dispatch and health gate went green; the first e2e run failed ONE spec
(12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
unrelated to frank-secondary), green on rerun.
unrelated to frank-zealot), green on rerun.
- The unit's `Description=` (MiMo era) was stale at cutover time and was
updated in-repo afterward — the relinked unit picks it up at the next
deploy/restart, not retroactively.
@@ -320,13 +320,13 @@ recipes:
second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
**frank-secondary** (87721ab — scripts, unit, seam, runbooks,
**frank-zealot** (87721ab — scripts, unit, seam, runbooks,
identifiers; historical references to the OLD FreeToken/CUDA
frank-small, and to the Dankserver service that became frank-medium,
stay frank-small) and the per-slot window cut 131,072 → 114,688 to
fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand
before the deploy: `systemctl stop frank-small` + `systemctl disable
frank-small`, then `deploy.sh dankclaw --service frank-secondary`.
frank-small`, then `deploy.sh dankclaw --service frank-zealot`.
1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
config.yml retargeted.
- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
@@ -0,0 +1,15 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
description: "The RETIRED frank-zealot runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-zealot
- dankclaw
- llama.cpp
- rtx3090
- qwen35
- mtp
- vision
timestamp: 2026-09-16
---
@@ -1,10 +1,10 @@
---
type: reference
title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night."
title: "frank-zealot on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
description: "The RETIRED frank-zealot runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-zealot-llama-mimo-q6k]] the same night."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- sglang
- rtx3090
@@ -15,10 +15,10 @@ tags:
timestamp: 2026-10-01
---
# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
# frank-zealot — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
> payload was staged to
@@ -28,13 +28,13 @@ timestamp: 2026-10-01
## Identity
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
- **Endpoint:** `frank-zealot` at `192.168.0.143:7771` (Dankclaw).
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
systemd unit `scripts/frank-secondary.service` (`User=dank`,
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with
tunables from `scripts/frank-secondary.env`.
systemd unit `scripts/frank-zealot.service` (`User=dank`,
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-zealot.sh` with
tunables from `scripts/frank-zealot.env`.
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
@@ -48,14 +48,14 @@ timestamp: 2026-10-01
(`--context-length 262144`, `--max-running-requests 2`) against a
**pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
~82k tokens of pool slack, and a compaction request (frank-primary's
~82k tokens of pool slack, and a compaction request (frank-archon's
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
one request. A single request may grow to the whole pool — a
170,019-token prompt was verified at the 4-slot shape — the grid is
the two-way concurrency contract. **That compaction arithmetic is
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
180,224, which 183,501 tokens already exceeds — see *What three slots
cost* in [[frank-secondary-llama-mimo-q5km]].
cost* in [[frank-zealot-llama-mimo-q5km]].
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
dim x 1 B), pool 9.31 GiB.
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
@@ -126,15 +126,15 @@ change lands.
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh`
all LFS sha256-verified by `old/download-frank-zealot-sglang-int8.sh`
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
cut over to llama.cpp + GGUF on 2026-10-01 night, and
`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The
`scripts/download-frank-zealot.sh` now fetches the GGUF quant instead. The
serve script refuses to start with a missing shard.
## Client contract
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
- **Alias:** `frank-zealot` (the `id` at `/v1/models`).
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
as llama.cpp; models.yml sets `reasoningContentField:
@@ -160,14 +160,14 @@ serve script refuses to start with a missing shard.
(216k tokens of live KV at once). 2-slot shape: two concurrent
50,000-token requests both answered (28.8 s wall).
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
well past frank-primary's 183,501-token compaction trigger (70% of its
well past frank-archon's 183,501-token compaction trigger (70% of its
262,144 window) — prefills and completes in ONE request.
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
## Rollback
[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the
practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at
[[frank-zealot-llama-mimo-q5km]] is the live shape two steps forward; the
practical rollback today is [[frank-zealot-llama-mimo-q6k]] (payload at
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
Gemma4-12B QAT payload, was staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
@@ -177,4 +177,4 @@ Gemma4-12B QAT payload, was staged to
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
- SGLang: https://github.com/sgl-project/sglang
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`
@@ -1,10 +1,10 @@
---
type: reference
title: frank-secondary on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
description: "The RETIRED frank-secondary runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-secondary-gemma4-12b]] on the same day."
title: frank-zealot on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
description: "The RETIRED frank-zealot runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-zealot-gemma4-12b]] on the same day."
tags:
- llm
- frank-secondary
- frank-zealot
- dankclaw
- vllm
- rtx3090
@@ -15,12 +15,12 @@ tags:
timestamp: 2026-09-19
---
# frank-secondary — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
> back to [[frank-secondary-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
> back to [[frank-zealot-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
> the same day it landed — this page is the rollback target. Same port
> (7771), same alias (`frank-secondary`), same client contract surface — the
> (7771), same alias (`frank-zealot`), same client contract surface — the
> runtime, the model payload, the drafter and the service shape (root
> systemd unit → rootless Quadlet) all changed.
@@ -28,12 +28,12 @@ timestamp: 2026-09-19
| what | value |
|---|---|
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-secondary` (`8dad46d7…`) |
| unit | **rootless Quadlet** `dankclaw/frank-secondary/quadlet/frank-secondary.container` (root systemd unit retired 2026-09-19) |
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-zealot` (`8dad46d7…`) |
| unit | **rootless Quadlet** `dankclaw/frank-zealot/quadlet/frank-zealot.container` (root systemd unit retired 2026-09-19) |
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
| port | 7771 → 8000 (alias `frank-secondary`; `frank-large` keeps 7770) |
| port | 7771 → 8000 (alias `frank-zealot`; `frank-dragoon` keeps 7770) |
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
@@ -113,7 +113,7 @@ trade for the native w4a16 format + audio + CUDA graphs.)
## The image
`dankclaw/frank-secondary/Dockerfile` (repo) — 16 comment lines + `FROM
`dankclaw/frank-zealot/Dockerfile` (repo) — 16 comment lines + `FROM
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
@@ -144,9 +144,9 @@ not a hardware mystery.
## Deployed config
Repo truth: `scripts/frank-secondary.env` + `scripts/serve-frank-secondary.sh`
Repo truth: `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
(mounted into the container at `/scripts/`, which is the entrypoint) +
`dankclaw/frank-secondary/quadlet/frank-secondary.container`. Key values:
`dankclaw/frank-zealot/quadlet/frank-zealot.container`. Key values:
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
@@ -186,27 +186,27 @@ container while it loads.
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
must be absent. 8.6 is correct.
- **Audio 400/500** — the container is running the *base* image, not the
derived `:frank-secondary` (missing `av`). Check the deployed image digest
derived `:frank-zealot` (missing `av`). Check the deployed image digest
is `8dad46d7…`.
- **CUDA not visible in container** — a `Device=` line or a lib mount
dropped out of the unit after an edit; the unit is the source of truth,
the host-side generated unit is not.
- Logs: `journalctl --user -u frank-secondary` (rootless Quadlet; `podman
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet; `podman
logs` covers only the current `--replace --rm` instance).
## Rollback
[[frank-secondary-gemma4-12b]] — the llama.cpp deployment. Its weights,
[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
mmproj and MTP sidecar are still on the host under
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
`scripts/.AGENTTRASH/frank-secondary.service.llama.cpp`. Restoring it means
`scripts/.AGENTTRASH/frank-zealot.service.llama.cpp`. Restoring it means
reinstalling that unit and stopping the Quadlet (one owner per port).
## References
- `scripts/serve-frank-secondary.sh` — the actual exec line
- `dankclaw/frank-secondary/Dockerfile` — the audio-derived image
- `dankclaw/frank-secondary/quadlet/frank-secondary.container` — the unit
- [[frank-secondary-gemma4-12b]] — RETIRED predecessor (rollback)
- `scripts/serve-frank-zealot.sh` — the actual exec line
- `dankclaw/frank-zealot/Dockerfile` — the audio-derived image
- `dankclaw/frank-zealot/quadlet/frank-zealot.container` — the unit
- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
@@ -6,7 +6,7 @@ tags:
- freetoken
- benchmark
- dankserver
- frank-secondary
- frank-zealot
- qwen3.6
timestamp: 2026-09-02
---
@@ -1,27 +1,27 @@
---
type: configuration
title: FreeToken frank-secondary Runbook
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-secondary on Dankserver (RTX 3090 era).
title: FreeToken frank-zealot Runbook
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-zealot on Dankserver (RTX 3090 era).
tags:
- freetoken
- dankserver
- frank-secondary
- frank-zealot
- qwen3.6
- moe
- fp8
timestamp: 2026-08-30
---
# FreeToken frank-secondary Runbook
# FreeToken frank-zealot Runbook
> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
> retirement record of the FreeToken/CUDA era; the unit was archived to
> `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/` and the repo files
> `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the repo files
> to `scripts/.AGENTTRASH/`.
The `frank-secondary` service on Dankserver is served by
The `frank-zealot` service on Dankserver is served by
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
expert weights in host RAM with an LRU GPU cache. Model history on this
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
@@ -134,8 +134,8 @@ modelopt block-fp8 remains the only FP8 path that loads.
## Sampling
**frank-secondary samples by default, and our config never said so.**
`scripts/serve-frank-secondary.sh` passes no sampling arguments at all — only
**frank-zealot samples by default, and our config never said so.**
`scripts/serve-frank-zealot.sh` passes no sampling arguments at all — only
`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
`--num-tokens` and `--max-running-requests`. The behaviour comes from the
runtime's own default:
@@ -158,9 +158,9 @@ sampling parameters sent at all** produced three different reasoning traces
(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding
would be byte-identical.
### This is the opposite of frank-large
### This is the opposite of frank-dragoon
| | frank-secondary (FreeToken) | frank-large (halogen) |
| | frank-zealot (FreeToken) | frank-dragoon (halogen) |
|---|---|---|
| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
| Deterministic across runs | No | Yes |
@@ -168,7 +168,7 @@ would be byte-identical.
A client that talks to both endpoints and sends no sampler gets
non-deterministic answers from :7770 on Dankserver and reproducible ones from
:7770 on Dankclaw. If a workflow needs determinism from frank-secondary it must
:7770 on Dankclaw. If a workflow needs determinism from frank-zealot it must
send `temperature: 0` explicitly, or the service must be relaunched with
`--sampling-defaults none` (framework defaults only).
@@ -188,9 +188,9 @@ send `temperature: 0` explicitly, or the service must be relaunched with
| File | Purpose |
|------|---------|
| `scripts/serve-frank-secondary.sh` | Serve script (launches `ft serve`) |
| `scripts/frank-secondary.env` | Tunables (model path, pool sizes) |
| `scripts/frank-secondary.service` | Systemd unit |
| `scripts/serve-frank-zealot.sh` | Serve script (launches `ft serve`) |
| `scripts/frank-zealot.env` | Tunables (model path, pool sizes) |
| `scripts/frank-zealot.service` | Systemd unit |
## Limitations
@@ -209,7 +209,7 @@ send `temperature: 0` explicitly, or the service must be relaunched with
- **`model` field is required** in `/v1/chat/completions` bodies —
FreeToken rejects requests without it with a 422 (`Field required`),
unlike llama.cpp's server which defaults to the loaded model. Send
`"model": "frank-secondary"`.
`"model": "frank-zealot"`.
- Load on this swap took well under the old llama.cpp numbers:
weights → experts (parallel build) → CUDA graph capture → Uvicorn
in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
@@ -1,12 +1,12 @@
---
type: reference
title: halogen-flash-server on Dankclaw (frank-large)
description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
title: halogen-flash-server on Dankclaw (frank-dragoon)
description: The active frank-dragoon runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
tags:
- llm
- halogen
- dankclaw
- frank-large
- frank-dragoon
- strix-halo
- gfx1151
- qwen4exp
@@ -14,16 +14,16 @@ tags:
timestamp: 2026-09-13
---
# halogen-flash-server on Dankclaw (frank-large)
# halogen-flash-server on Dankclaw (frank-dragoon)
> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
> [[frank-large-ik-uncensored]], the current rollback), and came back as the
> [[frank-dragoon-ik-uncensored]], the current rollback), and came back as the
> shape the fleet asked for: two 200k-token conversations rather than the
> native-context pair. Deployed as a rootless Quadlet container,
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
> it does not touch now belongs to [[frank-secondary-qwen38-gsq-rco]].
> `dankclaw/frank-dragoon/quadlet/frank-dragoon.container`, port 7770. The RTX 3090
> it does not touch now belongs to [[frank-zealot-qwen38-gsq-rco]].
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
@@ -45,7 +45,7 @@ The `all` mode is upstream's default and the shape a single-container host
wants: the front-end talks to the engine over loopback inside the container, so
there is one process tree, one log, and one published port.
`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
`HALOGEN_MODEL_ID=frank-dragoon` sets the id reported at `/v1/models`, in
`/health`, and in every response body. Requests are **never rejected for naming
a different model** — the server serves whatever asks. Keeping the fleet alias
is therefore free, and it means Hermes, the harness config, the OpenWebUI
@@ -94,13 +94,13 @@ upstream's defaults:
| flag | value | why |
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
| `HALOGEN_MODEL_ID` | `frank-dragoon` | fleet identity; no client rename |
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-dragoon` |
| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-secondary, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-zealot, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-dragoon` text-only to match |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -226,7 +226,7 @@ restart.
That is the failure mode to watch, and it is the least diagnosable of the
three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
looks like a hang, for the neighbours as much as for frank-large. `free` is
looks like a hang, for the neighbours as much as for frank-dragoon. `free` is
useless for spotting it — `MemAvailable` still reads ~74 GiB because the
page cache counts as reclaimable. Watch the `host memory left for everything
else` line and the contiguous-block count at boot instead.
@@ -374,7 +374,7 @@ tables above.
**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
the tower is a 1.97 GiB charge against the same device budget as the pool and
the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]],
the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]],
which loads the BF16 projector alongside its own weights. This server refuses
image content parts and says why, rather than ignoring them:
@@ -393,11 +393,11 @@ anywhere in the fleet — that left with [[frank-npu]].
## Deploy and operate
```bash
./scripts/deploy.sh dankclaw --service frank-large
./scripts/deploy.sh dankclaw --service frank-dragoon
```
The handler retires the ik-era **root** unit first (stop, disable, move
`/etc/systemd/system/frank-large.service` into
`/etc/systemd/system/frank-dragoon.service` into
`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
`disable` may delete the file outright, which the handler now reports), stops
the Quadlet's own unit if it is running, then pulls
@@ -419,8 +419,8 @@ file plus linger on the deploy user — generated units **cannot** be
`systemctl --user enable`d on this host.
```bash
systemctl --user status frank-large.service # the generated unit
podman logs frank-large | head -40 # version banner + memory ledger
systemctl --user status frank-dragoon.service # the generated unit
podman logs frank-dragoon | head -40 # version banner + memory ledger
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
curl -s http://127.0.0.1:7770/v1/models
```
@@ -447,13 +447,13 @@ steady-state noise on a shared host — raise it only for a live OOM report.
The lines worth grepping first:
```bash
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
podman logs frank-dragoon 2>&1 | grep -E '^(dmalloc|kv pool):'
```
Retrieval: `podman logs frank-large` covers the **current** instance only —
Retrieval: `podman logs frank-dragoon` covers the **current** instance only —
the Quadlet runs `--replace --rm`, so everything before the last recreate is
gone from the container log. `journalctl --user -u frank-large.service` is
the durable record. Both are rootless: `sudo podman logs frank-large` looks
gone from the container log. `journalctl --user -u frank-dragoon.service` is
the durable record. Both are rootless: `sudo podman logs frank-dragoon` looks
in root's storage and answers "no such container".
## Rollback
@@ -466,7 +466,7 @@ serve script and the Qwen-Sharp template are archived in the repo under
is a `git revert` of this change set plus a deploy — **but the Quadlet must be
removed from `~/.config/containers/systemd/` first**, or the restarted root
unit and the still-installed container both claim port 7770. The port gate in
the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
the handler now covers exactly that race. See [[frank-dragoon-ik-uncensored]].
The `amd_iommu=off` kernel argument that came with this change is separate and
reverts on its own — see [[dankclaw-iommu-translation-off]].
@@ -1,7 +1,7 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-secondary on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
@@ -16,7 +16,7 @@ timestamp: 2026-08-26
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
> **Active 2026-08-26.** `frank-secondary` switched from upstream
> **Active 2026-08-26.** `frank-zealot` switched from upstream
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
> RTX 3090. This page documents the build, the mandatory FA build flags, and
> the MoE / CPU-offload runtime arguments.
@@ -101,8 +101,8 @@ a hard compile error.
## Runtime configuration
Files: `scripts/serve-frank-secondary.sh`, `scripts/frank-secondary.env`,
`scripts/frank-secondary.service` in the repo. The service just calls
Files: `scripts/serve-frank-zealot.sh`, `scripts/frank-zealot.env`,
`scripts/frank-zealot.service` in the repo. The service just calls
the serve script; the env file holds all tunables.
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
@@ -241,7 +241,7 @@ apply.
The previous runtime `upstream llama.cpp b10121` and its binary
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
rollback to the old stack, change `LLAMA_SERVER=` in
`scripts/frank-secondary.env` back to that path and
`scripts/frank-zealot.env` back to that path and
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
spec / `--spec-type` flags. The model file is not changed either way.
+67 -67
View File
@@ -1,6 +1,6 @@
# LLM Fleet — Knowledge Base
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
@@ -8,15 +8,15 @@ driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
and frank-primary-vllm-shapeG-20261003),
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
and frank-archon-vllm-shapeG-20261003),
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
ctx, vision ON)** and
**llama.cpp CUDA (frank-secondary, port 7771: HauhauCS Gemma4-26B-A4B QAT
**llama.cpp CUDA (frank-zealot, port 7771: HauhauCS Gemma4-26B-A4B QAT
Uncensored Q4_K_M GGUF — 26B MoE hybrid (25 sliding-window + 5 full-attention
layers), ONE 200,704-token conversation, MTP disabled,
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
@@ -35,7 +35,7 @@ graph LR
A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
B2[radiance StillDeadcode, digest-pinned image
podman run, frank-primary, port 7770]
podman run, frank-archon, port 7770]
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
@@ -46,11 +46,11 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 114 GiB GTT]
E[flash-server container gfx1151, digest-pinned image
frank-large, port 7770]
frank-dragoon, port 7770]
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
2 x 200192 ctx, MTP always-on, image input]
G[llama.cpp master CUDA sm_86
frank-secondary, port 7771]
frank-zealot, port 7771]
H[Gemma4-26B-A4B QAT Uncensored Q4_K_M GGUF
1 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
E2 --> E --> F
@@ -71,7 +71,7 @@ frank-secondary, port 7771]
## Active Deployment
- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
- [[frank-dragoon-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
(root systemd unit + podman, port 7770): the digest-pinned
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
@@ -88,7 +88,7 @@ frank-secondary, port 7771]
greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens.
Rollback: revert the swap commit; the 27B payload + fetch script stay
on disk
- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
(rollback target; payload + fetch script + image on disk)**: the
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
@@ -99,24 +99,24 @@ frank-secondary, port 7771]
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
foreground-`podman run` ledger capture and the deep-role rerouting
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
engine moved with the model
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
it, 21,890/2,235 without).
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
MTP back on since
@@ -139,7 +139,7 @@ frank-secondary, port 7771]
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
(77decb5..380aafa) but never deployed
- [[frank-secondary-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
@@ -148,15 +148,15 @@ frank-secondary, port 7771]
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
the LIVE payload when Swift cut over. The ubatch A/B table and the
probe-size trap (>=256 px images or the model says "video") live inside
- [[frank-secondary-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
target; payload on disk; owns the measured boot-pair table)**: the same
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
104.8 tok/s, 3,571 MiB true free
- [[frank-primary-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
- [[frank-archon-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
replaces the FP8-native payload when the gates pass
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
- [[frank-archon-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
@@ -171,12 +171,12 @@ frank-secondary, port 7771]
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
`/metrics` scraped. Rollback: git tag
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
`frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
rocm714_torch214) built on the host into the local image
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
`frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
curated --no-deps install per docs/install_amd.md), serving the SAME
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
@@ -198,8 +198,8 @@ frank-secondary, port 7771]
and a mounted compile cache. Why `--max-num-seqs`
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
the tcclaviger RDNA4 vLLM fork
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
@@ -215,10 +215,10 @@ frank-secondary, port 7771]
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; payload and R9V state dirs intact on disk, git tag
frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(same-day swap to the Q8_0 payload; rollback target, state + payload
intact on disk): the R9V radiance-vLLM fork serving
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
@@ -226,8 +226,8 @@ frank-secondary, port 7771]
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
from llama.cpp the same day; rollback target
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
[[frank-archon-llamacpp-qwen38-flash-next-q8]]
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
@@ -236,7 +236,7 @@ frank-secondary, port 7771]
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
conversation, `reasoning_content`, the placement law, the cutover and
rollback
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
rollback target; active for hours): the Lucebox DS4.1 engine
(localhost/lucebox-rocm:gfx1201) serving
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
@@ -259,8 +259,8 @@ frank-secondary, port 7771]
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
[[frank-primary-lucebox-ds4]], then
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
[[frank-archon-lucebox-ds4]], then
[[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
@@ -288,15 +288,15 @@ frank-secondary, port 7771]
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
runtime it replaced
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
frank-large: the FreeToken v0.1.2 frank-secondary runbook (batsclamp
frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
[[frank-secondary-qwen38-gsq-rco]]
[[frank-zealot-qwen38-gsq-rco]]
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
10-05 (rollback target; payload, fork build and build script on disk)**:
the pinned ROCmFPX fork serving the uncensored protected-heads
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
@@ -304,21 +304,21 @@ frank-secondary, port 7771]
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
architecture.
- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
payload dirs were removed from disk 2026-10-03, so this rollback is a
re-download)** (root systemd unit, port 7770): gufo
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
workarounds, the client-contract delta, the measured prefill/decode
trade against llama.cpp
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
rollback by Frank after the autoresearch loop): the digest-pinned
kyuz0/vllm-therock-gfx1151 image serving
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
@@ -327,68 +327,68 @@ frank-secondary, port 7771]
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
v22.4.0 template, the 114 GiB GTT memory model, client contract,
troubleshooting
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
114 GiB GTT memory model, client contract, troubleshooting
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
head, the 114 GiB GTT memory model, client contract, troubleshooting
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
v4.4.1 runtime (private host build, QSA hotfix) serving
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
gate, client contract, troubleshooting (2026-09-19 → 09-21)
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
[[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract,
troubleshooting (2026-09-16 → 09-19)
- [[frank-secondary-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
[[frank-secondary-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
[[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
build) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
GGUF has no vision), reasoning_content, no spec decode; payload still at
`/home/dank/models/MiMo-Q6K/`
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
by [[frank-secondary-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
express; venv still on disk, INT8 payload staged to
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
83.9 tok/s decode
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
[[frank-secondary-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
[[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
token pool, 2 concurrent full-context sessions, native text+image+audio.
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
f16 KV at 131k
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
@@ -398,7 +398,7 @@ frank-secondary, port 7771]
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
that invalidated the first round, and the iGPU-only result that matches
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
@@ -407,7 +407,7 @@ frank-secondary, port 7771]
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
what it cost (the NPU endpoint), what was deliberately not copied from
the reference machine, and how to revert
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
@@ -419,9 +419,9 @@ frank-secondary, port 7771]
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
on Dankclaw (retired)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-secondary, retired 2026-09-02)
Dankserver (frank-zealot, retired 2026-09-02)
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
frank-secondary (2026-08-26): q8_0 + Hadamard KV cache,
frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
@@ -429,9 +429,9 @@ frank-secondary, port 7771]
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
the flash-server v2 runtime since 2026-10-06, frank-secondary on the RTX 3090)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
the retirement record — the IOMMU/SVA err=-19 root cause and where
@@ -455,7 +455,7 @@ frank-secondary, port 7771]
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (this architecture family is what the retired frank-secondary
notes and sizing (this architecture family is what the retired frank-zealot
served in the Huihui abliterated FP8 variant — see
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
that preceded it is archived at [[dealign-ornith-crack]])
@@ -471,7 +471,7 @@ frank-secondary, port 7771]
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
@@ -529,7 +529,7 @@ These sit alongside the operational pages under
## Related
- [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
git-based deploy with inference isolation
+4 -4
View File
@@ -17,10 +17,10 @@ timestamp: 2026-08-22
> **Retired 2026-08-22.** This experiment was deployed, benchmarks were
> collected, and the service reverted to v3 the same day (operator
> decision). `frank-large` serves v3 again. This page is kept as
> decision). `frank-dragoon` serves v3 again. This page is kept as
> the retired experiment's build runbook and reference.
`frank-large` (port 52625) briefly served
`frank-dragoon` (port 52625) briefly served
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` through the Kairic Edge CIRU ROCmFPX
llama.cpp runtime. The runtime routes qualified prompt and multi-token
verification shapes through the RDNA 3.5 native
@@ -215,7 +215,7 @@ service restart confirms the fresh tree, then it is a deletion candidate
The service runs the pinned runner profile plus the local chat
deviations below. Tunables live in
`scripts/frank-large.env`; flags in `scripts/serve-frank-large.sh`.
`scripts/frank-dragoon.env`; flags in `scripts/serve-frank-dragoon.sh`.
See [[deployment]] for the update process.
| Setting | Value | Origin |
@@ -301,7 +301,7 @@ ubatches do not. Production runs the runner's qualified 2048/512.
The v3 payload and binary stay on the machine for rollback.
1. Stop the service: `sudo systemctl stop frank-large`
1. Stop the service: `sudo systemctl stop frank-dragoon`
2. Restore the v3 scripts:
`git revert 4c43839 7aee1df 40d50fe` (or checkout the script pair from
commit `132e147`)
@@ -8,7 +8,7 @@ tags:
timestamp: 2026-09-15
---
# frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
# frank-dragoon: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
> **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the
> separate `draft-mtp` head) now serves Dankserver as **frank-medium** on the
@@ -21,7 +21,7 @@ the `set_rows` build bug** — the model loads but crashes in
`ggml_compute_forward_set_rows` before it can serve. All config goals are met;
a build fix is required for the model to run.
**Alias:** `frank-large`, port **7770**.
**Alias:** `frank-dragoon`, port **7770**.
The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF
from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a
@@ -42,8 +42,8 @@ fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` |
| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on |
| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` |
| Tunables | `scripts/frank-large.env` |
| Serve script | `scripts/serve-frank-large.sh` |
| Tunables | `scripts/frank-dragoon.env` |
| Serve script | `scripts/serve-frank-dragoon.sh` |
## Serving status: the `set_rows` build bug (blocks serving)
@@ -123,16 +123,16 @@ dl mtp-Qwen3.8-27B-Q4_0.gguf
```
The serve script checks each artifact is present and readable before launch
(`scripts/frank-large.env` -> the four `MODEL_*` files plus the chat
(`scripts/frank-dragoon.env` -> the four `MODEL_*` files plus the chat
template). A missing file fails the artifact check with `missing or
unreadable required artifact`.
## Serving profile
`scripts/serve-frank-large.sh` builds the `llama-server` call from the env:
`scripts/serve-frank-dragoon.sh` builds the `llama-server` call from the env:
```
/dev/llama-server -m <main> --alias frank-large
/dev/llama-server -m <main> --alias frank-dragoon
--host 0.0.0.0 --port 7770 --jinja
-dev ROCm0 -ngl 999 -np 1 -c 262144
-b 512 -ub 512 -fa on
@@ -213,7 +213,7 @@ and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the
## Env tunables
`scripts/frank-large.env`. Serving: `CTX_SIZE=262144`,
`scripts/frank-dragoon.env`. Serving: `CTX_SIZE=262144`,
`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`,
`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`,
`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`,
@@ -1,7 +1,7 @@
---
type: reference
title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-large.
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-dragoon.
tags:
- llama.cpp
- rocm
@@ -15,7 +15,7 @@ timestamp: 2026-08-24
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
> **RETIRED 2026-08-29.** This payload was `frank-large` from 2026-08-25
> **RETIRED 2026-08-29.** This payload was `frank-dragoon` from 2026-08-25
> until the Q8_0 AGENT experiment on 2026-08-29 (it was restored again
> 2026-09-01 and served until 2026-08-30, when [[agentionai-fast-imatrix]]
> took over). The Q8_0 model was retired because it crashed before
@@ -24,7 +24,7 @@ timestamp: 2026-08-24
> This page documents a retired model and its runbook.
`frank-large` (port 7770) serves
`frank-dragoon` (port 7770) serves
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
@@ -1,7 +1,7 @@
---
type: reference
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
description: PRIOR frank-large runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
description: PRIOR frank-dragoon runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
tags:
- llama.cpp
- rocm
@@ -16,7 +16,7 @@ timestamp: 2026-08-30
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-large moved to
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-dragoon moved to
> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
> runtime's prefill. Everything below documents the llama.cpp era and stays
@@ -33,10 +33,10 @@ timestamp: 2026-08-30
> (the card ships no spec flags; the Flash-Next MTP heads do not load on
> this loader family — see MTP Draft Head).
`frank-large` (port 7770) serves
`frank-dragoon` (port 7770) serves
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
via `scripts/serve-frank-large.sh` + `scripts/frank-large.env`
via `scripts/serve-frank-dragoon.sh` + `scripts/frank-dragoon.env`
and the Qwen-Sharp chat template.
## Model Architecture
@@ -188,7 +188,7 @@ cmake --build build --target llama-server llama-quantize -j16
## Serving Profile
Deployed args (`scripts/frank-large.env` — our validated
Deployed args (`scripts/frank-dragoon.env` — our validated
profile, unchanged from the FAST deployment except the binary and model):
```
@@ -282,9 +282,9 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
1. Build the runtime (see Runtime build) BEFORE touching the service.
2. Point the model file's directory at the canonical path (a plain `mv`
— never delete a 113 GiB payload to re-fetch it).
3. Edit `scripts/frank-large.env` (+ serve script defaults) via
3. Edit `scripts/frank-dragoon.env` (+ serve script defaults) via
git: commit, push, `./scripts/deploy.sh dankclaw --service
frank-large`.
frank-dragoon`.
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
`/v1/chat/completions` probe.
@@ -294,8 +294,8 @@ binary and the agentionai payload remain on disk.
### Config files
- Env: `scripts/frank-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Env: `scripts/frank-dragoon.env`
- Serve: `scripts/serve-frank-dragoon.sh`
- Chat template: `scripts/chat-template.jinja`
## Known Issues
@@ -8,11 +8,11 @@ timestamp: 2026-09-03
# Ling-3.0-flash Q6_K Trial on Dankclaw
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. At this page's retirement date `frank-large` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
## Status
- Service: `frank-large-ling` (systemd).
- Service: `frank-dragoon-ling` (systemd).
- Model: `inclusionAI/Ling-3.0-flash` Q6_K (MoE, ~118B total params).
- Runtime: llama.cpp **fork** `aetherbird/llama.cpp` branch
`bailingmoe3-support` (commit `db4480bc8`, build b10411) at
@@ -77,14 +77,14 @@ point varies (385-1588 tokens). Single streams never freeze.
`SPEC_TYPE=none`, `PARALLEL=2`. All concurrent streams complete (EOS).
Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with
MTP. Fallback chain if spec-off regresses: A) `PARALLEL=1` keep MTP,
B) switch back to Qwen3.6-27B (frank-large, MTP n_max 3).
B) switch back to Qwen3.6-27B (frank-dragoon, MTP n_max 3).
## Files
- `scripts/frank-large-ling.service` - systemd unit.
- `scripts/frank-dragoon-ling.service` - systemd unit.
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
draft-mtp).
- `scripts/frank-large.ling.env` - tunables.
- `scripts/frank-dragoon.ling.env` - tunables.
Passwordless management is installed:
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
@@ -92,9 +92,9 @@ Passwordless management is installed:
## Switch Back (obsolete)
The switch-back commands below are obsolete. At retirement time
`frank-large` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
re-enable `frank-large-ling` from the repo unit
(`scripts/frank-large-ling.service`) and serve script
`frank-dragoon` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
re-enable `frank-dragoon-ling` from the repo unit
(`scripts/frank-dragoon-ling.service`) and serve script
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
## Caveats
@@ -111,6 +111,6 @@ re-enable `frank-large-ling` from the repo unit
## See Also
- [[frank-large-benchmark]]
- [[frank-dragoon-benchmark]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
@@ -15,6 +15,6 @@ This page was the plan for a vLLM-ROCm deployment of
Dankclaw. The deployment never shipped. The actual Ling work used the
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
The trial service (`frank-large-ling`) is retired and disabled under the
boot policy (2026-08-17). `frank-large` is the Qwen3.8-27B PP
The trial service (`frank-dragoon-ling`) is retired and disabled under the
boot policy (2026-08-17). `frank-dragoon` is the Qwen3.8-27B PP
head, not the Ling path. See git history for the full decision table.
@@ -14,7 +14,7 @@ timestamp: 2026-09-03
# Qwen3.8-27B Speculative-Decoding Matrix
This page records the speculative-decoding benchmark history and the
production tuning for the Qwen3.8-27B service (`frank-large`),
production tuning for the Qwen3.8-27B service (`frank-dragoon`),
now a pipeline-parallel head with a Dankserver worker.
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
@@ -154,7 +154,7 @@ that is the long-context attention penalty, not a spec artifact.
## Production Configuration
The live tunables live in `scripts/frank-large.env` (head) and
The live tunables live in `scripts/frank-dragoon.env` (head) and
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
state section at the top of this page records the current values: Q6_K
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
@@ -162,6 +162,6 @@ q8_0/f16 KV.
## See Also
- [[frank-large-benchmark]]
- [[frank-dragoon-benchmark]]
- [[ling-3.0-flash-trial]]
- [[deployment]]
@@ -15,7 +15,7 @@ and the test matrix planned for 2026-08-16 morning.
Related pages:
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
- [Backend and spec benchmark](frank-large-benchmark.md)
- [Backend and spec benchmark](frank-dragoon-benchmark.md)
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
@@ -66,7 +66,7 @@ Consequences of the hybrid design:
## Current production state (baseline)
`frank-large` (Dankclaw, Vulkan0, 2026-08-15):
`frank-dragoon` (Dankclaw, Vulkan0, 2026-08-15):
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
@@ -233,7 +233,7 @@ tensor cache warm).
| E | A with n_max 3 and 5 | n_max curve on the split |
| F | A, plain Q8_0 model file | same-bit A/B |
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
| Ref | production frank-large (Vulkan, 2 slots) | the bar to beat |
| Ref | production frank-dragoon (Vulkan, 2 slots) | the bar to beat |
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
@@ -253,5 +253,5 @@ production @16k) with the same 8-bit model.
`-dio` for RPC).
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
- Local benchmarks: [frank-large-benchmark](frank-large-benchmark.md),
- Local benchmarks: [frank-dragoon-benchmark](frank-dragoon-benchmark.md),
[qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
@@ -4,9 +4,9 @@ Campaign branch `swift15-radiance-w4dn8`. Goal: convert
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB) to
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (GPTQ gate_up w4nl64 + block-fp8 down +
10 protected bf16 experts + int8 trunk + fixed-scale PLE) and swap it into
frank-primary on the 1.0.13 pin. Design settled (25-question grill); this log
frank-archon on the 1.0.13 pin. Design settled (25-question grill); this log
carries execution facts only. The cutover runbook
`frank-primary-radiance-swift15-w4dn8.md` replaces this page at cutover.
`frank-archon-radiance-swift15-w4dn8.md` replaces this page at cutover.
## 2026-10-06 evening
@@ -44,7 +44,7 @@ carries execution facts only. The cutover runbook
≥126,000 MiB), `apply-protected-cast.sh` (derive → recipe PROTECTED
section, refuses double-derive), `kld-gate-swift15.sh` (candidate vs
reference on 1.0.13, gate mean KL ≤ 0.03 / top-1 ≥ 95 %).
- `frank-primary` showing `failed` during the campaign is the 143-on-stop
- `frank-archon` showing `failed` during the campaign is the 143-on-stop
signature of the foreground podman unit — cosmetic, not a crash.
## 2026-10-07 morning
@@ -1,6 +1,6 @@
---
type: benchmark plan
title: frank-large ubatch sweep
title: frank-dragoon ubatch sweep
description: Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment.
tags:
- llama.cpp
@@ -10,9 +10,9 @@ tags:
timestamp: 2026-09-01
---
# frank-large ubatch sweep
# frank-dragoon ubatch sweep
This benchmark uses the active `frank-large` model on Dankclaw with
This benchmark uses the active `frank-dragoon` model on Dankclaw with
`llama-bench`. It does not use `llama-server`, HTTP requests, or server-log
scraping. Stop the production service before running it and start the service
again afterward.
@@ -29,9 +29,9 @@ Each point runs three 128-token repetitions by default.
Run on Dankclaw from the repository checkout:
```bash
sudo systemctl stop frank-large
sudo systemctl stop frank-dragoon
./scripts/bench-ubatch.sh
sudo systemctl start frank-large
sudo systemctl start frank-dragoon
```
The script uses the active Q6 AGENT model, an 8K prompt workload, equal
@@ -8,7 +8,7 @@ timestamp: 2026-10-05T00:00:00-07:00
# vLLM Boot Forensics and Config Interrogation
For serve-script vLLM endpoints (frank-primary/qwen-fp8 on dankserver, any
For serve-script vLLM endpoints (frank-archon/qwen-fp8 on dankserver, any
`podman rm -f` + restart service).
## 1. Capture the true cause — the journal, not `podman logs`
@@ -204,4 +204,4 @@ clipping at depth) starts with sections 1–3 here before touching anything.
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — probe/ladder hygiene on live hosts
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — probe/ladder hygiene on live hosts
@@ -302,7 +302,7 @@ consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
### Window discipline and verification
- Scratch boots on dankserver MUST stop + runtime-mask production
`frank-primary` first — the expert-offload planner reads `MemAvailable` at
`frank-archon` first — the expert-offload planner reads `MemAvailable` at
plan time, and beside-prod boots die at plan with
`ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
@@ -249,13 +249,13 @@ distinguishes a real split from a mislabelled sweep.
The fleet's answer is unchanged — **two independent endpoints**
([[halogen-flash-server]] on the iGPU via halogen,
[[frank-secondary-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
[[frank-zealot-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
claims from the first round of this experiment are now corrected:
1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode),
and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint
(`frank-secondary` holds 22.4 of the card's 24.6 GiB). An operator who wants the
(`frank-zealot` holds 22.4 of the card's 24.6 GiB). An operator who wants the
7 % must give up the fleet's only image-capable endpoint.
2. **Per-tensor placement across two devices is impossible while `-ot` is used**,
for a scheduler-mechanical reason, regardless of driver or vendor mix.
@@ -309,11 +309,11 @@ Operational notes for the next run:
`GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
host's co-tenants.
- `systemctl mask` is **refused** for a relinked root unit: frank-secondary's
`/etc/systemd/system/frank-secondary.service` is a symlink into the repo, so
- `systemctl mask` is **refused** for a relinked root unit: frank-zealot's
`/etc/systemd/system/frank-zealot.service` is a symlink into the repo, so
`mask` exits with "File ... already exists". Stop it and rely on the absence
of a restart trigger, or move the symlink aside.
- Restarting frank-large needs the quadlet source back in
- Restarting frank-dragoon needs the quadlet source back in
`~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
the unit from that file only. After the restore (and the host reboot, see the
landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB