llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
@@ -115,7 +115,7 @@ unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
|
||||
- Internal order is fixed (ngram types first, then drafters); CLI order is
|
||||
irrelevant.
|
||||
- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
|
||||
on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
|
||||
on the qwen35 hybrid (frank-dragoon era) measured −15 % prose / −7 %
|
||||
block-copy (2026-10-03): adding the ngram layer logs `recurrent state
|
||||
rollback is not compatible with 'ngram-mod' - disabling rollback support`,
|
||||
and MTP verification depends on that rollback, so acceptance drops on ALL
|
||||
@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
The Dankserver llama.cpp deployment (`frank-zealot`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-zealot` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
|
||||
|
||||
@@ -250,7 +250,7 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
`frank-large` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
|
||||
`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
|
||||
|
||||
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
@@ -315,7 +315,7 @@ not switch without re-measuring".
|
||||
|
||||
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
|
||||
slot split, quant rung — not just spec flags) is the transient systemd
|
||||
drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
|
||||
drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
|
||||
|
||||
## Is it worth it? (arithmetic before deployment)
|
||||
|
||||
@@ -419,4 +419,4 @@ measurement that would settle it.
|
||||
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
|
||||
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
|
||||
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle
|
||||
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
|
||||
@@ -22,7 +22,7 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
||||
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
|
||||
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
|
||||
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
|
||||
hold on the 8060S. See [[frank-large-benchmark]].
|
||||
hold on the 8060S. See [[frank-dragoon-benchmark]].
|
||||
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
||||
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
||||
`performance`, forced level `auto`. This item is N/A.
|
||||
@@ -48,7 +48,7 @@ lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
|
||||
different, driver-reported figure that has always been larger than the kernel
|
||||
cap.
|
||||
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
|
||||
limits documented in [[frank-large-benchmark]].
|
||||
limits documented in [[frank-dragoon-benchmark]].
|
||||
|
||||
## 1. Self-Hosted Agentic Tools
|
||||
|
||||
@@ -106,7 +106,7 @@ All require root and a reboot for the modprobe change.
|
||||
|
||||
## Sources
|
||||
|
||||
- [[frank-large-benchmark]] — measured backend/spec numbers
|
||||
- [[frank-dragoon-benchmark]] — measured backend/spec numbers
|
||||
- [[deployment]] — deploy and update process
|
||||
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
|
||||
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
|
||||
@@ -115,7 +115,7 @@ All require root and a reboot for the modprobe change.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-large-benchmark]]
|
||||
- [[frank-dragoon-benchmark]]
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[deployment]]
|
||||
- [[index]]
|
||||
|
||||
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
|
||||
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
|
||||
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
|
||||
|
||||
The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
|
||||
The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-zealot-gemma4-12b]]).
|
||||
|
||||
### ROCm
|
||||
|
||||
@@ -101,7 +101,7 @@ cable between claw and server is required for 10G.
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]].
|
||||
- The active service is `frank-dragoon` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-dragoon-gufo-qwen38-27b]].
|
||||
- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
|
||||
@@ -17,7 +17,7 @@ timestamp: 2026-09-20
|
||||
|
||||
Dankserver is a media/library server and the `frank-medium` LLM host. The
|
||||
host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
|
||||
FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
|
||||
FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and replaced with
|
||||
two AMD Radeon AI PRO R9700 cards.
|
||||
|
||||
## Hardware Specifications
|
||||
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
|
||||
The active inference service is `frank-medium.service`: mainline llama-server on
|
||||
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
|
||||
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
|
||||
`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
|
||||
its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
|
||||
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
|
||||
its unit is archived in `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`,
|
||||
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
|
||||
See [[deployment]].
|
||||
|
||||
|
||||
@@ -13,12 +13,12 @@ timestamp: 2026-09-03
|
||||
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
|
||||
> The `frank-secondary` name and port 52625 then belonged to the vLLM
|
||||
> The `frank-zealot` name and port 52625 then belonged to the vLLM
|
||||
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
|
||||
|
||||
# FastFlowLM NPU Inference
|
||||
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-zealot` (port 52625) serves in production. See Promotion.
|
||||
|
||||
## Verdict (2026-08-16)
|
||||
|
||||
@@ -167,7 +167,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
|
||||
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
|
||||
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
|
||||
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
|
||||
- The `llama-server` (frank-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
|
||||
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
|
||||
|
||||
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
|
||||
|
||||
@@ -185,7 +185,7 @@ Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens)
|
||||
|
||||
## Model Bakeoff (measured 2026-08-16, 4 models)
|
||||
|
||||
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
|
||||
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
|
||||
|
||||
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|
||||
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
|
||||
@@ -203,21 +203,21 @@ Notes:
|
||||
|
||||
## Promotion (2026-08-16)
|
||||
|
||||
`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
|
||||
### Boot policy (2026-08-17)
|
||||
|
||||
The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable frank-secondary
|
||||
sudo systemctl enable frank-zealot
|
||||
```
|
||||
|
||||
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
|
||||
|
||||
The PP split was renamed `frank-pp-head` → `frank-large` (2026-08-16, commit `48142f5`).
|
||||
The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
|
||||
|
||||
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
|
||||
|
||||
@@ -353,7 +353,7 @@ installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
|
||||
|
||||
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
|
||||
@@ -390,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
|
||||
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
|
||||
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
|
||||
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
|
||||
- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
- **NPU production role (2026-08-16).** `frank-zealot` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
|
||||
## Citations
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ read the returned content:
|
||||
```python
|
||||
import json, urllib.request
|
||||
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
||||
json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
|
||||
json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
|
||||
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
||||
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
||||
```
|
||||
|
||||
@@ -119,5 +119,5 @@ re-download.
|
||||
|
||||
- [[Llama Cpp Runtimes]] — which fork ships which custom types
|
||||
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
|
||||
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
|
||||
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|
||||
|
||||
@@ -13,7 +13,7 @@ Fleet law first: **STOP the production service before any GPU experiment**
|
||||
GPU and corrupts results; an unisolated GPU experiment has precedent for
|
||||
wedging a host until power cycle. Reference harnesses:
|
||||
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
|
||||
`scripts/autoresearch/frank-primary-arms.sh`.
|
||||
`scripts/autoresearch/frank-archon-arms.sh`.
|
||||
|
||||
## 1. The window structure (one config per arm, a driver owns the window)
|
||||
|
||||
@@ -27,7 +27,7 @@ wedging a host until power cycle. Reference harnesses:
|
||||
with `container state improper`.
|
||||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||||
LLM hosts — one card, one server). The serve script takes its config via
|
||||
env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
|
||||
pointing at a staged copy; the checkout is never mutated.
|
||||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||||
@@ -245,17 +245,17 @@ nearly shipped the wrong setting.
|
||||
the comment); verify multi-line command construction with `bash -n` AND a
|
||||
render test.
|
||||
|
||||
## 8. Results location and recovery (frank-primary arms)
|
||||
## 8. Results location and recovery (frank-archon arms)
|
||||
|
||||
| copy | path |
|
||||
|---|---|
|
||||
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
|
||||
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
|
||||
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
|
||||
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
|
||||
|
||||
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
constants 99.9/88.8 identical across arms by construction; **identical
|
||||
numbers across arms == stub or bug, never a measurement.** After compaction,
|
||||
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
|
||||
|
||||
@@ -161,8 +161,8 @@ of the same RAM, not extra memory.
|
||||
## 7. IBS model-access attribution (AMD CPUs)
|
||||
|
||||
Real CPU access patterns for mmap'd weights, live, no stop (established on
|
||||
frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
|
||||
`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
|
||||
frank-archon, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
|
||||
`scripts/frank-archon-expert-activity.py` (self-contained dump + sample +
|
||||
analyze; `--analyze-only`).
|
||||
|
||||
1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
|
||||
|
||||
@@ -80,7 +80,7 @@ co-tenants.
|
||||
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
|
||||
"move layers to the card" converts reclaimable cache into device memory and
|
||||
frees nothing for co-tenants. A served model's footprint ≠ weight size
|
||||
(frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
|
||||
(frank-zealot: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
|
||||
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
|
||||
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
|
||||
- **Context lengths: derive, never remember.** `llama-bench` computes
|
||||
@@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed
|
||||
## Related
|
||||
|
||||
- [[GGUF Compatibility]] — what a payload is before it is priced
|
||||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
|
||||
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
|
||||
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
|
||||
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
|
||||
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
|
||||
|
||||
@@ -8,8 +8,8 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
# Model Swap and Rollback
|
||||
|
||||
Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
|
||||
frank-primary). Repo law: all config flows through git; the live models are
|
||||
Applies to every endpoint swap on the fleet (frank-zealot / frank-dragoon /
|
||||
frank-archon). Repo law: all config flows through git; the live models are
|
||||
host state under `/home/dank/models/<name>/`.
|
||||
|
||||
## 1. Fit preflight — never deploy a swap without confirming it FITS
|
||||
|
||||
@@ -26,7 +26,7 @@ conclusions.
|
||||
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
|
||||
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
|
||||
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
|
||||
| ubatch | 256 | `UBATCH_SIZE` in `frank-large.env` |
|
||||
| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
|
||||
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
|
||||
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
|
||||
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
|
||||
|
||||
@@ -20,7 +20,7 @@ morning-of runbook. Research background:
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
This split is the production `frank-large` service. Current
|
||||
This split is the production `frank-dragoon` service. Current
|
||||
state, verified against the live journal and output smoke tests
|
||||
(2026-08-17, load 02:06):
|
||||
|
||||
@@ -41,17 +41,17 @@ state, verified against the live journal and output smoke tests
|
||||
Vulkan).
|
||||
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
|
||||
stable; all allocations are upfront).
|
||||
- Unit `frank-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-large.env`.
|
||||
- Unit `frank-dragoon.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-dragoon.env`.
|
||||
|
||||
## Service enablement (2026-08-17)
|
||||
|
||||
The PP pair is the only inference service enabled for boot. Both units
|
||||
(`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
|
||||
enabled. The NPU unit (`frank-secondary` on Dankclaw) and all retired
|
||||
(`frank-dragoon` on Dankclaw, `frank-pp-worker` on Dankserver) are
|
||||
enabled. The NPU unit (`frank-zealot` on Dankclaw) and all retired
|
||||
units are disabled. Restart order: worker first, then head. The Dankserver
|
||||
helper `frank-restart-small.sh` is obsolete (it restarts the removed
|
||||
`frank-secondary` unit); restart the worker with
|
||||
`frank-zealot` unit); restart the worker with
|
||||
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
|
||||
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
|
||||
head. `deploy.sh dankserver` is broken until retargeted. See
|
||||
@@ -173,9 +173,9 @@ llama-server b10433 ggml-rpc-server b10433
|
||||
|------|---------|---------|
|
||||
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
|
||||
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
|
||||
| `scripts/frank-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
|
||||
| `scripts/frank-dragoon.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
|
||||
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
|
||||
| `scripts/frank-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
|
||||
| `scripts/frank-dragoon.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
|
||||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||||
|
||||
@@ -229,7 +229,7 @@ FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
|
||||
```
|
||||
|
||||
Config variants: edit `scripts/frank-large.env` on the claw (git commit +
|
||||
Config variants: edit `scripts/frank-dragoon.env` on the claw (git commit +
|
||||
pull if the bench runs from the repo), then re-run `start` + `bench`:
|
||||
|
||||
- No spec baseline: `SPEC_NONE=1`
|
||||
|
||||
@@ -65,6 +65,6 @@ graph LR
|
||||
## See Also
|
||||
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[frank-secondary-benchmark]]
|
||||
- [[frank-zealot-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[Dankclaw System Specs]]
|
||||
|
||||
@@ -78,12 +78,12 @@ Laws that steer everything here:
|
||||
variants **[verified (tool existence: heretic; rest survey)]**.
|
||||
- Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm
|
||||
weights, so it composes with any expert quantization.
|
||||
- Fleet status: frank-large runs an abliterated base **by Frank's choice**
|
||||
- Fleet status: frank-dragoon runs an abliterated base **by Frank's choice**
|
||||
(AGENTS: censored-base refusals were the accepted trade of the 27B era);
|
||||
frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.
|
||||
|
||||
### HauhauCS QX_K_P "Perfect" quants
|
||||
- Our frank-secondary publisher. Method, from the publisher's own cards
|
||||
- Our frank-zealot publisher. Method, from the publisher's own cards
|
||||
**[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced"),
|
||||
(2) quantize with **imatrix calibration on the ABLITERATED weights**,
|
||||
(3) K_P = a **per-model, per-tensor mixed-precision profile** —
|
||||
@@ -106,7 +106,7 @@ Laws that steer everything here:
|
||||
|
||||
## Per-card playbooks
|
||||
|
||||
### RTX 3090 — the unlocked card (frank-secondary runs llama.cpp BY CHOICE)
|
||||
### RTX 3090 — the unlocked card (frank-zealot runs llama.cpp BY CHOICE)
|
||||
The 3090 is the only fleet device with no format lock: radiance needs
|
||||
`.rad`, halogen needs `.hgn`, and the v2 checkpoint loads in NO other
|
||||
engine. sm_86 + CUDA is the industry's default target, so the runtime
|
||||
@@ -179,7 +179,7 @@ Q8_K_P 40.6 GiB + mmproj 0.84) — Q4_K_P IS the protected-tensor blend.
|
||||
The self-made delta: calibration on OUR corpora, our hot-expert law,
|
||||
and a KLD gate scored at the serving shape.
|
||||
|
||||
### R9700 — frank-primary (radiance)
|
||||
### R9700 — frank-archon (radiance)
|
||||
The full self-quant pipeline already exists end to end: fetch BF16 →
|
||||
record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap,
|
||||
`RADIANCE_CALIB_DIR/TOKENS`) → derive protected experts (energy-share) →
|
||||
@@ -219,7 +219,7 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
|
||||
until proven), and does its MTP cast to the fp8/u2 encodings like
|
||||
qwen4exp. The MNBT-2048 disk-tier law is MOOT for a resident model.
|
||||
|
||||
### Strix Halo — frank-large (halogen) and friends
|
||||
### Strix Halo — frank-dragoon (halogen) and friends
|
||||
- `.hgn`: **closed production door.** No public quantizer exists
|
||||
(verified absence across peonist-ai repos); the format's lock is the
|
||||
engine's lock. We consume vendor artifacts; the repo-side tooling is
|
||||
@@ -255,6 +255,6 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
|
||||
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
|
||||
`scripts/build-kairic-edge-gfx1151.sh`, [[agentionai-fast-imatrix]],
|
||||
[[kingjones-strix-rocmfp4]], [[qwen3.8-strix-optimizations]],
|
||||
[[frank-primary-radiance-swift15-w4dn8]],
|
||||
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]],
|
||||
[[frank-large-flash-server-v2-swift15]], [[GGUF Compatibility]].
|
||||
[[frank-archon-radiance-swift15-w4dn8]],
|
||||
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]],
|
||||
[[frank-dragoon-flash-server-v2-swift15]], [[GGUF Compatibility]].
|
||||
|
||||
@@ -247,4 +247,4 @@ see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
|
||||
measured campaigns
|
||||
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — window discipline on live hosts
|
||||
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — window discipline on live hosts
|
||||
|
||||
@@ -8,7 +8,7 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
# RDNA4 MoE Expert Offload VRAM Budget
|
||||
|
||||
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-primary: Qwen3.8-
|
||||
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-archon: Qwen3.8-
|
||||
Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
|
||||
Tuning history and the deployed shape live in
|
||||
[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
|
||||
@@ -198,7 +198,7 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||||
`17.`):
|
||||
|
||||
```
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
|
||||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||||
```
|
||||
|
||||
@@ -206,7 +206,7 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||||
3. Verify after deploy: plan lines show new budget/graphs/arena;
|
||||
`GPU KV cache size` matches the ask; graph-estimate vs post-capture;
|
||||
`rocm-smi` high-water per card = the real margin;
|
||||
`systemctl show frank-primary -p NRestarts --value` = 0; a real chat
|
||||
`systemctl show frank-archon -p NRestarts --value` = 0; a real chat
|
||||
completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
|
||||
`localhost` inside an ssh session on the host — the latter returns `000`
|
||||
and looks like an outage).
|
||||
|
||||
@@ -55,7 +55,7 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
|
||||
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-dragoon.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
|
||||
|
||||
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
|
||||
|
||||
|
||||
@@ -122,7 +122,7 @@ export HF_HOME=/mnt/entropy/huggingface
|
||||
|
||||
### Active Service Conflict
|
||||
|
||||
The retired `frank-secondary` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
||||
The retired `frank-zealot` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
||||
|
||||
```
|
||||
nvidia-smi # verify GPU is free
|
||||
|
||||
@@ -10,9 +10,9 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||||
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
|
||||
first-class expert-offload planner) served `frank-primary` at shape G until
|
||||
first-class expert-offload planner) served `frank-archon` at shape G until
|
||||
the 2026-10-05 FreeToken cutover; it remains the rollback target
|
||||
(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
|
||||
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
|
||||
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
|
||||
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
@@ -213,7 +213,7 @@ through two image tags to bring an overlay up was entirely wasted work.
|
||||
## Compile-cache persistence (survives `podman rm -f`)
|
||||
|
||||
Serve scripts that recreate the container discard the writable layer, so
|
||||
every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
|
||||
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
|
||||
cold, of which 173 s was one graph).
|
||||
|
||||
1. **Prove it and locate it:** fresh container vs live
|
||||
@@ -241,7 +241,7 @@ cold, of which 173 s was one graph).
|
||||
## Plan-line verification checklist (after any fork deploy)
|
||||
|
||||
```
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
|
||||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||||
```
|
||||
|
||||
@@ -250,7 +250,7 @@ these columns to `17.`. Verify: new budget/graphs/arena in the plan;
|
||||
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
|
||||
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
|
||||
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
|
||||
real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
|
||||
real margin; `systemctl show frank-archon -p NRestarts --value` = 0;
|
||||
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
|
||||
completion from a PEER — `curl localhost:7770/health` inside an ssh session
|
||||
on the host can return `000` and looks like an outage that isn't.
|
||||
|
||||
@@ -244,7 +244,7 @@ vLLM supports multiple parallelism strategies:
|
||||
already uses them.** The tcclaviger fork is a vLLM fork, so the standard
|
||||
scheduler flags are present in the shipped image (verified by reading the
|
||||
`vllm.engine.arg_utils` and `vllm.config.scheduler` source in the live
|
||||
container `frank-primary-qwen`, not from a stub):
|
||||
container `frank-archon-qwen`, not from a stub):
|
||||
|
||||
| Flag | Default | What it controls |
|
||||
|---|---|---|
|
||||
@@ -268,14 +268,14 @@ ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
|
||||
|
||||
**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
|
||||
llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
|
||||
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-secondary
|
||||
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-zealot
|
||||
was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
|
||||
counterpart here, and the two backends should not be reasoned about by analogy.
|
||||
|
||||
**MNBT is not a lever on this shape either.** The deployed 2048 is the fork's
|
||||
default, so a change is a deliberate departure from the shipped value, not a
|
||||
tune. The one thing MNBT does cost: every change re-enters the compile RANGE
|
||||
path, so `FRANK_PRIMARY_CACHE_DIR` pays one recompilation.
|
||||
path, so `FRANK_ARCHON_CACHE_DIR` pays one recompilation.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
|
||||
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-large.
|
||||
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-dragoon.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
@@ -26,7 +26,7 @@ timestamp: 2026-08-29
|
||||
> `--flash-attn on` work on the charlie build-qwen38 tree — all still
|
||||
> true for that binary + model pair, which remains the rollback target.
|
||||
|
||||
`frank-large` (port 7770) served
|
||||
`frank-dragoon` (port 7770) served
|
||||
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
|
||||
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
|
||||
with the author's own MTP draft head
|
||||
@@ -166,8 +166,8 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
|
||||
|
||||
## Config files
|
||||
|
||||
- Env: `scripts/frank-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Env: `scripts/frank-dragoon.env`
|
||||
- Serve: `scripts/serve-frank-dragoon.sh`
|
||||
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
|
||||
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
|
||||
any clean fork rebuild — see Serving profile)
|
||||
@@ -175,10 +175,10 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
|
||||
## Swap procedure
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-large
|
||||
# edit scripts/frank-large.env + scripts/serve-frank-large.sh, commit, push
|
||||
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-large
|
||||
sudo systemctl start frank-large # or via deploy.sh
|
||||
sudo systemctl stop frank-dragoon
|
||||
# edit scripts/frank-dragoon.env + scripts/serve-frank-dragoon.sh, commit, push
|
||||
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-dragoon
|
||||
sudo systemctl start frank-dragoon # or via deploy.sh
|
||||
curl http://192.168.0.143:7770/health
|
||||
```
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: CIRU ActiveFPX PromptForge on Dankclaw
|
||||
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-large service.
|
||||
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-dragoon service.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
@@ -14,13 +14,13 @@ timestamp: 2026-08-22
|
||||
|
||||
# CIRU ActiveFPX PromptForge on Dankclaw
|
||||
|
||||
> **RETIRED 2026-08-24.** `frank-large` served this v3 model from
|
||||
> **RETIRED 2026-08-24.** `frank-dragoon` served this v3 model from
|
||||
> 2026-08-22 (restored after a same-day Kairic Edge IU4 experiment was
|
||||
> reverted — operator decision, commit `c60156f`; see [[kairic-edge]])
|
||||
> until [[eemin-rocmi4-mtp]] took over on 2026-08-24. Kept as the
|
||||
> historical record of the v3 deployment.
|
||||
|
||||
`frank-large` (port 7770, moved from 52625 on 2026-08-22) serves
|
||||
`frank-dragoon` (port 7770, moved from 52625 on 2026-08-22) serves
|
||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
|
||||
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
|
||||
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
|
||||
@@ -136,7 +136,7 @@ graph TD
|
||||
C --> D[Build llama-server]
|
||||
D --> E[Verify --list-devices shows gfx1151]
|
||||
E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
|
||||
F --> G[Deploy frank-large]
|
||||
F --> G[Deploy frank-dragoon]
|
||||
G --> H[Health check + chat completion]
|
||||
```
|
||||
|
||||
@@ -151,8 +151,8 @@ against the model card before the service restart.
|
||||
|
||||
## Service
|
||||
|
||||
- Unit: `frank-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
|
||||
- Unit: `frank-dragoon.service`, port 7770
|
||||
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
|
||||
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
|
||||
the system ROCm. `ROCM_PATH` points at the devel SDK.
|
||||
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
---
|
||||
type: research
|
||||
title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record
|
||||
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-large), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
|
||||
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-dragoon), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
|
||||
tags:
|
||||
- dankclaw
|
||||
- iommu
|
||||
- kernel
|
||||
- grub
|
||||
- prefill
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- halogen
|
||||
- frank-npu
|
||||
timestamp: 2026-09-11
|
||||
@@ -54,7 +54,7 @@ takes the NPU with it.
|
||||
1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and
|
||||
`/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure
|
||||
documented in [[frank-npu]]. Audio input left the fleet with it; image input
|
||||
survives on `frank-large` through halogen's vision tower.
|
||||
survives on `frank-dragoon` through halogen's vision tower.
|
||||
2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge
|
||||
and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the
|
||||
observability stack. DMA translation is now off for all of them. This was
|
||||
@@ -80,7 +80,7 @@ so only that one was taken:
|
||||
cat /proc/cmdline # expect amd_iommu=off, no iommu=pt
|
||||
ls /sys/kernel/iommu_groups/ | wc -l # expect 0 groups (was 30)
|
||||
systemctl is-active frank-npu # expect inactive (masked)
|
||||
curl -s http://127.0.0.1:7770/health # frank-large must still answer 200
|
||||
curl -s http://127.0.0.1:7770/health # frank-dragoon must still answer 200
|
||||
```
|
||||
|
||||
**To revert:** put `iommu=pt` back in `GRUB_CMDLINE_LINUX_DEFAULT` in place of
|
||||
|
||||
@@ -168,7 +168,7 @@ Not advisable.
|
||||
| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
|
||||
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
|
||||
| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
|
||||
| **radiance-vLLM** (drwolfen) | yes — served frank-primary 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
|
||||
| **radiance-vLLM** (drwolfen) | yes — served frank-archon 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
|
||||
| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
|
||||
| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
|
||||
| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
|
||||
@@ -203,7 +203,7 @@ reproducible **from a bf16 source**, not from our file. Uncensored-family
|
||||
bf16 source candidates: `windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF`
|
||||
(ships the bf16 GGUF) and `mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF`
|
||||
(i1 quants only). 622 G free fits source + container. A trial is:
|
||||
download → `rad-convert` (CPU-side) → stop+mask frank-primary window to
|
||||
download → `rad-convert` (CPU-side) → stop+mask frank-archon window to
|
||||
serve. Caveats on record: `--tp-wire wht6` is a lossy int8 all-reduce
|
||||
wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
|
||||
an opt-in f16 that changes generated text; every pool is budgeted
|
||||
@@ -221,7 +221,7 @@ gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-primary-lucebox-ds4]] — the retired Lucebox frank-primary runbook
|
||||
- [[frank-archon-lucebox-ds4]] — the retired Lucebox frank-archon runbook
|
||||
- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
|
||||
- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
|
||||
- [[frank-archon-r9v-radiance-qwen38-q8]] — the ACTIVE frank-archon
|
||||
|
||||
@@ -30,7 +30,7 @@ timestamp: 2026-09-29
|
||||
|
||||
This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
|
||||
image `docker.io/tcclaviger/vllm:latest`, engine
|
||||
`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-primary` systemd service —
|
||||
`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-archon` systemd service —
|
||||
the R9V production unit is untouched, so no stop-and-mask was required for
|
||||
any of this. All numbers are decode tok/s measured from the server's own
|
||||
`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
|
||||
@@ -58,7 +58,7 @@ Published to the host LAN via the serve script's `PORTMAP` hook
|
||||
(`PORTMAP=7773:7770`): health and completions answer on
|
||||
`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
|
||||
decode 36.49/34.55/35.98 through the published path). Host port 7773 was
|
||||
chosen over 7770 because `frank-primary` (7770) is stopped but ENABLED —
|
||||
chosen over 7770 because `frank-archon` (7770) is stopped but ENABLED —
|
||||
any future start would collide with a published 7770.
|
||||
|
||||
Device notes carried from the bring-up: `--device /dev/dri` must be the
|
||||
@@ -249,8 +249,8 @@ re-copies them; a recreate wipes container-local files).
|
||||
Two same-day cutovers moved this page's findings into production:
|
||||
|
||||
1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
|
||||
window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
|
||||
`frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
|
||||
window is documented in [[frank-archon-glm53-flash-w4a16]]; git tag
|
||||
`frank-archon-fp8-mtp-20261001` is the last commit of that shape). The
|
||||
production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
|
||||
measured 37.4-40.6 tok/s at ~54% acceptance.
|
||||
2. **The overnight redo campaign re-measured EVERY lever at matched
|
||||
@@ -278,20 +278,20 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=200704 200704-token window
|
||||
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
|
||||
FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
|
||||
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
|
||||
FRANK_PRIMARY_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
|
||||
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
|
||||
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit)
|
||||
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
|
||||
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
|
||||
FRANK_ARCHON_MAXLEN=200704 200704-token window
|
||||
FRANK_ARCHON_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
|
||||
FRANK_ARCHON_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
|
||||
FRANK_ARCHON_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
|
||||
FRANK_ARCHON_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
|
||||
FRANK_ARCHON_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
|
||||
FRANK_ARCHON_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
|
||||
FRANK_ARCHON_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
|
||||
FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit)
|
||||
FRANK_ARCHON_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
|
||||
FRANK_ARCHON_CACHE_DIR=/home/dank/.cache/frank-archon mounted at /root/.cache
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
FRANK_PRIMARY_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
|
||||
FRANK_ARCHON_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
|
||||
```
|
||||
|
||||
### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
|
||||
@@ -299,7 +299,7 @@ FRANK_PRIMARY_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the Q
|
||||
**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
|
||||
~40k tokens into a long conversation; short sessions stayed clean.
|
||||
|
||||
**Evidence (live `podman logs frank-primary-qwen`, the 2026-10-04 18:19
|
||||
**Evidence (live `podman logs frank-archon-qwen`, the 2026-10-04 18:19
|
||||
boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
|
||||
running request, and the engine's own 18:46:08 log line reading
|
||||
"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
|
||||
@@ -343,7 +343,7 @@ load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
|
||||
overclock); and a two-head, single-slot NaN state is a state-path fault,
|
||||
not scattered finite bit corruption.
|
||||
|
||||
**Fix.** `FRANK_PRIMARY_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
|
||||
**Fix.** `FRANK_ARCHON_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
|
||||
(the fork-named mitigation; the flag renders in both directions from the
|
||||
knob so a fork default change cannot silently flip the contract). Cost:
|
||||
no prefix reuse — every turn re-prefills the whole history instead of at
|
||||
@@ -388,7 +388,7 @@ conversation cannot distinguish fixed from unfixed; (2) a second
|
||||
long-context mechanism with no guard coverage — fp8 KV degradation at
|
||||
depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
|
||||
and would settle it). Prefix caching is **back ON**
|
||||
(`FRANK_PRIMARY_NO_PREFIX_CACHING` commented out — the cost bought no
|
||||
(`FRANK_ARCHON_NO_PREFIX_CACHING` commented out — the cost bought no
|
||||
fix). The knob and the serve branch stay for a future fork build with a
|
||||
fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
|
||||
it still breaks at ~40k with zero faults, A/B the KV dtype.
|
||||
@@ -472,7 +472,7 @@ running** — the bound tracks whatever the co-tenants hold at that moment. The
|
||||
demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
|
||||
boot where the bound drops near it silently converts next-hottest experts to
|
||||
VRAM-only (the `pinned_extra` warning is the only trace).
|
||||
`FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
|
||||
`FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
|
||||
on a host with 246.1 GiB total. Semantics verified in the shipped image
|
||||
(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
|
||||
warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
|
||||
@@ -484,7 +484,7 @@ was itself fitted at nbt 4096, which means 2048 is covered by an activation
|
||||
figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
|
||||
(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
|
||||
SEQS property). The costs are a one-time recompile (the compile **range** is
|
||||
part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
|
||||
part of the cache key, so `FRANK_ARCHON_CACHE_DIR` pays once) and a wider
|
||||
prefill chunk, which the 09-29 sweep found decode-interchangeable across
|
||||
1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
|
||||
|
||||
@@ -656,7 +656,7 @@ read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
|
||||
only available while the container lives).
|
||||
|
||||
Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
|
||||
`frank-archon-fp8-mtp-20261001` (env trio + seam re-pinned there).
|
||||
|
||||
## Parallelism options on this box (2026-10-01 survey)
|
||||
|
||||
@@ -696,9 +696,9 @@ more cards.
|
||||
run in the engine's namespace and the files must be copied in first:
|
||||
|
||||
```
|
||||
podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
|
||||
podman cp /home/dank/bench_iter.py frank-primary-qwen:/
|
||||
podman exec frank-primary-qwen python3 /mtp_measure.py
|
||||
podman cp /home/dank/mtp_measure.py frank-archon-qwen:/
|
||||
podman cp /home/dank/bench_iter.py frank-archon-qwen:/
|
||||
podman exec frank-archon-qwen python3 /mtp_measure.py
|
||||
```
|
||||
|
||||
Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
|
||||
@@ -717,7 +717,7 @@ reproducible across boots.
|
||||
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
|
||||
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
|
||||
V4.1 scale)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
|
||||
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
|
||||
fork era; production has been the tcclaviger fork since)
|
||||
- [[x4-pair-experiment]] — the same residency law measured on a narrower
|
||||
link: streaming over PCIe loses, computing in RAM does not care
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Dankserver VRAM and RAM Growth Under Load
|
||||
description: Investigation of the recurring cuBLAS crash on frank-secondary. What grows, what does not, and what to do next.
|
||||
description: Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next.
|
||||
tags:
|
||||
- dankserver
|
||||
- llamacpp
|
||||
@@ -18,7 +18,7 @@ timestamp: 2026-09-03
|
||||
# Dankserver VRAM and RAM Growth Under Load
|
||||
|
||||
This page records the investigation of the recurring cuBLAS crash on
|
||||
`frank-secondary` (llama.cpp, RTX 3090 24 GiB). The service aborts
|
||||
`frank-zealot` (llama.cpp, RTX 3090 24 GiB). The service aborts
|
||||
with `CUDA error: an unsupported value or parameter was passed to the
|
||||
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
|
||||
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
|
||||
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
|
||||
|
||||
## Mitigation Options
|
||||
|
||||
The documentation for the benchmark ([[frank-secondary-benchmark]])
|
||||
The documentation for the benchmark ([[frank-zealot-benchmark]])
|
||||
prescribes the first option for this crash class. None of these change
|
||||
the model.
|
||||
|
||||
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-secondary-benchmark]]
|
||||
- [[frank-zealot-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[deployment]]
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-secondary.
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-zealot.
|
||||
tags:
|
||||
- llamacpp
|
||||
- cuda
|
||||
@@ -14,12 +14,12 @@ timestamp: 2026-08-24
|
||||
|
||||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
|
||||
> **Active 2026-08-25.** `frank-secondary` serves this model. It
|
||||
> **Active 2026-08-25.** `frank-zealot` serves this model. It
|
||||
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
||||
> The Q6_K file was downloaded from the model repository and validated with
|
||||
> `llama-gguf` before installation.
|
||||
|
||||
`frank-secondary` (port 7770) serves
|
||||
`frank-zealot` (port 7770) serves
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
||||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||||
attention architecture, abliterated (uncensored). The GGUF also contains a
|
||||
|
||||
@@ -20,8 +20,8 @@ SearXNG, Caddy, or another container service does **not** restart inference.
|
||||
| Target | Host | Service | Port |
|
||||
|---|---|---|---:|
|
||||
| Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-secondary` | 7771 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-dragoon` | 7770 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-zealot` | 7771 |
|
||||
|
||||
## Scoped workflow
|
||||
|
||||
@@ -40,7 +40,7 @@ Run the unified deployer from the repository root:
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankserver --service frank-medium
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
./scripts/deploy.sh dankclaw --service frank-dragoon
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
```
|
||||
|
||||
@@ -51,23 +51,23 @@ on the selected host. The following files select the corresponding service:
|
||||
entrypoint and its healthcheck) selects `frank-medium` on Dankserver —
|
||||
the container mounts the entrypoint and healthcheck straight out of the
|
||||
checkout, so any file there changes deployed state.
|
||||
- `scripts/frank-large.env`, `frank-large.service`, and
|
||||
`serve-frank-large.sh` select `frank-large` on Dankclaw.
|
||||
- `scripts/frank-secondary.env`, `frank-secondary.service`, and
|
||||
`serve-frank-secondary.sh` select `frank-secondary` on Dankclaw.
|
||||
- `scripts/frank-dragoon.env`, `frank-dragoon.service`, and
|
||||
`serve-frank-dragoon.sh` select `frank-dragoon` on Dankclaw.
|
||||
- `scripts/frank-zealot.env`, `frank-zealot.service`, and
|
||||
`serve-frank-zealot.sh` select `frank-zealot` on Dankclaw.
|
||||
- Documentation, tests, deployment tooling, and unrelated container changes
|
||||
do not select any inference service.
|
||||
|
||||
The target-side `scripts/deploy-remote.sh` installs the selected service.
|
||||
`frank-large` and `frank-secondary` are **root units**: the handler relinks the
|
||||
`frank-dragoon` and `frank-zealot` are **root units**: the handler relinks the
|
||||
repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
|
||||
enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
|
||||
handler retires its predecessors (the replica Quadlets and the retired llama
|
||||
root unit), stops the service's own unit, waits for the published port to
|
||||
clear, installs the `.container` and restarts the generated unit. Each
|
||||
handler then waits on that service's health endpoint —
|
||||
`http://127.0.0.1:7770/health` for frank-large and frank-medium,
|
||||
`http://127.0.0.1:7771/health` for frank-secondary. The wait budget is 20 minutes
|
||||
`http://127.0.0.1:7770/health` for frank-dragoon and frank-medium,
|
||||
`http://127.0.0.1:7771/health` for frank-zealot. The wait budget is 20 minutes
|
||||
because cold model loads are slow. A failed handler does not advance its
|
||||
per-service state.
|
||||
|
||||
@@ -75,8 +75,8 @@ per-service state.
|
||||
|
||||
The root units point into the stable target checkout:
|
||||
```text
|
||||
/home/dank/serverAppConfigs/scripts/frank-large.service
|
||||
/home/dank/serverAppConfigs/scripts/frank-secondary.service
|
||||
/home/dank/serverAppConfigs/scripts/frank-dragoon.service
|
||||
/home/dank/serverAppConfigs/scripts/frank-zealot.service
|
||||
```
|
||||
`frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
|
||||
(`frank-medium.container`) rather than a unit in `scripts/`.
|
||||
@@ -110,7 +110,7 @@ change in that service's inputs.
|
||||
|
||||
If the LLM handler fails after the target pull:
|
||||
|
||||
1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-secondary
|
||||
1. Inspect `journalctl -u frank-dragoon -n 200` or `journalctl -u frank-zealot
|
||||
-n 200` (root units), or `journalctl --user -u frank-medium -n 200`
|
||||
(rootless Quadlet).
|
||||
2. Confirm the model files, the runtime binary and (for frank-medium) the
|
||||
@@ -119,14 +119,14 @@ If the LLM handler fails after the target pull:
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankserver --service frank-medium
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
./scripts/deploy.sh dankclaw --service frank-secondary
|
||||
./scripts/deploy.sh dankclaw --service frank-dragoon
|
||||
./scripts/deploy.sh dankclaw --service frank-zealot
|
||||
```
|
||||
4. Verify the matching health endpoint before declaring recovery.
|
||||
|
||||
Do not stop the other host's inference service during recovery. The two
|
||||
Dankclaw endpoints are independent processes on different devices: restarting
|
||||
frank-large does not disturb frank-secondary, and the reverse holds too — with one
|
||||
frank-dragoon does not disturb frank-zealot, and the reverse holds too — with one
|
||||
exception worth knowing, that the deployer waits for port 7770 to be released
|
||||
before it starts the replacement container. For benchmark work, follow the
|
||||
separate stop/benchmark/restart procedure in [[Testing Policy]] and the
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
title: DFlash2 on Dankclaw (frank-large)
|
||||
title: DFlash2 on Dankclaw (frank-dragoon)
|
||||
description: ARCHIVED - unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Retired 2026-08-21 for speed; kept as a runbook.
|
||||
tags:
|
||||
- llama.cpp
|
||||
@@ -13,14 +13,14 @@ tags:
|
||||
timestamp: 2026-08-21
|
||||
---
|
||||
|
||||
# DFlash2 on Dankclaw (frank-large) — ARCHIVED
|
||||
# DFlash2 on Dankclaw (frank-dragoon) — ARCHIVED
|
||||
|
||||
> **Archived 2026-08-21.** This DFlash2 Vulkan deployment was retired the
|
||||
> same day it shipped because it was slower than the CIRU runtime it
|
||||
> replaced. `frank-large` is back on CIRU. This page stays as the
|
||||
> replaced. `frank-dragoon` is back on CIRU. This page stays as the
|
||||
> runbook and provenance for the DFlash2 attempt.
|
||||
|
||||
`frank-large` (port 52625) briefly served
|
||||
`frank-dragoon` (port 52625) briefly served
|
||||
`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
|
||||
runtime was upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
|
||||
|
||||
@@ -114,14 +114,14 @@ graph TD
|
||||
C --> D[Build llama-server]
|
||||
D --> E[Verify Vulkan0 device]
|
||||
E --> F[Add DFlash2 drafter]
|
||||
F --> G[Swap frank-large]
|
||||
F --> G[Swap frank-dragoon]
|
||||
G --> H[Health check + completion]
|
||||
```
|
||||
|
||||
## Service
|
||||
|
||||
- Unit: `frank-large.service`, port 52625
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
|
||||
- Unit: `frank-dragoon.service`, port 52625
|
||||
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
|
||||
- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
|
||||
|
||||
### Launch profile
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-large.
|
||||
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-dragoon.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
@@ -15,14 +15,14 @@ timestamp: 2026-08-25
|
||||
|
||||
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
|
||||
> **RETIRED 2026-08-25.** `frank-large` served this model from 2026-08-18
|
||||
> **RETIRED 2026-08-25.** `frank-dragoon` served this model from 2026-08-18
|
||||
> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
|
||||
> on the same port and the same Strix Halo iGPU, running the new
|
||||
> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
|
||||
> the page it replaced), and was itself replaced by
|
||||
> [[kingjones-strix-rocmfp4]].
|
||||
|
||||
`frank-large` (port 7770) serves
|
||||
`frank-dragoon` (port 7770) serves
|
||||
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
|
||||
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
|
||||
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
|
||||
@@ -98,7 +98,7 @@ graph TD
|
||||
D --> E[Sharp chat template<br/>reasoning deepseek]
|
||||
```
|
||||
|
||||
Key flags (see `scripts/serve-frank-large.sh`):
|
||||
Key flags (see `scripts/serve-frank-dragoon.sh`):
|
||||
|
||||
```
|
||||
--spec-type draft-mtp --spec-mtp-strict-qwen
|
||||
@@ -129,8 +129,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
|
||||
### Service
|
||||
|
||||
- Unit: `frank-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-large.env`
|
||||
- Unit: `frank-dragoon.service`, port 7770
|
||||
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
|
||||
|
||||
## Tuning MTP
|
||||
|
||||
@@ -144,14 +144,14 @@ Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
|
||||
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
|
||||
|
||||
```
|
||||
sudo systemctl stop frank-large
|
||||
sudo systemctl stop frank-dragoon
|
||||
./scripts/bench-ubatch.sh
|
||||
sudo systemctl start frank-large
|
||||
sudo systemctl start frank-dragoon
|
||||
```
|
||||
|
||||
The script stops `frank-large` for a clean GPU, boots one server
|
||||
The script stops `frank-dragoon` for a clean GPU, boots one server
|
||||
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
|
||||
and restarts the service. See [[frank-large-benchmark]].
|
||||
and restarts the service. See [[frank-dragoon-benchmark]].
|
||||
|
||||
## Known Issues
|
||||
|
||||
@@ -185,4 +185,4 @@ The ubatch sweep above gives the production ubatch choice.
|
||||
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
|
||||
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
|
||||
- [[Dankclaw System Specs]] — the box
|
||||
- [[frank-large-benchmark]] — backend/spec benchmark history
|
||||
- [[frank-dragoon-benchmark]] — backend/spec benchmark history
|
||||
|
||||
+14
-14
@@ -1,7 +1,7 @@
|
||||
# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||||
# frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||||
|
||||
**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
|
||||
cutover — see [[frank-primary-radiance-qwen38-native-fp8]]; kept as the
|
||||
cutover — see [[frank-archon-radiance-qwen38-native-fp8]]; kept as the
|
||||
rollback runbook: the image + orcarouter payload stay on disk until the
|
||||
window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
|
||||
**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||||
@@ -9,11 +9,11 @@ window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
|
||||
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
|
||||
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
|
||||
on the host into the local image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
|
||||
alias (`frank-primary`), same 131-shard payload
|
||||
`localhost/frank-archon-freetoken:pr587-77a7201` by
|
||||
`scripts/docker/frank-archon-freetoken.Dockerfile`. Same port, same served
|
||||
alias (`frank-archon`), same 131-shard payload
|
||||
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
|
||||
git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
|
||||
git tag `frank-archon-vllm-shapeG-20261003` (the vLLM trio + its env lore),
|
||||
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
## Why FreeToken
|
||||
@@ -31,8 +31,8 @@ deterministic pass — no per-boot estimator to fight.
|
||||
|
||||
```bash
|
||||
cd /home/dank/serverAppConfigs
|
||||
podman build -t frank-primary-freetoken:pr587-77a7201 \
|
||||
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
|
||||
podman build -t frank-archon-freetoken:pr587-77a7201 \
|
||||
-f scripts/docker/frank-archon-freetoken.Dockerfile scripts/docker/
|
||||
```
|
||||
|
||||
Laws baked into the Dockerfile (each one cost a failure):
|
||||
@@ -129,7 +129,7 @@ native 262,144-token conversation + full budget. `models.yml`
|
||||
PLE ngram table in host RAM → the unit carries
|
||||
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
|
||||
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
|
||||
container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
|
||||
container's ~170 GiB MUST be down first; `systemctl stop frank-archon`
|
||||
frees it (measured: 207 GB free right after the stop).
|
||||
|
||||
## Resize live
|
||||
@@ -156,11 +156,11 @@ ft ctl stats # pool occupancy, VRAM, throughput
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
git checkout frank-primary-vllm-shapeG-20261003 -- \
|
||||
scripts/frank-primary.env scripts/serve-frank-primary.sh \
|
||||
scripts/frank-primary.service
|
||||
./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
|
||||
&& git push && ./scripts/deploy.sh dankserver --service frank-primary
|
||||
git checkout frank-archon-vllm-shapeG-20261003 -- \
|
||||
scripts/frank-archon.env scripts/serve-frank-archon.sh \
|
||||
scripts/frank-archon.service
|
||||
./tests/run.sh && git commit -m "frank-archon: rollback to vLLM shape G" \
|
||||
&& git push && ./scripts/deploy.sh dankserver --service frank-archon
|
||||
```
|
||||
|
||||
The tcclaviger image and the FreeToken image coexist in dank's rootless
|
||||
+13
-13
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: runbook
|
||||
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||||
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
|
||||
title: frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||||
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-archon cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
|
||||
tags:
|
||||
- dankserver
|
||||
- r9700
|
||||
@@ -16,7 +16,7 @@ tags:
|
||||
timestamp: 2026-09-30
|
||||
---
|
||||
|
||||
# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||||
# frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||||
|
||||
> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
|
||||
@@ -118,7 +118,7 @@ register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
|
||||
spec sets through the patched grouping/layout/builder functions. A 7-minute
|
||||
boot per hypothesis is the alternative.
|
||||
|
||||
**Window discipline:** stop + mask `frank-primary` FIRST and gate on
|
||||
**Window discipline:** stop + mask `frank-archon` FIRST and gate on
|
||||
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
|
||||
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
|
||||
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
|
||||
@@ -152,7 +152,7 @@ by this page's kernel-time measurement),
|
||||
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
|
||||
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
|
||||
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
|
||||
| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
|
||||
| unit | `scripts/frank-archon.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
|
||||
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
|
||||
|
||||
```mermaid
|
||||
@@ -698,14 +698,14 @@ newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
|
||||
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
|
||||
# dirs are untouched on disk):
|
||||
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
./scripts/deploy.sh dankserver --service frank-archon
|
||||
|
||||
# payload inventory (34 shards + 2 extras, weight-map index):
|
||||
bash scripts/download-frank-primary.sh --verify-only
|
||||
bash scripts/download-frank-archon.sh --verify-only
|
||||
|
||||
# live telemetry (needs the profiler window; with_stack=false is mandatory —
|
||||
# with stack the stop path hangs and the trace buffer grows to tens of GB):
|
||||
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
|
||||
podman logs frank-archon-glm 2>&1 | grep "expert_lru.py"
|
||||
```
|
||||
|
||||
**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
|
||||
@@ -719,11 +719,11 @@ mask symlink **and** the container is owned by the unit and not `Stopping`. The
|
||||
repair recipe, all four steps needed:
|
||||
|
||||
```bash
|
||||
sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
|
||||
sudo mv /run/systemd/system/frank-archon.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl reset-failed frank-primary.service # a failed unit refuses to start
|
||||
podman rm -f frank-primary-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
|
||||
sudo systemctl start frank-primary.service # then ~7 min to health 200
|
||||
sudo systemctl reset-failed frank-archon.service # a failed unit refuses to start
|
||||
podman rm -f frank-archon-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
|
||||
sudo systemctl start frank-archon.service # then ~7 min to health 200
|
||||
```
|
||||
|
||||
Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
|
||||
@@ -795,4 +795,4 @@ toggling on the mini itself).
|
||||
| state | where |
|
||||
| --- | --- |
|
||||
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
|
||||
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
|
||||
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-archon-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
|
||||
+20
-20
@@ -1,9 +1,9 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
|
||||
description: ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
|
||||
title: frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
|
||||
description: ACTIVE frank-archon runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
|
||||
tags:
|
||||
- frank-primary
|
||||
- frank-archon
|
||||
- dankserver
|
||||
- llamacpp
|
||||
- unsloth
|
||||
@@ -19,15 +19,15 @@ tags:
|
||||
timestamp: 2026-09-26
|
||||
---
|
||||
|
||||
# frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
|
||||
# frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
|
||||
|
||||
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**):
|
||||
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-archon**):
|
||||
unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
|
||||
(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
|
||||
with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
|
||||
**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the
|
||||
Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|
||||
([[frank-primary-lucebox-ds4]]), which itself replaced
|
||||
([[frank-archon-lucebox-ds4]]), which itself replaced
|
||||
[[frank-medium-radiance-mxfp4]].
|
||||
|
||||
## Why this shape
|
||||
@@ -89,20 +89,20 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|
||||
|
||||
|Piece|Value|
|
||||
|---|---|
|
||||
|Unit|`scripts/frank-primary.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|
||||
|Serve script|`scripts/serve-frank-primary.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|
||||
|Env tunables|`scripts/frank-primary.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|
||||
|Unit|`scripts/frank-archon.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|
||||
|Serve script|`scripts/serve-frank-archon.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|
||||
|Env tunables|`scripts/frank-archon.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|
||||
|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`|
|
||||
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-primary.sh` (sha256 + byte-count verified)|
|
||||
|Port / alias|7770 / `frank-primary` — unchanged from the lucebox and radiance eras; no client moves|
|
||||
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-archon.sh` (sha256 + byte-count verified)|
|
||||
|Port / alias|7770 / `frank-archon` — unchanged from the lucebox and radiance eras; no client moves|
|
||||
|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)|
|
||||
|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request|
|
||||
|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)|
|
||||
|Observability|`frank-primary` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
|
||||
|Observability|`frank-archon` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
|
||||
|
||||
### Cutover mechanics
|
||||
|
||||
The deployer (`deploy_llm_primary` in `scripts/deploy-remote.sh`) gates on
|
||||
The deployer (`deploy_llm_archon` in `scripts/deploy-remote.sh`) gates on
|
||||
every artifact at its declared byte count (`stat -c %s`, never `wc -c` —
|
||||
which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
|
||||
retires the lucebox user quadlet to dated home trash
|
||||
@@ -138,7 +138,7 @@ between slots — it does not multiply.
|
||||
|
||||
## Verification (2026-09-26 deploy)
|
||||
|
||||
- `/v1/models` reports `frank-primary`.
|
||||
- `/v1/models` reports `frank-archon`.
|
||||
- Chat completion returns `content` + `reasoning_content`.
|
||||
- Tool call returns well-formed `tool_calls` (`--jinja`).
|
||||
- Real image request answered from the projector (not a health check —
|
||||
@@ -218,7 +218,7 @@ vision and the target on the same battery class.
|
||||
|
||||
**Which blocks, measured (same day):** asked whether the `-ot` regex
|
||||
could target the *most active* blocks, I instrumented the live server
|
||||
instead of guessing — `scripts/frank-primary-expert-activity.py`
|
||||
instead of guessing — `scripts/frank-archon-expert-activity.py`
|
||||
samples llama-server with AMD IBS (`perf record -e ibs_op// --data`)
|
||||
while 4 mixed concurrent generations run, maps each sampled load
|
||||
address through `/proc/<pid>/maps` back to its backing tensor (extents
|
||||
@@ -235,7 +235,7 @@ per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of
|
||||
samples; hottest cell 0.1%), below the reach of a tensor-level
|
||||
override. Not a benchmark: it instruments production serving in place
|
||||
at negligible IBS overhead; re-measure any future checkpoint or
|
||||
traffic mix with `frank-primary-expert-activity.py <pid>` (or
|
||||
traffic mix with `frank-archon-expert-activity.py <pid>` (or
|
||||
`--analyze-only` against the saved sample file in `/tmp`).
|
||||
|
||||
## Rollback
|
||||
@@ -243,8 +243,8 @@ traffic mix with `frank-primary-expert-activity.py <pid>` (or
|
||||
The lucebox DS4.1 material is fully retained on the host: image
|
||||
`localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era
|
||||
repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
|
||||
[[frank-primary-lucebox-ds4]]). Restore = move the `.container` back,
|
||||
uninstall the root unit (`systemctl disable --now frank-primary` — the
|
||||
[[frank-archon-lucebox-ds4]]). Restore = move the `.container` back,
|
||||
uninstall the root unit (`systemctl disable --now frank-archon` — the
|
||||
link, not the file; never `systemctl disable` a unit whose file must
|
||||
survive), `daemon-reload`, start the user quadlet. The Qwen payload
|
||||
(~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a
|
||||
@@ -252,9 +252,9 @@ refetch.
|
||||
|
||||
## Fleet context
|
||||
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
|
||||
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
|
||||
b11030-mix binary family on gfx1151.
|
||||
- [[frank-secondary-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
|
||||
- [[frank-zealot-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
|
||||
3090; the only audio-capable endpoint.
|
||||
- The harness contract lives in `/home/blank/.omp/agent/models.yml`
|
||||
(`llamacpp (dankserver)` provider, `contextWindow 262144`,
|
||||
+15
-15
@@ -1,9 +1,9 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-primary Lucebox DeepSeek V4 Runbook
|
||||
description: RETIRED 2026-09-26 frank-primary runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-primary-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
|
||||
title: frank-archon Lucebox DeepSeek V4 Runbook
|
||||
description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
|
||||
tags:
|
||||
- frank-primary
|
||||
- frank-archon
|
||||
- dankserver
|
||||
- lucebox
|
||||
- deepseek-v4
|
||||
@@ -16,16 +16,16 @@ tags:
|
||||
timestamp: 2026-09-26
|
||||
---
|
||||
|
||||
# frank-primary Lucebox DeepSeek V4 Runbook
|
||||
# frank-archon Lucebox DeepSeek V4 Runbook
|
||||
|
||||
> [!warning] RETIRED 2026-09-26 (same day it went live)
|
||||
> Replaced by [[frank-primary-llamacpp-qwen38-flash-next-q8]] — unsloth
|
||||
> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
|
||||
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
|
||||
> served for hours; the rollback material (image `localhost/lucebox-rocm:
|
||||
> gfx1201` and the 378 GiB payload) is retained on the host, and the
|
||||
> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`.
|
||||
|
||||
The dankserver LLM endpoint (:7770, alias **frank-primary**) as it stood on
|
||||
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
|
||||
2026-09-26: the
|
||||
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
|
||||
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
|
||||
@@ -62,7 +62,7 @@ mode here, not an error.
|
||||
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
|
||||
cd lucebox
|
||||
# Three local modifications (repo-tracked at
|
||||
# dankserver/frank-primary/lucebox-gfx1201.patch):
|
||||
# dankserver/frank-archon/lucebox-gfx1201.patch):
|
||||
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
|
||||
# base already ships them in /opt/rocm; the apt repo does not serve
|
||||
# them) and the three runtime math libs.
|
||||
@@ -94,11 +94,11 @@ podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
|
||||
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
|
||||
400 against HF's xethub CDN for this repo; see the fetch note in
|
||||
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
|
||||
`deploy_llm_primary`.
|
||||
`deploy_llm_archon`.
|
||||
|
||||
## Deployed flags (2026-09-26)
|
||||
|
||||
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (the unit
|
||||
From `dankserver/frank-archon/frank-archon-entrypoint.sh` (the unit
|
||||
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
|
||||
flag set and cannot express the DS4 flags):
|
||||
|
||||
@@ -113,7 +113,7 @@ flag set and cannot express the DS4 flags):
|
||||
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
|
||||
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
|
||||
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
|
||||
| Identity | `--model-name frank-primary`, `--host 0.0.0.0 --port 7770` |
|
||||
| Identity | `--model-name frank-archon`, `--host 0.0.0.0 --port 7770` |
|
||||
|
||||
## API surface
|
||||
|
||||
@@ -123,11 +123,11 @@ metadata — the fleet contract is unchanged from frank-medium.
|
||||
|
||||
## Cutover procedure
|
||||
|
||||
1. Image built and model fetched (gates in `deploy_llm_primary`).
|
||||
2. `deploy.sh` → `deploy_llm_primary`: stops and **masks** the radiance
|
||||
1. Image built and model fetched (gates in `deploy_llm_archon`).
|
||||
2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
|
||||
unit (`frank-medium`), waits for :7770 to free, installs
|
||||
`dankserver/frank-primary/quadlet/frank-primary.container`, starts
|
||||
`frank-primary.service`, waits for health (1800 s start period — first
|
||||
`dankserver/frank-archon/quadlet/frank-archon.container`, starts
|
||||
`frank-archon.service`, waits for health (1800 s start period — first
|
||||
start page-ins the dense weights and builds the graphs).
|
||||
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
|
||||
call, and an Open WebUI completion.
|
||||
@@ -137,7 +137,7 @@ metadata — the fleet contract is unchanged from frank-medium.
|
||||
## Rollback
|
||||
|
||||
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
|
||||
systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
||||
systemctl --user start frank-medium`) and stop/mask frank-archon. The
|
||||
`localhost/radiance-vllm:0.13.0-e060895` image and the
|
||||
`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
|
||||
|
||||
+17
-17
@@ -1,25 +1,25 @@
|
||||
---
|
||||
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
|
||||
service: frank-primary
|
||||
title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
|
||||
service: frank-archon
|
||||
host: dankserver
|
||||
port: 7770
|
||||
status: active
|
||||
deployed: 2026-09-27
|
||||
supersedes: "[[frank-primary-r9v-radiance-qwen38-uncensored]]"
|
||||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, q8]
|
||||
supersedes: "[[frank-archon-r9v-radiance-qwen38-uncensored]]"
|
||||
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, q8]
|
||||
---
|
||||
|
||||
# frank-primary — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)
|
||||
# frank-archon — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)
|
||||
|
||||
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
|
||||
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
|
||||
`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
|
||||
(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
|
||||
MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
|
||||
/ `0000:07:00.0`) as `frank-primary` on **:7770**.
|
||||
/ `0000:07:00.0`) as `frank-archon` on **:7770**.
|
||||
|
||||
Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
|
||||
direction. Rollback target: [[frank-primary-r9v-radiance-qwen38-uncensored]]
|
||||
direction. Rollback target: [[frank-archon-r9v-radiance-qwen38-uncensored]]
|
||||
— the uncensored state dir (`/home/dank/r9v-state/uncensored/`, `ready:
|
||||
true`), its frozen config (`restore-config-20260927.env`) and payload stay
|
||||
on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.
|
||||
@@ -39,10 +39,10 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
|
||||
|
||||
| Piece | File | Law |
|
||||
|---|---|---|
|
||||
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
|
||||
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
|
||||
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
|
||||
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
|
||||
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
|
||||
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
|
||||
|
||||
**Why the tuned shape lives on-host:** the q8 profile's setup run
|
||||
qualified the `R9V_*` config (tiered-expert placement, per-rank offload
|
||||
@@ -55,7 +55,7 @@ would drift from the qualification — don't.
|
||||
## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
|
||||
|
||||
llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
|
||||
frank-secondary/large). The fork's `speculative_config` is a **single dict**
|
||||
frank-zealot/large). The fork's `speculative_config` is a **single dict**
|
||||
(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
|
||||
mutually exclusive — no chaining seam exists in the fork. What IS active:
|
||||
|
||||
@@ -67,7 +67,7 @@ mutually exclusive — no chaining seam exists in the fork. What IS active:
|
||||
- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram
|
||||
proposer) — swapping MTP4 for it is an either/or, not a chain.
|
||||
|
||||
## Sharp chat template (fleet parity with frank-large)
|
||||
## Sharp chat template (fleet parity with frank-dragoon)
|
||||
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in
|
||||
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is
|
||||
@@ -176,7 +176,7 @@ Laws from the matrix:
|
||||
after a bench teardown; a production boot failed on it the same morning
|
||||
(unit "Deactivated successfully", no error in journal — the serve script
|
||||
exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
|
||||
`frank-primary.env` (forwarded by launch.sh `2de0357`).
|
||||
`frank-archon.env` (forwarded by launch.sh `2de0357`).
|
||||
|
||||
Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
|
||||
VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
|
||||
@@ -256,7 +256,7 @@ quoting through ssh+zsh layers.
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
|
||||
script re-gates everything, relaunch through the shim, waits).
|
||||
- **First start after boot** re-warms ~175 GiB of page cache; expect
|
||||
minutes to /health; `TimeoutStartSec=1800`.
|
||||
@@ -266,11 +266,11 @@ quoting through ssh+zsh layers.
|
||||
produces a false-green deploy (the 2026-09-26 lesson, guarded by the
|
||||
MainPID check).
|
||||
- **Rollback to uncensored**: point
|
||||
`scripts/frank-primary.env` `R9V_STATE_FILE` back at
|
||||
`scripts/frank-archon.env` `R9V_STATE_FILE` back at
|
||||
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
|
||||
(adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
|
||||
uncensored state dir and payload are intact on disk.
|
||||
- **Models on the wire**: served alias `frank-primary`; thinking channel
|
||||
- **Models on the wire**: served alias `frank-archon`; thinking channel
|
||||
is `reasoning` (vLLM); tool parser `qwen3_coder`.
|
||||
|
||||
## Known costs (accepted at cutover)
|
||||
+16
-16
@@ -1,19 +1,19 @@
|
||||
---
|
||||
title: frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||||
service: frank-primary
|
||||
title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
|
||||
service: frank-archon
|
||||
host: dankserver
|
||||
port: 7770
|
||||
status: retired
|
||||
retired: 2026-09-27
|
||||
superseded_by: "[[frank-primary-r9v-radiance-qwen38-q8]]"
|
||||
supersedes: "[[frank-primary-llamacpp-qwen38-flash-next-q8]]"
|
||||
tags: [llm, frank-primary, r9v, vllm, radiance, gfx1201, retired]
|
||||
superseded_by: "[[frank-archon-r9v-radiance-qwen38-q8]]"
|
||||
supersedes: "[[frank-archon-llamacpp-qwen38-flash-next-q8]]"
|
||||
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, retired]
|
||||
---
|
||||
|
||||
# frank-primary — R9V radiance-vLLM (dankserver, RETIRED)
|
||||
# frank-archon — R9V radiance-vLLM (dankserver, RETIRED)
|
||||
|
||||
> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover,
|
||||
> [[frank-primary-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook:
|
||||
> [[frank-archon-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook:
|
||||
> the uncensored state dir, frozen config and payload are intact on disk;
|
||||
> the rollback procedure lives in that page's Ops section.
|
||||
|
||||
@@ -22,11 +22,11 @@ The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
|
||||
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
|
||||
Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
|
||||
MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
|
||||
`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770**.
|
||||
`0000:03:00.0` / `0000:07:00.0`) as `frank-archon` on **:7770**.
|
||||
|
||||
Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
|
||||
direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
|
||||
Rollback target: [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the
|
||||
Rollback target: [[frank-archon-llamacpp-qwen38-flash-next-q8]] — the
|
||||
llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
|
||||
wiring; host unit was the relinked one) and the GGUF payload stays on disk,
|
||||
so a rollback is a redeploy.
|
||||
@@ -37,10 +37,10 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
|
||||
|
||||
| Piece | File | Law |
|
||||
|---|---|---|
|
||||
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
|
||||
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs |
|
||||
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
|
||||
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
|
||||
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs |
|
||||
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
|
||||
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
|
||||
|
||||
**Why the tuned shape lives on-host:** the profile's setup run qualified a
|
||||
104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
|
||||
@@ -83,7 +83,7 @@ unit. `docker wait` cannot stop the container child, hence `ExecStop`.
|
||||
- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
|
||||
`runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
|
||||
all-reduce verified inside serving workers** on both BDFs.
|
||||
- Wire: `/v1/models` → `frank-primary`; `/health` + `/metrics` 200
|
||||
- Wire: `/v1/models` → `frank-archon`; `/health` + `/metrics` 200
|
||||
(Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
|
||||
thinking requests populate `reasoning` (vLLM channel).
|
||||
- Memlock: the full tiered-expert load ran to healthy **under a plain dank
|
||||
@@ -107,7 +107,7 @@ unit. `docker wait` cannot stop the container child, hence `ExecStop`.
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
|
||||
script re-gates everything, relaunch through the shim, waits).
|
||||
- **First start after boot** re-warms ~93 GiB of page cache; expect several
|
||||
minutes to /health; `TimeoutStartSec=1800`.
|
||||
@@ -119,7 +119,7 @@ unit. `docker wait` cannot stop the container child, hence `ExecStop`.
|
||||
- **Qualification artifacts** (kept on-host):
|
||||
`/home/dank/r9v-state/uncensored/qualification-*`,
|
||||
failure bundles `failure-*` (review before sharing — raw logs).
|
||||
- **Models on the wire**: served alias `frank-primary`; thinking channel is
|
||||
- **Models on the wire**: served alias `frank-archon`; thinking channel is
|
||||
`reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
|
||||
vllm030 wiring had planned).
|
||||
|
||||
+31
-31
@@ -1,4 +1,4 @@
|
||||
# frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
|
||||
# frank-archon — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
|
||||
|
||||
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
|
||||
|
||||
@@ -6,12 +6,12 @@ The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
|
||||
libr4d kernels for gfx1201, no Python in the serving path) driving the
|
||||
digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
|
||||
(v1.0.7 line, 287 MiB, `libr4d@0.5.0`) as a plain `podman run` through
|
||||
`scripts/serve-frank-primary.sh` under the **root systemd unit**
|
||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
|
||||
same served alias (`frank-primary`), same container identity
|
||||
(`frank-primary` = the unit name). Rollback: git tag
|
||||
`frank-primary-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201`, payload
|
||||
`scripts/serve-frank-archon.sh` under the **root systemd unit**
|
||||
(`frank-archon.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
|
||||
same served alias (`frank-archon`), same container identity
|
||||
(`frank-archon` = the unit name). Rollback: git tag
|
||||
`frank-archon-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
|
||||
`localhost/frank-archon-freetoken:pr587-77a7201`, payload
|
||||
`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`.
|
||||
|
||||
## Why radiance
|
||||
@@ -33,14 +33,14 @@ same served alias (`frank-primary`), same container identity
|
||||
|
||||
- **Raw checkpoint**: `/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
(ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision
|
||||
configs). Fetched by `scripts/download-frank-primary.sh` (vendored
|
||||
configs). Fetched by `scripts/download-frank-archon.sh` (vendored
|
||||
`hfget.sh`, curl+jq, no HF token needed) and **verified byte-for-byte**:
|
||||
all 131 shards sha256-match the Hub LFS oids (2026-10-05, log
|
||||
`/home/dank/logs/verify-qwen38-official-fp8.log`).
|
||||
- **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad`
|
||||
(177,109 MiB measured = 173.0 GiB; floor
|
||||
`FRANK_PRIMARY_RAD_MIN_MIB=177000` in the env gates a
|
||||
truncated convert). Built by `scripts/build-frank-primary-rad.sh`:
|
||||
`FRANK_ARCHON_RAD_MIN_MIB=177000` in the env gates a
|
||||
truncated convert). Built by `scripts/build-frank-archon-rad.sh`:
|
||||
1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with
|
||||
**`quantization_config` dropped** + relative symlinks into the raw
|
||||
checkout;
|
||||
@@ -99,12 +99,12 @@ same served alias (`frank-primary`), same container identity
|
||||
(declare = CPU-side layout queries); add `--device` only if a plugin
|
||||
load ever demands it.
|
||||
|
||||
## The shape (`scripts/frank-primary.env`, every value commented there)
|
||||
## The shape (`scripts/frank-archon.env`, every value commented there)
|
||||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| Image | `stilldeadcode/radiance@sha256:4c3f4845…` | digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH |
|
||||
| Container | `frank-primary` | identity = service name across runtimes; legacy list retires `frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe` |
|
||||
| Container | `frank-archon` | identity = service name across runtimes; legacy list retires `frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe` |
|
||||
| `--tp 2 --tp-wire wht6` | both R9700s | fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB) |
|
||||
| `--max-num-seqs 2` | first boot | Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8` |
|
||||
| `--max-model-len 200000` | served window | native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated |
|
||||
@@ -113,7 +113,7 @@ same served alias (`frank-primary`), same container identity
|
||||
| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
|
||||
| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
|
||||
| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
|
||||
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
|
||||
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-archon --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
|
||||
| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
|
||||
| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
|
||||
|
||||
@@ -122,7 +122,7 @@ same served alias (`frank-primary`), same container identity
|
||||
spare ≈ **134 GiB**. `ttm.pages_limit` (this kernel has NO `pages_max` —
|
||||
the two-knob story was stale) caps GPU-mappable pinned pages (default = half
|
||||
of RAM): the pre-cutover cap was **123 GiB** — BELOW the ledger — so the
|
||||
serve script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
|
||||
serve script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=140000`) refuses to
|
||||
start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
|
||||
**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
|
||||
verified read-back; raising at runtime is harmless and allocates nothing).
|
||||
@@ -175,16 +175,16 @@ KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom.
|
||||
|
||||
```bash
|
||||
# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
|
||||
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh --plan-only' # declare + plan, no bytes
|
||||
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh' # full convert + rad-info
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh --plan-only' # declare + plan, no bytes
|
||||
ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh' # full convert + rad-info
|
||||
./scripts/deploy.sh dankserver --service frank-archon
|
||||
```
|
||||
|
||||
Post-cutover checklist (each is observable, not assumed):
|
||||
|
||||
1. Unit: `is-active` active, `is-enabled` enabled, container answers
|
||||
`/health` with 200 JSON.
|
||||
2. Boot ledger (engine stdout → `/home/dank/logs/frank_primary.log`):
|
||||
2. Boot ledger (engine stdout → `/home/dank/logs/frank_archon.log`):
|
||||
P2P probe per rank pair, TP=2 shard placement, expert pool coverage
|
||||
line ("backs N % of the worst case … M full-length sessions"), KV pool
|
||||
bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
|
||||
@@ -194,7 +194,7 @@ Post-cutover checklist (each is observable, not assumed):
|
||||
(tool-call + thinking turn both exercise it).
|
||||
5. Vision: a ≥256 px solid-color PNG answers the exact color through an
|
||||
`image_url` content part.
|
||||
6. Prometheus target `frank-primary` green (not never-green).
|
||||
6. Prometheus target `frank-archon` green (not never-green).
|
||||
|
||||
## Cutover measured (2026-10-06, all values observed live)
|
||||
|
||||
@@ -225,25 +225,25 @@ Post-cutover checklist (each is observable, not assumed):
|
||||
converter's scale-name law, the PLE file-shadow and the
|
||||
symlink-through-raw-index bug are in the payload section above.
|
||||
- Prometheus: `/metrics` answers 200 with vLLM-compatible families
|
||||
(verified from Dankclaw); the `frank-primary` job needs a
|
||||
(verified from Dankclaw); the `frank-archon` job needs a
|
||||
prometheus-service deploy to go green (config pushed, reload
|
||||
deferred behind a live parallel writer's dirty tree).
|
||||
|
||||
## Tuning levers (in order of likely use)
|
||||
|
||||
- **`FRANK_PRIMARY_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
|
||||
- **`FRANK_ARCHON_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
|
||||
headroom line and real slot occupancy agree; each slot buys a
|
||||
recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
|
||||
- **`FRANK_PRIMARY_HOST_POOL_MIB`**: the ledger's coverage line is the
|
||||
- **`FRANK_ARCHON_HOST_POOL_MIB`**: the ledger's coverage line is the
|
||||
oracle — it prints what the pool backs at the worst case and how many
|
||||
full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of
|
||||
the host's 246.
|
||||
- **`FRANK_PRIMARY_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
|
||||
- **`FRANK_ARCHON_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
|
||||
point; 0.87 zeroes it. Move only with a measured hit-rate reason.
|
||||
- **`FRANK_PRIMARY_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
|
||||
- **`FRANK_ARCHON_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
|
||||
+5.6 % prefill for bit-exact reductions; the fused one-shot forms are
|
||||
declared over the seqs × (1+spec) row bound.
|
||||
- **`FRANK_PRIMARY_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
|
||||
- **`FRANK_ARCHON_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
|
||||
unquoted.
|
||||
- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
|
||||
the tunable space at this model's shapes on this machine, writes
|
||||
@@ -252,14 +252,14 @@ Post-cutover checklist (each is observable, not assumed):
|
||||
--num-speculative-tokens 3`), or you measure a program the serve never
|
||||
runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
|
||||
`--rm` scratch container's cache dies with it: the serve script
|
||||
bind-mounts `$FRANK_PRIMARY_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
|
||||
bind-mounts `$FRANK_ARCHON_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
|
||||
at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
|
||||
`gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
|
||||
stopped-service window) survives and is read at every production boot.
|
||||
The two attention requests have no shape recipe (`attn_paged`,
|
||||
`attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
|
||||
`RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
|
||||
- **`FRANK_PRIMARY_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
|
||||
- **`FRANK_ARCHON_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
|
||||
stopped-service window (real-text probes: 20,417-token cold prefill,
|
||||
1,985-token warm prefill, 256-token decode). The prefill ladder is
|
||||
NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
|
||||
@@ -274,14 +274,14 @@ Post-cutover checklist (each is observable, not assumed):
|
||||
publisher's `scripts/` harnesses (`fnpf`, `fnconc`, `cstep2`, `ident`,
|
||||
`mctools`) — note `mcserve/fnstop`-family scripts `pkill` the engine
|
||||
binary by name; on this host the service is a systemd unit, so use
|
||||
`sudo systemctl stop frank-primary` instead.
|
||||
`sudo systemctl stop frank-archon` instead.
|
||||
|
||||
## Rollback
|
||||
|
||||
| Layer | Path |
|
||||
|---|---|
|
||||
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-primary-freetoken-pr587-20261005 -- scripts/frank-primary.env scripts/frank-primary.service scripts/serve-frank-primary.sh scripts/docker/ tests/frank_primary_unit.bats` + redeploy; image `localhost/frank-primary-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
|
||||
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-primary-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
|
||||
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-archon-freetoken-pr587-20261005 -- scripts/frank-archon.env scripts/frank-archon.service scripts/serve-frank-archon.sh scripts/docker/ tests/frank_archon_unit.bats` + redeploy; image `localhost/frank-archon-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
|
||||
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-archon-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
|
||||
| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` + `--host-pool-mib 12288` — the documented fallback if the native container ever proves unbootable |
|
||||
|
||||
Retire the FreeToken-era orcarouter payload to `~/.AGENTTRASH/` only after
|
||||
@@ -290,7 +290,7 @@ the rollback window closes (trash policy, never `rm`).
|
||||
## Links
|
||||
|
||||
- [[LLM/index]] — fleet index
|
||||
- [[frank-primary-freetoken-pr587]] — the retired FreeToken era
|
||||
- [[frank-archon-freetoken-pr587]] — the retired FreeToken era
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM era
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
- [[RDNA4 Container Serving]] — rootless-podman GPU laws on these hosts
|
||||
+22
-22
@@ -1,4 +1,4 @@
|
||||
# frank-primary — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
|
||||
# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
|
||||
|
||||
**Status: LIVE — deployed 2026-10-08 (main `3ca64e1`, e2e 33/33).** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
|
||||
|
||||
@@ -10,7 +10,7 @@ finetune) converted with the recipe
|
||||
**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
|
||||
the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
|
||||
Same port, alias, container name, systemd unit. Rollback: revert the
|
||||
`MODEL_FILE` line in `scripts/frank-primary.env` + redeploy —
|
||||
`MODEL_FILE` line in `scripts/frank-archon.env` + redeploy —
|
||||
`/home/dank/models/rad/qwen4exp-fp8-native.rad` stays on disk (trash
|
||||
policy; 173 GiB, purge when the window closes).
|
||||
|
||||
@@ -22,7 +22,7 @@ policy; 173 GiB, purge when the window closes).
|
||||
host and VRAM budget for KV and slots at equal-or-better measured
|
||||
decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control).
|
||||
- **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's
|
||||
fleet-personality call; frank-large serves the same family).
|
||||
fleet-personality call; frank-dragoon serves the same family).
|
||||
- **GPTQ closes the precision gap**: RTN alone on BF16 weights lands
|
||||
measurably worse; GPTQ with Hessians from a 10M-token corpus of this
|
||||
model's own distribution recovers most of it, verified end-to-end by a
|
||||
@@ -30,7 +30,7 @@ policy; 173 GiB, purge when the window closes).
|
||||
|
||||
## The payload chain
|
||||
|
||||
1. **Fetch**: `scripts/download-frank-primary-swift15.sh` →
|
||||
1. **Fetch**: `scripts/download-frank-archon-swift15.sh` →
|
||||
`/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3
|
||||
GiB, per-shard LFS sha256 verified, no token on the command line).
|
||||
48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560,
|
||||
@@ -95,7 +95,7 @@ policy; 173 GiB, purge when the window closes).
|
||||
| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
|
||||
| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |
|
||||
|
||||
## The shape (`scripts/frank-primary.env`)
|
||||
## The shape (`scripts/frank-archon.env`)
|
||||
|
||||
Unchanged from the FP8-native era: TP=2 + `wht6`, `--max-num-seqs 3`,
|
||||
`--max-model-len 200000`, MNBT 2048 (the MTP resonance law — 2560
|
||||
@@ -103,7 +103,7 @@ collapses decode to 2.6–7.7 t/s), fp8 KV, `expert_tiered` host pool
|
||||
40960 (the load-bearing floor), `--ngram-placement ram`, froggeric
|
||||
v22.5 template override. `MODEL_FILE` points at
|
||||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416
|
||||
MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
|
||||
MiB measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
|
||||
177000 → **110000**.
|
||||
|
||||
## Gates (measured at cutover)
|
||||
@@ -149,8 +149,8 @@ MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
|
||||
budget and `content` comes back empty (the answer lands in the
|
||||
reasoning channel).
|
||||
- **Deployed verification (2026-10-08)**: `deploy.sh dankserver
|
||||
--service frank-primary` green incl. Playwright e2e 33/33; unit
|
||||
`active`, `/health` 200, `/v1/models` id `frank-primary` (the harness
|
||||
--service frank-archon` green incl. Playwright e2e 33/33; unit
|
||||
`active`, `/health` 200, `/v1/models` id `frank-archon` (the harness
|
||||
alias is the served-model-name, independent of the file — models.yml
|
||||
untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2,
|
||||
seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert
|
||||
@@ -159,23 +159,23 @@ MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
|
||||
|
||||
### Wire rename + the bench drop-in trap (2026-10-09)
|
||||
|
||||
`FRANK_PRIMARY_SERVED_NAME` moved `frank-primary` → **frank-archon**; the
|
||||
`FRANK_ARCHON_SERVED_NAME` moved `frank-archon` → **frank-archon**; the
|
||||
service, unit, scripts and env file keep the old name. Radiance is the only
|
||||
fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
|
||||
**requested** model name in chat responses — so a chat probe that returns 200
|
||||
proves nothing about the served id. Always read `/v1/models`.
|
||||
|
||||
The first deploy after the edit came up green and `/v1/models` still said
|
||||
`frank-primary`. Cause: a campaign leftover drop-in
|
||||
`/etc/systemd/system/frank-primary.service.d/bench.conf` (2026-10-08 04:09)
|
||||
overrode BOTH `Environment=FRANK_PRIMARY_ENV_FILE` and `ExecStart` to the
|
||||
`frank-archon`. Cause: a campaign leftover drop-in
|
||||
`/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09)
|
||||
overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the
|
||||
staging worktree `/home/dank/serverAppConfigs-swift15`. Every
|
||||
`deploy.sh dankserver --service frank-primary` since then restarted the
|
||||
`deploy.sh dankserver --service frank-archon` since then restarted the
|
||||
staging tree: green health gate, old configuration, silently. Verified before
|
||||
removing: the staging `serve-frank-primary.sh` and `frank-primary.service`
|
||||
are byte-identical to the checkout, its `frank-primary.env` differed only by
|
||||
removing: the staging `serve-frank-archon.sh` and `frank-archon.service`
|
||||
are byte-identical to the checkout, its `frank-archon.env` differed only by
|
||||
the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
|
||||
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-primary-bench-dropin/`.
|
||||
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`.
|
||||
After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
|
||||
healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
|
||||
26071/3543).
|
||||
@@ -190,9 +190,9 @@ run. After an LLM deploy, verify the served configuration behaviourally
|
||||
## Deploy + verify
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-primary # drained window
|
||||
sudo systemctl stop frank-archon # drained window
|
||||
bash convert-swift15-w4dn8.sh # if not already run
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
./scripts/deploy.sh dankserver --service frank-archon
|
||||
```
|
||||
|
||||
Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger
|
||||
@@ -206,9 +206,9 @@ probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
|
||||
|
||||
| Layer | Path |
|
||||
|---|---|
|
||||
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-primary.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
|
||||
| FreeToken PR #587 | git tag `frank-primary-freetoken-pr587-20261005` |
|
||||
| tcclaviger vLLM shape G | git tag `frank-primary-vllm-shapeG-20261003` |
|
||||
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-archon.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
|
||||
| FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` |
|
||||
| tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` |
|
||||
|
||||
Campaign artifacts on disk: BF16 checkpoint (335 GiB), `kld-ref` (24
|
||||
GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, logs
|
||||
@@ -218,7 +218,7 @@ re-convert source if the window ever reopens.
|
||||
|
||||
## Links
|
||||
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — the replaced FP8-native era
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the replaced FP8-native era
|
||||
- [[LLM/index]] — fleet index
|
||||
- [[swift15-campaign-worklog]] — campaign record
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
description: "This page records the benchmarks that chose the runtime for `frank-large` on Dankclaw."
|
||||
description: "This page records the benchmarks that chose the runtime for `frank-dragoon` on Dankclaw."
|
||||
tags:
|
||||
- llm
|
||||
- frank
|
||||
@@ -11,10 +11,10 @@ timestamp: 2026-09-03
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
# frank-large — Backend and Spec Benchmark
|
||||
# frank-dragoon — Backend and Spec Benchmark
|
||||
|
||||
This page records the benchmarks that chose the runtime for
|
||||
`frank-large` on Dankclaw. Every number came from the live
|
||||
`frank-dragoon` on Dankclaw. Every number came from the live
|
||||
llama.cpp build on that machine (upstream, b10133+).
|
||||
|
||||
## Summary
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
type: reference
|
||||
title: "CIRU v4.4.1 Orca on Dankclaw (frank-dragoon)"
|
||||
description: "The retired frank-dragoon runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- ciru
|
||||
- dankclaw
|
||||
- frank-dragoon
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
- rocm
|
||||
- mtp
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
+15
-15
@@ -1,14 +1,14 @@
|
||||
# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
|
||||
# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
|
||||
|
||||
**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
|
||||
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
|
||||
`scripts/frank-large.service`); the retired 27B payload stays on disk at
|
||||
(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
|
||||
`scripts/frank-dragoon.service`); the retired 27B payload stays on disk at
|
||||
`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
|
||||
(`scripts/download-frank-large-halogen.sh`) stays in the repo.
|
||||
(`scripts/download-frank-dragoon-halogen.sh`) stays in the repo.
|
||||
|
||||
## Why this shape exists
|
||||
|
||||
Frank's ask: replace the frank-large model with
|
||||
Frank's ask: replace the frank-dragoon model with
|
||||
`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
|
||||
That payload is an **abliterated Swift1.5 finetune of peonist's
|
||||
Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
|
||||
@@ -70,13 +70,13 @@ Payload notes worth keeping:
|
||||
dangle inside the container mount; both the serve gate and the deployer
|
||||
refuse them).
|
||||
|
||||
Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
|
||||
Fetch: `scripts/download-frank-dragoon-flashnext.sh` (`--verify-only` to
|
||||
re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
|
||||
resume law throughout; no token — the repos are public).
|
||||
|
||||
## Shape — the knobs that are not free
|
||||
|
||||
`scripts/frank-large.env` carries the full rationale inline; the table:
|
||||
`scripts/frank-dragoon.env` carries the full rationale inline; the table:
|
||||
|
||||
| Env | Value | Why |
|
||||
|---|---|---|
|
||||
@@ -88,7 +88,7 @@ resume law throughout; no token — the repos are public).
|
||||
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
|
||||
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
|
||||
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
|
||||
| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-large`); engine port 8730 never published |
|
||||
| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-dragoon`); engine port 8730 never published |
|
||||
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
|
||||
|
||||
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
|
||||
@@ -112,7 +112,7 @@ instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
|
||||
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
|
||||
mandatory.
|
||||
- Vision: OpenAI `image_url` content parts only (the fleet's standard
|
||||
contract; the legacy top-level `images` field is the frank-secondary trap).
|
||||
contract; the legacy top-level `images` field is the frank-zealot trap).
|
||||
≥256 px probe images or the model answers "video".
|
||||
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
|
||||
truncation. `max_tokens` INCLUDES thinking.
|
||||
@@ -133,12 +133,12 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
|
||||
## Mechanics
|
||||
|
||||
- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||||
- **Root systemd unit** (`scripts/frank-dragoon.service`, `User=dank`,
|
||||
`LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
|
||||
frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
|
||||
frank-dragoon-halogen`), driven by `scripts/serve-frank-dragoon.sh` — the
|
||||
fleet's `podman run`-direct pattern. The script runs the container
|
||||
FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
|
||||
prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
|
||||
prompt cache, any clamp) lands in `/home/dank/logs/frank_dragoon.log`.
|
||||
- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
|
||||
image digest in podman storage, byte-exact checkpoint + n-gram table +
|
||||
vision tower, flat tokenizer with all six files as regular files,
|
||||
@@ -151,7 +151,7 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
|
||||
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
|
||||
when the dying 27B engine freed RAM). The serve script now checks
|
||||
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
|
||||
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_DRAGOON_RAM_WAIT`
|
||||
(default 600 s) for the RAM to clear, so a transient dip boots once
|
||||
instead of crash-looping; if it never clears it refuses loudly. The 16
|
||||
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
|
||||
@@ -162,7 +162,7 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
|
||||
## Verification checklist (run after any redeploy)
|
||||
|
||||
1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
|
||||
1. Boot ledger greps in `/home/dank/logs/frank_dragoon.log`: resident
|
||||
weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
|
||||
**no clamp named**.
|
||||
2. `curl http://192.168.0.143:7770/health` → 200.
|
||||
@@ -183,7 +183,7 @@ BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
|
||||
`maxTokens: 45056`, `input: [text, image]`,
|
||||
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
|
||||
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
|
||||
frank-secondary (the `compactionModel` steering is unchanged; the input
|
||||
frank-zealot (the `compactionModel` steering is unchanged; the input
|
||||
≈ 170k fits its 200,704 slot).
|
||||
|
||||
## Rollback
|
||||
+29
-29
@@ -1,13 +1,13 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
|
||||
description: "The RETIRED frank-dragoon runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
- swift-1.5
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- q8
|
||||
@@ -16,12 +16,12 @@ tags:
|
||||
timestamp: 2026-09-28
|
||||
---
|
||||
|
||||
# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
|
||||
# frank-dragoon — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
|
||||
|
||||
> **RETIRED 2026-10-02 — the rollback target.** Superseded by
|
||||
> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
|
||||
> [[frank-dragoon-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
|
||||
> MTP4, 17.0 t/s measured). This era was itself a return after one day of
|
||||
> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
|
||||
> [[frank-dragoon-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
|
||||
> ≥6-bit vLLM quant path dead on gfx1151). The shape
|
||||
> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
|
||||
> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
|
||||
@@ -30,10 +30,10 @@ timestamp: 2026-09-28
|
||||
> Swift-1.5 Q8 history below is retained for the engine/template law,
|
||||
> which still governs.
|
||||
> **Originally ACTIVE 2026-09-28.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
> alias (`frank-large`).
|
||||
> [[frank-dragoon-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
> alias (`frank-dragoon`).
|
||||
>
|
||||
> **The payload also changed on 2026-09-28**, later the same day: the target
|
||||
> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
|
||||
@@ -44,16 +44,16 @@ timestamp: 2026-09-28
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
|
||||
inference engine", a vertical engine built for exactly this hardware
|
||||
(Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
|
||||
**v0.1.1** (commit `b0f8673`), built from source on the host to
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/`.
|
||||
- **Service shape:** root systemd unit `scripts/frank-large.service`
|
||||
- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
|
||||
(`User=dank`, `LimitMEMLOCK=infinity`), launched by
|
||||
`scripts/serve-frank-large.sh` with tunables from
|
||||
`scripts/frank-large.env`.
|
||||
`scripts/serve-frank-dragoon.sh` with tunables from
|
||||
`scripts/frank-dragoon.env`.
|
||||
|
||||
gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
|
||||
are empty) and no Nix on this host, so the runtime is a source build. It is
|
||||
@@ -77,7 +77,7 @@ gufo's own measurement of the two engines on the same Qwen3.8-27B targets
|
||||
|
||||
Read that honestly: **prefill, load time and short-context decode improve;
|
||||
deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
|
||||
around 32k of cached depth. frank-large runs one 262,144-token conversation,
|
||||
around 32k of cached depth. frank-dragoon runs one 262,144-token conversation,
|
||||
so a long agent session spends its life on the losing side of that line
|
||||
while a short turn wins. The Q4_K_XL target does not show the regression
|
||||
(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
|
||||
@@ -116,7 +116,7 @@ architecture, driver and queue budget.
|
||||
- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
|
||||
file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
|
||||
sha256-verified (`9076f99a…`, the Hub LFS oid) by
|
||||
`scripts/download-frank-large-swift.sh`. It is an **abliterated
|
||||
`scripts/download-frank-dragoon-swift.sh`. It is an **abliterated
|
||||
(uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
|
||||
`mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
|
||||
- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
|
||||
@@ -160,7 +160,7 @@ were confirmed against gufo's source rather than assumed:
|
||||
from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
|
||||
drafter's filename rather than the target's.
|
||||
- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
|
||||
frank-large served. The harness overrides per request via `extraBody`
|
||||
frank-dragoon served. The harness overrides per request via `extraBody`
|
||||
(1.0 / 0.95 / top-k 20).
|
||||
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
|
||||
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
|
||||
@@ -227,8 +227,8 @@ tokens or `inter_token` is first-token noise.
|
||||
## The template law: Qwen-Sharp is gone
|
||||
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
|
||||
wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was
|
||||
`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no
|
||||
wired into `frank-archon` as `R9V_CHAT_TEMPLATE_FILE` and it was
|
||||
`frank-dragoon`'s `--chat-template-file` under llama.cpp. **frank-dragoon can no
|
||||
longer use it**, and the reason is stronger than "gufo lacks the flag":
|
||||
|
||||
> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
|
||||
@@ -265,7 +265,7 @@ second is the same class of change that produced the 2026-09-21
|
||||
|
||||
## Client contract — what actually changed
|
||||
|
||||
Everything the fleet depended on still holds: port 7770, alias `frank-large`,
|
||||
Everything the fleet depended on still holds: port 7770, alias `frank-dragoon`,
|
||||
`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
|
||||
auto-discovers a `mmproj` beside the model; this payload has none).
|
||||
|
||||
@@ -300,9 +300,9 @@ The differences worth knowing:
|
||||
## Failure modes worth knowing
|
||||
|
||||
- **A missing runtime is a deploy error, not a restart loop.** The handler
|
||||
gates on `LLM_LARGE_RUNTIME` *before* it stops anything, and the serve
|
||||
gates on `LLM_DRAGOON_RUNTIME` *before* it stops anything, and the serve
|
||||
script exits with a `build-gufo.sh` pointer rather than exec'ing.
|
||||
- **Host memory is the ceiling, and it is shared.** frank-large and ComfyUI
|
||||
- **Host memory is the ceiling, and it is shared.** frank-dragoon and ComfyUI
|
||||
both hold gfx1151 compute queues. gufo reads
|
||||
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
|
||||
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
|
||||
@@ -321,13 +321,13 @@ The differences worth knowing:
|
||||
|
||||
Two independent rollbacks, because two things changed on 2026-09-28:
|
||||
|
||||
1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-large.env` at
|
||||
1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-dragoon.env` at
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
|
||||
and redeploy. **The 29.3 GiB target was reclaimed on 2026-09-28**, so this
|
||||
path now needs a re-fetch first: `scripts/download-frank-large.sh`, then
|
||||
path now needs a re-fetch first: `scripts/download-frank-dragoon.sh`, then
|
||||
the env edit. The DFlash2 drafter was never touched and still lives in
|
||||
that directory.
|
||||
2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
|
||||
2. **Engine** — [[frank-dragoon-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
|
||||
prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
|
||||
Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
|
||||
script's llama.cpp command, and redeploy. This is also the only way to
|
||||
@@ -343,8 +343,8 @@ Two independent rollbacks, because two things changed on 2026-09-28:
|
||||
- GGUF header evidence for the payload swap: both files' headers were parsed
|
||||
and diffed directly (architecture, all 866 tensor names in order, per-tensor
|
||||
`ggml_type` and dims) before either was loaded
|
||||
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
|
||||
tunables: `scripts/frank-large.env`
|
||||
- Payload: `scripts/download-frank-large-swift.sh` (target),
|
||||
`scripts/download-frank-large.sh` (DFlash2 drafter)
|
||||
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
|
||||
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-dragoon.sh`;
|
||||
tunables: `scripts/frank-dragoon.env`
|
||||
- Payload: `scripts/download-frank-dragoon-swift.sh` (target),
|
||||
`scripts/download-frank-dragoon.sh` (DFlash2 drafter)
|
||||
- Predecessor: [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
|
||||
+21
-21
@@ -1,14 +1,14 @@
|
||||
# frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
|
||||
# frank-dragoon — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
|
||||
|
||||
frank-large is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
|
||||
frank-dragoon is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
|
||||
2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
|
||||
pinned digest, launched by `scripts/serve-frank-large.sh` under a root
|
||||
pinned digest, launched by `scripts/serve-frank-dragoon.sh` under a root
|
||||
systemd unit as a podman container — serving
|
||||
**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
|
||||
35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
|
||||
**DFlash2** drafter at **one slot x 200,700 ctx**.
|
||||
|
||||
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-large-rocmfpx-q6agent-uncensored-mtp]]),
|
||||
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]),
|
||||
which remains the rollback target with its payload on disk.
|
||||
|
||||
## Why swap
|
||||
@@ -25,21 +25,21 @@ confirmed by Frank before deploy:
|
||||
|
||||
1. **Text-only.** halogen loads the language tower only; README "honest
|
||||
limits": *"The vision encoder is ignored; image inputs are not supported."*
|
||||
The mmproj-era vision role moves wholly to frank-secondary.
|
||||
The mmproj-era vision role moves wholly to frank-zealot.
|
||||
2. **Censored base.** The checkpoint is a finetune of the censored
|
||||
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
|
||||
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
|
||||
|
||||
## Shape and why
|
||||
|
||||
`scripts/frank-large.env` carries every tunable with its evidence; the load-
|
||||
`scripts/frank-dragoon.env` carries every tunable with its evidence; the load-
|
||||
bearing numbers:
|
||||
|
||||
- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
|
||||
speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
|
||||
a step batches when more than one conversation is running, so the drafter
|
||||
stays off with more than one slot"). Slots>1 silently loses DFlash2; the
|
||||
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-large
|
||||
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-dragoon
|
||||
was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
|
||||
- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
|
||||
prompt is a 400 naming the limit, never a truncation. The harness side is
|
||||
@@ -83,7 +83,7 @@ bearing numbers:
|
||||
|thinking field|`reasoning_content` (same as before)|
|
||||
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|
||||
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|
||||
|image input|ignored/error — route to frank-secondary|
|
||||
|image input|ignored/error — route to frank-zealot|
|
||||
|
||||
## Deploy mechanics
|
||||
|
||||
@@ -91,7 +91,7 @@ bearing numbers:
|
||||
(published tag 0.1.4). The deployer gates the exact digest exists in podman
|
||||
storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
|
||||
was the Flash-Next w4b era, different image, retired.
|
||||
- **Payload:** `scripts/download-frank-large-halogen.sh` (fetch law: `curl -f`,
|
||||
- **Payload:** `scripts/download-frank-dragoon-halogen.sh` (fetch law: `curl -f`,
|
||||
exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
|
||||
token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
|
||||
tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
|
||||
@@ -101,20 +101,20 @@ bearing numbers:
|
||||
docker-run parity). The deployer and serve script refuse a missing file or
|
||||
ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
|
||||
those links dangle inside the container mount.
|
||||
- **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
|
||||
- **Unit:** `scripts/frank-dragoon.service` (root, `User=dank`, relinked into
|
||||
`/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
|
||||
stop -t 120 frank-large-halogen`, script runs `podman run` FOREGROUND
|
||||
stop -t 120 frank-dragoon-halogen`, script runs `podman run` FOREGROUND
|
||||
(2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
|
||||
only the container ID — the engine ledger existed nowhere on disk and
|
||||
the post-deploy grep below read a one-line hex file; foreground run
|
||||
returns the container's exit code so `Restart=always` is real and the
|
||||
ledger lands in the log). Engine stdout goes to
|
||||
`/home/dank/logs/frank_large.log` (the boot ledger lives there, not in
|
||||
`/home/dank/logs/frank_dragoon.log` (the boot ledger lives there, not in
|
||||
the journal).
|
||||
- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
|
||||
- **Cutover:** `deploy_llm_dragoon` in `scripts/deploy-remote.sh` sweeps the
|
||||
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
|
||||
the legacy `frank-large` container name by NAME (never `pkill -f
|
||||
llama-server` — that hits frank-secondary), retires the old quadlet if the 2026-09
|
||||
the legacy `frank-dragoon` container name by NAME (never `pkill -f
|
||||
llama-server` — that hits frank-zealot), retires the old quadlet if the 2026-09
|
||||
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
|
||||
- **Bench law:** stop+mask the unit before any GPU experiment on this box
|
||||
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
|
||||
@@ -122,7 +122,7 @@ bearing numbers:
|
||||
|
||||
## Verification (post-deploy checklist)
|
||||
|
||||
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_large.log`
|
||||
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_dragoon.log`
|
||||
— ledger shows the pool at the asked shape, DFlash2 active, the
|
||||
prompt cache line naming the pinned budget, NO clamp line. EXPECT the
|
||||
WARNING "smaller than ONE full-context entry (18.4 GB at
|
||||
@@ -136,7 +136,7 @@ bearing numbers:
|
||||
`reasoning_content` populated; decode rate from the timings.
|
||||
4. Spec decode fires: multi-token step behavior visible in rate; compare
|
||||
against DRAFTER=0 arm if in doubt (never against gufo — bench law).
|
||||
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-large`
|
||||
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-dragoon`
|
||||
incl. a tool call (V1TOOLS).
|
||||
6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
|
||||
|
||||
@@ -172,10 +172,10 @@ Measured on real agent traffic (serve_api lines) the day after cutover:
|
||||
one with public evidence, and it moved nothing that matters here.
|
||||
- **Routing is the applied mitigation (2026-10-06):** the harness's
|
||||
deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
|
||||
moved to frank-primary (radiance on Dankserver: fp8 KV at
|
||||
moved to frank-archon (radiance on Dankserver: fp8 KV at
|
||||
7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
|
||||
200k window, prefix-cache host+disk tiers). frank-large keeps OpenWebUI
|
||||
and shallow bursts, where it runs 30-50 t/s. frank-primary's
|
||||
200k window, prefix-cache host+disk tiers). frank-dragoon keeps OpenWebUI
|
||||
and shallow bursts, where it runs 30-50 t/s. frank-archon's
|
||||
depth-decode rate is designed-for but not yet measured on record.
|
||||
- **The lever that existed was the prompt cache, and it was OFF** (see
|
||||
the shape bullet). The pin turns the 85-180 s re-prefill per deep
|
||||
@@ -192,7 +192,7 @@ Measured on real agent traffic (serve_api lines) the day after cutover:
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:
|
||||
Revert the frank-dragoon swap commit and redeploy `dankclaw --service frank-dragoon`:
|
||||
the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
|
||||
the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
|
||||
`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
|
||||
+9
-9
@@ -1,11 +1,11 @@
|
||||
---
|
||||
type: reference
|
||||
description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
|
||||
tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance]
|
||||
description: frank-dragoon on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
|
||||
tags: [llm, frank-dragoon, ik_llama, dankclaw, rtx3090, strix-halo, performance]
|
||||
timestamp: 2026-09-16T03:30:00-07:00
|
||||
---
|
||||
|
||||
# frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
|
||||
# frank-dragoon — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
|
||||
|
||||
**RETIRED 2026-09-16, after one day.** Replaced by [[halogen-flash-server]],
|
||||
which was the runtime before it as well — the fleet went ik → halogen → back.
|
||||
@@ -23,10 +23,10 @@ hardware, and they are why the fleet went back. It served
|
||||
`/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
|
||||
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
|
||||
- Unit: **root systemd unit** `/etc/systemd/system/frank-large.service`
|
||||
symlinked to `scripts/frank-large.service` in the repo; enabled at boot;
|
||||
`Restart=always`. Serve script `scripts/serve-frank-large.sh` + env
|
||||
`scripts/frank-large.env`.
|
||||
- Unit: **root systemd unit** `/etc/systemd/system/frank-dragoon.service`
|
||||
symlinked to `scripts/frank-dragoon.service` in the repo; enabled at boot;
|
||||
`Restart=always`. Serve script `scripts/serve-frank-dragoon.sh` + env
|
||||
`scripts/frank-dragoon.env`.
|
||||
- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
|
||||
dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
|
||||
~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
|
||||
@@ -108,9 +108,9 @@ decoding is unavailable for this GGUF — mradermacher stripped the NextN head
|
||||
- **The systemd unit self-heals through manual tests**: `pkill` the server and
|
||||
the unit respawns it ~4 s later (Restart=always) — two instances, port
|
||||
races, phantom measurements. For manual experiments:
|
||||
`sudo systemctl stop frank-large.service` first (it is a root unit; plain
|
||||
`sudo systemctl stop frank-dragoon.service` first (it is a root unit; plain
|
||||
`systemctl --user` lines in older docs are stale), and restore with
|
||||
`sudo systemctl start frank-large.service`.
|
||||
`sudo systemctl start frank-dragoon.service`.
|
||||
- Verify "HTTP server listening" in the CURRENT log and
|
||||
`pgrep -xc llama-server` == 1 before trusting any measurement.
|
||||
- FA must stay ON — FA-off crashes the loader on this arch.
|
||||
+8
-8
@@ -1,13 +1,13 @@
|
||||
---
|
||||
type: reference
|
||||
title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
|
||||
title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-dragoon)"
|
||||
description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- kairic-edge
|
||||
- rocmfpx
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- iu4
|
||||
@@ -18,11 +18,11 @@ tags:
|
||||
timestamp: 2026-09-21
|
||||
---
|
||||
|
||||
# frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)
|
||||
# frank-dragoon — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)
|
||||
|
||||
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
|
||||
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
|
||||
> runbook on 2026-09-21 before it was replaced by
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] later the same day. The payload
|
||||
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] later the same day. The payload
|
||||
> and runtime remain on disk as the rollback target.
|
||||
|
||||
## What it served
|
||||
@@ -38,8 +38,8 @@ timestamp: 2026-09-21
|
||||
|
||||
## Rollback
|
||||
|
||||
Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
|
||||
for the current ACTIVE runbook and [[frank-large-unsloth-qwen38-flash-next]]
|
||||
Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
|
||||
for the current ACTIVE runbook and [[frank-dragoon-unsloth-qwen38-flash-next]]
|
||||
for the predecessor.
|
||||
|
||||
# Citations
|
||||
+22
-22
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
|
||||
description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
|
||||
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
|
||||
description: "The active frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rocmfpx
|
||||
@@ -15,14 +15,14 @@ tags:
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
|
||||
# frank-dragoon — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
|
||||
|
||||
> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
|
||||
> build script all on disk).** Replaced by [[frank-large-halogen27b-dflash2]]
|
||||
> build script all on disk).** Replaced by [[frank-dragoon-halogen27b-dflash2]]
|
||||
> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
|
||||
> remains the contract record for the rollback. Original banner follows.
|
||||
> ACTIVE since 2026-10-03. Cut over from
|
||||
> [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
|
||||
> [[frank-dragoon-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
|
||||
> on Frank's request: the *uncensored* variant of the same protected-heads
|
||||
> family, one quant rung down. The swap turned out to be a strict upgrade on
|
||||
> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
|
||||
@@ -34,15 +34,15 @@ timestamp: 2026-10-03
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
|
||||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
|
||||
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
|
||||
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
|
||||
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
|
||||
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
|
||||
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
|
||||
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
|
||||
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
|
||||
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
|
||||
| Fetch | [`scripts/download-frank-large-kingjones-uncensored.sh`](../../scripts/download-frank-large-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
|
||||
| Seam | `tests/frank_large_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
|
||||
| Fetch | [`scripts/download-frank-dragoon-kingjones-uncensored.sh`](../../scripts/download-frank-dragoon-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
|
||||
| Seam | `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
|
||||
|
||||
## Why this file
|
||||
|
||||
@@ -78,7 +78,7 @@ Unchanged from the Q8 era — the swap was payload-only:
|
||||
| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
|
||||
| `--fit off` | on | the auto-fitter would silently resize the window |
|
||||
| `--flash-attn on` | on | required for the long-window attention path |
|
||||
| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-large-rocmfpx-q8agent-mtp]] |
|
||||
| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-dragoon-rocmfpx-q8agent-mtp]] |
|
||||
|
||||
Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
|
||||
`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
|
||||
@@ -138,37 +138,37 @@ models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
|
||||
## Client contract
|
||||
|
||||
Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
|
||||
`frank-large`, native tools, thinking in `reasoning_content`,
|
||||
`frank-dragoon`, native tools, thinking in `reasoning_content`,
|
||||
`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
|
||||
card-recipe sampling defaults server-side, vision through the BF16 projector
|
||||
(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
|
||||
`content: null` when reasoning eats the budget, the self-recovering MTP
|
||||
boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
|
||||
in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
|
||||
in [[frank-dragoon-rocmfpx-q8agent-mtp]] and still applies.
|
||||
|
||||
Sweep key for scratch servers is now the payload name:
|
||||
`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
|
||||
llama-server` (frank-secondary is also one).
|
||||
llama-server` (frank-zealot is also one).
|
||||
|
||||
## Operations
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
systemctl status frank-large
|
||||
journalctl -u frank-large -n 200 --no-pager
|
||||
scripts/download-frank-large-kingjones-uncensored.sh --verify-only
|
||||
./scripts/deploy.sh dankclaw --service frank-dragoon
|
||||
systemctl status frank-dragoon
|
||||
journalctl -u frank-dragoon -n 200 --no-pager
|
||||
scripts/download-frank-dragoon-kingjones-uncensored.sh --verify-only
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
| Target | How |
|
||||
|---|---|
|
||||
| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_LARGE_MODEL_FILE` + `LLM_LARGE_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-large-kingjones.sh --verify-only` |
|
||||
| gufo Q4_K_XL era | git tag `frank-large-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-large-gufo-qwen38-27b]] |
|
||||
| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_DRAGOON_MODEL_FILE` + `LLM_DRAGOON_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-dragoon-kingjones.sh --verify-only` |
|
||||
| gufo Q4_K_XL era | git tag `frank-dragoon-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-dragoon-gufo-qwen38-27b]] |
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
Same decision tree as [[frank-large-rocmfpx-q8agent-mtp]]:
|
||||
Same decision tree as [[frank-dragoon-rocmfpx-q8agent-mtp]]:
|
||||
`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
|
||||
`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
|
||||
error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
|
||||
+26
-26
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
|
||||
description: "The retired frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
|
||||
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
|
||||
description: "The retired frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rocmfpx
|
||||
@@ -15,17 +15,17 @@ tags:
|
||||
timestamp: 2026-10-02
|
||||
---
|
||||
|
||||
# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
|
||||
# frank-dragoon — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
|
||||
|
||||
> **RETIRED 2026-10-03** — superseded by
|
||||
> [[frank-large-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
|
||||
> [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
|
||||
> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
|
||||
> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
|
||||
> runbook stays live as the rollback reference and the trap catalogue; the
|
||||
> payload it documents remains on disk at
|
||||
> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
|
||||
>
|
||||
> Cut over 2026-10-02 from [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
|
||||
> Cut over 2026-10-02 from [[frank-dragoon-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
|
||||
> on two requests from Frank: *highest quality that runs*, and *one full
|
||||
> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
|
||||
> could load — at 17.0 t/s, so quality was not bought with speed.
|
||||
@@ -39,15 +39,15 @@ timestamp: 2026-10-02
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
|
||||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
|
||||
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
|
||||
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
|
||||
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
|
||||
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
|
||||
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
|
||||
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
|
||||
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
|
||||
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
|
||||
| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
|
||||
| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
|
||||
| Fetch | [`scripts/download-frank-dragoon-kingjones.sh`](../../scripts/download-frank-dragoon-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
|
||||
| Seam | `tests/frank_dragoon_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
|
||||
|
||||
## Why this file, and why only this runtime
|
||||
|
||||
@@ -117,7 +117,7 @@ there is no second 26 GiB copy in memory. See the fleet law in
|
||||
`AGENTS.md → llama.cpp serving`.
|
||||
|
||||
**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
|
||||
`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
|
||||
`cache_prompt: false`, `frank-dragoon.service` stopped for every scratch arm (the
|
||||
benchmarking law), baseline arm stripped of `--model-draft` **and** every
|
||||
`--spec-*` flag:
|
||||
|
||||
@@ -144,7 +144,7 @@ re-litigates this:
|
||||
|
||||
| Behavior | Observation |
|
||||
|---|---|
|
||||
| Model id | `/v1/models` → `frank-large` |
|
||||
| Model id | `/v1/models` → `frank-dragoon` |
|
||||
| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
|
||||
| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
|
||||
| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
|
||||
@@ -162,7 +162,7 @@ Three traps, all hit for real:
|
||||
enough for the vision probe).
|
||||
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
|
||||
appears in the log under draft resets. Self-recovering; not an outage signal.
|
||||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-secondary is also a
|
||||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-zealot is also a
|
||||
`llama-server`. The sweep key here is the payload name:
|
||||
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
|
||||
|
||||
@@ -170,24 +170,24 @@ Three traps, all hit for real:
|
||||
|
||||
```bash
|
||||
# deploy (planner selects this endpoint only via its own unit/env/serve trio)
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
./scripts/deploy.sh dankclaw --service frank-dragoon
|
||||
|
||||
# status / logs / cause of a crash loop
|
||||
systemctl status frank-large
|
||||
journalctl -u frank-large -n 200 --no-pager
|
||||
systemctl status frank-dragoon
|
||||
journalctl -u frank-dragoon -n 200 --no-pager
|
||||
|
||||
# rebuild the pinned fork (only when deliberately moving the pin)
|
||||
scripts/build-rocmfpx.sh
|
||||
|
||||
# re-verify the payload without re-downloading
|
||||
scripts/download-frank-large-kingjones.sh --verify-only
|
||||
scripts/download-frank-dragoon-kingjones.sh --verify-only
|
||||
```
|
||||
|
||||
`serve-frank-large.sh` refuses to start — before touching the port — unless the
|
||||
fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
|
||||
`serve-frank-dragoon.sh` refuses to start — before touching the port — unless the
|
||||
fork binary, the weights, the MTP head and (unless `FRANK_DRAGOON_MMPROJ=""`) the
|
||||
projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
|
||||
broken payload never stops the running unit. Caller environment **overrides**
|
||||
`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
|
||||
`frank-dragoon.env`, and defaults use `:=` so an explicitly empty value surfaces
|
||||
as the error it reads as.
|
||||
|
||||
## Rollback (as of retirement, 2026-10-03)
|
||||
@@ -196,16 +196,16 @@ as the error it reads as.
|
||||
payload and remains on disk** at
|
||||
`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
|
||||
2026-10-03 swap commit (unit/env/serve trio) or repoint
|
||||
`FRANK_LARGE_MODEL_FILE`/`LLM_LARGE_MODEL_FILE`, verify with
|
||||
`scripts/download-frank-large-kingjones.sh --verify-only`, redeploy.
|
||||
`FRANK_DRAGOON_MODEL_FILE`/`LLM_DRAGOON_MODEL_FILE`, verify with
|
||||
`scripts/download-frank-dragoon-kingjones.sh --verify-only`, redeploy.
|
||||
|
||||
git tag **`frank-large-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
|
||||
git tag **`frank-dragoon-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
|
||||
DFlash2) is now a **re-download** rollback: the gufo payloads
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
|
||||
removed from disk on 2026-10-03 with Frank's explicit approval to free space.
|
||||
`scripts/build-gufo.sh` remains on disk; re-fetch per
|
||||
[[frank-large-gufo-qwen38-27b]].
|
||||
[[frank-dragoon-gufo-qwen38-27b]].
|
||||
|
||||
The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
|
||||
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
|
||||
@@ -215,7 +215,7 @@ recoverable until the trash is purged, and never part of a rollback plan.
|
||||
## When it misbehaves
|
||||
|
||||
- **`unknown tensor type` / ftype 115 refused** — something moved off the
|
||||
ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
|
||||
ROCmFPX binary. Check `readlink -f $(systemctl show frank-dragoon -p ExecStart)`
|
||||
and the fork commit; this file cannot run on stock llama.cpp or gufo.
|
||||
- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
|
||||
older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
|
||||
+18
-18
@@ -1,13 +1,13 @@
|
||||
---
|
||||
type: reference
|
||||
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (2026-09-21 → 2026-09-28; replaced by [[frank-large-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
|
||||
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
|
||||
description: "The RETIRED frank-dragoon runbook (2026-09-21 → 2026-09-28; replaced by [[frank-dragoon-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- llama.cpp
|
||||
- unsloth
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- q8
|
||||
@@ -17,31 +17,31 @@ tags:
|
||||
timestamp: 2026-09-21
|
||||
---
|
||||
|
||||
# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||||
# frank-dragoon — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||||
|
||||
> **RETIRED 2026-09-28**, replaced by [[frank-large-gufo-qwen38-27b]] (the
|
||||
> **RETIRED 2026-09-28**, replaced by [[frank-dragoon-gufo-qwen38-27b]] (the
|
||||
> gufo engine, same weights, same port and alias); the prebuilt is intact on
|
||||
> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
|
||||
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
|
||||
> [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
|
||||
> [[frank-dragoon-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
|
||||
> preceded).
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
|
||||
(`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
|
||||
A mainline build, not a fork: it supports the `qwen3_5` architecture and the
|
||||
DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
|
||||
`common_speculative_impl_draft_dflash`).
|
||||
- **Service shape:** root systemd unit `scripts/frank-large.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
|
||||
`scripts/frank-large.env`.
|
||||
- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-dragoon.sh` with tunables from
|
||||
`scripts/frank-dragoon.env`.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
|
||||
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
|
||||
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-dragoon.sh`).
|
||||
- **Context:** one stream, full 262,144 tokens (the model's native window),
|
||||
`PARALLEL_SLOTS=1`.
|
||||
- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
|
||||
@@ -75,18 +75,18 @@ harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
|
||||
|
||||
## Payload
|
||||
|
||||
`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
|
||||
`scripts/download-frank-dragoon.sh` fetches the main GGUF and the DFlash2
|
||||
drafter, each verified against its declared byte count + sha256
|
||||
(`--verify-only` re-checks). The serve script refuses to start if the binary,
|
||||
the main GGUF, or the drafter is missing.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
||||
- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
|
||||
`--reasoning-format deepseek`, unlimited budget).
|
||||
- **Tools:** supported (`--jinja` is pinned).
|
||||
- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
|
||||
- **Vision:** none — text-only (no mmproj), matching the prior frank-dragoon.
|
||||
- **Sampling:** greedy by default; a client that sends a temperature gets its
|
||||
own, and the harness applies per-model overrides from `models.yml`.
|
||||
|
||||
@@ -105,8 +105,8 @@ the main GGUF, or the drafter is missing.
|
||||
|
||||
## Rollback
|
||||
|
||||
Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
|
||||
IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
|
||||
Payload and runtime for [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge
|
||||
IU4 W4A4) remain intact on disk. See [[frank-dragoon-unsloth-qwen38-flash-next]]
|
||||
for the Flash-Next predecessor it replaced on 2026-09-21.
|
||||
|
||||
# Citations
|
||||
@@ -114,4 +114,4 @@ for the Flash-Next predecessor it replaced on 2026-09-21.
|
||||
- unsloth prebuilts: https://huggingface.co/unsloth
|
||||
- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
|
||||
- DFlash2 drafter: https://huggingface.co/z-lab
|
||||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||||
- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
|
||||
+8
-8
@@ -1,12 +1,12 @@
|
||||
---
|
||||
type: reference
|
||||
title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
|
||||
title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-dragoon)"
|
||||
description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- unsloth
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
@@ -15,11 +15,11 @@ tags:
|
||||
timestamp: 2026-09-21
|
||||
---
|
||||
|
||||
# frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
|
||||
# frank-dragoon — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
|
||||
|
||||
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
|
||||
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
|
||||
> runbook on 2026-09-21 before it was replaced by
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
|
||||
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
|
||||
> same day. The payload and runtime remain on disk as the rollback target.
|
||||
|
||||
## What it served
|
||||
@@ -37,8 +37,8 @@ timestamp: 2026-09-21
|
||||
|
||||
## Rollback
|
||||
|
||||
Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
|
||||
for the current ACTIVE runbook and [[frank-large-ciru-orca]] for the
|
||||
Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
|
||||
for the current ACTIVE runbook and [[frank-dragoon-ciru-orca]] for the
|
||||
predecessor.
|
||||
|
||||
# Citations
|
||||
+26
-26
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
|
||||
description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
|
||||
title: "frank-dragoon on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
|
||||
description: "The RETIRED frank-dragoon runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-dragoon-gufo-qwen38-27b]] by Frank the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- dankclaw
|
||||
- vllm
|
||||
- strix-halo
|
||||
@@ -15,27 +15,27 @@ tags:
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||||
# frank-dragoon — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||||
|
||||
> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
|
||||
> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
|
||||
> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
|
||||
> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
|
||||
> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
|
||||
> torchao config unwired). Back to [[frank-dragoon-gufo-qwen38-27b]]
|
||||
> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
|
||||
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
|
||||
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
|
||||
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
|
||||
`podman run` driven by `scripts/serve-frank-large.sh` under a root unit
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`podman run` driven by `scripts/serve-frank-dragoon.sh` under a root unit
|
||||
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
|
||||
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
|
||||
(17.53 GiB, 4 shards) — the same checkpoint frank-secondary serves as an INT8
|
||||
(17.53 GiB, 4 shards) — the same checkpoint frank-zealot serves as an INT8
|
||||
W8A16 AutoRound conversion on the 3090, here unquantized.
|
||||
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
|
||||
are loaded into the draft pass).
|
||||
@@ -67,7 +67,7 @@ scheduler instead of carved per session.
|
||||
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
|
||||
|
||||
One session, temperature 0, 28,647-token prefill / 512-token decode, same
|
||||
client for all three arms (2026-10-01 bench window, frank-large stopped per
|
||||
client for all three arms (2026-10-01 bench window, frank-dragoon stopped per
|
||||
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
|
||||
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
|
||||
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
|
||||
@@ -82,7 +82,7 @@ Qwen3.5-9B that preserves the hidden-state geometry — identical
|
||||
per-position acceptance is the direct measurement that the distillation
|
||||
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
|
||||
(verified in both the BF16 base and the INT8 conversion) — which is exactly
|
||||
why frank-secondary's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||||
why frank-zealot's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||||
reads the head from the TARGET checkpoint, vLLM reads the draft from a
|
||||
separate path. That difference is the whole trick:
|
||||
|
||||
@@ -91,8 +91,8 @@ separate path. That difference is the whole trick:
|
||||
```
|
||||
|
||||
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
|
||||
[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
|
||||
`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
|
||||
[1,5,…,29]) stays fetched on disk; flip `FRANK_DRAGOON_SPEC=dflash` and repoint
|
||||
`FRANK_DRAGOON_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
|
||||
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
|
||||
|
||||
## The numerics trap (greedy text is NOT preserved by spec decode)
|
||||
@@ -109,14 +109,14 @@ chase text drift between spec and no-spec arms as a regression.
|
||||
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
|
||||
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
|
||||
byte-count + LFS sha256-verified by
|
||||
`old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
|
||||
`old/download-frank-dragoon-mimo.sh` (`--verify-only` re-checks; no token on
|
||||
the curl command line).
|
||||
- MTP source: `old/download-frank-large-qwen35.sh` →
|
||||
- MTP source: `old/download-frank-dragoon-qwen35.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
|
||||
`model.safetensors.index.json`; the shards carry the odd HF asset names
|
||||
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
|
||||
the engine boots from exactly this tree).
|
||||
- DFlash drafter: `old/download-frank-large-dflash.sh` →
|
||||
- DFlash drafter: `old/download-frank-dragoon-dflash.sh` →
|
||||
`/home/dank/models/Qwen3.5-9B-DFlash/`.
|
||||
- **Payload status 2026-10-02:** all three trees moved to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
|
||||
@@ -131,7 +131,7 @@ chase text drift between spec and no-spec arms as a regression.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
||||
- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
|
||||
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
|
||||
`models.yml` compat block for this provider sets
|
||||
@@ -146,7 +146,7 @@ chase text drift between spec and no-spec arms as a regression.
|
||||
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
|
||||
`chat_template_kwargs` form is what vLLM reads; a top-level
|
||||
`preserve_thinking` is ignored there.
|
||||
- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
|
||||
- **Env hygiene:** every tunable is `FRANK_DRAGOON_`-prefixed because HIP
|
||||
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
|
||||
the env file with `set -a`.
|
||||
|
||||
@@ -154,23 +154,23 @@ chase text drift between spec and no-spec arms as a regression.
|
||||
|
||||
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
|
||||
- The serve script sweeps stale container identities before starting:
|
||||
`frank-large-mimo` itself and the bench-window container `vllm-mimo`.
|
||||
`frank-dragoon-mimo` itself and the bench-window container `vllm-mimo`.
|
||||
- Container death → `podman wait` returns non-zero from ExecStart →
|
||||
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
|
||||
frank-primary lesson).
|
||||
- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
|
||||
frank-archon lesson).
|
||||
- Bench windows: `systemctl stop frank-dragoon` is enough (mask fails on the
|
||||
relinked unit; the deploy restores).
|
||||
- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
|
||||
boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
|
||||
262,144 tokens per request: 3.13x` — four sessions share the pool
|
||||
(the scheduler, not a carve). Live battery green: alias `frank-large`,
|
||||
(the scheduler, not a carve). Live battery green: alias `frank-dragoon`,
|
||||
reasoning split into `reasoning`, tool_calls with correct args, and
|
||||
`/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
|
||||
matching the bench window.
|
||||
|
||||
## Rollback
|
||||
|
||||
Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
|
||||
Git tag **`frank-dragoon-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
|
||||
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
|
||||
payload + DFlash2 drafter, all still on the host. Restoring the tag's
|
||||
unit/env/serve trio and redeploying returns the 27B endpoint.
|
||||
@@ -181,6 +181,6 @@ unit/env/serve trio and redeploying returns the 27B endpoint.
|
||||
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
|
||||
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
|
||||
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
|
||||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||||
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
|
||||
[[frank-secondary-sglang-mimo-v26]]
|
||||
- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
|
||||
- Predecessor: [[frank-dragoon-gufo-qwen38-27b]]; sibling on the same arch:
|
||||
[[frank-zealot-sglang-mimo-v26]]
|
||||
@@ -1,16 +0,0 @@
|
||||
---
|
||||
type: reference
|
||||
title: "CIRU v4.4.1 Orca on Dankclaw (frank-large)"
|
||||
description: "The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- ciru
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
- rocm
|
||||
- mtp
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
@@ -211,7 +211,7 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
|
||||
|
||||
- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
|
||||
history
|
||||
- [[frank-secondary-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
|
||||
- [[frank-zealot-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
|
||||
at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
|
||||
difference: with no KV to attend over, per-pass overhead dominates and a
|
||||
bigger micro-batch wins; at 64k of resident KV it does not.
|
||||
|
||||
@@ -312,7 +312,7 @@ came up healthy and *text-only*: image requests answered 500 "image input is
|
||||
not supported — hint: … provide the mmproj" and the load log had no mmproj
|
||||
line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the
|
||||
model directory. The serve script now passes `--mmproj` explicitly (the
|
||||
frank-secondary pattern), refuses to start without the file when vision is on, and
|
||||
frank-zealot pattern), refuses to start without the file when vision is on, and
|
||||
the deploy handler's preflight checks for it. Verified end-to-end after the
|
||||
fix: a generated 64x32 test image was described from its pixels — with
|
||||
thinking on the model reasoned about "the three horizontal bars" (its reading
|
||||
@@ -724,7 +724,7 @@ fleet.
|
||||
2026-09-15, whose `--split-mode layer` measurements this service reuses
|
||||
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
|
||||
collective, and every software escape that is closed
|
||||
- [[frank-secondary-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
|
||||
- [[frank-zealot-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
|
||||
3090, MTP, the image path)
|
||||
- [[Dankserver System Specs]] — hardware inventory
|
||||
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
|
||||
|
||||
@@ -602,7 +602,7 @@ but not both. That is the wall, and it is capacity, not scheduling.
|
||||
### Watching it instead of measuring it by hand
|
||||
|
||||
Everything above was diagnosed with ad-hoc probes. The same signals are now
|
||||
scraped — Prometheus has `frank-medium` and `frank-large` jobs
|
||||
scraped — Prometheus has `frank-medium` and `frank-dragoon` jobs
|
||||
(`shared/prometheus/prometheus.yml`), so the numbers that matter have history:
|
||||
`vllm:kv_cache_usage_perc` (per-card pool fill), `vllm:num_requests_waiting`
|
||||
(queue depth), `rate(vllm:prefix_cache_hits_total[5m]) /
|
||||
|
||||
@@ -144,7 +144,7 @@ ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
|
||||
|
||||
First deploy (2026-09-15) also ran `retire_root_llm_small()` in
|
||||
`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
|
||||
disabled and moved to `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units`
|
||||
disabled and moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units`
|
||||
(the host's `systemctl disable` has been observed to delete unit files,
|
||||
2026-08-26 incident, so both outcomes are tolerated).
|
||||
|
||||
@@ -156,7 +156,7 @@ Deployed flags (see `scripts/frank-medium.env` for the full set):
|
||||
hidden state is produced. Reasoning on, deepseek format, unlimited budget;
|
||||
sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8,
|
||||
top-k 20, min-p 0, presence 1.5). Text only — vision stays on
|
||||
`frank-large` (Dankclaw).
|
||||
`frank-dragoon` (Dankclaw).
|
||||
|
||||
## Verified serving behavior (2026-09-15)
|
||||
|
||||
@@ -181,7 +181,7 @@ are projected into `reasoning_content` (deepseek format); requests that send
|
||||
no sampler fields get the server's CLI defaults, not greedy. Text only.
|
||||
Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default`
|
||||
→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small
|
||||
routes) → `frank-medium`, vision stays `frank-large`.
|
||||
routes) → `frank-medium`, vision stays `frank-dragoon`.
|
||||
|
||||
## Host ROCm posture (post-cleanup)
|
||||
|
||||
|
||||
@@ -48,17 +48,17 @@ upstream project on this exact silicon. That single flag is the whole decision:
|
||||
|
||||
Consequences accepted at retirement: **audio input left the fleet with no
|
||||
replacement** (halogen reads images only, no audio or video), and the surviving
|
||||
image path is halogen's vision tower on `frank-large`.
|
||||
image path is halogen's vision tower on `frank-dragoon`.
|
||||
|
||||
## Where its routes went
|
||||
|
||||
| consumer | setting | now |
|
||||
|---|---|---|
|
||||
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) |
|
||||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-secondary` |
|
||||
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` |
|
||||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-secondary` |
|
||||
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` |
|
||||
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-dragoon` (halogen vision tower) |
|
||||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-zealot` |
|
||||
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-dragoon` |
|
||||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-zealot` |
|
||||
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-dragoon:medium` |
|
||||
|
||||
The `flm (dankclaw-npu)` provider block was removed from the harness model
|
||||
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
|
||||
@@ -104,7 +104,7 @@ all live-verified on the host:
|
||||
carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID
|
||||
state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`.
|
||||
3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain
|
||||
identity-mapped (no DMA-translation overhead for `frank-large`) while
|
||||
identity-mapped (no DMA-translation overhead for `frank-dragoon`) while
|
||||
enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
|
||||
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
|
||||
zero `SVA bind device failed` entries since boot, `accel0` recreated at
|
||||
|
||||
@@ -17,15 +17,15 @@ The old scripts moved to `old/scripts/` in this repository.
|
||||
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-secondary | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
| frank-dragoon | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-zealot | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
|
||||
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
|
||||
tensor split, no NPU.
|
||||
|
||||
## Models
|
||||
|
||||
### frank-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||||
### frank-dragoon — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||||
|
||||
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
|
||||
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
|
||||
@@ -40,7 +40,7 @@ tensor split, no NPU.
|
||||
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
|
||||
(48/64/24).
|
||||
|
||||
### frank-secondary — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
### frank-zealot — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
|
||||
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
|
||||
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
|
||||
@@ -55,7 +55,7 @@ tensor split, no NPU.
|
||||
instead of the Qwen3 XML structure). The Quark quant follows the format
|
||||
cleanly; swapped in the evening of 2026-08-17.
|
||||
|
||||
## Context Math (frank-large, RTX 3090)
|
||||
## Context Math (frank-dragoon, RTX 3090)
|
||||
|
||||
KV per token: 16 attention layers x 4 KV heads x 256 dim.
|
||||
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
|
||||
@@ -72,11 +72,11 @@ Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
|
||||
|
||||
Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
|
||||
`scripts/frank-large.env` to 122880 for ~1.8 GiB headroom.
|
||||
`scripts/frank-dragoon.env` to 122880 for ~1.8 GiB headroom.
|
||||
|
||||
## Install Runbook
|
||||
|
||||
### Dankserver (frank-large)
|
||||
### Dankserver (frank-dragoon)
|
||||
|
||||
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
|
||||
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
|
||||
@@ -96,17 +96,17 @@ Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
|
||||
4. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-large.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-large
|
||||
sudo systemctl restart frank-large
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-dragoon.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-dragoon
|
||||
sudo systemctl restart frank-dragoon
|
||||
```
|
||||
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
|
||||
6. Confirm VRAM headroom with `nvidia-smi`.
|
||||
|
||||
### Dankclaw (frank-secondary)
|
||||
### Dankclaw (frank-zealot)
|
||||
|
||||
1. Stop the old services: `sudo systemctl stop frank-large
|
||||
frank-secondary` (retired PP head and NPU endpoint).
|
||||
1. Stop the old services: `sudo systemctl stop frank-dragoon
|
||||
frank-zealot` (retired PP head and NPU endpoint).
|
||||
2. Install uv and create the venv:
|
||||
```
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
@@ -131,9 +131,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
vision encoder falls back to torch SDPA and engine init OOMs.
|
||||
5. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-secondary
|
||||
sudo systemctl restart frank-secondary
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-zealot
|
||||
sudo systemctl restart frank-zealot
|
||||
```
|
||||
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
|
||||
|
||||
@@ -159,7 +159,7 @@ on gfx1151; newer vLLM releases may already include it.
|
||||
|
||||
Three changes landed after the initial deployment:
|
||||
|
||||
### 1. Tool calling on frank-secondary (HTTP 400 fix)
|
||||
### 1. Tool calling on frank-zealot (HTTP 400 fix)
|
||||
|
||||
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
|
||||
unless the server enables tool calling. Fixes applied:
|
||||
@@ -184,7 +184,7 @@ model behavior, which motivated the swap.
|
||||
|
||||
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
|
||||
|
||||
`frank-secondary` now serves
|
||||
`frank-zealot` now serves
|
||||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
|
||||
vLLM-native expert layout). The Avesed compressed-tensors quant's
|
||||
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
|
||||
description: "The RETIRED frank-secondary runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-09-16
|
||||
---
|
||||
+4
-4
@@ -1,7 +1,7 @@
|
||||
---
|
||||
timestamp: 2026-09-03
|
||||
type: reference
|
||||
title: frank-secondary Benchmark — ncmoe and Batch Tuning
|
||||
title: frank-zealot Benchmark — ncmoe and Batch Tuning
|
||||
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
|
||||
tags:
|
||||
- benchmark
|
||||
@@ -11,11 +11,11 @@ tags:
|
||||
- dankserver
|
||||
---
|
||||
|
||||
# frank-secondary Benchmark — ncmoe and Batch/Ubatch Tuning
|
||||
# frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning
|
||||
|
||||
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
|
||||
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
|
||||
`--n-cpu-moe` and `--ubatch-size` values for `frank-secondary`.
|
||||
`--n-cpu-moe` and `--ubatch-size` values for `frank-zealot`.
|
||||
|
||||
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
|
||||
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
|
||||
@@ -106,7 +106,7 @@ service runs 700416 cells (13,680 MiB).
|
||||
|
||||
## Chosen State
|
||||
|
||||
> **Live drift (2026-08-22 audit):** the deployed `frank-secondary` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
|
||||
> **Live drift (2026-08-22 audit):** the deployed `frank-zealot` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
|
||||
|
||||
Prefill speed is the priority, so the state keeps the top prefill tier and
|
||||
moves one expert layer to CPU for stability:
|
||||
+16
-16
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
|
||||
description: "The active frank-secondary runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
|
||||
description: "The active frank-zealot runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
@@ -14,25 +14,25 @@ tags:
|
||||
timestamp: 2026-09-17
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
|
||||
# frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
|
||||
|
||||
> **ACTIVE since 2026-09-17.** Replaces [[frank-secondary-qwen38-gsq-rco]]
|
||||
> **ACTIVE since 2026-09-17.** Replaces [[frank-zealot-qwen38-gsq-rco]]
|
||||
> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
|
||||
> [[frank-secondary-vllm-gemma4]] (the same-day vLLM experiment, also the
|
||||
> rollback target). Same port (7771), same alias (`frank-secondary`), same
|
||||
> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment, also the
|
||||
> rollback target). Same port (7771), same alias (`frank-zealot`), same
|
||||
> client contract surface.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Endpoint:** `frank-zealot` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
|
||||
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
|
||||
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
|
||||
collapses prefill ~12x on CUDA.
|
||||
- **Service shape:** root systemd unit `scripts/frank-secondary.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-secondary.sh` with tunables from
|
||||
`scripts/frank-secondary.env`.
|
||||
- **Service shape:** root systemd unit `scripts/frank-zealot.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-zealot.sh` with tunables from
|
||||
`scripts/frank-zealot.env`.
|
||||
|
||||
## Shape
|
||||
|
||||
@@ -51,7 +51,7 @@ timestamp: 2026-09-17
|
||||
the card. 4 is the measured max.
|
||||
- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
|
||||
only tokenizes and samples (its CPU belongs to the halogen engine serving
|
||||
frank-large).
|
||||
frank-dragoon).
|
||||
|
||||
## Spec method
|
||||
|
||||
@@ -90,7 +90,7 @@ fault).
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
|
||||
- **Alias:** `frank-zealot` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
|
||||
deepseek`, unlimited budget).
|
||||
- **Tools:** supported (`--jinja` is pinned even though this build defaults
|
||||
@@ -109,12 +109,12 @@ pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
|
||||
|
||||
## Rollback
|
||||
|
||||
Payload and runtime for [[frank-secondary-qwen38-gsq-rco]] (ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-secondary-vllm-gemma4]]
|
||||
Payload and runtime for [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-zealot-vllm-gemma4]]
|
||||
is the same-day vLLM experiment (also a rollback target).
|
||||
|
||||
# Citations
|
||||
|
||||
- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
|
||||
- llama.cpp: https://github.com/ggml-org/llama.cpp
|
||||
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
|
||||
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
|
||||
# frank-zealot — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
|
||||
|
||||
**RETIRED 2026-10-07** — same day it went live. Superseded by
|
||||
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
|
||||
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
|
||||
2 × 200,704 with exact KV); the payload stays at
|
||||
`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the
|
||||
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
|
||||
@@ -108,4 +108,4 @@ free).
|
||||
Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
|
||||
`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
|
||||
payload remains on disk. See
|
||||
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]].
|
||||
[[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]].
|
||||
+21
-21
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
|
||||
description: "The retired 2026-10-02/03 frank-secondary runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
|
||||
description: "The retired 2026-10-02/03 frank-zealot runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
@@ -14,17 +14,17 @@ tags:
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
|
||||
# frank-zealot — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
|
||||
|
||||
> **RETIRED 2026-10-03** — superseded by
|
||||
> [[frank-secondary-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
|
||||
> [[frank-zealot-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
|
||||
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
|
||||
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
|
||||
> decode is the measured price of the higher rung on this bandwidth-bound
|
||||
> card. This file remains the rollback reference and the owner of the
|
||||
> measured boot-pair table.
|
||||
|
||||
> Its own history: one quant step down from [[frank-secondary-llama-mimo-q6k]]
|
||||
> Its own history: one quant step down from [[frank-zealot-llama-mimo-q6k]]
|
||||
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
|
||||
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
|
||||
> conversations of 200,704 tokens with the KV VALUES unquantized** —
|
||||
@@ -36,11 +36,11 @@ timestamp: 2026-10-03
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-secondary.sh`|
|
||||
|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
|
||||
|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-zealot.sh`|
|
||||
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|
||||
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
@@ -100,13 +100,13 @@ The per-slot window is the whole price, in both directions:
|
||||
- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
|
||||
sets `contextWindow: 200704` for this endpoint — the harness compacts before
|
||||
it reaches the ceiling instead of discovering it.
|
||||
- **frank-large's tallest pre-compaction prompt now fits here.** Compaction
|
||||
for frank-large sessions runs on this endpoint (`compactionModel` in
|
||||
`models.yml`); frank-large compacts at 70 % of 262,144 = 183,501 tokens,
|
||||
- **frank-dragoon's tallest pre-compaction prompt now fits here.** Compaction
|
||||
for frank-dragoon sessions runs on this endpoint (`compactionModel` in
|
||||
`models.yml`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
|
||||
which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
|
||||
tokens to spare. The 2026-10-02 window trim had restored the fit for
|
||||
frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens
|
||||
the fit to frank-large itself. Nothing routes compaction automatically —
|
||||
frank-archon's smaller 140,492-token trigger; the 2026-10-03 reslice widens
|
||||
the fit to frank-dragoon itself. Nothing routes compaction automatically —
|
||||
the point is the ceiling no longer forbids it.
|
||||
- Two simultaneous generations are the slot ceiling; a third request queues
|
||||
behind a released slot (the mechanism measured on the 3-slot shape).
|
||||
@@ -128,7 +128,7 @@ The per-slot window is the whole price, in both directions:
|
||||
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
|
||||
but refuses a request without `messages` in this build, so chat is the only
|
||||
vision path.
|
||||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-secondary.env` drops
|
||||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-zealot.env` drops
|
||||
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
|
||||
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
|
||||
refusal, never a silent fallback to blindness.
|
||||
@@ -145,15 +145,15 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-secondary.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|
||||
|Q6_K, same runtime|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-secondary-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|
||||
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-zealot.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|
||||
|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|
||||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SECONDARY_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-secondary-sglang-int8.sh`|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_ZEALOT_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script exits 1
|
||||
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script exits 1
|
||||
naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
+18
-18
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
|
||||
description: "The RETIRED frank-secondary runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
|
||||
description: "The RETIRED frank-zealot runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
@@ -12,10 +12,10 @@ tags:
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
# frank-zealot — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **RETIRED 2026-10-05 (evening).** Superseded by
|
||||
> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
|
||||
> [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
|
||||
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
|
||||
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
|
||||
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
|
||||
@@ -29,14 +29,14 @@ timestamp: 2026-10-03
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-zealot`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||||
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
|
||||
| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
|
||||
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
|
||||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
@@ -62,12 +62,12 @@ MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
|
||||
under the floor — and the calibrated bandwidth law (it predicted this file's
|
||||
94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
|
||||
−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
|
||||
pool to 2 × 182,272 — which also breaks frank-large's 183,501-token
|
||||
pool to 2 × 182,272 — which also breaks frank-dragoon's 183,501-token
|
||||
compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
|
||||
were declined. Re-open only on evidence the Q6_K residual matters.
|
||||
|
||||
**Pool prices on this card come from boots, never formulas** — the measured
|
||||
boot-pair table lives in [[frank-secondary-llama-mimo-q5km]] (*Shape and KV
|
||||
boot-pair table lives in [[frank-zealot-llama-mimo-q5km]] (*Shape and KV
|
||||
contract*) and still prices every rollback shape.
|
||||
|
||||
## Compute-buffer geometry (2026-10-05)
|
||||
@@ -78,7 +78,7 @@ measured rather than assumed, because the predecessor shape recorded **"ubatch
|
||||
|
||||
**That failure was a different shape, and it did not reproduce here.** The
|
||||
service itself is the bench (one card holds one server, so no scratch port):
|
||||
the old rung was re-booted through a transient `FRANK_SECONDARY_ENV_FILE` drop-in
|
||||
the old rung was re-booted through a transient `FRANK_ZEALOT_ENV_FILE` drop-in
|
||||
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
|
||||
2,849 MiB free — which makes the ubatch the only variable in the comparison.
|
||||
|
||||
@@ -99,7 +99,7 @@ Three readings:
|
||||
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
|
||||
inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
|
||||
and the pool's 12,288-token (384 MiB) ladder in
|
||||
[[frank-secondary-llama-mimo-q5km]] is the other way to buy room back.
|
||||
[[frank-zealot-llama-mimo-q5km]] is the other way to buy room back.
|
||||
|
||||
**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
|
||||
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
|
||||
@@ -125,7 +125,7 @@ that is the pair the curve was measured at.
|
||||
|
||||
## Client contract
|
||||
|
||||
llama.cpp surface, live-verified on this model family: model id `frank-secondary`,
|
||||
llama.cpp surface, live-verified on this model family: model id `frank-zealot`,
|
||||
native tools via `--jinja` (`tool_calls` in the response), thinking in
|
||||
**`reasoning_content`**, vision via `image_url` content parts. Two vision
|
||||
traps, both measured here:
|
||||
@@ -147,13 +147,13 @@ so the harness compacts before llama.cpp answers a longer prompt with a 400.
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q5_K_M at the same shape|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|
||||
|Q5_K_M at the same shape|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|
||||
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|
||||
|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|
||||
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
Same decision tree as [[frank-secondary-llama-mimo-q5km]] (*When it misbehaves*):
|
||||
Same decision tree as [[frank-zealot-llama-mimo-q5km]] (*When it misbehaves*):
|
||||
`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
|
||||
200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
|
||||
a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
|
||||
+10
-10
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||||
description: "The retired 2026-10-01 frank-secondary runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||||
description: "The retired 2026-10-01 frank-zealot runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
@@ -14,10 +14,10 @@ tags:
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
# frank-zealot — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
|
||||
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
|
||||
> [[frank-secondary-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||||
> target.
|
||||
>
|
||||
@@ -29,11 +29,11 @@ timestamp: 2026-10-01
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|
||||
|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
|
||||
|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
|
||||
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|
||||
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
@@ -95,12 +95,12 @@ cmake --build build-cuda -j 16 --target llama-server
|
||||
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
|
||||
AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
|
||||
stay on disk as the rollback: repoint the unit's serve script and env from
|
||||
git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
|
||||
git history (`git show frank-dragoon-gufo-q4kxl-20261001` era is BEFORE this;
|
||||
use the parent of the 2026-10-01 night swap commit).
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script
|
||||
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script
|
||||
exits 1 naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
|
||||
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
+18
-18
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
@@ -12,10 +12,10 @@ tags:
|
||||
timestamp: 2026-10-07
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
|
||||
# frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
Live since 2026-10-07, replacing the
|
||||
[[frank-secondary-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
|
||||
[[frank-zealot-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
|
||||
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
|
||||
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
|
||||
conversation slot and unquantized KV back.
|
||||
@@ -25,14 +25,14 @@ conversation slot and unquantized KV back.
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-secondary` name), `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||||
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
|
||||
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine) |
|
||||
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
|
||||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
|
||||
|
||||
## Payload geometry (read from the GGUF header, not the model card)
|
||||
|
||||
@@ -68,7 +68,7 @@ that is what makes two 200k slots affordable where the 26B MoE fit only one.
|
||||
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
|
||||
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
|
||||
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
|
||||
2,279** at ubatch 2048 — see [[frank-secondary-llama-mimo-q6k-2x200704]].
|
||||
2,279** at ubatch 2048 — see [[frank-zealot-llama-mimo-q6k-2x200704]].
|
||||
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
|
||||
so the projection is ~21,430 used / ~2,700 free.
|
||||
|
||||
@@ -80,7 +80,7 @@ fallback rung if a future shape lands under the 2 GiB floor.
|
||||
|
||||
`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
|
||||
projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
|
||||
(`--log-verbosity 4`, `journalctl -u frank-secondary`):
|
||||
(`--log-verbosity 4`, `journalctl -u frank-zealot`):
|
||||
|
||||
|Line|Value|
|
||||
|---|---|
|
||||
@@ -98,7 +98,7 @@ safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
|
||||
**past** its window). Weights load in ~6 s from page cache; `/health` answers
|
||||
`{"status":"ok"}` and `/v1/models` serves the alias `frank-zealot`.
|
||||
|
||||
**Wire rename (2026-10-09):** `ALIAS` moved `frank-secondary` → `frank-zealot`,
|
||||
**Wire rename (2026-10-09):** `ALIAS` moved `frank-zealot` → `frank-zealot`,
|
||||
so the served id matches the harness model id and the family-facing name.
|
||||
Consumers re-keyed in the same change: OpenWebUI (`model` row, `access_grant`
|
||||
row, `openai.api_configs` index-2 `model_ids`) and `models.yml` (already
|
||||
@@ -111,7 +111,7 @@ unchanged.
|
||||
|
||||
## Client contract
|
||||
|
||||
llama.cpp surface: model id `frank-secondary`, native tools via `--jinja`,
|
||||
llama.cpp surface: model id `frank-zealot`, native tools via `--jinja`,
|
||||
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
|
||||
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
|
||||
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
|
||||
@@ -128,7 +128,7 @@ behind a live generation (the Gemma4 era's `--parallel 1` law), and the
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-secondary-llama-gemma4-q4km-200704]]|
|
||||
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-zealot-llama-gemma4-q4km-200704]]|
|
||||
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|
||||
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
|
||||
|
||||
@@ -138,13 +138,13 @@ Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
|
||||
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
|
||||
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
|
||||
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
|
||||
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
|
||||
`journalctl -u frank-zealot` (KV MiB, compute buffers) before touching
|
||||
flags — the engine prints the exact figures the shape needs.
|
||||
|
||||
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-secondary.sh`
|
||||
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-zealot.sh`
|
||||
shipped without its `#!/usr/bin/env bash` line, and systemd failed the
|
||||
`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
|
||||
`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
|
||||
`Failed at step EXEC spawning /home/dank/…/serve-frank-secondary.sh`. A
|
||||
`Failed at step EXEC spawning /home/dank/…/serve-frank-zealot.sh`. A
|
||||
shebang is not decoration on this host: every LLM serve script is a direct
|
||||
`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).
|
||||
+19
-19
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-secondary runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-zealot runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
@@ -15,7 +15,7 @@ tags:
|
||||
timestamp: 2026-10-06
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
|
||||
# frank-zealot — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening); reshaped three times on
|
||||
> 2026-10-06, renamed from frank-small the same night.** A 27B replaces
|
||||
@@ -32,7 +32,7 @@ timestamp: 2026-10-06
|
||||
> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
|
||||
> measured prediction). This supersedes BOTH earlier 2026-10-05
|
||||
> payloads: the live MiMo-V2.6 Q6_K shape
|
||||
> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> ([[frank-zealot-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
> never deployed.
|
||||
|
||||
@@ -41,14 +41,14 @@ timestamp: 2026-10-06
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-zealot`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||||
| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
|
||||
| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
|
||||
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
|
||||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
|
||||
|
||||
## Why this payload (2026-10-06 evening)
|
||||
|
||||
@@ -90,7 +90,7 @@ built against the target's FULL context (262,144 — the draft window, not
|
||||
the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 −
|
||||
22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter
|
||||
quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free`
|
||||
field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor
|
||||
field) — the tightest fit in frank-zealot's history, the 2–3 GiB floor
|
||||
knowingly violated, and the 512×512 vision transient (34 MiB) still fits
|
||||
and recovers.
|
||||
|
||||
@@ -114,7 +114,7 @@ drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
|
||||
depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
|
||||
ngram speculation logs *"recurrent state rollback is not compatible with
|
||||
'ngram-mod' — disabling rollback support"* (fleet lesson from
|
||||
frank-large's chain measurement, 2026-10-03). A rejected draft on Gated
|
||||
frank-dragoon's chain measurement, 2026-10-03). A rejected draft on Gated
|
||||
DeltaNet layers cannot be rewound by KV truncation — without the rollback
|
||||
checkpoints the DeltaNet state carries phantom transitions and output
|
||||
degrades silently. The flags move as a set: the serve script emits BOTH
|
||||
@@ -216,7 +216,7 @@ leave room for thinking.
|
||||
a clean A/B needs stop + mask (bench policy).
|
||||
- **Smoke**: `READY` completion + `reasoning_content` populated; the
|
||||
harness reached :7771 under the new alias (models.yml provider/id
|
||||
`frank-secondary`, config.yml modelRoles rewritten; per-slot
|
||||
`frank-zealot`, config.yml modelRoles rewritten; per-slot
|
||||
`contextWindow: 114688`).
|
||||
|
||||
## Measured: ubatch 2048 (2026-10-06 midnight, live)
|
||||
@@ -284,7 +284,7 @@ recipes:
|
||||
`/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
|
||||
2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||||
the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
[[frank-zealot-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||||
3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||||
@@ -300,12 +300,12 @@ recipes:
|
||||
read from the Hub file by HTTP range + struct parse — no download needed
|
||||
— before the payload was fetched.
|
||||
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
|
||||
label-committed both sets at Frank's direction — 1919ad7 (frank-primary
|
||||
label-committed both sets at Frank's direction — 1919ad7 (frank-archon
|
||||
radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
|
||||
Deploy ran `deploy.sh dankclaw --service frank-secondary`; the service
|
||||
Deploy ran `deploy.sh dankclaw --service frank-zealot`; the service
|
||||
dispatch and health gate went green; the first e2e run failed ONE spec
|
||||
(12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
|
||||
unrelated to frank-secondary), green on rerun.
|
||||
unrelated to frank-zealot), green on rerun.
|
||||
- The unit's `Description=` (MiMo era) was stale at cutover time and was
|
||||
updated in-repo afterward — the relinked unit picks it up at the next
|
||||
deploy/restart, not retroactively.
|
||||
@@ -320,13 +320,13 @@ recipes:
|
||||
second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
|
||||
decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
|
||||
- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
|
||||
**frank-secondary** (87721ab — scripts, unit, seam, runbooks,
|
||||
**frank-zealot** (87721ab — scripts, unit, seam, runbooks,
|
||||
identifiers; historical references to the OLD FreeToken/CUDA
|
||||
frank-small, and to the Dankserver service that became frank-medium,
|
||||
stay frank-small) and the per-slot window cut 131,072 → 114,688 to
|
||||
fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand
|
||||
before the deploy: `systemctl stop frank-small` + `systemctl disable
|
||||
frank-small`, then `deploy.sh dankclaw --service frank-secondary`.
|
||||
frank-small`, then `deploy.sh dankclaw --service frank-zealot`.
|
||||
1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
|
||||
config.yml retargeted.
|
||||
- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
|
||||
description: "The RETIRED frank-zealot runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-09-16
|
||||
---
|
||||
+18
-18
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night."
|
||||
title: "frank-zealot on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-zealot runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-zealot-llama-mimo-q6k]] the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- sglang
|
||||
- rtx3090
|
||||
@@ -15,10 +15,10 @@ tags:
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
# frank-zealot — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
|
||||
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
|
||||
> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
|
||||
> payload was staged to
|
||||
@@ -28,13 +28,13 @@ timestamp: 2026-10-01
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Endpoint:** `frank-zealot` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
|
||||
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
|
||||
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
|
||||
systemd unit `scripts/frank-secondary.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with
|
||||
tunables from `scripts/frank-secondary.env`.
|
||||
systemd unit `scripts/frank-zealot.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-zealot.sh` with
|
||||
tunables from `scripts/frank-zealot.env`.
|
||||
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
|
||||
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
|
||||
|
||||
@@ -48,14 +48,14 @@ timestamp: 2026-10-01
|
||||
(`--context-length 262144`, `--max-running-requests 2`) against a
|
||||
**pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
|
||||
reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
|
||||
~82k tokens of pool slack, and a compaction request (frank-primary's
|
||||
~82k tokens of pool slack, and a compaction request (frank-archon's
|
||||
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
|
||||
one request. A single request may grow to the whole pool — a
|
||||
170,019-token prompt was verified at the 4-slot shape — the grid is
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
|
||||
180,224, which 183,501 tokens already exceeds — see *What three slots
|
||||
cost* in [[frank-secondary-llama-mimo-q5km]].
|
||||
cost* in [[frank-zealot-llama-mimo-q5km]].
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
@@ -126,15 +126,15 @@ change lands.
|
||||
|
||||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||||
all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh`
|
||||
all LFS sha256-verified by `old/download-frank-zealot-sglang-int8.sh`
|
||||
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
|
||||
cut over to llama.cpp + GGUF on 2026-10-01 night, and
|
||||
`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The
|
||||
`scripts/download-frank-zealot.sh` now fetches the GGUF quant instead. The
|
||||
serve script refuses to start with a missing shard.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
|
||||
- **Alias:** `frank-zealot` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
|
||||
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
|
||||
as llama.cpp; models.yml sets `reasoningContentField:
|
||||
@@ -160,14 +160,14 @@ serve script refuses to start with a missing shard.
|
||||
(216k tokens of live KV at once). 2-slot shape: two concurrent
|
||||
50,000-token requests both answered (28.8 s wall).
|
||||
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
|
||||
well past frank-primary's 183,501-token compaction trigger (70% of its
|
||||
well past frank-archon's 183,501-token compaction trigger (70% of its
|
||||
262,144 window) — prefills and completes in ONE request.
|
||||
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at
|
||||
[[frank-zealot-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-zealot-llama-mimo-q6k]] (payload at
|
||||
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
|
||||
Gemma4-12B QAT payload, was staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
|
||||
@@ -177,4 +177,4 @@ Gemma4-12B QAT payload, was staged to
|
||||
|
||||
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
- SGLang: https://github.com/sgl-project/sglang
|
||||
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
|
||||
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`
|
||||
+20
-20
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-secondary on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||||
description: "The RETIRED frank-secondary runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-secondary-gemma4-12b]] on the same day."
|
||||
title: frank-zealot on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||||
description: "The RETIRED frank-zealot runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-zealot-gemma4-12b]] on the same day."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- vllm
|
||||
- rtx3090
|
||||
@@ -15,12 +15,12 @@ tags:
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
|
||||
# frank-secondary — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||||
# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||||
|
||||
> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
|
||||
> back to [[frank-secondary-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
|
||||
> back to [[frank-zealot-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
|
||||
> the same day it landed — this page is the rollback target. Same port
|
||||
> (7771), same alias (`frank-secondary`), same client contract surface — the
|
||||
> (7771), same alias (`frank-zealot`), same client contract surface — the
|
||||
> runtime, the model payload, the drafter and the service shape (root
|
||||
> systemd unit → rootless Quadlet) all changed.
|
||||
|
||||
@@ -28,12 +28,12 @@ timestamp: 2026-09-19
|
||||
|
||||
| what | value |
|
||||
|---|---|
|
||||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-secondary` (`8dad46d7…`) |
|
||||
| unit | **rootless Quadlet** `dankclaw/frank-secondary/quadlet/frank-secondary.container` (root systemd unit retired 2026-09-19) |
|
||||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-zealot` (`8dad46d7…`) |
|
||||
| unit | **rootless Quadlet** `dankclaw/frank-zealot/quadlet/frank-zealot.container` (root systemd unit retired 2026-09-19) |
|
||||
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
|
||||
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
|
||||
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
|
||||
| port | 7771 → 8000 (alias `frank-secondary`; `frank-large` keeps 7770) |
|
||||
| port | 7771 → 8000 (alias `frank-zealot`; `frank-dragoon` keeps 7770) |
|
||||
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
|
||||
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
|
||||
|
||||
@@ -113,7 +113,7 @@ trade for the native w4a16 format + audio + CUDA graphs.)
|
||||
|
||||
## The image
|
||||
|
||||
`dankclaw/frank-secondary/Dockerfile` (repo) — 16 comment lines + `FROM
|
||||
`dankclaw/frank-zealot/Dockerfile` (repo) — 16 comment lines + `FROM
|
||||
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
|
||||
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
|
||||
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
|
||||
@@ -144,9 +144,9 @@ not a hardware mystery.
|
||||
|
||||
## Deployed config
|
||||
|
||||
Repo truth: `scripts/frank-secondary.env` + `scripts/serve-frank-secondary.sh`
|
||||
Repo truth: `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
|
||||
(mounted into the container at `/scripts/`, which is the entrypoint) +
|
||||
`dankclaw/frank-secondary/quadlet/frank-secondary.container`. Key values:
|
||||
`dankclaw/frank-zealot/quadlet/frank-zealot.container`. Key values:
|
||||
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
|
||||
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
|
||||
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
|
||||
@@ -186,27 +186,27 @@ container while it loads.
|
||||
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
|
||||
must be absent. 8.6 is correct.
|
||||
- **Audio 400/500** — the container is running the *base* image, not the
|
||||
derived `:frank-secondary` (missing `av`). Check the deployed image digest
|
||||
derived `:frank-zealot` (missing `av`). Check the deployed image digest
|
||||
is `8dad46d7…`.
|
||||
- **CUDA not visible in container** — a `Device=` line or a lib mount
|
||||
dropped out of the unit after an edit; the unit is the source of truth,
|
||||
the host-side generated unit is not.
|
||||
- Logs: `journalctl --user -u frank-secondary` (rootless Quadlet; `podman
|
||||
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet; `podman
|
||||
logs` covers only the current `--replace --rm` instance).
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-secondary-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||||
[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||||
mmproj and MTP sidecar are still on the host under
|
||||
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
|
||||
`scripts/.AGENTTRASH/frank-secondary.service.llama.cpp`. Restoring it means
|
||||
`scripts/.AGENTTRASH/frank-zealot.service.llama.cpp`. Restoring it means
|
||||
reinstalling that unit and stopping the Quadlet (one owner per port).
|
||||
|
||||
## References
|
||||
|
||||
- `scripts/serve-frank-secondary.sh` — the actual exec line
|
||||
- `dankclaw/frank-secondary/Dockerfile` — the audio-derived image
|
||||
- `dankclaw/frank-secondary/quadlet/frank-secondary.container` — the unit
|
||||
- [[frank-secondary-gemma4-12b]] — RETIRED predecessor (rollback)
|
||||
- `scripts/serve-frank-zealot.sh` — the actual exec line
|
||||
- `dankclaw/frank-zealot/Dockerfile` — the audio-derived image
|
||||
- `dankclaw/frank-zealot/quadlet/frank-zealot.container` — the unit
|
||||
- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
|
||||
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
|
||||
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
|
||||
@@ -6,7 +6,7 @@ tags:
|
||||
- freetoken
|
||||
- benchmark
|
||||
- dankserver
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- qwen3.6
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
@@ -1,27 +1,27 @@
|
||||
---
|
||||
type: configuration
|
||||
title: FreeToken frank-secondary Runbook
|
||||
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-secondary on Dankserver (RTX 3090 era).
|
||||
title: FreeToken frank-zealot Runbook
|
||||
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-zealot on Dankserver (RTX 3090 era).
|
||||
tags:
|
||||
- freetoken
|
||||
- dankserver
|
||||
- frank-secondary
|
||||
- frank-zealot
|
||||
- qwen3.6
|
||||
- moe
|
||||
- fp8
|
||||
timestamp: 2026-08-30
|
||||
---
|
||||
|
||||
# FreeToken frank-secondary Runbook
|
||||
# FreeToken frank-zealot Runbook
|
||||
|
||||
> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
|
||||
> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
|
||||
> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
|
||||
> retirement record of the FreeToken/CUDA era; the unit was archived to
|
||||
> `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/` and the repo files
|
||||
> `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the repo files
|
||||
> to `scripts/.AGENTTRASH/`.
|
||||
|
||||
The `frank-secondary` service on Dankserver is served by
|
||||
The `frank-zealot` service on Dankserver is served by
|
||||
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
|
||||
expert weights in host RAM with an LRU GPU cache. Model history on this
|
||||
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
|
||||
@@ -134,8 +134,8 @@ modelopt block-fp8 remains the only FP8 path that loads.
|
||||
|
||||
## Sampling
|
||||
|
||||
**frank-secondary samples by default, and our config never said so.**
|
||||
`scripts/serve-frank-secondary.sh` passes no sampling arguments at all — only
|
||||
**frank-zealot samples by default, and our config never said so.**
|
||||
`scripts/serve-frank-zealot.sh` passes no sampling arguments at all — only
|
||||
`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
|
||||
`--num-tokens` and `--max-running-requests`. The behaviour comes from the
|
||||
runtime's own default:
|
||||
@@ -158,9 +158,9 @@ sampling parameters sent at all** produced three different reasoning traces
|
||||
(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding
|
||||
would be byte-identical.
|
||||
|
||||
### This is the opposite of frank-large
|
||||
### This is the opposite of frank-dragoon
|
||||
|
||||
| | frank-secondary (FreeToken) | frank-large (halogen) |
|
||||
| | frank-zealot (FreeToken) | frank-dragoon (halogen) |
|
||||
|---|---|---|
|
||||
| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
|
||||
| Deterministic across runs | No | Yes |
|
||||
@@ -168,7 +168,7 @@ would be byte-identical.
|
||||
|
||||
A client that talks to both endpoints and sends no sampler gets
|
||||
non-deterministic answers from :7770 on Dankserver and reproducible ones from
|
||||
:7770 on Dankclaw. If a workflow needs determinism from frank-secondary it must
|
||||
:7770 on Dankclaw. If a workflow needs determinism from frank-zealot it must
|
||||
send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
`--sampling-defaults none` (framework defaults only).
|
||||
|
||||
@@ -188,9 +188,9 @@ send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `scripts/serve-frank-secondary.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-secondary.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-secondary.service` | Systemd unit |
|
||||
| `scripts/serve-frank-zealot.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-zealot.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-zealot.service` | Systemd unit |
|
||||
|
||||
## Limitations
|
||||
|
||||
@@ -209,7 +209,7 @@ send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
- **`model` field is required** in `/v1/chat/completions` bodies —
|
||||
FreeToken rejects requests without it with a 422 (`Field required`),
|
||||
unlike llama.cpp's server which defaults to the loaded model. Send
|
||||
`"model": "frank-secondary"`.
|
||||
`"model": "frank-zealot"`.
|
||||
- Load on this swap took well under the old llama.cpp numbers:
|
||||
weights → experts (parallel build) → CUDA graph capture → Uvicorn
|
||||
in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
---
|
||||
type: reference
|
||||
title: halogen-flash-server on Dankclaw (frank-large)
|
||||
description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
|
||||
title: halogen-flash-server on Dankclaw (frank-dragoon)
|
||||
description: The active frank-dragoon runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
|
||||
tags:
|
||||
- llm
|
||||
- halogen
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- frank-dragoon
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
@@ -14,16 +14,16 @@ tags:
|
||||
timestamp: 2026-09-13
|
||||
---
|
||||
|
||||
# halogen-flash-server on Dankclaw (frank-large)
|
||||
# halogen-flash-server on Dankclaw (frank-dragoon)
|
||||
|
||||
> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
|
||||
> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
|
||||
> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
|
||||
> [[frank-large-ik-uncensored]], the current rollback), and came back as the
|
||||
> [[frank-dragoon-ik-uncensored]], the current rollback), and came back as the
|
||||
> shape the fleet asked for: two 200k-token conversations rather than the
|
||||
> native-context pair. Deployed as a rootless Quadlet container,
|
||||
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
|
||||
> it does not touch now belongs to [[frank-secondary-qwen38-gsq-rco]].
|
||||
> `dankclaw/frank-dragoon/quadlet/frank-dragoon.container`, port 7770. The RTX 3090
|
||||
> it does not touch now belongs to [[frank-zealot-qwen38-gsq-rco]].
|
||||
|
||||
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
|
||||
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
|
||||
@@ -45,7 +45,7 @@ The `all` mode is upstream's default and the shape a single-container host
|
||||
wants: the front-end talks to the engine over loopback inside the container, so
|
||||
there is one process tree, one log, and one published port.
|
||||
|
||||
`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
|
||||
`HALOGEN_MODEL_ID=frank-dragoon` sets the id reported at `/v1/models`, in
|
||||
`/health`, and in every response body. Requests are **never rejected for naming
|
||||
a different model** — the server serves whatever asks. Keeping the fleet alias
|
||||
is therefore free, and it means Hermes, the harness config, the OpenWebUI
|
||||
@@ -94,13 +94,13 @@ upstream's defaults:
|
||||
|
||||
| flag | value | why |
|
||||
|---|---|---|
|
||||
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
|
||||
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
|
||||
| `HALOGEN_MODEL_ID` | `frank-dragoon` | fleet identity; no client rename |
|
||||
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-dragoon` |
|
||||
| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
|
||||
| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
|
||||
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-secondary, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
|
||||
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-zealot, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
|
||||
| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
|
||||
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
|
||||
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-dragoon` text-only to match |
|
||||
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
|
||||
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
|
||||
|
||||
@@ -226,7 +226,7 @@ restart.
|
||||
|
||||
That is the failure mode to watch, and it is the least diagnosable of the
|
||||
three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
|
||||
looks like a hang, for the neighbours as much as for frank-large. `free` is
|
||||
looks like a hang, for the neighbours as much as for frank-dragoon. `free` is
|
||||
useless for spotting it — `MemAvailable` still reads ~74 GiB because the
|
||||
page cache counts as reclaimable. Watch the `host memory left for everything
|
||||
else` line and the contiguous-block count at boot instead.
|
||||
@@ -374,7 +374,7 @@ tables above.
|
||||
|
||||
**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
|
||||
the tower is a 1.97 GiB charge against the same device budget as the pool and
|
||||
the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]],
|
||||
the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]],
|
||||
which loads the BF16 projector alongside its own weights. This server refuses
|
||||
image content parts and says why, rather than ignoring them:
|
||||
|
||||
@@ -393,11 +393,11 @@ anywhere in the fleet — that left with [[frank-npu]].
|
||||
## Deploy and operate
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
./scripts/deploy.sh dankclaw --service frank-dragoon
|
||||
```
|
||||
|
||||
The handler retires the ik-era **root** unit first (stop, disable, move
|
||||
`/etc/systemd/system/frank-large.service` into
|
||||
`/etc/systemd/system/frank-dragoon.service` into
|
||||
`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
|
||||
`disable` may delete the file outright, which the handler now reports), stops
|
||||
the Quadlet's own unit if it is running, then pulls
|
||||
@@ -419,8 +419,8 @@ file plus linger on the deploy user — generated units **cannot** be
|
||||
`systemctl --user enable`d on this host.
|
||||
|
||||
```bash
|
||||
systemctl --user status frank-large.service # the generated unit
|
||||
podman logs frank-large | head -40 # version banner + memory ledger
|
||||
systemctl --user status frank-dragoon.service # the generated unit
|
||||
podman logs frank-dragoon | head -40 # version banner + memory ledger
|
||||
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
|
||||
curl -s http://127.0.0.1:7770/v1/models
|
||||
```
|
||||
@@ -447,13 +447,13 @@ steady-state noise on a shared host — raise it only for a live OOM report.
|
||||
The lines worth grepping first:
|
||||
|
||||
```bash
|
||||
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
podman logs frank-dragoon 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
```
|
||||
|
||||
Retrieval: `podman logs frank-large` covers the **current** instance only —
|
||||
Retrieval: `podman logs frank-dragoon` covers the **current** instance only —
|
||||
the Quadlet runs `--replace --rm`, so everything before the last recreate is
|
||||
gone from the container log. `journalctl --user -u frank-large.service` is
|
||||
the durable record. Both are rootless: `sudo podman logs frank-large` looks
|
||||
gone from the container log. `journalctl --user -u frank-dragoon.service` is
|
||||
the durable record. Both are rootless: `sudo podman logs frank-dragoon` looks
|
||||
in root's storage and answers "no such container".
|
||||
|
||||
## Rollback
|
||||
@@ -466,7 +466,7 @@ serve script and the Qwen-Sharp template are archived in the repo under
|
||||
is a `git revert` of this change set plus a deploy — **but the Quadlet must be
|
||||
removed from `~/.config/containers/systemd/` first**, or the restarted root
|
||||
unit and the still-installed container both claim port 7770. The port gate in
|
||||
the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
|
||||
the handler now covers exactly that race. See [[frank-dragoon-ik-uncensored]].
|
||||
|
||||
The `amd_iommu=off` kernel argument that came with this change is separate and
|
||||
reverts on its own — see [[dankclaw-iommu-translation-off]].
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-secondary on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
tags:
|
||||
- ikllama
|
||||
- llamacpp
|
||||
@@ -16,7 +16,7 @@ timestamp: 2026-08-26
|
||||
|
||||
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
||||
|
||||
> **Active 2026-08-26.** `frank-secondary` switched from upstream
|
||||
> **Active 2026-08-26.** `frank-zealot` switched from upstream
|
||||
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
||||
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
||||
> the MoE / CPU-offload runtime arguments.
|
||||
@@ -101,8 +101,8 @@ a hard compile error.
|
||||
|
||||
## Runtime configuration
|
||||
|
||||
Files: `scripts/serve-frank-secondary.sh`, `scripts/frank-secondary.env`,
|
||||
`scripts/frank-secondary.service` in the repo. The service just calls
|
||||
Files: `scripts/serve-frank-zealot.sh`, `scripts/frank-zealot.env`,
|
||||
`scripts/frank-zealot.service` in the repo. The service just calls
|
||||
the serve script; the env file holds all tunables.
|
||||
|
||||
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
|
||||
@@ -241,7 +241,7 @@ apply.
|
||||
The previous runtime `upstream llama.cpp b10121` and its binary
|
||||
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
||||
rollback to the old stack, change `LLAMA_SERVER=` in
|
||||
`scripts/frank-secondary.env` back to that path and
|
||||
`scripts/frank-zealot.env` back to that path and
|
||||
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
|
||||
spec / `--spec-type` flags. The model file is not changed either way.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||||
@@ -8,15 +8,15 @@ driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||||
and frank-primary-vllm-shapeG-20261003),
|
||||
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
|
||||
and frank-archon-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
|
||||
ctx, vision ON)** and
|
||||
**llama.cpp CUDA (frank-secondary, port 7771: HauhauCS Gemma4-26B-A4B QAT
|
||||
**llama.cpp CUDA (frank-zealot, port 7771: HauhauCS Gemma4-26B-A4B QAT
|
||||
Uncensored Q4_K_M GGUF — 26B MoE hybrid (25 sliding-window + 5 full-attention
|
||||
layers), ONE 200,704-token conversation, MTP disabled,
|
||||
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
|
||||
@@ -35,7 +35,7 @@ graph LR
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-primary, port 7770]
|
||||
podman run, frank-archon, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
@@ -46,11 +46,11 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-large, port 7770]
|
||||
frank-dragoon, port 7770]
|
||||
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
|
||||
2 x 200192 ctx, MTP always-on, image input]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-secondary, port 7771]
|
||||
frank-zealot, port 7771]
|
||||
H[Gemma4-26B-A4B QAT Uncensored Q4_K_M GGUF
|
||||
1 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
||||
E2 --> E --> F
|
||||
@@ -71,7 +71,7 @@ frank-secondary, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
|
||||
- [[frank-dragoon-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
|
||||
(root systemd unit + podman, port 7770): the digest-pinned
|
||||
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
|
||||
Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
|
||||
@@ -88,7 +88,7 @@ frank-secondary, port 7771]
|
||||
greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens.
|
||||
Rollback: revert the swap commit; the 27B payload + fetch script stay
|
||||
on disk
|
||||
- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||||
- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||||
(rollback target; payload + fetch script + image on disk)**: the
|
||||
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
|
||||
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
|
||||
@@ -99,24 +99,24 @@ frank-secondary, port 7771]
|
||||
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
|
||||
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
|
||||
foreground-`podman run` ledger capture and the deep-role rerouting
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
|
||||
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
|
||||
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
|
||||
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
|
||||
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||||
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||||
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
|
||||
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
|
||||
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
|
||||
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
|
||||
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
|
||||
it, 21,890/2,235 without).
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
|
||||
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
|
||||
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
|
||||
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
|
||||
MTP back on since
|
||||
@@ -139,7 +139,7 @@ frank-secondary, port 7771]
|
||||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||||
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
|
||||
(77decb5..380aafa) but never deployed
|
||||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||||
@@ -148,15 +148,15 @@ frank-secondary, port 7771]
|
||||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||||
probe-size trap (>=256 px images or the model says "video") live inside
|
||||
- [[frank-secondary-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-primary-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
replaces the FP8-native payload when the gates pass
|
||||
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
@@ -171,12 +171,12 @@ frank-secondary, port 7771]
|
||||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||||
`/metrics` scraped. Rollback: git tag
|
||||
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||||
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
|
||||
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||||
`frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||||
payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
|
||||
- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||||
rocm714_torch214) built on the host into the local image
|
||||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
`frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||||
@@ -198,8 +198,8 @@ frank-secondary, port 7771]
|
||||
and a mounted compile cache. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
|
||||
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
|
||||
the tcclaviger RDNA4 vLLM fork
|
||||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||||
@@ -215,10 +215,10 @@ frank-secondary, port 7771]
|
||||
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
|
||||
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
|
||||
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
|
||||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; payload and R9V state dirs intact on disk, git tag
|
||||
frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
|
||||
[[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||||
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
|
||||
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||||
intact on disk): the R9V radiance-vLLM fork serving
|
||||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||||
@@ -226,8 +226,8 @@ frank-secondary, port 7771]
|
||||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
|
||||
from llama.cpp the same day; rollback target
|
||||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||||
[[frank-archon-llamacpp-qwen38-flash-next-q8]]
|
||||
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
@@ -236,7 +236,7 @@ frank-secondary, port 7771]
|
||||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||||
conversation, `reasoning_content`, the placement law, the cutover and
|
||||
rollback
|
||||
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||||
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||||
rollback target; active for hours): the Lucebox DS4.1 engine
|
||||
(localhost/lucebox-rocm:gfx1201) serving
|
||||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||||
@@ -259,8 +259,8 @@ frank-secondary, port 7771]
|
||||
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
|
||||
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||||
[[frank-primary-lucebox-ds4]], then
|
||||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||||
[[frank-archon-lucebox-ds4]], then
|
||||
[[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||||
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
|
||||
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
|
||||
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
|
||||
@@ -288,15 +288,15 @@ frank-secondary, port 7771]
|
||||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||||
runtime it replaced
|
||||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||||
frank-large: the FreeToken v0.1.2 frank-secondary runbook (batsclamp
|
||||
frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||||
[[frank-secondary-qwen38-gsq-rco]]
|
||||
[[frank-zealot-qwen38-gsq-rco]]
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
|
||||
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
|
||||
10-05 (rollback target; payload, fork build and build script on disk)**:
|
||||
the pinned ROCmFPX fork serving the uncensored protected-heads
|
||||
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
|
||||
@@ -304,21 +304,21 @@ frank-secondary, port 7771]
|
||||
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
|
||||
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
|
||||
architecture.
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
|
||||
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
|
||||
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
|
||||
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
|
||||
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
|
||||
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
|
||||
- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
|
||||
payload dirs were removed from disk 2026-10-03, so this rollback is a
|
||||
re-download)** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||||
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
|
||||
of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode
|
||||
trade against llama.cpp
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||||
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||||
@@ -327,68 +327,68 @@ frank-secondary, port 7771]
|
||||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
|
||||
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
|
||||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
|
||||
v22.4.0 template, the 114 GiB GTT memory model, client contract,
|
||||
troubleshooting
|
||||
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
|
||||
swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
|
||||
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
|
||||
swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
|
||||
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
|
||||
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
|
||||
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
|
||||
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
|
||||
114 GiB GTT memory model, client contract, troubleshooting
|
||||
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
|
||||
swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
|
||||
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
|
||||
swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
|
||||
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
|
||||
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
|
||||
head, the 114 GiB GTT memory model, client contract, troubleshooting
|
||||
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||||
[[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||||
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||||
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||||
v4.4.1 runtime (private host build, QSA hotfix) serving
|
||||
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
|
||||
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
|
||||
gate, client contract, troubleshooting (2026-09-19 → 09-21)
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||||
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||||
[[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-secondary-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
[[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||||
`/home/dank/models/MiMo-Q6K/`
|
||||
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-secondary-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv still on disk, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-secondary-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||||
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
f16 KV at 131k
|
||||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||||
@@ -398,7 +398,7 @@ frank-secondary, port 7771]
|
||||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||||
that invalidated the first round, and the iGPU-only result that matches
|
||||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||||
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||||
- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||||
@@ -407,7 +407,7 @@ frank-secondary, port 7771]
|
||||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||||
the reference machine, and how to revert
|
||||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
|
||||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||||
@@ -419,9 +419,9 @@ frank-secondary, port 7771]
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-secondary, retired 2026-09-02)
|
||||
Dankserver (frank-zealot, retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||||
frank-secondary (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||||
@@ -429,9 +429,9 @@ frank-secondary, port 7771]
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||||
the flash-server v2 runtime since 2026-10-06, frank-secondary on the RTX 3090)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
|
||||
the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
@@ -455,7 +455,7 @@ frank-secondary, port 7771]
|
||||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||||
notes and sizing (this architecture family is what the retired frank-secondary
|
||||
notes and sizing (this architecture family is what the retired frank-zealot
|
||||
served in the Huihui abliterated FP8 variant — see
|
||||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||||
that preceded it is archived at [[dealign-ornith-crack]])
|
||||
@@ -471,7 +471,7 @@ frank-secondary, port 7771]
|
||||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||||
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||||
@@ -529,7 +529,7 @@ These sit alongside the operational pages under
|
||||
## Related
|
||||
|
||||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
|
||||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
|
||||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||||
git-based deploy with inference isolation
|
||||
@@ -17,10 +17,10 @@ timestamp: 2026-08-22
|
||||
|
||||
> **Retired 2026-08-22.** This experiment was deployed, benchmarks were
|
||||
> collected, and the service reverted to v3 the same day (operator
|
||||
> decision). `frank-large` serves v3 again. This page is kept as
|
||||
> decision). `frank-dragoon` serves v3 again. This page is kept as
|
||||
> the retired experiment's build runbook and reference.
|
||||
|
||||
`frank-large` (port 52625) briefly served
|
||||
`frank-dragoon` (port 52625) briefly served
|
||||
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` through the Kairic Edge CIRU ROCmFPX
|
||||
llama.cpp runtime. The runtime routes qualified prompt and multi-token
|
||||
verification shapes through the RDNA 3.5 native
|
||||
@@ -215,7 +215,7 @@ service restart confirms the fresh tree, then it is a deletion candidate
|
||||
|
||||
The service runs the pinned runner profile plus the local chat
|
||||
deviations below. Tunables live in
|
||||
`scripts/frank-large.env`; flags in `scripts/serve-frank-large.sh`.
|
||||
`scripts/frank-dragoon.env`; flags in `scripts/serve-frank-dragoon.sh`.
|
||||
See [[deployment]] for the update process.
|
||||
|
||||
| Setting | Value | Origin |
|
||||
@@ -301,7 +301,7 @@ ubatches do not. Production runs the runner's qualified 2048/512.
|
||||
|
||||
The v3 payload and binary stay on the machine for rollback.
|
||||
|
||||
1. Stop the service: `sudo systemctl stop frank-large`
|
||||
1. Stop the service: `sudo systemctl stop frank-dragoon`
|
||||
2. Restore the v3 scripts:
|
||||
`git revert 4c43839 7aee1df 40d50fe` (or checkout the script pair from
|
||||
commit `132e147`)
|
||||
|
||||
@@ -8,7 +8,7 @@ tags:
|
||||
timestamp: 2026-09-15
|
||||
---
|
||||
|
||||
# frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
|
||||
# frank-dragoon: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
|
||||
|
||||
> **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the
|
||||
> separate `draft-mtp` head) now serves Dankserver as **frank-medium** on the
|
||||
@@ -21,7 +21,7 @@ the `set_rows` build bug** — the model loads but crashes in
|
||||
`ggml_compute_forward_set_rows` before it can serve. All config goals are met;
|
||||
a build fix is required for the model to run.
|
||||
|
||||
**Alias:** `frank-large`, port **7770**.
|
||||
**Alias:** `frank-dragoon`, port **7770**.
|
||||
|
||||
The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF
|
||||
from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a
|
||||
@@ -42,8 +42,8 @@ fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
|
||||
| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` |
|
||||
| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on |
|
||||
| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` |
|
||||
| Tunables | `scripts/frank-large.env` |
|
||||
| Serve script | `scripts/serve-frank-large.sh` |
|
||||
| Tunables | `scripts/frank-dragoon.env` |
|
||||
| Serve script | `scripts/serve-frank-dragoon.sh` |
|
||||
|
||||
## Serving status: the `set_rows` build bug (blocks serving)
|
||||
|
||||
@@ -123,16 +123,16 @@ dl mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
```
|
||||
|
||||
The serve script checks each artifact is present and readable before launch
|
||||
(`scripts/frank-large.env` -> the four `MODEL_*` files plus the chat
|
||||
(`scripts/frank-dragoon.env` -> the four `MODEL_*` files plus the chat
|
||||
template). A missing file fails the artifact check with `missing or
|
||||
unreadable required artifact`.
|
||||
|
||||
## Serving profile
|
||||
|
||||
`scripts/serve-frank-large.sh` builds the `llama-server` call from the env:
|
||||
`scripts/serve-frank-dragoon.sh` builds the `llama-server` call from the env:
|
||||
|
||||
```
|
||||
/dev/llama-server -m <main> --alias frank-large
|
||||
/dev/llama-server -m <main> --alias frank-dragoon
|
||||
--host 0.0.0.0 --port 7770 --jinja
|
||||
-dev ROCm0 -ngl 999 -np 1 -c 262144
|
||||
-b 512 -ub 512 -fa on
|
||||
@@ -213,7 +213,7 @@ and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the
|
||||
|
||||
## Env tunables
|
||||
|
||||
`scripts/frank-large.env`. Serving: `CTX_SIZE=262144`,
|
||||
`scripts/frank-dragoon.env`. Serving: `CTX_SIZE=262144`,
|
||||
`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`,
|
||||
`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`,
|
||||
`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-large.
|
||||
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-dragoon.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
@@ -15,7 +15,7 @@ timestamp: 2026-08-24
|
||||
|
||||
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
|
||||
> **RETIRED 2026-08-29.** This payload was `frank-large` from 2026-08-25
|
||||
> **RETIRED 2026-08-29.** This payload was `frank-dragoon` from 2026-08-25
|
||||
> until the Q8_0 AGENT experiment on 2026-08-29 (it was restored again
|
||||
> 2026-09-01 and served until 2026-08-30, when [[agentionai-fast-imatrix]]
|
||||
> took over). The Q8_0 model was retired because it crashed before
|
||||
@@ -24,7 +24,7 @@ timestamp: 2026-08-24
|
||||
|
||||
> This page documents a retired model and its runbook.
|
||||
|
||||
`frank-large` (port 7770) serves
|
||||
`frank-dragoon` (port 7770) serves
|
||||
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
|
||||
Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
|
||||
the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
|
||||
description: PRIOR frank-large runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
|
||||
description: PRIOR frank-dragoon runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
@@ -16,7 +16,7 @@ timestamp: 2026-08-30
|
||||
|
||||
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
|
||||
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-large moved to
|
||||
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-dragoon moved to
|
||||
> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
|
||||
> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
|
||||
> runtime's prefill. Everything below documents the llama.cpp era and stays
|
||||
@@ -33,10 +33,10 @@ timestamp: 2026-08-30
|
||||
> (the card ships no spec flags; the Flash-Next MTP heads do not load on
|
||||
> this loader family — see MTP Draft Head).
|
||||
|
||||
`frank-large` (port 7770) serves
|
||||
`frank-dragoon` (port 7770) serves
|
||||
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
|
||||
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
|
||||
via `scripts/serve-frank-large.sh` + `scripts/frank-large.env`
|
||||
via `scripts/serve-frank-dragoon.sh` + `scripts/frank-dragoon.env`
|
||||
and the Qwen-Sharp chat template.
|
||||
|
||||
## Model Architecture
|
||||
@@ -188,7 +188,7 @@ cmake --build build --target llama-server llama-quantize -j16
|
||||
|
||||
## Serving Profile
|
||||
|
||||
Deployed args (`scripts/frank-large.env` — our validated
|
||||
Deployed args (`scripts/frank-dragoon.env` — our validated
|
||||
profile, unchanged from the FAST deployment except the binary and model):
|
||||
|
||||
```
|
||||
@@ -282,9 +282,9 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
|
||||
1. Build the runtime (see Runtime build) BEFORE touching the service.
|
||||
2. Point the model file's directory at the canonical path (a plain `mv`
|
||||
— never delete a 113 GiB payload to re-fetch it).
|
||||
3. Edit `scripts/frank-large.env` (+ serve script defaults) via
|
||||
3. Edit `scripts/frank-dragoon.env` (+ serve script defaults) via
|
||||
git: commit, push, `./scripts/deploy.sh dankclaw --service
|
||||
frank-large`.
|
||||
frank-dragoon`.
|
||||
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
|
||||
`/v1/chat/completions` probe.
|
||||
|
||||
@@ -294,8 +294,8 @@ binary and the agentionai payload remain on disk.
|
||||
|
||||
### Config files
|
||||
|
||||
- Env: `scripts/frank-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Env: `scripts/frank-dragoon.env`
|
||||
- Serve: `scripts/serve-frank-dragoon.sh`
|
||||
- Chat template: `scripts/chat-template.jinja`
|
||||
|
||||
## Known Issues
|
||||
|
||||
@@ -8,11 +8,11 @@ timestamp: 2026-09-03
|
||||
|
||||
# Ling-3.0-flash Q6_K Trial on Dankclaw
|
||||
|
||||
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. At this page's retirement date `frank-large` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
|
||||
> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
|
||||
|
||||
## Status
|
||||
|
||||
- Service: `frank-large-ling` (systemd).
|
||||
- Service: `frank-dragoon-ling` (systemd).
|
||||
- Model: `inclusionAI/Ling-3.0-flash` Q6_K (MoE, ~118B total params).
|
||||
- Runtime: llama.cpp **fork** `aetherbird/llama.cpp` branch
|
||||
`bailingmoe3-support` (commit `db4480bc8`, build b10411) at
|
||||
@@ -77,14 +77,14 @@ point varies (385-1588 tokens). Single streams never freeze.
|
||||
`SPEC_TYPE=none`, `PARALLEL=2`. All concurrent streams complete (EOS).
|
||||
Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with
|
||||
MTP. Fallback chain if spec-off regresses: A) `PARALLEL=1` keep MTP,
|
||||
B) switch back to Qwen3.6-27B (frank-large, MTP n_max 3).
|
||||
B) switch back to Qwen3.6-27B (frank-dragoon, MTP n_max 3).
|
||||
|
||||
## Files
|
||||
|
||||
- `scripts/frank-large-ling.service` - systemd unit.
|
||||
- `scripts/frank-dragoon-ling.service` - systemd unit.
|
||||
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
|
||||
draft-mtp).
|
||||
- `scripts/frank-large.ling.env` - tunables.
|
||||
- `scripts/frank-dragoon.ling.env` - tunables.
|
||||
|
||||
Passwordless management is installed:
|
||||
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
|
||||
@@ -92,9 +92,9 @@ Passwordless management is installed:
|
||||
## Switch Back (obsolete)
|
||||
|
||||
The switch-back commands below are obsolete. At retirement time
|
||||
`frank-large` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
|
||||
re-enable `frank-large-ling` from the repo unit
|
||||
(`scripts/frank-large-ling.service`) and serve script
|
||||
`frank-dragoon` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
|
||||
re-enable `frank-dragoon-ling` from the repo unit
|
||||
(`scripts/frank-dragoon-ling.service`) and serve script
|
||||
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
|
||||
|
||||
## Caveats
|
||||
@@ -111,6 +111,6 @@ re-enable `frank-large-ling` from the repo unit
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-large-benchmark]]
|
||||
- [[frank-dragoon-benchmark]]
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[deployment]]
|
||||
|
||||
@@ -15,6 +15,6 @@ This page was the plan for a vLLM-ROCm deployment of
|
||||
Dankclaw. The deployment never shipped. The actual Ling work used the
|
||||
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
|
||||
|
||||
The trial service (`frank-large-ling`) is retired and disabled under the
|
||||
boot policy (2026-08-17). `frank-large` is the Qwen3.8-27B PP
|
||||
The trial service (`frank-dragoon-ling`) is retired and disabled under the
|
||||
boot policy (2026-08-17). `frank-dragoon` is the Qwen3.8-27B PP
|
||||
head, not the Ling path. See git history for the full decision table.
|
||||
|
||||
@@ -14,7 +14,7 @@ timestamp: 2026-09-03
|
||||
# Qwen3.8-27B Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark history and the
|
||||
production tuning for the Qwen3.8-27B service (`frank-large`),
|
||||
production tuning for the Qwen3.8-27B service (`frank-dragoon`),
|
||||
now a pipeline-parallel head with a Dankserver worker.
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
@@ -154,7 +154,7 @@ that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
The live tunables live in `scripts/frank-large.env` (head) and
|
||||
The live tunables live in `scripts/frank-dragoon.env` (head) and
|
||||
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
|
||||
state section at the top of this page records the current values: Q6_K
|
||||
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
|
||||
@@ -162,6 +162,6 @@ q8_0/f16 KV.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-large-benchmark]]
|
||||
- [[frank-dragoon-benchmark]]
|
||||
- [[ling-3.0-flash-trial]]
|
||||
- [[deployment]]
|
||||
|
||||
@@ -15,7 +15,7 @@ and the test matrix planned for 2026-08-16 morning.
|
||||
|
||||
Related pages:
|
||||
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
|
||||
- [Backend and spec benchmark](frank-large-benchmark.md)
|
||||
- [Backend and spec benchmark](frank-dragoon-benchmark.md)
|
||||
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
|
||||
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||||
|
||||
@@ -66,7 +66,7 @@ Consequences of the hybrid design:
|
||||
|
||||
## Current production state (baseline)
|
||||
|
||||
`frank-large` (Dankclaw, Vulkan0, 2026-08-15):
|
||||
`frank-dragoon` (Dankclaw, Vulkan0, 2026-08-15):
|
||||
|
||||
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
|
||||
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
|
||||
@@ -233,7 +233,7 @@ tensor cache warm).
|
||||
| E | A with n_max 3 and 5 | n_max curve on the split |
|
||||
| F | A, plain Q8_0 model file | same-bit A/B |
|
||||
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
|
||||
| Ref | production frank-large (Vulkan, 2 slots) | the bar to beat |
|
||||
| Ref | production frank-dragoon (Vulkan, 2 slots) | the bar to beat |
|
||||
|
||||
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
|
||||
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
|
||||
@@ -253,5 +253,5 @@ production @16k) with the same 8-bit model.
|
||||
`-dio` for RPC).
|
||||
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
|
||||
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
|
||||
- Local benchmarks: [frank-large-benchmark](frank-large-benchmark.md),
|
||||
- Local benchmarks: [frank-dragoon-benchmark](frank-dragoon-benchmark.md),
|
||||
[qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
|
||||
|
||||
@@ -4,9 +4,9 @@ Campaign branch `swift15-radiance-w4dn8`. Goal: convert
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB) to
|
||||
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (GPTQ gate_up w4nl64 + block-fp8 down +
|
||||
10 protected bf16 experts + int8 trunk + fixed-scale PLE) and swap it into
|
||||
frank-primary on the 1.0.13 pin. Design settled (25-question grill); this log
|
||||
frank-archon on the 1.0.13 pin. Design settled (25-question grill); this log
|
||||
carries execution facts only. The cutover runbook
|
||||
`frank-primary-radiance-swift15-w4dn8.md` replaces this page at cutover.
|
||||
`frank-archon-radiance-swift15-w4dn8.md` replaces this page at cutover.
|
||||
|
||||
## 2026-10-06 evening
|
||||
|
||||
@@ -44,7 +44,7 @@ carries execution facts only. The cutover runbook
|
||||
≥126,000 MiB), `apply-protected-cast.sh` (derive → recipe PROTECTED
|
||||
section, refuses double-derive), `kld-gate-swift15.sh` (candidate vs
|
||||
reference on 1.0.13, gate mean KL ≤ 0.03 / top-1 ≥ 95 %).
|
||||
- `frank-primary` showing `failed` during the campaign is the 143-on-stop
|
||||
- `frank-archon` showing `failed` during the campaign is the 143-on-stop
|
||||
signature of the foreground podman unit — cosmetic, not a crash.
|
||||
|
||||
## 2026-10-07 morning
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: benchmark plan
|
||||
title: frank-large ubatch sweep
|
||||
title: frank-dragoon ubatch sweep
|
||||
description: Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment.
|
||||
tags:
|
||||
- llama.cpp
|
||||
@@ -10,9 +10,9 @@ tags:
|
||||
timestamp: 2026-09-01
|
||||
---
|
||||
|
||||
# frank-large ubatch sweep
|
||||
# frank-dragoon ubatch sweep
|
||||
|
||||
This benchmark uses the active `frank-large` model on Dankclaw with
|
||||
This benchmark uses the active `frank-dragoon` model on Dankclaw with
|
||||
`llama-bench`. It does not use `llama-server`, HTTP requests, or server-log
|
||||
scraping. Stop the production service before running it and start the service
|
||||
again afterward.
|
||||
@@ -29,9 +29,9 @@ Each point runs three 128-token repetitions by default.
|
||||
Run on Dankclaw from the repository checkout:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-large
|
||||
sudo systemctl stop frank-dragoon
|
||||
./scripts/bench-ubatch.sh
|
||||
sudo systemctl start frank-large
|
||||
sudo systemctl start frank-dragoon
|
||||
```
|
||||
|
||||
The script uses the active Q6 AGENT model, an 8K prompt workload, equal
|
||||
|
||||
@@ -8,7 +8,7 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
# vLLM Boot Forensics and Config Interrogation
|
||||
|
||||
For serve-script vLLM endpoints (frank-primary/qwen-fp8 on dankserver, any
|
||||
For serve-script vLLM endpoints (frank-archon/qwen-fp8 on dankserver, any
|
||||
`podman rm -f` + restart service).
|
||||
|
||||
## 1. Capture the true cause — the journal, not `podman logs`
|
||||
@@ -204,4 +204,4 @@ clipping at depth) starts with sections 1–3 here before touching anything.
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — probe/ladder hygiene on live hosts
|
||||
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — probe/ladder hygiene on live hosts
|
||||
|
||||
@@ -302,7 +302,7 @@ consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
|
||||
### Window discipline and verification
|
||||
|
||||
- Scratch boots on dankserver MUST stop + runtime-mask production
|
||||
`frank-primary` first — the expert-offload planner reads `MemAvailable` at
|
||||
`frank-archon` first — the expert-offload planner reads `MemAvailable` at
|
||||
plan time, and beside-prod boots die at plan with
|
||||
`ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
|
||||
loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
|
||||
|
||||
@@ -249,13 +249,13 @@ distinguishes a real split from a mislabelled sweep.
|
||||
|
||||
The fleet's answer is unchanged — **two independent endpoints**
|
||||
([[halogen-flash-server]] on the iGPU via halogen,
|
||||
[[frank-secondary-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
|
||||
[[frank-zealot-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
|
||||
claims from the first round of this experiment are now corrected:
|
||||
|
||||
1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
|
||||
tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode),
|
||||
and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint
|
||||
(`frank-secondary` holds 22.4 of the card's 24.6 GiB). An operator who wants the
|
||||
(`frank-zealot` holds 22.4 of the card's 24.6 GiB). An operator who wants the
|
||||
7 % must give up the fleet's only image-capable endpoint.
|
||||
2. **Per-tensor placement across two devices is impossible while `-ot` is used**,
|
||||
for a scheduler-mechanical reason, regardless of driver or vendor mix.
|
||||
@@ -309,11 +309,11 @@ Operational notes for the next run:
|
||||
`GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
|
||||
pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
|
||||
host's co-tenants.
|
||||
- `systemctl mask` is **refused** for a relinked root unit: frank-secondary's
|
||||
`/etc/systemd/system/frank-secondary.service` is a symlink into the repo, so
|
||||
- `systemctl mask` is **refused** for a relinked root unit: frank-zealot's
|
||||
`/etc/systemd/system/frank-zealot.service` is a symlink into the repo, so
|
||||
`mask` exits with "File ... already exists". Stop it and rely on the absence
|
||||
of a restart trigger, or move the symlink aside.
|
||||
- Restarting frank-large needs the quadlet source back in
|
||||
- Restarting frank-dragoon needs the quadlet source back in
|
||||
`~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
|
||||
the unit from that file only. After the restore (and the host reboot, see the
|
||||
landmines) the ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB
|
||||
|
||||
Reference in New Issue
Block a user