docs(dragoon,zealot): record both runtime pins as they now run

frank-dragoon: engine 0.18.20 (image e35133b2…), the 18-model NPU catalog, the
re-cut qwen3-embedding-0.6b device programs, and the wire surface the new engine
actually answers (GET /tokenize is 405, /props is 200, /completion and /ping
stay 404). Eras gains the 2026-10-30 row.

frank-zealot: fork build 12118 (0.7.0-dev, 024474448) with the payload-load
evidence (202 pq2_0 tensors, 202 Hadamard-folded weights), plus the law a
rebuild pays for: cmake --install strips the rpath and installs bin/+lib/, so
the configure needs CMAKE_INSTALL_RPATH and LLAMA_BIN names …/bin/llama-server.
This commit is contained in:
Frank committed 2026-10-30 15:27:21 -07:00
1 parent acbb539b97
commit e1d4d59773
4 files changed
+42 -27

No files matched your search

+8 -8
View File
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Payload | Port | | Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------| |--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 | | Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 | | Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 | | Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
Payload identity, shape, eras, laws and rollback live in ONE page each: Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env` [[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -293,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
2. Bench against the service itself. No scratch ports on an LLM host — one 2. Bench against the service itself. No scratch ports on an LLM host — one
process owns the card. Use the runtime's own harness (`llama-bench`; a process owns the card. Use the runtime's own harness (`llama-bench`; a
live llama.cpp server also reports honest `timings` per request, including live llama.cpp server also reports honest `timings` per request, including
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
`ignore_eos` run on the raw `/completion` path there answers **500 "The `ignore_eos` run on the raw `/completion` path there answers **500 "The
model produced output that does not match the expected Content-only model produced output that does not match the expected Content-only
format"** (measured 2026-10-18). halogen has no `/completion` path at all, format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -439,16 +439,16 @@ disagrees with it is the bug.
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771 ### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`, - Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY `scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
loader (the payload carries the fork's custom PQ2_0 tensor type plus is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
loader ignoring that metadata dequantizes wrong); payload loader ignoring that metadata dequantizes wrong); payload
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16 `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
tower can crash the server with concurrent image requests) + the rank-1 tower can crash the server with concurrent image requests) + the rank-1
refusal-direction adapter applied at load by `--lora-scaled …:1`. The refusal-direction adapter applied at load by `--lora-scaled …:1`. The
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
— and build 10777 is also the gate that lets it load: older forks refuse it — and the fork BUILD is also the gate that lets it load: anything older than
with `unsupported prism.hadamard.version: 2`. b10777 refuses it with `unsupported prism.hadamard.version: 2`.
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at - Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088 200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024, B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
## Server Overview ## Server Overview
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771). Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
## Hardware Specifications ## Hardware Specifications
### CPU ### CPU
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM. - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget. - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`. - The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either. - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18. - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage. - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
+21 -14
View File
@@ -1,6 +1,6 @@
--- ---
type: reference type: reference
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)" title: "frank-dragoon on Dankclaw — halogen-flash-server 0.18.20, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it." description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
tags: tags:
- llm - llm
@@ -10,10 +10,10 @@ tags:
- gfx1151 - gfx1151
- halogen - halogen
- endpoint - endpoint
timestamp: 2026-10-21 timestamp: 2026-10-30
--- ---
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU) # frank-dragoon — halogen-flash-server 0.18.20, native v2 abliterated (Dankclaw iGPU)
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
@@ -33,8 +33,8 @@ flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, ver
| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON | | `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked | | `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a` Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0`
(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's (0.18.20, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
unauthenticated token port 8730 stays on container loopback. Engine stdout → unauthenticated token port 8730 stays on container loopback. Engine stdout →
`/home/dank/logs/frank_dragoon.log`, **not** the journal. `/home/dank/logs/frank_dragoon.log`, **not** the journal.
@@ -74,6 +74,7 @@ rollback plan — recoverable only until the trash is purged.
| Window | Payload | Runtime | Shape | Replaced because | Rollback | | Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 2026-10-30 → | same v2 abliterated `.hgn` trio (payload untouched) | flash-server **0.18.20** `e35133b2…` | unchanged (2 slots, 262,144 ceiling, pool 401,408, MTP always-on) | vendor moved two minor versions past the 0.16.2 quickstart pin; the payload lock **held** — the same boot served chat, a `tool_calls` turn, vision ("Red", 132 prompt tokens) and `/v1/embeddings`, with the NPU lane's 19-file set verified against the new catalog | env line back to `0c61bf84…` + redeploy |
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` | | 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage | | 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` | | 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
@@ -190,9 +191,11 @@ created it — earlier topologies in .
## Contract (wire) ## Contract (wire)
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props` - Live: `/v1/models`, `/health`, `/metrics`, `/props` (model_alias, total_slots, chat_template),
`/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure POST `/v1/chat/completions`, POST `/v1/embeddings` when the NPU lane is armed; `404`: `/ping`,
wall-clock plus `usage.prompt_tokens`. `/completion`; `/tokenize` answers **405 to a GET** (the route exists and wants POST). **No
`Server-Timing` header and no `/completion` timing object** — measure wall-clock plus
`usage.prompt_tokens`.
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies - Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476 vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
@@ -232,11 +235,15 @@ created it — earlier topologies in .
XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the ([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog is exactly what runs today. The pinned 0.18.20 image carries an 18-model catalog
(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`, (`/opt/halogen/npu/models.txt`: `decider-0.8b`, `decider-4b`, `qwen3-embedding-0.6b`, `qwen3-embedding-4b`,
`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts `qwen3-reranker-0.6b`, `qwen3-reranker-4b`, `qwen3guard-gen-0.6b`, `qwen3guard-stream-0.6b`,
away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2` `embeddinggemma-2`, `wemm-embedding-4b`, `qwen3.5-2b`, `flux2-klein-4b`, `qwen3-asr-0.6b`, `qwen3-asr-1.7b`,
(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3. `d3-lite`, `d3-nano`, `qwen3-tts-0.6b-customvoice`, `qwen3-tts-1.7b-customvoice`), so **`/v1/embeddings` is one
env var and three mounts away — no runtime bump needed.** The catalog is **not** version-stable: 0.18.20
re-cut `qwen3-embedding-0.6b`'s device programs (14 `u*.elf` + an 8,495,512 B `consts.hnpb`, where 0.16.2
had 10 and 5,086,496 B) and the entrypoint verifies size+sha256 per file before it starts, so **an engine
bump re-fetches the staged set** — `scripts/download-frank-dragoon-npu.sh` carries the catalog block to do it.
Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts** Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
(not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string (not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
@@ -282,7 +289,7 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
size and sha256 against the catalog and the payload mount carries the tree in. size and sha256 against the catalog and the payload mount carries the tree in.
Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `15 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0…u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector). Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `19 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0, u1, u10, u11, u12, u13, u2, u3, u4, u5, u6, u7, u8, u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus, set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,
+11 -3
View File
@@ -1,6 +1,6 @@
--- ---
type: reference type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed" title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 12118), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws." description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
tags: tags:
- llm - llm
@@ -31,7 +31,7 @@ is the bug.
| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) | | Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
| Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws | | Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws |
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file | | Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file |
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload | | Runtime | `/home/dank/GitPackages/llama-prism-0244744/bin/llama-server` — PrismML **fork**, `0.7.0-dev (build 12118, commit 024474448)`; the only loader for this payload |
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` | | Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) | | Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` | | Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
@@ -136,6 +136,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
| Window | Payload | Runtime | Shape | Replaced because | Rollback | | Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 2026-10-30 → | same 9B MTP file, Q8_0 mmproj and refusal adapter (payload untouched) | PrismML fork **12118** (`0.7.0-dev`, `024474448`) | unchanged (3 × 200,704; f16 K + q8_0 V; MTP depth 2) | fork rebuilt from `origin/prism`; the payload gate held — 202 `pq2_0` tensors and 202 Hadamard-folded weights loaded, MTP drafting engaged, chat/tools/vision answered | `LLAMA_BIN` back to `llama-prism-b10777/llama-server` (kept on disk) + redeploy |
| 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads | | 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads |
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move | | 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move | | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
@@ -201,7 +202,14 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check. - `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change. - `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
- **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason. - **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason.
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. - **A fork rebuild needs an install RPATH and a bin/lib-aware `LLAMA_BIN`.** `cmake --install` strips the
build-tree rpath, so the installed `llama-server` dies on `libllama-server-impl.so` unless the configure
carries `-DCMAKE_INSTALL_RPATH='$ORIGIN;$ORIGIN/../lib'`; the installed layout is `bin/` + `lib/`, not the
flat directory b10777 shipped, so the env's `LLAMA_BIN` has to name `…/bin/llama-server`. Build 12118 was
made from `origin/prism` with the 10777 configure (Release, `GGML_CUDA=ON`, `sm_86`,
`CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13`, Ninja) — gcc 15 is the box default and nvcc 12.4 rejects it.
- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. Build 12118 clears all
three.
## Contract ## Contract