diff --git a/AGENTS.md b/AGENTS.md index 1db3f7a..a146aa1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|---------|------| | Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 | | Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 | -| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj | 7771 | +| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 | Payload identity, shape, eras, laws and rollback live in ONE page each: [[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/.env` @@ -440,7 +440,11 @@ disagrees with it is the bug. `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a loader ignoring that metadata dequantizes wrong); payload `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16 - tower can crash the server with concurrent image requests). + tower can crash the server with concurrent image requests) + the rank-1 + refusal-direction adapter applied at load by `--lora-scaled …:1`. The + adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild + — and build 10777 is also the gate that lets it load: older forks refuse it + with `unsupported prism.hadamard.version: 2`. - Shape: `--ctx-size 602112 --parallel 3` = **three conversations at 200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088 B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024, diff --git a/serverAppConfigs-docs/LLM/frank-zealot.md b/serverAppConfigs-docs/LLM/frank-zealot.md index 5132dd7..5de21fd 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot.md +++ b/serverAppConfigs-docs/LLM/frank-zealot.md @@ -1,6 +1,6 @@ --- type: reference -title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 3 × 200704, f16 K + q8_0 V, MTP armed" +title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed" description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws." tags: - llm @@ -10,10 +10,10 @@ tags: - gguf - rtx3090 - endpoint -timestamp: 2026-10-18 +timestamp: 2026-10-19 --- -# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP, 3 × 200,704, f16 K + q8_0 V, MTP armed +# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200,704, f16 K + q8_0 V, MTP armed Dankclaw's RTX 3090 endpoint — `http://192.168.0.143:7771`, wire id `frank-zealot`, the fleet's **compaction hub** (three conversations at 200,704 each, the window the other two endpoints also cap at). One page @@ -29,11 +29,12 @@ is the bug. |---|---| | Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, LFS sha256 `c79fcdfaf6b7bf47…` | | Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) | -| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte count, LFS sha256 per file) | +| Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled :1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws | +| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file | | Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload | | Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` | | Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) | -| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` | +| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` | **Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block) `full_attention_interval 4` ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, `nextn_predict_layers 1` @@ -57,6 +58,17 @@ and `UBATCH_SIZE=2048` beside an armed drafter is still the shape that halved de `--spec-draft-model` — the draft rides the main model's weights, so a draft file would load the trunk twice. `SPEC_TYPE=none` is the rollback and strips the whole set. +**The adapter is behaviour, not shape.** It folds 3.2 MB of rank-1 tensors into the trunk at +load, so it moves nothing in the budget equation above — no KV, no slot, no window. What it +changes is what the endpoint answers, and the only instrument for that is a refusal probe: +the same two prompts the plain pack answered `I cannot provide…` (captured on the live +endpoint before the swap, `temperature 0.1`), re-asked after it. Budget **≥ 2 048 tokens** for +that probe on this payload — the thinking channel swallows a 600-token request whole and +`content` comes back empty with `finish_reason: length`, which reads as a refusal that never +happened. The adapter's own card table is **not** evidence here: it was measured on the PTQ1_0 +packing. Rollback is `LORA_FILE=` (empty) in the env and one redeploy — no rebuild, no payload +move; `LORA_SCALE=0` is the same result through the flag. + **Harness contract** (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw-3090)`, id `frank-zealot`): `contextWindow: 200704` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]` @@ -113,6 +125,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c | Window | Payload | Runtime | Shape | Replaced because | Rollback | |---|---|---|---|---|---| +| 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads | | 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move | | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move | | 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | — | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` | @@ -169,13 +182,15 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c - MTP is a property of the **file**: the plain PQ2_0 sibling could not draft at any flag value; only the `-MTP` export carries `blk.N.nextn.*`. - **The format is the runtime lock.** PQ2_0 is `general.file_type 141` over 202 tensors of custom GGML type **142** plus `prism.hadamard.*` metadata: stock llama.cpp refuses the type, and a loader ignoring the Hadamard transform would dequant **wrong, silently** (same lock class as ROCmFPX). Build ≥ b10776 is additionally the floor that pairs MTP with a quantized V cache. - Check arch support in the source tree: `grep -c` on the packed binary returns 0 for every arch, including the one it serves. -- **Base alignment.** Bonsai-2 is the censored base lineage. `/home/dank/models/Bonsai2-CRACK/` matches the **27B** geometry (a one-line `MODEL_FILE` swap there, identical tensor census); no dealigned 9B exists, so for this payload the refusal escape hatch is a fetch, not a revert. +- **Base alignment.** Bonsai-2 is the censored base lineage. `/home/dank/models/Bonsai2-CRACK/` matches the **27B** geometry (a one-line `MODEL_FILE` swap there, identical tensor census); no dealigned 9B exists, so for this payload the refusal escape hatch is a fetch, not a revert — which is exactly what was taken on 2026-10-19: the rank-1 adapter in the Adapter row above, behind `LORA_FILE`. - **A stale bench drop-in hijacks the production boot and no health check sees it.** An `Environment=FRANK_ZEALOT_ENV_FILE=/tmp/.env` in `frank-zealot.service.d/` booted a 27B PTQ1_0 payload while `/health` answered 200, `/v1/models` echoed the alias flag and the e2e gate passed 33/33 — none of them read the payload. - After any bench window: `systemctl show -p DropInPaths` → `tr '\0' ' ' -/`, and payload decisions land on this page the moment they are made, or an approved arm becomes indistinguishable from a rogue one a day later. - **An arm env must use the serve script's own variable names** (`CACHE_TYPE_K`/`CACHE_TYPE_V`, `BIND_ADDR`, `NGL`): an invented `KV_CACHE_TYPE` is swallowed by `: "${CACHE_TYPE_K:=f16}"` and the arm boots f16 — an OOM that reads like a model fault. - **Stop, shebang, GPU node.** llama.cpp rides out the default stop timeout (`TimeoutStopSec=180` is on the unit for that reason) and a hand-booted arm keeps answering 7771 behind a `failed` unit — kill it **by PID**, never `pkill -f llama-server`. - `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check. - `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change. +- **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason. +- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. ## Contract @@ -210,12 +225,12 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c boot ledger and the KV derivation all live on this page now; the header used to hold them, and two of its rung figures were wrong (`CACHE_TYPE_V=q8_0` and `CTX_SIZE=401408` both free **3,840 MiB** at the 524,288 pool, not 8,192 and not ~6.7 GiB — the Fallback rungs table above is the corrected statement). -- `scripts/serve-frank-zealot.sh` — the exec line and its rationale comments; honors `FRANK_ZEALOT_ENV_FILE` for arm boots; exits 1 naming a missing binary, GGUF or projector rather than half-loading (a restart policy would turn an unfinished download into a crash loop that reads like a runtime fault). +- `scripts/serve-frank-zealot.sh` — the exec line and its rationale comments; honors `FRANK_ZEALOT_ENV_FILE` for arm boots; exits 1 naming a missing binary, GGUF, projector or adapter rather than half-loading (a restart policy would turn an unfinished download into a crash loop that reads like a runtime fault). - `scripts/frank-zealot.service` — root unit, relinked into `/etc/systemd/system/`. -- `scripts/download-frank-zealot-bonsai9.sh` (active) · `…-bonsai2.sh` · `…-bonsai2-ptq1.sh` · +- `scripts/download-frank-zealot-bonsai9.sh` (active payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh` (the refusal adapter; its header is the provenance record) · `…-bonsai2.sh` · `…-bonsai2-ptq1.sh` · `scripts/download-frank-zealot.sh` (master-era GGUFs) · `old/download-frank-zealot-sglang-int8.sh` (SGLang). - `tests/frank_zealot_unit.bats` — stub-argv seam plus the VRAM fit gate. -- `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates, MainPID-hardened health); +- `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates including the adapter, MainPID-hardened health); - `shared/prometheus/prometheus.yml`, `shared/prometheus/rules/llm-fleet.yml` `shared/grafana/provisioning/dashboards/llm-fleet.json` — the scrape, the alert, the dashboard. - Live state, not repo files: the OpenWebUI picker row (back up `webui.db` before re-keying) and `~/.omp/agent/models.yml`.