diff --git a/AGENTS.md b/AGENTS.md index 14af787..8362f26 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server - container on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the + container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack @@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 | -| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771| +| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771| Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -73,7 +73,7 @@ All edits flow through git. Never edit files on a remote machine directly. The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by -their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`) +their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`) by its unit/env/serve trio in `scripts/` — so unrelated container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless @@ -229,7 +229,7 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp -endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench +endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service @@ -266,8 +266,8 @@ merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - Dankclaw `frank-large` (root unit, halogen podman on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): - `sudo systemctl stop frank-small && sudo systemctl mask frank-small` + - Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090): + `sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary` - Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` 2. Bench. @@ -287,7 +287,7 @@ The harness reaches the fleet through `.omp/agent/models.yml` in the operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is where the harness runs; there is no copy on the servers. Each provider is one HOST:PORT; each model `id` is the served alias: `frank-primary` → -`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` → +`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-secondary` → `modelRoles`. Resolution rules: the default model resolves as a `defaultProvider` + @@ -307,9 +307,9 @@ image]`) — `frank-large` returned to vision with the flash-server v2 swap `frank-primary` takes them natively (radiance runs the checkpoint's 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; `--mm-max-patches` available), -and `frank-small` through +and `frank-secondary` through `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo -GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is +GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is **200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the model's native 262144: the harness must compact before llama.cpp answers a longer prompt with a 400. Note also @@ -358,7 +358,7 @@ sidecar is deliberately not fetched). **Product facts vs the 27B era, both regressions FIXED:** the base is **abliterated** (the 27B was a censored-base finetune; refusals were an accepted trade there) and **vision is ON** (the 27B engine silently -ignored its encoder — frank-small owned images for exactly one day at this +ignored its encoder — frank-secondary owned images for exactly one day at this endpoint). **Engine laws that shape the config** (all in `scripts/frank-large.env` @@ -411,12 +411,12 @@ Rollback: revert the swap commit and redeploy — the 27B payload halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched. -### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) +### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) **llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`, `GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** -(`scripts/frank-small.service`, `relink_unit`ed into +(`scripts/frank-secondary.service`, `relink_unit`ed into `/etc/systemd/system/`), serving `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **RCO-IQ3_XXS -mtp** (10,442,827,392 B, sha256 `e13d3e74…`) **plus the publisher's own `mmproj-BF16.gguf`** (931,145,888 B, sha256 @@ -485,11 +485,11 @@ payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's 98/100, KL 0.0884). Runbook: -`serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`. +`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md`. Rollback: `/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload live until the 2026-10-06 evening swap — revert da36fa3 for the two-slot no-spec shape) or `/home/dank/models/MiMo-Q6K/` (revert to the -2×200,704 f16 shape — [[frank-small-llama-mimo-q6k-2x200704]]) or +2×200,704 f16 shape — [[frank-secondary-llama-mimo-q6k-2x200704]]) or `/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0; revert to 380aafa's tree). @@ -784,7 +784,7 @@ linked runbooks. 121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect. Record the child pid and kill it in a `finally`, then prove `free` came back before believing a number. Kill by pid — never `pkill -f llama-server`, - which also hits frank-small. + which also hits frank-secondary. **llama.cpp serving** @@ -850,7 +850,7 @@ linked runbooks. free. That was a CONTIGUITY failure, not a total-bytes one, and no totals model — the sweep's or the seam's — can see it.) - **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so - frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the + frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 → 2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a @@ -859,9 +859,9 @@ linked runbooks. than the ubatch cannot see it. The logical `--batch-size` was again free. And the predecessor's "ubatch 2048 failed the scratch boot" was a shape observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench - it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE` + it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE` drop-in, never a scratch port — one card holds one server. See - [[frank-small-llama-mimo-q6k-2x200704]]. + [[frank-secondary-llama-mimo-q6k-2x200704]]. - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one @@ -883,7 +883,7 @@ linked runbooks. CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT (`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)`) even when the pool has room. So - frank-small's 2 → 3 reslice moved each conversation from the model's native + frank-secondary's 2 → 3 reslice moved each conversation from the model's native 262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or the harness would discover the ceiling as a 400 on a long conversation. Side effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs @@ -894,9 +894,9 @@ linked runbooks. day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung itself is a further measured −10 %, weight-bandwidth, not slot geometry). All moves measured; both slot directions cost or buy window, never pool. - See [[frank-small-llama-mimo-q6k-2x200704]]. + See [[frank-secondary-llama-mimo-q6k-2x200704]]. - **Price a VRAM budget from two boots, never from a formula — and an mmproj - costs ~1.5x its file size.** The frank-small 3090 fit was solved from two + costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒ **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV 14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file @@ -904,12 +904,12 @@ linked runbooks. ride with it. Subtracting the two rows is exact; extrapolating weights+KV+ "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of - `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize + `PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize starts from measurement. - **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the client used the legacy top-level - `images` field on `/v1/chat/completions` — frank-small answered "Brown." for + `images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The failure is silent because a hallucinated color is a plausible answer. Probe vision with a solid color and expect an exact word. @@ -921,7 +921,7 @@ linked runbooks. returns `tool_calls`. Check tool support before routing a role at an endpoint. - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same option and the same ~12x prefill collapse described in the ROCm note above. - f16 KV needs nothing; frank-small ran f16 for that reason until + f16 KV needs nothing; frank-secondary ran f16 for that reason until 2026-09-17, when `build-cuda` was rebuilt with the option ON so the endpoint could take q8_0 KV. - **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both @@ -939,7 +939,7 @@ linked runbooks. raises on anything else — HTTP 500, not a clamp — so the harness's levels must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request, and the failure - names Jinja, not the model: every subagent on `frank-small` died with + names Jinja, not the model: every subagent on `frank-secondary` died with `Unexpected reasoning effort high` until `high: xhigh` was added on 2026-09-16. Halogen accepts the wider set, so the two entries legitimately differ. @@ -1014,7 +1014,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` | -| frank-small (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` | +| frank-secondary (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md` | | frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -1027,14 +1027,14 @@ Retired (rollback targets and history): | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` | | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` | -| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-small-llama-mimo-q6k-2x200704.md` | -| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-small`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` | +| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | -| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` | -| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` | -| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` | +| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` | |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`| | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` | | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` | @@ -1047,16 +1047,16 @@ Retired (rollback targets and history): | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` | | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` | | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` | -| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | -| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | -| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` | +| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` | +| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` | +| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | | CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` | | DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` | | FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` | -| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` | +| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` | | FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` | | ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` | | llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` | diff --git a/old/download-frank-large-mimo.sh b/old/download-frank-large-mimo.sh index b49573a..57c73b9 100755 --- a/old/download-frank-large-mimo.sh +++ b/old/download-frank-large-mimo.sh @@ -4,7 +4,7 @@ # command line (a Bearer token in argv is world-readable via /proc). # # Per-file FRESH downloads, no resume: the house rule (see -# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the +# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the # Hub LFS oid is what makes an access error fail instead of "succeeding". # # Usage: diff --git a/old/download-frank-large-qwen35.sh b/old/download-frank-large-qwen35.sh index 4b2dcb5..53c452c 100755 --- a/old/download-frank-large-qwen35.sh +++ b/old/download-frank-large-qwen35.sh @@ -2,7 +2,7 @@ # Download the source Qwen3.5-9B checkpoint — the ONLY place this model # family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the # vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero -# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00 +# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00 # acceptance). Public repo — no auth header on the command line. # # NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...). diff --git a/old/download-frank-small-sglang-int8.sh b/old/download-frank-secondary-sglang-int8.sh similarity index 97% rename from old/download-frank-small-sglang-int8.sh rename to old/download-frank-secondary-sglang-int8.sh index d054864..ceb6433 100755 --- a/old/download-frank-small-sglang-int8.sh +++ b/old/download-frank-secondary-sglang-int8.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small +# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary # (SGLang, RTX 3090). Public repo — no auth header on the command line. # # Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB @@ -7,7 +7,7 @@ # the house rule. # # Usage: -# scripts/download-frank-small.sh [DEST_DIR] [--verify-only] +# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only] # # LFS shards are verified against the Hub LFS sha256; non-LFS text files # against their declared byte size. Idempotent: complete files are skipped. diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index 4e0afac..e65e7c1 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -62,7 +62,7 @@ SERVICE_ORDER = ( "lidarr", "readarr", "frank-large", - "frank-small", + "frank-secondary", "frank-primary", ) @@ -94,7 +94,7 @@ DANKCLAW_SERVICES = { "caddy", "homepage", "frank-large", - "frank-small", + "frank-secondary", } DANKSERVER_SERVICES = { @@ -400,12 +400,12 @@ def services_for_path(path: str, target: str) -> set[str]: # change to it redeploying alertmanager is the only path that # reaches it on the host. _add(services, "alertmanager") - if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}: - # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve - # files here; each one selects frank-small only. The ik_llama.cpp + if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}: + # llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve + # files here; each one selects frank-secondary only. The ik_llama.cpp # trio that lived here for two days is in scripts/.AGENTTRASH/, # which is archive material and plans nothing. - _add(services, "frank-small") + _add(services, "frank-secondary") if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}: # CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and # serve files here the same way: a root unit on a host-built diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index ab8091c..46c3a7f 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -90,7 +90,7 @@ service_is_allowed() { dankclaw:authentik-postgres|dankclaw:authentik-server|\ dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\ dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\ - dankclaw:frank-large|dankclaw:frank-small|\ + dankclaw:frank-large|dankclaw:frank-secondary|\ dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\ dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\ dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\ @@ -1202,7 +1202,7 @@ deploy_llm_large() { # `podman run --name frank-large-halogen` / `podman wait` but NOT in # this deployer's own command line (`bash deploy-remote.sh # frank-large`) — so it cannot shoot itself in the foot (the fleet - # pkill law), and it can never match frank-small's llama-server. + # pkill law), and it can never match frank-secondary's llama-server. run_privileged systemctl stop frank-large.service 2>/dev/null || true pkill -f 'frank-large-halogen' 2>/dev/null || true podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true @@ -1216,57 +1216,57 @@ deploy_llm_large() { wait_for_health frank-large http://127.0.0.1:7770/health 1200 } -# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on +# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on # the RTX 3090 as a ROOT unit: the runtime is a native binary tree under # ~/GitPackages/llama-master/build-cuda, so there is no container image and # no generated unit to avoid. relink_unit points -# /etc/systemd/system/frank-small.service at the repo file and boot +# /etc/systemd/system/frank-secondary.service at the repo file and boot # persistence is that unit's [Install] section. # # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is # not shared: frank-large's engine runs on the gfx1151 iGPU. -LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}" -LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}" +LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}" +LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}" # The projector IS the vision capability here: the GGUF carries no vision # tensors, and the publisher ships mmproj-BF16.gguf in the same repo, so a # deploy that starts without this file silently ships a text-only # endpoint. Checked unconditionally, like frank-large's. -LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf}" +LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf}" -deploy_llm_small() { +deploy_llm_secondary() { # Refuse rather than restart-loop: a unit whose runtime or payload is # missing never becomes healthy, and a green deploy would leave the # endpoint dark with nothing in the log to say why. - if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then - echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2 - echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2 - echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2 - echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2 + if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then + echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2 + echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2 + echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2 + echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2 return 1 fi - for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do + for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do if [[ ! -f "$f" ]]; then - echo "ERROR: frank-small payload file absent at $f" >&2 - echo "frank-small: run scripts/download-frank-small.sh first;" >&2 - echo "frank-small: refusing to start a unit that cannot load." >&2 + echo "ERROR: frank-secondary payload file absent at $f" >&2 + echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2 + echo "frank-secondary: refusing to start a unit that cannot load." >&2 return 1 fi done - relink_unit frank-small + relink_unit frank-secondary run_privileged systemctl daemon-reload - run_privileged systemctl enable frank-small.service - run_privileged systemctl restart frank-small.service - wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1 + run_privileged systemctl enable frank-secondary.service + run_privileged systemctl restart frank-secondary.service + wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1 # False-green hardening (frank-large's lesson): the port answering # proves nothing about WHO answers — the unit must be active with a # live MainPID, or a stray SGLang holdover is masquerading as deployed. local main_pid - main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)" + main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)" if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then - echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2 + echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2 return 1 fi - echo "frank-small: health answered by unit MainPID $main_pid" + echo "frank-secondary: health answered by unit MainPID $main_pid" } # deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode, @@ -1567,7 +1567,7 @@ deploy_service() { seerr) deploy_seerr ;; marreta) deploy_marreta ;; frank-large) deploy_llm_large ;; - frank-small) deploy_llm_small ;; + frank-secondary) deploy_llm_secondary ;; frank-primary) deploy_llm_primary ;; hermes|hermes-dashboard|hermes-webui) deploy_hermes ;; romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;; diff --git a/scripts/deploy.sh b/scripts/deploy.sh index f0b62f5..3eb6700 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -141,7 +141,7 @@ valid_service() { dankclaw:alertmanager|\ dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\ dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\ - dankclaw:frank-large|dankclaw:frank-small|\ + dankclaw:frank-large|dankclaw:frank-secondary|\ dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\ dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\ dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\ diff --git a/scripts/download-frank-small.sh b/scripts/download-frank-secondary.sh similarity index 95% rename from scripts/download-frank-small.sh rename to scripts/download-frank-secondary.sh index 1447b89..183b7cf 100755 --- a/scripts/download-frank-small.sh +++ b/scripts/download-frank-secondary.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Download the frank-small payload for llama.cpp on the RTX 3090: +# Download the frank-secondary payload for llama.cpp on the RTX 3090: # ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS # **with the MTP head in the file** (the -mtp variant), plus the # PUBLISHER's OWN vision projector (mmproj-BF16.gguf, same repo). Public @@ -24,7 +24,7 @@ # re-fetch recipes are in git history (predecessors of this script). # # Usage: -# scripts/download-frank-small.sh [DEST_DIR] [--verify-only] +# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only] set -euo pipefail DEST="" diff --git a/scripts/frank-large.env b/scripts/frank-large.env index 4092685..2282583 100644 --- a/scripts/frank-large.env +++ b/scripts/frank-large.env @@ -42,7 +42,7 @@ # weights ~62 GiB resident + KV 10.7 + arena (HALOGEN_MAX_TOK=8192 => # ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2 # = ~82.7 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host -# honest; baseline co-tenancy (frank-small, ComfyUI, Caddy/observ) +# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ) # ~10-15. ~18 GiB margin. The boot ledger is the referee — read it # after deploy and re-size if the engine prints a clamp. # HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy diff --git a/scripts/frank-large.service b/scripts/frank-large.service index dc9aba7..7c83350 100644 --- a/scripts/frank-large.service +++ b/scripts/frank-large.service @@ -23,7 +23,7 @@ TimeoutStopSec=120 Restart=always RestartSec=10 # The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set -# on a box that also hosts frank-small, ComfyUI and the Caddy/observability +# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability # stack; make it a late OOM candidate, not the first. OOMScoreAdjust=-500 diff --git a/scripts/frank-small.env b/scripts/frank-secondary.env similarity index 95% rename from scripts/frank-small.env rename to scripts/frank-secondary.env index a4312bc..87f6f14 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-secondary.env @@ -1,4 +1,4 @@ -# frank-small — llama.cpp (master, CUDA sm_86) serving +# frank-secondary — llama.cpp (master, CUDA sm_86) serving # ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS # **with the MTP head in the file** on the RTX 3090, :7771. # @@ -45,15 +45,15 @@ MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored- # Vision: the publisher SHIPS the projector — mmproj-BF16.gguf # (931,145,888 B) from the same repo, fetched by -# scripts/download-frank-small.sh. No substitution needed for this +# scripts/download-frank-secondary.sh. No substitution needed for this # payload (the unsloth base-tower proof of 2026-10-05 remains the # fallback argument for the IQ3_S rollback payload). # Empty string = text-only on purpose. MMPROJ_FILE=/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf -# Fleet endpoint: same port and alias as every frank-small before it, so the +# Fleet endpoint: same port and alias as every frank-secondary before it, so the # harness models.yml id and the homepage probe do not move. -ALIAS=frank-small +ALIAS=frank-secondary BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 diff --git a/scripts/frank-small.service b/scripts/frank-secondary.service similarity index 66% rename from scripts/frank-small.service rename to scripts/frank-secondary.service index a897f9c..2c3af17 100644 --- a/scripts/frank-small.service +++ b/scripts/frank-secondary.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp Swift-1.5-Qwen3.8-27B Uncensored RCO IQ3_XXS GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, draft-mtp depth 3, mmproj vision) on the RTX 3090 (dankclaw, frank-small) +Description=llama.cpp Swift-1.5-Qwen3.8-27B Uncensored RCO IQ3_XXS GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, draft-mtp depth 3, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary) After=network-online.target Wants=network-online.target @@ -10,8 +10,8 @@ Group=dank # CUDA pins its staging buffers, so the process needs the memlock budget even # though the weights themselves are copied into VRAM. LimitMEMLOCK=infinity -Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env" -ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh +Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env" +ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh Restart=on-failure RestartSec=10 diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-secondary.sh similarity index 89% rename from scripts/serve-frank-small.sh rename to scripts/serve-frank-secondary.sh index 727bc43..9a3e182 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-secondary.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Serve ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF -# (RCO-IQ3_XXS, **MTP head in the file**) as frank-small on Dankclaw's +# (RCO-IQ3_XXS, **MTP head in the file**) as frank-secondary on Dankclaw's # RTX 3090 via llama.cpp (master build, CUDA sm_86). # # 2026-10-06 (Frank): TWO 131,072 conversations (262144 / 2) at the 128k @@ -24,11 +24,11 @@ # Sampling: llama.cpp takes it per request (the fleet harness sends its # recipe from models.yml); greedy is the harness default. # -# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides. +# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides. set -euo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}" +ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}" if [[ -f "$ENV_FILE" ]]; then set -a # shellcheck disable=SC1090 @@ -39,13 +39,13 @@ fi # --- defaults (used when the env file does not set them) ------------------- : "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}" : "${MODEL_FILE:=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}" -: "${ALIAS:=frank-small}" +: "${ALIAS:=frank-secondary}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7771}" # ctx-size is the POOL llama.cpp divides across slots: 262144 / 2 gives # TWO 131,072 conversations — Frank's 128k cap each. 2026-10-06 (Frank): # drop MTP, drop ubatch to 512, take two slots; the fit is tight by his -# explicit order (frank-small.env carries the arithmetic and the floor +# explicit order (frank-secondary.env carries the arithmetic and the floor # violation). : "${CTX_SIZE:=262144}" : "${PARALLEL:=2}" @@ -76,11 +76,11 @@ fi # has not finished, and Restart=always would turn it into a crash loop that # reads like a runtime fault. for path in "$LLAMA_BIN" "$MODEL_FILE"; do - [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; } + [[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; } done EXTRA_ARGS=() if [[ -n "$MMPROJ_FILE" ]]; then - [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; } + [[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; } EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE") fi @@ -97,7 +97,7 @@ nvidia-modprobe -u -c=0 2>/dev/null || true # Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0), # and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so -# the q8_0 KV contract in frank-small.env is a real path — without this +# the q8_0 KV contract in frank-secondary.env is a real path — without this # flag or that build option the engine silently falls back and the # contract becomes a lie. ARGS=( diff --git a/scripts/test-changed.sh b/scripts/test-changed.sh index 198e456..043fef6 100755 --- a/scripts/test-changed.sh +++ b/scripts/test-changed.sh @@ -204,12 +204,12 @@ for path in "${CHANGED[@]}"; do scripts/qwen-sharp-v22.4.0.jinja) add_test tests/frank_large_unit.bats ;; - # frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA + # frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA # binary, so no generated quadlet is involved. Its env, unit and serve # script all live in scripts/, which is why this case has to precede # the generic scripts/*.service and scripts/* fallbacks below. - scripts/frank-small.*|scripts/serve-frank-small.sh) - add_test tests/frank_small_unit.bats + scripts/frank-secondary.*|scripts/serve-frank-secondary.sh) + add_test tests/frank_secondary_unit.bats add_test tests/llm_units.bats add_test tests/deployment_scoping.bats add_test tests/scripts_syntax.bats diff --git a/serverAppConfigs-docs/Agent/OMP Harness.md b/serverAppConfigs-docs/Agent/OMP Harness.md index 2486790..f9f6b97 100644 --- a/serverAppConfigs-docs/Agent/OMP Harness.md +++ b/serverAppConfigs-docs/Agent/OMP Harness.md @@ -35,12 +35,12 @@ encodes): 1. **Source trigger** = `thresholdPercent × source contextWindow` (0.70 × 200,704 = 140,492 for frank-primary on the current shape) 2. **Output budget** = capped at the target model's `maxTokens` - (32,768 for frank-small; 36,864 frank-primary; 57,344 frank-large — the + (32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it) 3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤ 200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700 - (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-small). + (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary). Procedure (recompute from the LIVE files every time — windows rotate with every swap; the numbers above are the 2026-10-05 state): diff --git a/serverAppConfigs-docs/Architecture/Deployment Flow.md b/serverAppConfigs-docs/Architecture/Deployment Flow.md index 12efb8d..f527aa1 100644 --- a/serverAppConfigs-docs/Architecture/Deployment Flow.md +++ b/serverAppConfigs-docs/Architecture/Deployment Flow.md @@ -63,7 +63,7 @@ service on the other host. | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. | | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. | | `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. | -| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. | +| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. | | `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. | | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver//quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/` to `/home/dank/Containers/` (mapped-uid roots move without chown). Steady state is restart-only. | diff --git a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md index bc33310..557c892 100644 --- a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md +++ b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md @@ -95,7 +95,7 @@ host-side `workflows/` dir and the repo copy. ## Recommended additions Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is -frank-small's). All repos verified 2026-10-05. None are on disk yet. +frank-secondary's). All repos verified 2026-10-05. None are on disk yet. | # | What | Repo | Grab | Size | Why | |---|---|---|---|---|---| diff --git a/serverAppConfigs-docs/Dankclaw/Hermes.md b/serverAppConfigs-docs/Dankclaw/Hermes.md index 26de54c..e9d941a 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes.md @@ -189,7 +189,7 @@ What actually breaks, and the fix: auxiliary: title_generation: provider: dankserver-small # names track the live config; diff first - model: frank-small + model: frank-secondary api_mode: codex_responses # real per-task key extra_body: reasoning: diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 2f1e0f3..c3484d8 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -24,7 +24,7 @@ https://github.com/open-webui/open-webui | Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). | | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. | | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` | -| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) | +| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-secondary` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) | | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) | | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` | | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` | @@ -39,14 +39,14 @@ discovery — so the picker shows exactly the models seeded there. |--------------|----------------|---------|------| | `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw | | `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver | -| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw | +| `frank-secondary` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw | Two gotchas make the wiring non-obvious: - **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns `{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses - the llama.cpp endpoints (frank-large, frank-small). The fix is to set + the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set `model_ids` in each connection's `api_config` — when non-empty, OpenWebUI builds the model list directly and never fetches `/v1/models`. frank-medium (vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids` @@ -218,13 +218,13 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad. - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].) - The default admin account is created on first login; secure it immediately. - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready. -- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`. +- The on-host LLM services (frank-large `:7770`, frank-secondary `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`. ## History - **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns. -- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`. +- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`. - **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`. diff --git a/serverAppConfigs-docs/Infrastructure/Homepage.md b/serverAppConfigs-docs/Infrastructure/Homepage.md index 19a3a34..a05ad12 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage.md @@ -23,7 +23,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat page (`shared/homepage/docs.html`) — access walkthrough, the running-model guide, media how-tos, and troubleshooting. Per-service `/docs//` pages remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`, -`frank-small`) moved from admin-only to the **user** page with live model +`frank-secondary`) moved from admin-only to the **user** page with live model descriptions and a `docs` link to `/docs#models`. **Sections + WIP split (2026-09-21):** both pages now group their panels @@ -111,7 +111,7 @@ flowchart TD ## Service classification **User services** (root page), in section order: -- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-small — status + docs link, no Open link). +- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-secondary — status + docs link, no Open link). - **Web Apps** — SearXNG, Gitea, Copyparty, Marreta. - **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr. diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index ac43cd0..5644cf9 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works | | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` | | `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works | -| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway | +| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway | **Three jobs read down, and they are not outages** (audited 2026-09-16): diff --git a/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md b/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md index bb0b2b6..35e9460 100644 --- a/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md +++ b/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md @@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model. ## Deployment on Dankserver (NVIDIA RTX 3090) -The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware. +The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware. -> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only. +> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only. The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: @@ -315,7 +315,7 @@ not switch without re-measuring". The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type, slot split, quant rung — not just spec flags) is the transient systemd -drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]]. +drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]]. ## Is it worth it? (arithmetic before deployment) @@ -419,4 +419,4 @@ measurement that would settle it. - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic - [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands -- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle \ No newline at end of file +- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md index 6cce340..f86013a 100644 --- a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md @@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run | Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) | | Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] | -The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]). +The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]). ### ROCm diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 9573e3c..a342371 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -17,7 +17,7 @@ timestamp: 2026-09-20 Dankserver is a media/library server and the `frank-medium` LLM host. The host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the -FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with +FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with two AMD Radeon AI PRO R9700 cards. ## Hardware Specifications @@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see The active inference service is `frank-medium.service`: mainline llama-server on port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former -`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap; -its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`, +`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap; +its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`, and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`. See [[deployment]]. diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md index cd8e3f9..28a3ed2 100644 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md @@ -13,12 +13,12 @@ timestamp: 2026-09-03 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology. > See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration. -> The `frank-small` name and port 52625 then belonged to the vLLM +> The `frank-secondary` name and port 52625 then belonged to the vLLM > service on Dankclaw (nameistoken Quark W8A8); this page is archive. # FastFlowLM NPU Inference -FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion. +FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion. ## Verdict (2026-08-16) @@ -203,16 +203,16 @@ Notes: ## Promotion (2026-08-16) -`frank-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch. +`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch. ### Boot policy (2026-08-17) -The `frank-small` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw: +The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw: ``` -sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/frank-small.service +sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service sudo systemctl daemon-reload -sudo systemctl enable frank-small +sudo systemctl enable frank-secondary ``` Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom). @@ -353,7 +353,7 @@ installed) against `/opt/fastflowlm/share/flm/xclbins/` (available). Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window. 1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`. -2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-small' is not supported.` Each such request also triggered the unknown-tag fallback below. +2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below. 3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load. Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16. @@ -390,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso - **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step. - **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this. - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags. -- **NPU production role (2026-08-16).** `frank-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded. +- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded. ## Citations diff --git a/serverAppConfigs-docs/LLM/FreeToken Builds.md b/serverAppConfigs-docs/LLM/FreeToken Builds.md index cf83a1d..39f0df5 100644 --- a/serverAppConfigs-docs/LLM/FreeToken Builds.md +++ b/serverAppConfigs-docs/LLM/FreeToken Builds.md @@ -23,7 +23,7 @@ read the returned content: ```python import json, urllib.request req = urllib.request.Request("http://:/v1/chat/completions", - json.dumps({"model": "frank-small", "messages": [{"role": "user", + json.dumps({"model": "frank-secondary", "messages": [{"role": "user", "content": "Hi"}]}).encode(), {"Content-Type": "application/json"}) print(json.load(urllib.request.urlopen(req, timeout=600))) ``` diff --git a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md index 76e3d2c..0e7b768 100644 --- a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md +++ b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md @@ -27,7 +27,7 @@ wedging a host until power cycle. Reference harnesses: with `container state improper`. 2. **Boot the arm as the service, not a scratch port** (no scratch ports on LLM hosts — one card, one server). The serve script takes its config via - env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`) + env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`) pointing at a staged copy; the checkout is never mutated. 3. **Fast-fail the health wait**: poll `podman ps | grep -qx ` inside the loop — a refused boot dies in one poll instead of 20 min. Capture the diff --git a/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md b/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md index 269126b..4e41767 100644 --- a/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md +++ b/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md @@ -80,7 +80,7 @@ co-tenants. - **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned — "move layers to the card" converts reclaimable cache into device memory and frees nothing for co-tenants. A served model's footprint ≠ weight size - (frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity + (frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV = 28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k. - **Context lengths: derive, never remember.** `llama-bench` computes @@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed ## Related - [[GGUF Compatibility]] — what a payload is before it is priced -- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape +- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape - [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured - [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp - [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side diff --git a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md index ae47760..bc4551a 100644 --- a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md +++ b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md @@ -8,7 +8,7 @@ timestamp: 2026-10-05T00:00:00-07:00 # Model Swap and Rollback -Applies to every endpoint swap on the fleet (frank-small / frank-large / +Applies to every endpoint swap on the fleet (frank-secondary / frank-large / frank-primary). Repo law: all config flows through git; the live models are host state under `/home/dank/models//`. diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md index cbbdfbe..9b5c508 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -48,10 +48,10 @@ state, verified against the live journal and output smoke tests The PP pair is the only inference service enabled for boot. Both units (`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are -enabled. The NPU unit (`frank-small` on Dankclaw) and all retired +enabled. The NPU unit (`frank-secondary` on Dankclaw) and all retired units are disabled. Restart order: worker first, then head. The Dankserver helper `frank-restart-small.sh` is obsolete (it restarts the removed -`frank-small` unit); restart the worker with +`frank-secondary` unit); restart the worker with `sudo systemctl restart frank-pp-worker`. The Dankclaw helper `frank-restart-large.sh` (sync + cache drop + restart) is correct for the head. `deploy.sh dankserver` is broken until retargeted. See diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md index ce513d5..68933fb 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md @@ -65,6 +65,6 @@ graph LR ## See Also - [[DFlash DSpark Speculative Decoding]] -- [[frank-small-benchmark]] +- [[frank-secondary-benchmark]] - [[Dankserver System Specs]] - [[Dankclaw System Specs]] diff --git a/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md b/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md index 91d771a..4c06f72 100644 --- a/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md +++ b/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md @@ -247,4 +247,4 @@ see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected); measured campaigns - [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it - [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive -- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts +- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — window discipline on live hosts diff --git a/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md b/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md index 65082e5..1b3f041 100644 --- a/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md +++ b/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md @@ -122,7 +122,7 @@ export HF_HOME=/mnt/entropy/huggingface ### Active Service Conflict -The retired `frank-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free: +The retired `frank-secondary` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free: ``` nvidia-smi # verify GPU is free diff --git a/serverAppConfigs-docs/LLM/VLLM Overview.md b/serverAppConfigs-docs/LLM/VLLM Overview.md index dbccf8b..ae0e27c 100644 --- a/serverAppConfigs-docs/LLM/VLLM Overview.md +++ b/serverAppConfigs-docs/LLM/VLLM Overview.md @@ -268,7 +268,7 @@ ad hoc qwen-fp8 container (the same fork, same payload, same TP=2): **There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger -fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-small +fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-secondary was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no counterpart here, and the two backends should not be reasoned about by analogy. diff --git a/serverAppConfigs-docs/LLM/dankserver-vram-growth.md b/serverAppConfigs-docs/LLM/dankserver-vram-growth.md index 31c2ea2..c306e44 100644 --- a/serverAppConfigs-docs/LLM/dankserver-vram-growth.md +++ b/serverAppConfigs-docs/LLM/dankserver-vram-growth.md @@ -1,7 +1,7 @@ --- type: reference title: Dankserver VRAM and RAM Growth Under Load -description: Investigation of the recurring cuBLAS crash on frank-small. What grows, what does not, and what to do next. +description: Investigation of the recurring cuBLAS crash on frank-secondary. What grows, what does not, and what to do next. tags: - dankserver - llamacpp @@ -18,7 +18,7 @@ timestamp: 2026-09-03 # Dankserver VRAM and RAM Growth Under Load This page records the investigation of the recurring cuBLAS crash on -`frank-small` (llama.cpp, RTX 3090 24 GiB). The service aborts +`frank-secondary` (llama.cpp, RTX 3090 24 GiB). The service aborts with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during `update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13 @@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode ## Mitigation Options -The documentation for the benchmark ([[frank-small-benchmark]]) +The documentation for the benchmark ([[frank-secondary-benchmark]]) prescribes the first option for this crash class. None of these change the model. @@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119 ## See Also -- [[frank-small-benchmark]] +- [[frank-secondary-benchmark]] - [[Dankserver System Specs]] - [[deployment]] \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/dealign-ornith-crack.md b/serverAppConfigs-docs/LLM/dealign-ornith-crack.md index bc5397e..6ee66f2 100644 --- a/serverAppConfigs-docs/LLM/dealign-ornith-crack.md +++ b/serverAppConfigs-docs/LLM/dealign-ornith-crack.md @@ -1,7 +1,7 @@ --- type: reference title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver -description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-small. +description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-secondary. tags: - llamacpp - cuda @@ -14,12 +14,12 @@ timestamp: 2026-08-24 # dealign Ornith-1.5-35B-A3B CRACK on Dankserver -> **Active 2026-08-25.** `frank-small` serves this model. It +> **Active 2026-08-25.** `frank-secondary` serves this model. It > replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121). > The Q6_K file was downloaded from the model repository and validated with > `llama-gguf` before installation. -`frank-small` (port 7770) serves +`frank-secondary` (port 7770) serves `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + attention architecture, abliterated (uncensored). The GGUF also contains a diff --git a/serverAppConfigs-docs/LLM/deployment.md b/serverAppConfigs-docs/LLM/deployment.md index ddbce22..853ffd9 100644 --- a/serverAppConfigs-docs/LLM/deployment.md +++ b/serverAppConfigs-docs/LLM/deployment.md @@ -21,7 +21,7 @@ SearXNG, Caddy, or another container service does **not** restart inference. |---|---|---|---:| | Dankserver | `dank@dankserver` | `frank-medium` | 7770 | | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 | -| Dankclaw | `dank@dankclaw` | `frank-small` | 7771 | +| Dankclaw | `dank@dankclaw` | `frank-secondary` | 7771 | ## Scoped workflow @@ -53,13 +53,13 @@ on the selected host. The following files select the corresponding service: checkout, so any file there changes deployed state. - `scripts/frank-large.env`, `frank-large.service`, and `serve-frank-large.sh` select `frank-large` on Dankclaw. -- `scripts/frank-small.env`, `frank-small.service`, and - `serve-frank-small.sh` select `frank-small` on Dankclaw. +- `scripts/frank-secondary.env`, `frank-secondary.service`, and + `serve-frank-secondary.sh` select `frank-secondary` on Dankclaw. - Documentation, tests, deployment tooling, and unrelated container changes do not select any inference service. The target-side `scripts/deploy-remote.sh` installs the selected service. -`frank-large` and `frank-small` are **root units**: the handler relinks the +`frank-large` and `frank-secondary` are **root units**: the handler relinks the repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd, enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the handler retires its predecessors (the replica Quadlets and the retired llama @@ -67,7 +67,7 @@ root unit), stops the service's own unit, waits for the published port to clear, installs the `.container` and restarts the generated unit. Each handler then waits on that service's health endpoint — `http://127.0.0.1:7770/health` for frank-large and frank-medium, -`http://127.0.0.1:7771/health` for frank-small. The wait budget is 20 minutes +`http://127.0.0.1:7771/health` for frank-secondary. The wait budget is 20 minutes because cold model loads are slow. A failed handler does not advance its per-service state. @@ -76,7 +76,7 @@ per-service state. The root units point into the stable target checkout: ```text /home/dank/serverAppConfigs/scripts/frank-large.service -/home/dank/serverAppConfigs/scripts/frank-small.service +/home/dank/serverAppConfigs/scripts/frank-secondary.service ``` `frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/` (`frank-medium.container`) rather than a unit in `scripts/`. @@ -110,7 +110,7 @@ change in that service's inputs. If the LLM handler fails after the target pull: -1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-small +1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-secondary -n 200` (root units), or `journalctl --user -u frank-medium -n 200` (rootless Quadlet). 2. Confirm the model files, the runtime binary and (for frank-medium) the @@ -120,13 +120,13 @@ If the LLM handler fails after the target pull: ```bash ./scripts/deploy.sh dankserver --service frank-medium ./scripts/deploy.sh dankclaw --service frank-large - ./scripts/deploy.sh dankclaw --service frank-small + ./scripts/deploy.sh dankclaw --service frank-secondary ``` 4. Verify the matching health endpoint before declaring recovery. Do not stop the other host's inference service during recovery. The two Dankclaw endpoints are independent processes on different devices: restarting -frank-large does not disturb frank-small, and the reverse holds too — with one +frank-large does not disturb frank-secondary, and the reverse holds too — with one exception worth knowing, that the deployer waits for port 7770 to be released before it starts the replacement container. For benchmark work, follow the separate stop/benchmark/restart procedure in [[Testing Policy]] and the diff --git a/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md b/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md index 90e0f2a..1b2e326 100644 --- a/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md +++ b/serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md @@ -112,7 +112,7 @@ instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays mandatory. - Vision: OpenAI `image_url` content parts only (the fleet's standard - contract; the legacy top-level `images` field is the frank-small trap). + contract; the legacy top-level `images` field is the frank-secondary trap). ≥256 px probe images or the model answers "video". - `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a truncation. `max_tokens` INCLUDES thinking. @@ -181,7 +181,7 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints. `maxTokens: 57344`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`, the `high→xhigh` effort fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides -frank-small (the `compactionModel` steering is unchanged). +frank-secondary (the `compactionModel` steering is unchanged). ## Rollback diff --git a/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md b/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md index ae85a9a..08fd65a 100644 --- a/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md @@ -25,7 +25,7 @@ confirmed by Frank before deploy: 1. **Text-only.** halogen loads the language tower only; README "honest limits": *"The vision encoder is ignored; image inputs are not supported."* - The mmproj-era vision role moves wholly to frank-small. + The mmproj-era vision role moves wholly to frank-secondary. 2. **Censored base.** The checkpoint is a finetune of the censored `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The ROCmFPX AGENT file was an *Uncensored* build; refusals can return. @@ -83,7 +83,7 @@ bearing numbers: |thinking field|`reasoning_content` (same as before)| |`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking| |tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy| -|image input|ignored/error — route to frank-small| +|image input|ignored/error — route to frank-secondary| ## Deploy mechanics @@ -114,7 +114,7 @@ bearing numbers: - **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the legacy bare `llama-server --model /home/dank/models/kingjones-*` process and the legacy `frank-large` container name by NAME (never `pkill -f - llama-server` — that hits frank-small), retires the old quadlet if the 2026-09 + llama-server` — that hits frank-secondary), retires the old quadlet if the 2026-09 halogen era's unit lingers, waits port 7770 free, relinks, health-gates. - **Bench law:** stop+mask the unit before any GPU experiment on this box (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md index d09bd07..65e3123 100644 --- a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md @@ -148,7 +148,7 @@ in [[frank-large-rocmfpx-q8agent-mtp]] and still applies. Sweep key for scratch servers is now the payload name: `pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f -llama-server` (frank-small is also one). +llama-server` (frank-secondary is also one). ## Operations diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md index c11dd0d..af029b8 100644 --- a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md @@ -162,7 +162,7 @@ Three traps, all hit for real: enough for the vision probe). 2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`** appears in the log under draft resets. Self-recovering; not an outage signal. -3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a +3. **Never `pkill -f llama-server` on Dankclaw** — frank-secondary is also a `llama-server`. The sweep key here is the payload name: `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`. diff --git a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md index 55428a3..8f331d2 100644 --- a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md +++ b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md @@ -35,7 +35,7 @@ timestamp: 2026-10-01 (`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace). - **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16** - (17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8 + (17.53 GiB, 4 shards) — the same checkpoint frank-secondary serves as an INT8 W8A16 AutoRound conversion on the 3090, here unquantized. - **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors are loaded into the draft pass). @@ -82,7 +82,7 @@ Qwen3.5-9B that preserves the hidden-state geometry — identical per-position acceptance is the direct measurement that the distillation preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors** (verified in both the BF16 base and the INT8 conversion) — which is exactly -why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang +why frank-secondary's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang reads the head from the TARGET checkpoint, vLLM reads the draft from a separate path. That difference is the whole trick: @@ -183,4 +183,4 @@ unit/env/serve trio and redeploying returns the 27B endpoint. - Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151 - Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env` - Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch: - [[frank-small-sglang-mimo-v26]] + [[frank-secondary-sglang-mimo-v26]] diff --git a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md index 1ed505b..c12f8b6 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md +++ b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md @@ -211,7 +211,7 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating. - [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split history -- [[frank-small-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt +- [[frank-secondary-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt at depth 0 made prefill climb all the way to ubatch 2048. Depth is the difference: with no KV to attend over, per-pass overhead dominates and a bigger micro-batch wins; at 64k of resident KV it does not. diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 3abaddf..84ad26d 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -312,7 +312,7 @@ came up healthy and *text-only*: image requests answered 500 "image input is not supported — hint: … provide the mmproj" and the load log had no mmproj line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the model directory. The serve script now passes `--mmproj` explicitly (the -frank-small pattern), refuses to start without the file when vision is on, and +frank-secondary pattern), refuses to start without the file when vision is on, and the deploy handler's preflight checks for it. Verified end-to-end after the fix: a generated 64x32 test image was described from its pixels — with thinking on the model reasoned about "the three horizontal bars" (its reading @@ -724,7 +724,7 @@ fleet. 2026-09-15, whose `--split-mode layer` measurements this service reuses - [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a collective, and every software escape that is closed -- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX +- [[frank-secondary-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX 3090, MTP, the image path) - [[Dankserver System Specs]] — hardware inventory - [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed diff --git a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md index afbf213..6f59385 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md +++ b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md @@ -144,7 +144,7 @@ ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed First deploy (2026-09-15) also ran `retire_root_llm_small()` in `scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped, -disabled and moved to `~/.AGENTTRASH/20260915-frank-small-retire/host-units` +disabled and moved to `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units` (the host's `systemctl disable` has been observed to delete unit files, 2026-08-26 incident, so both outcomes are tolerated). diff --git a/serverAppConfigs-docs/LLM/frank-npu.md b/serverAppConfigs-docs/LLM/frank-npu.md index b7a5761..bab3c5a 100644 --- a/serverAppConfigs-docs/LLM/frank-npu.md +++ b/serverAppConfigs-docs/LLM/frank-npu.md @@ -55,9 +55,9 @@ image path is halogen's vision tower on `frank-large`. | consumer | setting | now | |---|---|---| | Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) | -| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-small` | +| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-secondary` | | harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` | -| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-small` | +| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-secondary` | | harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` | The `flm (dankclaw-npu)` provider block was removed from the harness model diff --git a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md index ece22a3..4e92d92 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md @@ -254,7 +254,7 @@ refetch. - [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same b11030-mix binary family on gfx1151. -- [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the +- [[frank-secondary-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the 3090; the only audio-capable endpoint. - The harness contract lives in `/home/blank/.omp/agent/models.yml` (`llamacpp (dankserver)` provider, `contextWindow 262144`, diff --git a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md index 4f666d3..0a83acd 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md @@ -55,7 +55,7 @@ would drift from the qualification — don't. ## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on -frank-small/large). The fork's `speculative_config` is a **single dict** +frank-secondary/large). The fork's `speculative_config` is a **single dict** (one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are mutually exclusive — no chaining seam exists in the fork. What IS active: diff --git a/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md b/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md index 9a1feef..6543fe9 100644 --- a/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md +++ b/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md @@ -18,7 +18,7 @@ The old scripts moved to `old/scripts/` in this repository. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| | frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 | -| frank-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 | +| frank-secondary | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 | The RTX 3090 runs the whole 27B model standalone. No RPC worker, no tensor split, no NPU. @@ -40,7 +40,7 @@ tensor split, no NPU. `--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback (48/64/24). -### frank-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 +### frank-secondary — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 - Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40 layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx). @@ -103,10 +103,10 @@ Production choice: **131072 (128K)** — the maximum that keeps about 5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion. 6. Confirm VRAM headroom with `nvidia-smi`. -### Dankclaw (frank-small) +### Dankclaw (frank-secondary) 1. Stop the old services: `sudo systemctl stop frank-large - frank-small` (retired PP head and NPU endpoint). + frank-secondary` (retired PP head and NPU endpoint). 2. Install uv and create the venv: ``` curl -LsSf https://astral.sh/uv/install.sh | sh @@ -131,9 +131,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about vision encoder falls back to torch SDPA and engine init OOMs. 5. Symlink the unit, reload, enable, restart: ``` - sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/ - sudo systemctl daemon-reload && sudo systemctl enable frank-small - sudo systemctl restart frank-small + sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/ + sudo systemctl daemon-reload && sudo systemctl enable frank-secondary + sudo systemctl restart frank-secondary ``` 6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion. @@ -159,7 +159,7 @@ on gfx1151; newer vLLM releases may already include it. Three changes landed after the initial deployment: -### 1. Tool calling on frank-small (HTTP 400 fix) +### 1. Tool calling on frank-secondary (HTTP 400 fix) Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400 unless the server enables tool calling. Fixes applied: @@ -184,7 +184,7 @@ model behavior, which motivated the swap. ### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8 -`frank-small` now serves +`frank-secondary` now serves `nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB, vLLM-native expert layout). The Avesed compressed-tensors quant's Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain diff --git a/serverAppConfigs-docs/LLM/frank-small-benchmark.md b/serverAppConfigs-docs/LLM/frank-secondary-benchmark.md similarity index 94% rename from serverAppConfigs-docs/LLM/frank-small-benchmark.md rename to serverAppConfigs-docs/LLM/frank-secondary-benchmark.md index 6107acf..92418a6 100644 --- a/serverAppConfigs-docs/LLM/frank-small-benchmark.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-benchmark.md @@ -1,7 +1,7 @@ --- timestamp: 2026-09-03 type: reference -title: frank-small Benchmark — ncmoe and Batch Tuning +title: frank-secondary Benchmark — ncmoe and Batch Tuning description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P. tags: - benchmark @@ -11,11 +11,11 @@ tags: - dankserver --- -# frank-small Benchmark — ncmoe and Batch/Ubatch Tuning +# frank-secondary Benchmark — ncmoe and Batch/Ubatch Tuning This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final -`--n-cpu-moe` and `--ubatch-size` values for `frank-small`. +`--n-cpu-moe` and `--ubatch-size` values for `frank-secondary`. Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads, `--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to @@ -106,7 +106,7 @@ service runs 700416 cells (13,680 MiB). ## Chosen State -> **Live drift (2026-08-22 audit):** the deployed `frank-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation. +> **Live drift (2026-08-22 audit):** the deployed `frank-secondary` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation. Prefill speed is the priority, so the state keeps the top prefill tier and moves one expert layer to CPU for stability: diff --git a/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md b/serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md similarity index 75% rename from serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md rename to serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md index ffcab4e..42deb5b 100644 --- a/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md @@ -1,10 +1,10 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M" -description: "The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves." +title: "frank-secondary on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M" +description: "The active frank-secondary runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves." tags: - llm - - frank-small + - frank-secondary - dankclaw - llama.cpp - rtx3090 @@ -14,25 +14,25 @@ tags: timestamp: 2026-09-17 --- -# frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod) +# frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod) -> **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]] +> **ACTIVE since 2026-09-17.** Replaces [[frank-secondary-qwen38-gsq-rco]] > (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and -> [[frank-small-vllm-gemma4]] (the same-day vLLM experiment, also the -> rollback target). Same port (7771), same alias (`frank-small`), same +> [[frank-secondary-vllm-gemma4]] (the same-day vLLM experiment, also the +> rollback target). Same port (7771), same alias (`frank-secondary`), same > client contract surface. ## Identity -- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw). +- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw). - **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with `-DGGML_CUDA_FA_ALL_QUANTS=ON` (`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`). The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention collapses prefill ~12x on CUDA. -- **Service shape:** root systemd unit `scripts/frank-small.service` - (`User=dank`), launched by `scripts/serve-frank-small.sh` with tunables from - `scripts/frank-small.env`. +- **Service shape:** root systemd unit `scripts/frank-secondary.service` + (`User=dank`), launched by `scripts/serve-frank-secondary.sh` with tunables from + `scripts/frank-secondary.env`. ## Shape @@ -90,7 +90,7 @@ fault). ## Client contract -- **Alias:** `frank-small` (the `id` at `/v1/models`). +- **Alias:** `frank-secondary` (the `id` at `/v1/models`). - **Reasoning:** extracted to `reasoning_content` (`--reasoning-format deepseek`, unlimited budget). - **Tools:** supported (`--jinja` is pinned even though this build defaults @@ -109,12 +109,12 @@ pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches ## Rollback -Payload and runtime for [[frank-small-qwen38-gsq-rco]] (ISTA-DASLab -Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-small-vllm-gemma4]] +Payload and runtime for [[frank-secondary-qwen38-gsq-rco]] (ISTA-DASLab +Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-secondary-vllm-gemma4]] is the same-day vLLM experiment (also a rollback target). # Citations - Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS - llama.cpp: https://github.com/ggml-org/llama.cpp -- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env` +- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env` diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q5km.md similarity index 79% rename from serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md rename to serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q5km.md index 2112efb..a7a7f02 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q5km.md @@ -1,10 +1,10 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)" -description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree." +title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)" +description: "The retired 2026-10-02/03 frank-secondary runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree." tags: - llm - - frank-small + - frank-secondary - dankclaw - llama-cpp - rtx3090 @@ -14,17 +14,17 @@ tags: timestamp: 2026-10-03 --- -# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired) +# frank-secondary — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired) > **RETIRED 2026-10-03** — superseded by -> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free +> [[frank-secondary-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free > bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same > f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower > decode is the measured price of the higher rung on this bandwidth-bound > card. This file remains the rollback reference and the owner of the > measured boot-pair table. -> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]] +> Its own history: one quant step down from [[frank-secondary-llama-mimo-q6k]] > (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then > the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two > conversations of 200,704 tokens with the KV VALUES unquantized** — @@ -36,11 +36,11 @@ timestamp: 2026-10-03 |Field|Value| |---|---| -|Endpoint|`192.168.0.143:7771`, served alias `frank-small`| +|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`| |Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap| -|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`| -|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`| -|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`| +|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-secondary.sh`| +|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`| +|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`| ## Shape and the KV contract @@ -128,7 +128,7 @@ The per-slot window is the whole price, in both directions: "Brown.", i.e. it generated from text alone. `/completion` takes `images` but refuses a request without `messages` in this build, so chat is the only vision path. -- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops +- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-secondary.env` drops `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot refusal, never a silent fallback to blindness. @@ -145,15 +145,15 @@ traffic first. The bats seam fails if a draft flag ships unannounced. |Target|How| |---|---| -|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V| -|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)| +|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-secondary.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V| +|Q6_K, same runtime|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-secondary-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)| |Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window| -|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit| -|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`| +|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SECONDARY_MMPROJ` at any real projector file or drop the gate in the same commit| +|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-secondary-sglang-int8.sh`| ## When it misbehaves -- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1 +- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script exits 1 naming the missing file if the binary or GGUF moved. - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it). diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md similarity index 77% rename from serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md rename to serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md index e5f0840..d710239 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md @@ -1,10 +1,10 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)" -description: "The RETIRED frank-small runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap." +title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)" +description: "The RETIRED frank-secondary runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap." tags: - llm - - frank-small + - frank-secondary - dankclaw - llama-cpp - gguf @@ -12,10 +12,10 @@ tags: timestamp: 2026-10-03 --- -# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV +# frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV > **RETIRED 2026-10-05 (evening).** Superseded by -> [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B +> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B > RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail: > committed earlier the same day (77decb5..380aafa) was **never deployed** — > THIS MiMo shape was the live payload when Swift cut over; (2) the KV @@ -29,14 +29,14 @@ timestamp: 2026-10-03 | | | |---|---| | Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | -| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | +| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface | | Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) | -| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | -| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | +| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | +| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) | | Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) | | Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) | -| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy | -| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) | +| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy | +| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) | ## Shape and KV contract @@ -67,7 +67,7 @@ compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both were declined. Re-open only on evidence the Q6_K residual matters. **Pool prices on this card come from boots, never formulas** — the measured -boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV +boot-pair table lives in [[frank-secondary-llama-mimo-q5km]] (*Shape and KV contract*) and still prices every rollback shape. ## Compute-buffer geometry (2026-10-05) @@ -78,7 +78,7 @@ measured rather than assumed, because the predecessor shape recorded **"ubatch **That failure was a different shape, and it did not reproduce here.** The service itself is the bench (one card holds one server, so no scratch port): -the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in +the old rung was re-booted through a transient `FRANK_SECONDARY_ENV_FILE` drop-in and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used / 2,849 MiB free — which makes the ubatch the only variable in the comparison. @@ -99,7 +99,7 @@ Three readings: - **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung, and the pool's 12,288-token (384 MiB) ladder in - [[frank-small-llama-mimo-q5km]] is the other way to buy room back. + [[frank-secondary-llama-mimo-q5km]] is the other way to buy room back. **Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 % @@ -125,7 +125,7 @@ that is the pair the curve was measured at. ## Client contract -llama.cpp surface, live-verified on this model family: model id `frank-small`, +llama.cpp surface, live-verified on this model family: model id `frank-secondary`, native tools via `--jinja` (`tool_calls` in the response), thinking in **`reasoning_content`**, vision via `image_url` content parts. Two vision traps, both measured here: @@ -147,13 +147,13 @@ so the harness compacts before llama.cpp answers a longer prompt with a 400. |Target|How| |---|---| -|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk| +|Q5_K_M at the same shape|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk| |3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)| -|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`| +|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`| ## When it misbehaves -Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*): +Same decision tree as [[frank-secondary-llama-mimo-q5km]] (*When it misbehaves*): `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k.md similarity index 81% rename from serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md rename to serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k.md index f2cd2ef..20bd64b 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k.md @@ -1,10 +1,10 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)" -description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload." +title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)" +description: "The retired 2026-10-01 frank-secondary runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload." tags: - llm - - frank-small + - frank-secondary - dankclaw - llama-cpp - rtx3090 @@ -14,10 +14,10 @@ tags: timestamp: 2026-10-01 --- -# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090) +# frank-secondary — llama.cpp MiMo-V2.6 Q6_K (RTX 3090) > **RETIRED 2026-10-01 → 10-02** (one night): superseded by -> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung +> [[frank-secondary-llama-mimo-q5km]], the same runtime and shape one quant rung > down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback > target. > @@ -29,11 +29,11 @@ timestamp: 2026-10-01 |Field|Value| |---|---| -|Endpoint|`192.168.0.143:7771`, served alias `frank-small`| +|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`| |Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`| |Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`| -|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`| -|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`| +|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`| +|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`| ## Shape and the KV contract @@ -100,7 +100,7 @@ use the parent of the 2026-10-01 night swap commit). ## When it misbehaves -- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script +- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script exits 1 naming the missing file if the binary or GGUF moved. - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it). diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md similarity index 87% rename from serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md rename to serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md index 045d7e4..a0f2c73 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md @@ -1,10 +1,10 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV" -description: "The active frank-small runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic." +title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV" +description: "The active frank-secondary runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic." tags: - llm - - frank-small + - frank-secondary - dankclaw - llama-cpp - gguf @@ -15,7 +15,7 @@ tags: timestamp: 2026-10-06 --- -# frank-small — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV +# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV > **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A > 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit @@ -32,7 +32,7 @@ timestamp: 2026-10-06 > full context (the 262,144 draft window), not the 3-token proposal > depth. The 2–3 GiB floor is knowingly violated, as ordered. This > supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K -> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the +> shape ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the > Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never > deployed. @@ -41,14 +41,14 @@ timestamp: 2026-10-06 | | | |---|---| | Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | -| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | +| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface | | Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) | -| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | -| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | +| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | +| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) | | Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) | | Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) | -| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) | -| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) | +| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) | +| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) | ## Why this payload (2026-10-06 evening) @@ -89,7 +89,7 @@ context (262,144 — the draft window, not the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free` field) — the tightest fit in -frank-small's history, the 2–3 GiB floor knowingly violated, and the +frank-secondary's history, the 2–3 GiB floor knowingly violated, and the 512×512 vision transient (34 MiB) still fits and recovers. Compute buffers allocate at graph capture during BOOT, so a miss is loud and fast — the failure mode is a crash loop, not silent corruption. @@ -225,7 +225,7 @@ recipes: `/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above. 2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert - [[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or + [[frank-secondary-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or 380aafa's parent), one `MODEL_FILE` + shape edit. 3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap: `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native @@ -243,10 +243,10 @@ recipes: - Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label-committed both sets at Frank's direction — 1919ad7 (frank-primary radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs). - Deploy ran `deploy.sh dankclaw --service frank-small`; the service + Deploy ran `deploy.sh dankclaw --service frank-secondary`; the service dispatch and health gate went green; the first e2e run failed ONE spec (12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, - unrelated to frank-small), green on rerun. + unrelated to frank-secondary), green on rerun. - The unit's `Description=` (MiMo era) was stale at cutover time and was updated in-repo afterward — the relinked unit picks it up at the next deploy/restart, not retroactively. diff --git a/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md b/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md new file mode 100644 index 0000000..7c72457 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md @@ -0,0 +1,15 @@ +--- +type: reference +title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S" +description: "The RETIRED frank-secondary runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves." +tags: + - llm + - frank-secondary + - dankclaw + - llama.cpp + - rtx3090 + - qwen35 + - mtp + - vision +timestamp: 2026-09-16 +--- diff --git a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md b/serverAppConfigs-docs/LLM/frank-secondary-sglang-mimo-v26.md similarity index 83% rename from serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md rename to serverAppConfigs-docs/LLM/frank-secondary-sglang-mimo-v26.md index 5854678..a95ff6d 100644 --- a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-sglang-mimo-v26.md @@ -1,10 +1,10 @@ --- type: reference -title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)" -description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night." +title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)" +description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night." tags: - llm - - frank-small + - frank-secondary - dankclaw - sglang - rtx3090 @@ -15,10 +15,10 @@ tags: timestamp: 2026-10-01 --- -# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP) +# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP) > **RETIRED 2026-10-01, hours after cutover.** Replaced by -> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF, +> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF, > chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. > The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8 > payload was staged to @@ -28,13 +28,13 @@ timestamp: 2026-10-01 ## Identity -- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw). +- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw). - **Runtime:** SGLang (latest) in a self-contained python3.12 venv at `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130, transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root - systemd unit `scripts/frank-small.service` (`User=dank`, - `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with - tunables from `scripts/frank-small.env`. + systemd unit `scripts/frank-secondary.service` (`User=dank`, + `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with + tunables from `scripts/frank-secondary.env`. - **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric). @@ -55,7 +55,7 @@ timestamp: 2026-10-01 the two-way concurrency contract. **That compaction arithmetic is specific to a 262,144-token slot**: the ACTIVE shape runs three slots of 180,224, which 183,501 tokens already exceeds — see *What three slots - cost* in [[frank-small-llama-mimo-q5km]]. + cost* in [[frank-secondary-llama-mimo-q5km]]. - **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256 dim x 1 B), pool 9.31 GiB. - **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet @@ -126,15 +126,15 @@ change lands. Eight safetensors shards (12,891,718,440 B total) + tokenizer and config files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`, -all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh` +all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh` (`--verify-only` re-checks) — the script moved to `old/` when the endpoint cut over to llama.cpp + GGUF on 2026-10-01 night, and -`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The +`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The serve script refuses to start with a missing shard. ## Client contract -- **Alias:** `frank-small` (the `id` at `/v1/models`). +- **Alias:** `frank-secondary` (the `id` at `/v1/models`). - **Reasoning:** extracted to **`reasoning_content`** (live-verified 2026-10-01 — this SGLang build's qwen3 parser uses the same field name as llama.cpp; models.yml sets `reasoningContentField: @@ -166,8 +166,8 @@ serve script refuses to start with a missing shard. ## Rollback -[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the -practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at +[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the +practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at `/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp Gemma4-12B QAT payload, was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on @@ -177,4 +177,4 @@ Gemma4-12B QAT payload, was staged to - Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound - SGLang: https://github.com/sgl-project/sglang -- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env` +- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env` diff --git a/serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md b/serverAppConfigs-docs/LLM/frank-secondary-vllm-gemma4.md similarity index 82% rename from serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md rename to serverAppConfigs-docs/LLM/frank-secondary-vllm-gemma4.md index 11a1f10..aaae3d6 100644 --- a/serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-vllm-gemma4.md @@ -1,10 +1,10 @@ --- type: reference -title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors) -description: "The RETIRED frank-small runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-small-gemma4-12b]] on the same day." +title: frank-secondary on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors) +description: "The RETIRED frank-secondary runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-secondary-gemma4-12b]] on the same day." tags: - llm - - frank-small + - frank-secondary - dankclaw - vllm - rtx3090 @@ -15,12 +15,12 @@ tags: timestamp: 2026-09-19 --- -# frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16) +# frank-secondary — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16) > **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut -> back to [[frank-small-gemma4-12b]] (llama.cpp, now the ACTIVE runbook) +> back to [[frank-secondary-gemma4-12b]] (llama.cpp, now the ACTIVE runbook) > the same day it landed — this page is the rollback target. Same port -> (7771), same alias (`frank-small`), same client contract surface — the +> (7771), same alias (`frank-secondary`), same client contract surface — the > runtime, the model payload, the drafter and the service shape (root > systemd unit → rootless Quadlet) all changed. @@ -28,12 +28,12 @@ timestamp: 2026-09-19 | what | value | |---|---| -| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d7…`) | -| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (root systemd unit retired 2026-09-19) | +| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-secondary` (`8dad46d7…`) | +| unit | **rootless Quadlet** `dankclaw/frank-secondary/quadlet/frank-secondary.container` (root systemd unit retired 2026-09-19) | | model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) | | model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` | | drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` | -| port | 7771 → 8000 (alias `frank-small`; `frank-large` keeps 7770) | +| port | 7771 → 8000 (alias `frank-secondary`; `frank-large` keeps 7770) | | window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** | | multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) | @@ -113,7 +113,7 @@ trade for the native w4a16 format + audio + CUDA graphs.) ## The image -`dankclaw/frank-small/Dockerfile` (repo) — 16 comment lines + `FROM +`dankclaw/frank-secondary/Dockerfile` (repo) — 16 comment lines + `FROM localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone: @@ -144,9 +144,9 @@ not a hardware mystery. ## Deployed config -Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh` +Repo truth: `scripts/frank-secondary.env` + `scripts/serve-frank-secondary.sh` (mounted into the container at `/scripts/`, which is the entrypoint) + -`dankclaw/frank-small/quadlet/frank-small.container`. Key values: +`dankclaw/frank-secondary/quadlet/frank-secondary.container`. Key values: `KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by compressed-tensors checkpoints and fp8 is hardware-dead on sm_86), `ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via @@ -186,27 +186,27 @@ container while it loads. - **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH` must be absent. 8.6 is correct. - **Audio 400/500** — the container is running the *base* image, not the - derived `:frank-small` (missing `av`). Check the deployed image digest + derived `:frank-secondary` (missing `av`). Check the deployed image digest is `8dad46d7…`. - **CUDA not visible in container** — a `Device=` line or a lib mount dropped out of the unit after an edit; the unit is the source of truth, the host-side generated unit is not. -- Logs: `journalctl --user -u frank-small` (rootless Quadlet; `podman +- Logs: `journalctl --user -u frank-secondary` (rootless Quadlet; `podman logs` covers only the current `--replace --rm` instance). ## Rollback -[[frank-small-gemma4-12b]] — the llama.cpp deployment. Its weights, +[[frank-secondary-gemma4-12b]] — the llama.cpp deployment. Its weights, mmproj and MTP sidecar are still on the host under `/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at -`scripts/.AGENTTRASH/frank-small.service.llama.cpp`. Restoring it means +`scripts/.AGENTTRASH/frank-secondary.service.llama.cpp`. Restoring it means reinstalling that unit and stopping the Quadlet (one owner per port). ## References -- `scripts/serve-frank-small.sh` — the actual exec line -- `dankclaw/frank-small/Dockerfile` — the audio-derived image -- `dankclaw/frank-small/quadlet/frank-small.container` — the unit -- [[frank-small-gemma4-12b]] — RETIRED predecessor (rollback) +- `scripts/serve-frank-secondary.sh` — the actual exec line +- `dankclaw/frank-secondary/Dockerfile` — the audio-derived image +- `dankclaw/frank-secondary/quadlet/frank-secondary.container` — the unit +- [[frank-secondary-gemma4-12b]] — RETIRED predecessor (rollback) - [[halogen-flash-server]] — the sibling endpoint on the iGPU - [[VLLM Overview]] / [[VLLM Index]] — runtime background diff --git a/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md b/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md deleted file mode 100644 index 758ace7..0000000 --- a/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -type: reference -title: "frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S" -description: "The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves." -tags: - - llm - - frank-small - - dankclaw - - llama.cpp - - rtx3090 - - qwen35 - - mtp - - vision -timestamp: 2026-09-16 ---- diff --git a/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md b/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md index b89b775..7cb4d04 100644 --- a/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md +++ b/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md @@ -6,7 +6,7 @@ tags: - freetoken - benchmark - dankserver - - frank-small + - frank-secondary - qwen3.6 timestamp: 2026-09-02 --- diff --git a/serverAppConfigs-docs/LLM/freetoken-frank-small.md b/serverAppConfigs-docs/LLM/freetoken-frank-small.md index 8f32d70..892ee2b 100644 --- a/serverAppConfigs-docs/LLM/freetoken-frank-small.md +++ b/serverAppConfigs-docs/LLM/freetoken-frank-small.md @@ -1,27 +1,27 @@ --- type: configuration -title: FreeToken frank-small Runbook -description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090 era). +title: FreeToken frank-secondary Runbook +description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-secondary on Dankserver (RTX 3090 era). tags: - freetoken - dankserver - - frank-small + - frank-secondary - qwen3.6 - moe - fp8 timestamp: 2026-08-30 --- -# FreeToken frank-small Runbook +# FreeToken frank-secondary Runbook > **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor > [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI > PRO R9700) took the :7770 endpoint and the port. This page is kept as the > retirement record of the FreeToken/CUDA era; the unit was archived to -> `~/.AGENTTRASH/20260915-frank-small-retire/host-units/` and the repo files +> `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/` and the repo files > to `scripts/.AGENTTRASH/`. -The `frank-small` service on Dankserver is served by +The `frank-secondary` service on Dankserver is served by **FreeToken v0.1.2** — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache. Model history on this runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) → @@ -134,8 +134,8 @@ modelopt block-fp8 remains the only FP8 path that loads. ## Sampling -**frank-small samples by default, and our config never said so.** -`scripts/serve-frank-small.sh` passes no sampling arguments at all — only +**frank-secondary samples by default, and our config never said so.** +`scripts/serve-frank-secondary.sh` passes no sampling arguments at all — only `--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`, `--num-tokens` and `--max-running-requests`. The behaviour comes from the runtime's own default: @@ -160,7 +160,7 @@ would be byte-identical. ### This is the opposite of frank-large -| | frank-small (FreeToken) | frank-large (halogen) | +| | frank-secondary (FreeToken) | frank-large (halogen) | |---|---|---| | Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults | | Deterministic across runs | No | Yes | @@ -168,7 +168,7 @@ would be byte-identical. A client that talks to both endpoints and sends no sampler gets non-deterministic answers from :7770 on Dankserver and reproducible ones from -:7770 on Dankclaw. If a workflow needs determinism from frank-small it must +:7770 on Dankclaw. If a workflow needs determinism from frank-secondary it must send `temperature: 0` explicitly, or the service must be relaunched with `--sampling-defaults none` (framework defaults only). @@ -188,9 +188,9 @@ send `temperature: 0` explicitly, or the service must be relaunched with | File | Purpose | |------|---------| -| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) | -| `scripts/frank-small.env` | Tunables (model path, pool sizes) | -| `scripts/frank-small.service` | Systemd unit | +| `scripts/serve-frank-secondary.sh` | Serve script (launches `ft serve`) | +| `scripts/frank-secondary.env` | Tunables (model path, pool sizes) | +| `scripts/frank-secondary.service` | Systemd unit | ## Limitations @@ -209,7 +209,7 @@ send `temperature: 0` explicitly, or the service must be relaunched with - **`model` field is required** in `/v1/chat/completions` bodies — FreeToken rejects requests without it with a 422 (`Field required`), unlike llama.cpp's server which defaults to the loaded model. Send - `"model": "frank-small"`. + `"model": "frank-secondary"`. - Load on this swap took well under the old llama.cpp numbers: weights → experts (parallel build) → CUDA graph capture → Uvicorn in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index 8d7f374..4fd3fab 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -23,7 +23,7 @@ timestamp: 2026-09-13 > shape the fleet asked for: two 200k-token conversations rather than the > native-context pair. Deployed as a rootless Quadlet container, > `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090 -> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]]. +> it does not touch now belongs to [[frank-secondary-qwen38-gsq-rco]]. The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs. @@ -98,9 +98,9 @@ upstream's defaults: | `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` | | `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column | | `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well | -| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces | +| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-secondary, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces | | `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) | -| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match | +| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -374,7 +374,7 @@ tables above. **None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset: the tower is a 1.97 GiB charge against the same device budget as the pool and -the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]], +the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]], which loads the BF16 projector alongside its own weights. This server refuses image content parts and says why, rather than ignoring them: diff --git a/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md b/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md index 0f1dd4f..4cf702b 100644 --- a/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md +++ b/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md @@ -1,7 +1,7 @@ --- type: reference title: ik_llama.cpp on Dankserver (Ornith Q8_0) -description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. +description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-secondary on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. tags: - ikllama - llamacpp @@ -16,7 +16,7 @@ timestamp: 2026-08-26 # ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK) -> **Active 2026-08-26.** `frank-small` switched from upstream +> **Active 2026-08-26.** `frank-secondary` switched from upstream > llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the > RTX 3090. This page documents the build, the mandatory FA build flags, and > the MoE / CPU-offload runtime arguments. @@ -101,8 +101,8 @@ a hard compile error. ## Runtime configuration -Files: `scripts/serve-frank-small.sh`, `scripts/frank-small.env`, -`scripts/frank-small.service` in the repo. The service just calls +Files: `scripts/serve-frank-secondary.sh`, `scripts/frank-secondary.env`, +`scripts/frank-secondary.service` in the repo. The service just calls the serve script; the env file holds all tunables. ### KV cache: K default f16 / V q6_0 + Hadamard (V only) @@ -241,7 +241,7 @@ apply. The previous runtime `upstream llama.cpp b10121` and its binary (`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To rollback to the old stack, change `LLAMA_SERVER=` in -`scripts/frank-small.env` back to that path and +`scripts/frank-secondary.env` back to that path and `CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram spec / `--spec-type` flags. The model file is not changed either way. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 51153f3..cb44d34 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -16,7 +16,7 @@ digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine, MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192 ctx, vision ON)** and -**llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B +**llama.cpp CUDA (frank-secondary, port 7771: ajgazin Swift-1.5-Qwen3.8-27B Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's 128k cap (the tightest fit in the endpoint's history — 489 MiB free), @@ -51,7 +51,7 @@ frank-large, port 7770] F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower 2 x 200192 ctx, MTP always-on, image input] G[llama.cpp master CUDA sm_86 -frank-small, port 7771] +frank-secondary, port 7771] H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF 2 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision] E2 --> E --> F @@ -103,7 +103,7 @@ frank-small, port 7771] (`slow`/`advisor` → frank-primary) carried forward. Superseded by the MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the engine moved with the model -- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since +- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since 2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since 2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP- @@ -120,7 +120,7 @@ frank-small, port 7771] 98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed -- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening; +- [[frank-secondary-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening; rollback target; payload on disk)**: llama.cpp master (CUDA sm_86) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of 200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for @@ -129,7 +129,7 @@ frank-small, port 7771] (+4.6 % prefill, measured), 2,279 MiB of the card left free. This was the LIVE payload when Swift cut over. The ubatch A/B table and the probe-size trap (>=256 px images or the model says "video") live inside -- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback +- [[frank-secondary-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback target; payload on disk; owns the measured boot-pair table)**: the same runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower — 104.8 tok/s, 3,571 MiB true free @@ -265,10 +265,10 @@ frank-small, port 7771] kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it replaced - [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to - frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp + frank-large: the FreeToken v0.1.2 frank-secondary runbook (batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The NAME was reused on 2026-09-16 by a different service on Dankclaw — see - [[frank-small-qwen38-gsq-rco]] + [[frank-secondary-qwen38-gsq-rco]] - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers @@ -337,35 +337,35 @@ frank-small, port 7771] .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting (2026-09-16 → 09-19) -- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same +- [[frank-secondary-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape — - [[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86 + [[frank-secondary-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no vision), reasoning_content, no spec decode; payload still at `/home/dank/models/MiMo-Q6K/` -- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced - by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot +- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced + by [[frank-secondary-llama-mimo-q6k]] for the asymmetric KV SGLang cannot express; venv still on disk, INT8 payload staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8 W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision, 83.9 tok/s decode -- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by - [[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged +- [[frank-secondary-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by + [[frank-secondary-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX 3090, with the separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. -- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, +- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- token pool, 2 concurrent full-context sessions, native text+image+audio. -- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the - prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab +- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the + prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and f16 KV at 131k - [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and @@ -396,9 +396,9 @@ frank-small, port 7771] - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on - Dankserver (frank-small, retired 2026-09-02) + Dankserver (frank-secondary, retired 2026-09-02) - [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for - frank-small (2026-08-26): q8_0 + Hadamard KV cache, + frank-secondary (2026-08-26): q8_0 + Hadamard KV cache, split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags - [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01 after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27) @@ -408,7 +408,7 @@ frank-small, port 7771] (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the 2x R9700 box (frank-primary) - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on - the flash-server v2 runtime since 2026-10-06, frank-small on the RTX 3090) + the flash-server v2 runtime since 2026-10-06, frank-secondary on the RTX 3090) - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where @@ -432,7 +432,7 @@ frank-small, port 7771] - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware - notes and sizing (this architecture family is what the retired frank-small + notes and sizing (this architecture family is what the retired frank-secondary served in the Huihui abliterated FP8 variant — see [[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint that preceded it is archived at [[dealign-ornith-crack]]) diff --git a/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md b/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md index 4bc52ec..2bd9e74 100644 --- a/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md +++ b/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md @@ -204,4 +204,4 @@ clipping at depth) starts with sections 1–3 here before touching anything. - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs - [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history -- [[frank-large-benchmark]] and [[frank-small-benchmark]] — probe/ladder hygiene on live hosts +- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — probe/ladder hygiene on live hosts diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 16d05fb..1c147ef 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -249,13 +249,13 @@ distinguishes a real split from a mislabelled sweep. The fleet's answer is unchanged — **two independent endpoints** ([[halogen-flash-server]] on the iGPU via halogen, -[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two +[[frank-secondary-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two claims from the first round of this experiment are now corrected: 1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23 tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode), and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint - (`frank-small` holds 22.4 of the card's 24.6 GiB). An operator who wants the + (`frank-secondary` holds 22.4 of the card's 24.6 GiB). An operator who wants the 7 % must give up the fleet's only image-capable endpoint. 2. **Per-tensor placement across two devices is impossible while `-ot` is used**, for a scheduler-mechanical reason, regardless of driver or vendor mix. @@ -309,8 +309,8 @@ Operational notes for the next run: `GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this host's co-tenants. -- `systemctl mask` is **refused** for a relinked root unit: frank-small's - `/etc/systemd/system/frank-small.service` is a symlink into the repo, so +- `systemctl mask` is **refused** for a relinked root unit: frank-secondary's + `/etc/systemd/system/frank-secondary.service` is a symlink into the repo, so `mask` exits with "File ... already exists". Stop it and rely on the absence of a restart trigger, or move the symlink aside. - Restarting frank-large needs the quadlet source back in diff --git a/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md b/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md index c0af835..72f7609 100644 --- a/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md +++ b/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md @@ -217,7 +217,7 @@ done ``` Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw -= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix += AI stack + `frank-large` :7770 + `frank-secondary` :7771 (RTX 3090 + Strix Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary` :7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so diff --git a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md index f46533d..d840a29 100644 --- a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md +++ b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md @@ -12,7 +12,7 @@ timestamp: 2026-10-05T00:00:00-07:00 Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the service was not crashed. Applies to `frank-medium` (dankserver, root unit), -`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless +`frank-secondary` (dankclaw, root unit), `frank-large` (dankclaw, rootless Quadlet). **1. Classify the stop before touching anything** @@ -103,7 +103,7 @@ the sudo journal names who and when: `journalctl -b | grep sudo`. ## Scratch vLLM boot on a fleet GPU host — stop-first law Any scratch/experimental vLLM container on a host that also runs a production -LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the +LLM service (`frank-primary`, `frank-large`, `frank-secondary`) MUST stop the production unit first, and the window script must verify it. Two independent failure modes if skipped: diff --git a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md index 019040d..0e6dc0a 100644 --- a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md +++ b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md @@ -64,7 +64,7 @@ done Then **read every hit**; do not count them. Most are benign (comments, retired fetch-script destinations, `retire_*_quadlet` helpers that move a quadlet not a model) — but on Dankclaw this check found a live bug: -`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a +`serve-frank-secondary.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a payload retired a week earlier. The env file always overrode it, so it was invisible; a missing env file would have produced a "missing asset" naming a model that no longer exists. @@ -99,7 +99,7 @@ active). ## 5. Verify, then report — and do not purge ```sh -systemctl is-active frank-large frank-small +systemctl is-active frank-large frank-secondary curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health ``` diff --git a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md index 32ccc06..72d9aa4 100644 --- a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md +++ b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md @@ -191,7 +191,7 @@ Ports 8880–8883 avoid the fleet's LLM ports (7770/7771). licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT / Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set. - **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the - TTS engine shares the card with frank-small; move to the R9700s only if TTS + TTS engine shares the card with frank-secondary; move to the R9700s only if TTS throughput demand grows. - **Guard the endpoint like any other service:** if vhosted, it goes behind the Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the diff --git a/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md b/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md index 08b1054..07f9f86 100644 --- a/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md +++ b/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md @@ -107,7 +107,7 @@ TTS), [Kokoro-82M](https://github.com/hexgrad/kokoro) (8,933, Apache-2.0, TTS), the `Comfy-Org` org. Already running on this fleet (Qwen-Image 2.1 GGUF) and wired as the Open WebUI image engine. - **llama.cpp** — the reference GGUF inference engine; `llama-server` exposes an - OpenAI-compatible endpoint. Backbone of this fleet's frank-small/frank-large. + OpenAI-compatible endpoint. Backbone of this fleet's frank-secondary/frank-large. - **vLLM** — high-throughput PagedAttention serving with an OpenAI-compatible API; the multi-GPU option (frank-medium runs vLLM tensor-parallel across two R9700s). diff --git a/serverAppConfigs-docs/Research/Service Research Brief.md b/serverAppConfigs-docs/Research/Service Research Brief.md index 3d09d4b..3eb7cc9 100644 --- a/serverAppConfigs-docs/Research/Service Research Brief.md +++ b/serverAppConfigs-docs/Research/Service Research Brief.md @@ -1,6 +1,6 @@ --- type: reference -description: "Consolidated research from the `frank-small` research subagents (2026-08-14) plus the pi agent knowledge base." +description: "Consolidated research from the `frank-secondary` research subagents (2026-08-14) plus the pi agent knowledge base." tags: - research timestamp: 2026-09-03 @@ -8,7 +8,7 @@ timestamp: 2026-09-03 # Service Research Brief -Consolidated research from the `frank-small` research subagents (2026-08-14) plus the pi agent knowledge base. This is the source material behind the per-service docs pages. +Consolidated research from the `frank-secondary` research subagents (2026-08-14) plus the pi agent knowledge base. This is the source material behind the per-service docs pages. ## Method diff --git a/serverAppConfigs-docs/Research/popular-self-hosted-ai-apps.html b/serverAppConfigs-docs/Research/popular-self-hosted-ai-apps.html index 53ad7f6..4ffc914 100644 --- a/serverAppConfigs-docs/Research/popular-self-hosted-ai-apps.html +++ b/serverAppConfigs-docs/Research/popular-self-hosted-ai-apps.html @@ -227,7 +227,7 @@

llama.cpp EXCELLENT

129,149★ · MIT · LLM runtime
-

The reference GGUF inference engine; llama-server exposes an OpenAI-compatible endpoint. Backbone of this fleet's frank-small/frank-large endpoints. Runs on CPU, CUDA, ROCm, Vulkan, Metal.

+

The reference GGUF inference engine; llama-server exposes an OpenAI-compatible endpoint. Backbone of this fleet's frank-secondary/frank-large endpoints. Runs on CPU, CUDA, ROCm, Vulkan, Metal.

diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 4368355..84ca16a 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -372,15 +372,15 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE - frank-small runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS +- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE + frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS with MTP back on since 2026-10-06 evening**): llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (the MTP head in the file is drafted) on the RTX 3090 — TWO 131,072-token conversations at Frank's 128k cap, the tightest fit in the endpoint's history (489 MiB measured free), q8_0 K + q8_0 V, vision through the publisher's own projector, ~63 t/s recipe decode with MTP -- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening, +- [[frank-secondary-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening, superseded by swift15): llama.cpp master serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at 200,704 tokens each with fully unquantized f16 KV and mmproj vision; @@ -388,32 +388,32 @@ The LLM inference fleet on the two hosts. Full index: weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny images answer "video") -- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback +- [[frank-secondary-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback target, payload on disk; owns the measured boot-pair table): same runtime and shape one rung lower — 104.8 tok/s, 3,571 MiB true free -- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same +- [[frank-secondary-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see - [[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape + [[frank-secondary-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured -- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target): +- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target): SGLang in a self-contained python3.12 venv serving letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool, 2 x 262,144 slots, bf16 mamba states, eager prefill -- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target): +- [[frank-secondary-gemma4-12b]] — RETIRED 2026-10-01 (rollback target): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (MTP assistant sidecar + ngram-mod, q8_0 KV) on Dankclaw's RTX 3090, port 7771 -- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, +- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- token pool, 2 concurrent full-context sessions, native text+image+audio -- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): - the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab +- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): + the prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector -- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) +- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-secondary (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) - [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21) @@ -433,7 +433,7 @@ The LLM inference fleet on the two hosts. Full index: - [[frank-large-benchmark]] — backend/spec benchmark behind the old head - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results -- [[frank-small-benchmark]] — the small-model benchmark suite +- [[frank-secondary-benchmark]] — the small-model benchmark suite - [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the logical batch buys nothing measurable, decode never moves, and the ubatch's diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 3d3d5c4..45e6300 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2,7 +2,7 @@ ## 2026-10-06 -- **frank-small (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).** +- **frank-secondary (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).** Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb): 22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's @@ -19,7 +19,7 @@ uncensored model thinks more — 64 tokens is all `reasoning_content`). ngram-mod still skipped (recurrent-state rollback law on the qwen35 hybrid). Runbook renamed + rewritten: - `serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`. + `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md`. - **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server 0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the @@ -120,12 +120,12 @@ Live pass of the frankbook walk itself is deferred to the next deploy with an awake host. - **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had - left `frank_small_unit.bats` and the deployer's payload gate stale** — + left `frank_secondary_unit.bats` and the deployer's payload gate stale** — the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated `LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer loads (the anti-drift seam doing its job). Both repointed to the Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests). -- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win +- **frank-secondary: compute-buffer pair doubled to 4096/2048, and the prefill win measured rather than assumed.** Frank asked for the batch and ubatch doubled. The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048 failed the scratch boot"**, and this shape carries 616 MiB less free space @@ -135,7 +135,7 @@ `FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03 boot **byte-identically** (21,276 used / 2,849 free), which is what makes ubatch the only variable. Repo never saw a fake revert commit; scaffolding - staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`. + staged to `~/.AGENTTRASH/20261005-frank-secondary-ubatch-ab/`. - **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %, for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 — noise). The control that proves it is geometry and not drift: a 279-token @@ -158,7 +158,7 @@ edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a second burst and ~90 s). Docs: - [[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*). + [[frank-secondary-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*). ## 2026-10-03 - **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term @@ -504,8 +504,8 @@ **sparse** file, 3.3 G apparent but 4 K real, so removing it frees nothing. - **A live reference check came first, and it earned its keep.** - `serve-frank-small.sh`'s *fallback* `MODEL_DIR` still pointed at the - retired GSQ-RCO payload frank-small served on 2026-09-16/17. The env + `serve-frank-secondary.sh`'s *fallback* `MODEL_DIR` still pointed at the + retired GSQ-RCO payload frank-secondary served on 2026-09-16/17. The env file has always overridden it, so the drift was invisible — but a missing or unreadable env file would have turned a config problem into a "missing asset" naming a model that no longer exists. Repointed at @@ -519,7 +519,7 @@ - Pre-existing: **118 G of trash from 2026-09-21** was already staged (`20260921-frank-large-assembled` 88 G, `-redownload` 30 G) and never purged. Total now staged on this host: **627 G**. - - Verified after the moves: frank-large and frank-small both `active`, + - Verified after the moves: frank-large and frank-secondary both `active`, both `/health` 200, and all three live assets (Swift target, DFlash2 drafter, Gemma4 target) still present. - **frank-large payload swapped to Swift-1.5 (uncensored)** (production @@ -735,14 +735,14 @@ ## 2026-09-25 - `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy. -- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-small → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197). -- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-small = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified. +- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-secondary → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197). +- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-secondary = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified. - Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build). - `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e). ## 2026-09-22 -- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-small-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries. -- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B). +- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-secondary-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries. +- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-secondary (Gemma4-12B). - `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op. ## 2026-09-21 @@ -757,7 +757,7 @@ - `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape. - `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint. - `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`). -- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused). +- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-secondary/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused). - `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54). - Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs). - `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped. @@ -766,7 +766,7 @@ - `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5). - `/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` — New runbook. - `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9). -- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c). +- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c). - Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless). - AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green. @@ -775,9 +775,9 @@ - Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption. ## 2026-09-17 -- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level. -- `/scripts/frank-small.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx. -- `/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` — New ACTIVE runbook; `frank-small-qwen38-gsq-rco.md` retired as rollback target. +- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level. +- `/scripts/frank-secondary.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx. +- `/serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md` — New ACTIVE runbook; `frank-secondary-qwen38-gsq-rco.md` retired as rollback target. - `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0). - `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876). - Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval. @@ -786,10 +786,10 @@ - `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff). - `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion. - `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap. -- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp. -- `/scripts/frank-small.env`, `/scripts/frank-small.service`, `/scripts/serve-frank-small.sh` — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771. +- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-secondary); ledger 95.4 GiB held / 14.7 GiB spare, no clamp. +- `/scripts/frank-secondary.env`, `/scripts/frank-secondary.service`, `/scripts/serve-frank-secondary.sh` — New frank-secondary: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771. - `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added. -- `/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired. +- `/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired. - `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only). - Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened. - Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note. @@ -828,7 +828,7 @@ - `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere). - `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission. - `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`). -- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion. +- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion. - `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/` → `/home/dank/Containers/`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap). - `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers//` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied). - `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page. @@ -840,7 +840,7 @@ - `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541). - `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement. - `/dankclaw/frank-large/quadlet/frank-large.container` — frank-large: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s. -- `/scripts/frank-small.env` — frank-small (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified. +- `/scripts/frank-secondary.env` — frank-secondary (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified. - `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed. - `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5). @@ -1076,8 +1076,8 @@ any repo service. ## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP - **Question that started it:** can a Qwen3.5 MTP drafter attach to - frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: - not on frank-small — SGLang reads the MTP head from the TARGET checkpoint + frank-secondary, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: + not on frank-secondary — SGLang reads the MTP head from the TARGET checkpoint and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 / TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and @@ -1147,12 +1147,12 @@ bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` fetchers moved to `old/`. Runbook [[frank-large-rocmfpx-q8agent-mtp]]. -## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots +## 2026-10-02 — frank-secondary: Q5_K_M, then three contexts + vision, sized by two boots Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp with two contexts ("whatever context size fits the 3090"), then **three contexts** "as much as can fit while keeping overhead open space to 2-3gb on -the 3090", then **"frank-small should have vision"**. +the 3090", then **"frank-secondary should have vision"**. - **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V under flash attention, `--mmproj` vision, all layers on the 3090. Measured @@ -1190,7 +1190,7 @@ the 3090", then **"frank-small should have vision"**. `~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`, `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the - gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]]. + gufo DFlash2 drafter). Runbook [[frank-secondary-llama-mimo-q5km]]. ## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts @@ -1260,7 +1260,7 @@ top of the proven-good base, root cause first. - **Harness contract followed the window:** `models.yml` `contextWindow` 262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <= 200,704). Side effect: frank-primary's compaction request now fits inside a - frank-small slot again, which it had stopped doing when frank-small went to + frank-secondary slot again, which it had stopped doing when frank-secondary went to 3 x 180,224. - **Comment-only follow-up** `295d578` recorded the measured numbers and was deployed so the per-service revision marker sits at HEAD; e2e 33/33 again. diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index 7411d87..dbe75f0 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -1,7 +1,7 @@ { "uid": "llm-fleet", "title": "LLM fleet \u2014 capacity and reuse", - "description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* across all three endpoints — frank-small (Dankclaw, RTX 3090), frank-large (Dankclaw, gfx1151 iGPU) and frank-primary (Dankserver, 2x R9700, llama.cpp again since the 2026-09-26 swap back from Lucebox DS4.1). The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.", + "description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* across all three endpoints — frank-secondary (Dankclaw, RTX 3090), frank-large (Dankclaw, gfx1151 iGPU) and frank-primary (Dankserver, 2x R9700, llama.cpp again since the 2026-09-26 swap back from Lucebox DS4.1). The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.", "tags": [ "llm", "capacity" @@ -166,7 +166,7 @@ "id": 5, "type": "timeseries", "title": "Decode steps and batch size", - "description": "Per-step counters: steps per second and the busy-slot count per step (the effective batch). With frank-primary's --parallel 1 the slot count stays at 0 or 1 and says nothing about the card; with frank-small's 4 slots it tracks the agent-fleet burst shape.", + "description": "Per-step counters: steps per second and the busy-slot count per step (the effective batch). With frank-primary's --parallel 1 the slot count stays at 0 or 1 and says nothing about the card; with frank-secondary's 4 slots it tracks the agent-fleet burst shape.", "gridPos": { "h": 8, "w": 12, diff --git a/shared/homepage/docs.html b/shared/homepage/docs.html index 455549d..f002382 100644 --- a/shared/homepage/docs.html +++ b/shared/homepage/docs.html @@ -36,8 +36,8 @@ Pick a model in the chat's model selector — the three models the fleet runs are described below. Open WebUI keeps your chat history; the raw model endpoints do not.

-

Images go to frank-primary or frank-small - (both see them). Audio (WAV) goes to frank-small only.

+

Images go to frank-primary or frank-secondary + (both see them). Audio (WAV) goes to frank-secondary only.

@@ -62,7 +62,7 @@
  • Best for: software engineering, system administration, and instruction tasks.
    • -

    frank-small — the fast utility model

    +

    frank-secondary — the fast utility model

    • Model: Gemma4-12B QAT (Balanced, Q4_K_M, 7.4 GiB).
    • Capacity: one million tokens of pooled context across four simultaneous conversations (up to 262k each); fastest at images.
      • @@ -71,7 +71,7 @@

    Which one to pick? Small utility work, RAG, search, and - screenshots: frank-small. Software engineering, sysadmin, and + screenshots: frank-secondary. Software engineering, sysadmin, and instruction tasks: frank-primary. Cybersecurity and debugging: frank-large (with Frank's permission).

    diff --git a/shared/homepage/services.json b/shared/homepage/services.json index c412927..3c19e03 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -298,7 +298,7 @@ "showLink": false }, { - "name": "frank-small", + "name": "frank-secondary", "category": "user", "section": "AI", "details": { diff --git a/shared/prometheus/prometheus.yml b/shared/prometheus/prometheus.yml index ba5d1ac..d1db406 100644 --- a/shared/prometheus/prometheus.yml +++ b/shared/prometheus/prometheus.yml @@ -32,7 +32,7 @@ scrape_configs: # bridge container, and on this netavark host a container can only reach the # host's LAN IP at a PUBLISHED port — node-exporter's net=host bind on # 192.168.0.143:9100 is refused by construction (the same rule that sends - # frank-large and frank-small to the gateway below). node-dankserver is a + # frank-large and frank-secondary to the gateway below). node-dankserver is a # different machine, so its LAN address works. - job_name: "node-dankclaw" static_configs: @@ -79,20 +79,20 @@ scrape_configs: # this container, not the guarded vhosts: these are machine clients and # Prometheus cannot solve an Anubis challenge. # - # One metric namespace: llamacpp:* — frank-large, frank-small and + # One metric namespace: llamacpp:* — frank-large, frank-secondary and # frank-primary, all llama.cpp with /metrics (the family verified from # /metrics on 2026-09-16 and 2026-09-20) and all exporting # requests_deferred (the queue depth, the only form llama.cpp shows the # capacity wall in), spec_decode_num_drafts_total, tokens_predicted_total, # and the prompt_tokens_cached_total / prompt_tokens_total reuse pair. # - # frank-large and frank-small are scraped over host.containers.internal, + # frank-large and frank-secondary are scraped over host.containers.internal, # not their own LAN addresses: this Prometheus is a bridge-networked # container (Network=dankclaw_default), and on this netavark host a # container can only reach the host's LAN IP at a PUBLISHED port — a # host-process bind like llama-server's 0.0.0.0:7770 is refused at # 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is - # why frank-small was red here for days (up{job="frank-small"} == 0, + # why frank-secondary was red here for days (up{job="frank-secondary"} == 0, # "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and # exporting: the target address was unreachable by construction, not the # service down. Same host, same rule, so both local endpoints use the @@ -105,12 +105,12 @@ scrape_configs: # plus radiance:* families, a different namespace — hence the separate # job below, not a re-add to the llamacpp pair. The netavark quirk above # is same-host only; plain LAN routing REACHES 192.168.0.235:7770. - - job_name: "frank-small" + - job_name: "frank-secondary" metrics_path: "/metrics" static_configs: - targets: ["host.containers.internal:7771"] labels: - service: frank-small + service: frank-secondary - job_name: "frank-large" metrics_path: "/metrics" diff --git a/shared/prometheus/rules/llm-fleet.yml b/shared/prometheus/rules/llm-fleet.yml index 7d167df..e48e679 100644 --- a/shared/prometheus/rules/llm-fleet.yml +++ b/shared/prometheus/rules/llm-fleet.yml @@ -1,6 +1,6 @@ # Alerting and recording rules for the LLM fleet. # -# One metric namespace: llamacpp:* — frank-small and frank-large, both +# One metric namespace: llamacpp:* — frank-secondary and frank-large, both # llama.cpp with /metrics (the family verified from /metrics on 2026-09-16 # and 2026-09-20). frank-primary LEFT the family 2026-10-05: FreeToken (the # PR587 ROCm engine) has no /metrics endpoint, so its scrape job was @@ -110,8 +110,8 @@ groups: annotations: summary: "{{ $labels.job }}: scrape target down for 2 minutes" description: >- - Prometheus cannot reach the LLM endpoint. frank-small and + Prometheus cannot reach the LLM endpoint. frank-secondary and frank-large are root units on Dankclaw: `systemctl is-active - frank-small` / `frank-large`. Both scraped endpoints export + frank-secondary` / `frank-large`. Both scraped endpoints export the llamacpp:* family. (frank-primary left the scrape set 2026-10-05 — FreeToken exports no /metrics.) diff --git a/tests/deployment_flow.bats b/tests/deployment_flow.bats index 5582197..afde086 100644 --- a/tests/deployment_flow.bats +++ b/tests/deployment_flow.bats @@ -70,7 +70,7 @@ load helpers @test "remote runner has a handler per inference service" { grep -q 'deploy_llm_large' "$REPO_ROOT/scripts/deploy-remote.sh" grep -q 'deploy_llm_primary' "$REPO_ROOT/scripts/deploy-remote.sh" - grep -q 'deploy_llm_small' "$REPO_ROOT/scripts/deploy-remote.sh" + grep -q 'deploy_llm_secondary' "$REPO_ROOT/scripts/deploy-remote.sh" } @test "inference restarts are absent from the top-level dispatcher" { diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index b8df710..9a39f49 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -31,7 +31,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "the retired ik chat template plans nothing, and its archive is still here" { # scripts/chat-template.jinja existed only for the ik_llama.cpp runtime # (retired 2026-09-16 in favour of halogen-flash-server again). The - # llama.cpp frank-small runtime serves the template the GGUF itself + # llama.cpp frank-secondary runtime serves the template the GGUF itself # carries, which is the one with reasoning_effort, preserve_thinking and # the tool-call grammar. Editing the archive must deploy nothing -- and the # archive must still be present, so this cannot pass by the file having @@ -84,15 +84,15 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" @test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" { run python3 "$PLANNER" --target dankclaw --paths \ - scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh + scripts/frank-secondary.env scripts/frank-secondary.service scripts/serve-frank-secondary.sh [ "$status" -eq 0 ] - [ "$output" = "frank-small" ] + [ "$output" = "frank-secondary" ] - # frank-small runs on Dankclaw on the RTX 3090; the Dankserver endpoint of + # frank-secondary runs on Dankclaw on the RTX 3090; the Dankserver endpoint of # that name was retired with the FreeToken runtime on 2026-09-15 and must # not come back through a planner mapping. run python3 "$PLANNER" --target dankserver --paths \ - scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh + scripts/frank-secondary.env scripts/frank-secondary.service scripts/serve-frank-secondary.sh [ "$status" -eq 0 ] [ -z "$output" ] } diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index 75004ef..47b7fc4 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -142,7 +142,7 @@ good_env() { # ~28 KiB/position (FLAGS.md). Against it: ~62 GiB resident weights, # the arena scratch 3.8 + 8.4*(MAX_TOK/16384) GiB, the vision tower # ~2 GiB pool charge, the 20 GiB host reserve and the host baseline - # + co-tenancy (frank-small, ComfyUI, Caddy/observability) ~15. The + # + co-tenancy (frank-secondary, ComfyUI, Caddy/observability) ~15. The # box has 121 GiB unified. The n-gram table rides as mmap'd page # cache beside the checkpoint and is not a pinned charge (the card's # own quickstart arithmetic does not bill it). @@ -320,8 +320,8 @@ $(cat "$TEST_TMP/serve.log")" grep -qE 'podman pull' "$DEPLOYER" } -@test "the stray sweep cannot take frank-small or the deployer down" { - # frank-small also listens on this host (7771, llama-server) and the +@test "the stray sweep cannot take frank-secondary or the deployer down" { + # frank-secondary also listens on this host (7771, llama-server) and the # deployer's own command line contains "frank-large"; only the FULL # container name matches a stray manual flash-server run and nothing # else. diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 3a6c79c..61a09ed 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -44,7 +44,7 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja" } @test "frank-primary unit is boot-persistent, self-healing and unprivileged" { - # Root unit parity with frank-large/frank-small: enable is the [Install] + # Root unit parity with frank-large/frank-secondary: enable is the [Install] # section, the restart policy survives a crash, and the pinned host RAM # (expert pools + PLE table + prefix host tier, ~132 GiB) needs an # unbounded memlock (a root unit grants it via PID1; the dank session's diff --git a/tests/frank_small_unit.bats b/tests/frank_secondary_unit.bats similarity index 87% rename from tests/frank_small_unit.bats rename to tests/frank_secondary_unit.bats index 5f2f67e..b813345 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_secondary_unit.bats @@ -1,5 +1,5 @@ -# Seam: the frank-small llama.cpp root unit on dankclaw. -# frank-small is the RTX 3090 endpoint; since 2026-10-06 (evening) it +# Seam: the frank-secondary llama.cpp root unit on dankclaw. +# frank-secondary is the RTX 3090 endpoint; since 2026-10-06 (evening) it # serves ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at # RCO-IQ3_XXS (27B dense qwen3_5 hybrid) via a CUDA (sm_86) build of # llama.cpp master: TWO 131,072-token conversations at Frank's 128k cap @@ -13,9 +13,9 @@ load helpers -UNIT="$REPO_ROOT/scripts/frank-small.service" -SERVE="$REPO_ROOT/scripts/serve-frank-small.sh" -ENV_FILE="$REPO_ROOT/scripts/frank-small.env" +UNIT="$REPO_ROOT/scripts/frank-secondary.service" +SERVE="$REPO_ROOT/scripts/serve-frank-secondary.sh" +ENV_FILE="$REPO_ROOT/scripts/frank-secondary.env" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" ENV_LARGE="$REPO_ROOT/scripts/frank-large.env" TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf" @@ -49,7 +49,7 @@ serve_argv() { LLAMA_BIN=$RUNTIME_DIR/bin/llama-server MODEL_FILE=$MODEL_DIR/$TARGET_STUB MMPROJ_FILE=$MODEL_DIR/${mmproj##*/} -ALIAS=frank-small +ALIAS=frank-secondary BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 CTX_SIZE=$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE") @@ -60,27 +60,27 @@ CACHE_TYPE_K=$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE") CACHE_TYPE_V=$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE") NGL=$(sed -nE 's/^NGL=([0-9]+)$/\1/p' "$ENV_FILE") EOF - run env FRANK_SMALL_ENV_FILE="$envf" nvidia-modprobe=true bash "$SERVE" + run env FRANK_SECONDARY_ENV_FILE="$envf" nvidia-modprobe=true bash "$SERVE" } argv_value() { grep -A1 -xF -- "$1" <<<"$output" | tail -1 } -@test "frank-small unit, serve script and env exist in scripts/" { +@test "frank-secondary unit, serve script and env exist in scripts/" { [ -f "$UNIT" ] [ -f "$SERVE" ] [ -f "$ENV_FILE" ] } -@test "frank-small is boot-persistent, self-healing and unprivileged" { +@test "frank-secondary is boot-persistent, self-healing and unprivileged" { grep -q '^WantedBy=multi-user.target$' "$UNIT" grep -q '^Restart=on-failure$' "$UNIT" grep -q '^RestartSec=' "$UNIT" grep -q '^User=dank$' "$UNIT" grep -q '^LimitMEMLOCK=infinity$' "$UNIT" - grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT" - grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT" + grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh$' "$UNIT" + grep -q 'FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env' "$UNIT" } @test "the shipped env points at the llama.cpp runtime and the Swift-1.5-27B Uncensored RCO-IQ3_XXS GGUF" { @@ -95,7 +95,7 @@ argv_value() { # would be gating on a file nobody reads — and a missing projector would # ship a blind endpoint. local env_v dep_v path pair - for pair in "MODEL_FILE LLM_SMALL_MODEL_FILE" "MMPROJ_FILE LLM_SMALL_MMPROJ"; do + for pair in "MODEL_FILE LLM_SECONDARY_MODEL_FILE" "MMPROJ_FILE LLM_SECONDARY_MMPROJ"; do read -r env_v dep_v <<<"$pair" path="$(sed -nE "s/^${env_v}=(.+)\$/\1/p" "$ENV_FILE" | tail -1)" [ -n "$path" ] @@ -106,7 +106,7 @@ argv_value() { @test "the pool is exactly the native 262144 window, so each of the two conversations is 131072" { serve_argv [ "$status" -eq 0 ] - [ "$(argv_value --alias)" = "frank-small" ] + [ "$(argv_value --alias)" = "frank-secondary" ] [ "$(argv_value --port)" = "7771" ] [ "$(argv_value --host)" = "0.0.0.0" ] [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ] @@ -212,7 +212,7 @@ argv_value() { # The documented opt-out must reach the engine as ABSENCE of the flag, # not as an argument pointing at an empty string. sed -i 's|^MMPROJ_FILE=.*|MMPROJ_FILE=|' "$STAGE/test.env" - run env FRANK_SMALL_ENV_FILE="$STAGE/test.env" nvidia-modprobe=true bash "$SERVE" + run env FRANK_SECONDARY_ENV_FILE="$STAGE/test.env" nvidia-modprobe=true bash "$SERVE" [ "$status" -eq 0 ] ! grep -qxF -- '--mmproj' <<<"$output" } @@ -232,27 +232,27 @@ argv_value() { rm -f "$MODEL_DIR/$MM_STUB" serve_argv [ "$status" -ne 0 ] - [[ "$output" == *"frank-small: missing"* ]] + [[ "$output" == *"frank-secondary: missing"* ]] rm -f "$RUNTIME_DIR/bin/llama-server" serve_argv [ "$status" -ne 0 ] - [[ "$output" == *"frank-small: missing"* ]] + [[ "$output" == *"frank-secondary: missing"* ]] } @test "the deploy handler gates on binary, GGUF and projector, then MainPID-hardens health" { - grep -q 'deploy_llm_small' "$REMOTE_DEPLOY" - grep -q 'LLM_SMALL_RUNTIME' "$REMOTE_DEPLOY" - grep -q 'LLM_SMALL_MODEL_FILE' "$REMOTE_DEPLOY" - grep -q 'LLM_SMALL_MMPROJ' "$REMOTE_DEPLOY" - grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY" - grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY" + grep -q 'deploy_llm_secondary' "$REMOTE_DEPLOY" + grep -q 'LLM_SECONDARY_RUNTIME' "$REMOTE_DEPLOY" + grep -q 'LLM_SECONDARY_MODEL_FILE' "$REMOTE_DEPLOY" + grep -q 'LLM_SECONDARY_MMPROJ' "$REMOTE_DEPLOY" + grep -q 'relink_unit frank-secondary' "$REMOTE_DEPLOY" + grep -q 'systemctl enable frank-secondary.service' "$REMOTE_DEPLOY" grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY" - grep -q 'frank-small.service has no MainPID' "$REMOTE_DEPLOY" + grep -q 'frank-secondary.service has no MainPID' "$REMOTE_DEPLOY" } @test "the two endpoints keep separate ports and the large handler never pkills llama-server" { - # frank-large's gufo sweep must not take down frank-small, whose + # frank-large's gufo sweep must not take down frank-secondary, whose # process is llama-server on the SAME host since this swap. grep -q '^LISTEN_PORT=7771$' "$ENV_FILE" local large diff --git a/tests/homepage_config.bats b/tests/homepage_config.bats index 2c3baeb..6cf3293 100644 --- a/tests/homepage_config.bats +++ b/tests/homepage_config.bats @@ -72,8 +72,8 @@ by_probe = {s["probe"]: s["name"] for s in services} primary = by_probe.get("192.168.0.235:7770") assert primary == "frank-primary", f"Dankserver :7770 card is named {primary!r}" # Dankclaw serves two endpoints on two ports: frank-large (unsloth llama.cpp -# on the iGPU, 7770) and frank-small (llama.cpp on the RTX 3090, 7771). +# on the iGPU, 7770) and frank-secondary (llama.cpp on the RTX 3090, 7771). assert by_probe.get("127.0.0.1:7770") == "frank-large", by_probe.get("127.0.0.1:7770") -assert by_probe.get("127.0.0.1:7771") == "frank-small", by_probe.get("127.0.0.1:7771") +assert by_probe.get("127.0.0.1:7771") == "frank-secondary", by_probe.get("127.0.0.1:7771") PY } diff --git a/tests/observability_quadlet.bats b/tests/observability_quadlet.bats index ad66936..4202b5d 100755 --- a/tests/observability_quadlet.bats +++ b/tests/observability_quadlet.bats @@ -325,15 +325,15 @@ PAIRS # container, and on this netavark host a container can only reach the # host's LAN IP at a PUBLISHED port — a host-process bind like # llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it - # answers at 169.254.1.2:7770 (why frank-small stayed red by construction + # answers at 169.254.1.2:7770 (why frank-secondary stayed red by construction # for days until the 2026-09-20 /targets audit). local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml" - grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; } + grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-secondary not scraped"; return 1; } grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; } ! grep -q '"192.168.0.235:7770"' "$cfg" grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \ && { echo "frank-primary folded into a Dankclaw gateway target"; return 1; } - for job in frank-small frank-large; do + for job in frank-secondary frank-large; do grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; } done ! grep -q 'job_name: "frank-primary"' "$cfg" @@ -383,11 +383,11 @@ for want in ('job:llm_prefix_cache_reuse:rate5m', # Every rule must be keyed to a series that actually exists. A rule naming a # series nobody exports evaluates nothing forever and errors never — the # failure this seam exists for. Since the 2026-09-26 cutover the only scraped -# LLM runtime is llama.cpp (frank-small, frank-large); the vllm: family left +# LLM runtime is llama.cpp (frank-secondary, frank-large); the vllm: family left # with frank-medium's radiance runtime and must not survive in any expr. exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']] # llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20 -# (frank-small, frank-large). Anything else is a guess. +# (frank-secondary, frank-large). Anything else is a guess. live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total', 'llamacpp:requests_deferred', 'llamacpp:requests_processing', 'llamacpp:spec_decode_num_drafts_total',