rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
This commit is contained in:
@@ -35,12 +35,12 @@ encodes):
|
||||
1. **Source trigger** = `thresholdPercent × source contextWindow`
|
||||
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
|
||||
2. **Output budget** = capped at the target model's `maxTokens`
|
||||
(32,768 for frank-small; 36,864 frank-primary; 57,344 frank-large — the
|
||||
(32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
|
||||
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
|
||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
||||
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
|
||||
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
|
||||
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-small).
|
||||
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
|
||||
|
||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||
every swap; the numbers above are the 2026-10-05 state):
|
||||
|
||||
@@ -63,7 +63,7 @@ service on the other host.
|
||||
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
||||
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
||||
| `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
|
||||
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
||||
|
||||
|
||||
@@ -95,7 +95,7 @@ host-side `workflows/` dir and the repo copy.
|
||||
## Recommended additions
|
||||
|
||||
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
|
||||
frank-small's). All repos verified 2026-10-05. None are on disk yet.
|
||||
frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
|
||||
|
||||
| # | What | Repo | Grab | Size | Why |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
@@ -189,7 +189,7 @@ What actually breaks, and the fix:
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small # names track the live config; diff first
|
||||
model: frank-small
|
||||
model: frank-secondary
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
|
||||
@@ -24,7 +24,7 @@ https://github.com/open-webui/open-webui
|
||||
| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
|
||||
| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-secondary` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
|
||||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||||
@@ -39,14 +39,14 @@ discovery — so the picker shows exactly the models seeded there.
|
||||
|--------------|----------------|---------|------|
|
||||
| `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw |
|
||||
| `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver |
|
||||
| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
|
||||
| `frank-secondary` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
|
||||
|
||||
Two gotchas make the wiring non-obvious:
|
||||
|
||||
- **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
|
||||
`{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
|
||||
OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
|
||||
the llama.cpp endpoints (frank-large, frank-small). The fix is to set
|
||||
the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
|
||||
`model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
|
||||
builds the model list directly and never fetches `/v1/models`. frank-medium
|
||||
(vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
|
||||
@@ -218,13 +218,13 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||||
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
|
||||
- The default admin account is created on first login; secure it immediately.
|
||||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
||||
- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
|
||||
- The on-host LLM services (frank-large `:7770`, frank-secondary `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
|
||||
|
||||
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
|
||||
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
|
||||
|
||||
- **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
|
||||
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
|
||||
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
|
||||
remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
|
||||
`frank-small`) moved from admin-only to the **user** page with live model
|
||||
`frank-secondary`) moved from admin-only to the **user** page with live model
|
||||
descriptions and a `docs` link to `/docs#models`.
|
||||
|
||||
**Sections + WIP split (2026-09-21):** both pages now group their panels
|
||||
@@ -111,7 +111,7 @@ flowchart TD
|
||||
## Service classification
|
||||
|
||||
**User services** (root page), in section order:
|
||||
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-small — status + docs link, no Open link).
|
||||
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-secondary — status + docs link, no Open link).
|
||||
- **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
|
||||
- **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
|
||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
||||
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
|
||||
| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
|
||||
| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
|
||||
|
||||
**Three jobs read down, and they are not outages** (audited 2026-09-16):
|
||||
|
||||
|
||||
@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
|
||||
|
||||
@@ -315,7 +315,7 @@ not switch without re-measuring".
|
||||
|
||||
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
|
||||
slot split, quant rung — not just spec flags) is the transient systemd
|
||||
drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
|
||||
drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
|
||||
|
||||
## Is it worth it? (arithmetic before deployment)
|
||||
|
||||
@@ -419,4 +419,4 @@ measurement that would settle it.
|
||||
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
|
||||
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
|
||||
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
|
||||
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle
|
||||
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
|
||||
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
|
||||
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
|
||||
|
||||
The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]).
|
||||
The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
|
||||
|
||||
### ROCm
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ timestamp: 2026-09-20
|
||||
|
||||
Dankserver is a media/library server and the `frank-medium` LLM host. The
|
||||
host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
|
||||
FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with
|
||||
FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
|
||||
two AMD Radeon AI PRO R9700 cards.
|
||||
|
||||
## Hardware Specifications
|
||||
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
|
||||
The active inference service is `frank-medium.service`: mainline llama-server on
|
||||
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
|
||||
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
|
||||
`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
|
||||
its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
|
||||
`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
|
||||
its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
|
||||
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
|
||||
See [[deployment]].
|
||||
|
||||
|
||||
@@ -13,12 +13,12 @@ timestamp: 2026-09-03
|
||||
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
|
||||
> The `frank-small` name and port 52625 then belonged to the vLLM
|
||||
> The `frank-secondary` name and port 52625 then belonged to the vLLM
|
||||
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
|
||||
|
||||
# FastFlowLM NPU Inference
|
||||
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
|
||||
|
||||
## Verdict (2026-08-16)
|
||||
|
||||
@@ -203,16 +203,16 @@ Notes:
|
||||
|
||||
## Promotion (2026-08-16)
|
||||
|
||||
`frank-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
|
||||
### Boot policy (2026-08-17)
|
||||
|
||||
The `frank-small` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/frank-small.service
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable frank-small
|
||||
sudo systemctl enable frank-secondary
|
||||
```
|
||||
|
||||
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
|
||||
@@ -353,7 +353,7 @@ installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
|
||||
|
||||
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
|
||||
@@ -390,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
|
||||
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
|
||||
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
|
||||
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
|
||||
- **NPU production role (2026-08-16).** `frank-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
|
||||
## Citations
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ read the returned content:
|
||||
```python
|
||||
import json, urllib.request
|
||||
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
||||
json.dumps({"model": "frank-small", "messages": [{"role": "user",
|
||||
json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
|
||||
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
||||
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
||||
```
|
||||
|
||||
@@ -27,7 +27,7 @@ wedging a host until power cycle. Reference harnesses:
|
||||
with `container state improper`.
|
||||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||||
LLM hosts — one card, one server). The serve script takes its config via
|
||||
env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
pointing at a staged copy; the checkout is never mutated.
|
||||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||||
|
||||
@@ -80,7 +80,7 @@ co-tenants.
|
||||
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
|
||||
"move layers to the card" converts reclaimable cache into device memory and
|
||||
frees nothing for co-tenants. A served model's footprint ≠ weight size
|
||||
(frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
|
||||
(frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
|
||||
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
|
||||
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
|
||||
- **Context lengths: derive, never remember.** `llama-bench` computes
|
||||
@@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed
|
||||
## Related
|
||||
|
||||
- [[GGUF Compatibility]] — what a payload is before it is priced
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
|
||||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
|
||||
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
|
||||
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
|
||||
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
|
||||
|
||||
@@ -8,7 +8,7 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
# Model Swap and Rollback
|
||||
|
||||
Applies to every endpoint swap on the fleet (frank-small / frank-large /
|
||||
Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
|
||||
frank-primary). Repo law: all config flows through git; the live models are
|
||||
host state under `/home/dank/models/<name>/`.
|
||||
|
||||
|
||||
@@ -48,10 +48,10 @@ state, verified against the live journal and output smoke tests
|
||||
|
||||
The PP pair is the only inference service enabled for boot. Both units
|
||||
(`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
|
||||
enabled. The NPU unit (`frank-small` on Dankclaw) and all retired
|
||||
enabled. The NPU unit (`frank-secondary` on Dankclaw) and all retired
|
||||
units are disabled. Restart order: worker first, then head. The Dankserver
|
||||
helper `frank-restart-small.sh` is obsolete (it restarts the removed
|
||||
`frank-small` unit); restart the worker with
|
||||
`frank-secondary` unit); restart the worker with
|
||||
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
|
||||
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
|
||||
head. `deploy.sh dankserver` is broken until retargeted. See
|
||||
|
||||
@@ -65,6 +65,6 @@ graph LR
|
||||
## See Also
|
||||
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[frank-small-benchmark]]
|
||||
- [[frank-secondary-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[Dankclaw System Specs]]
|
||||
|
||||
@@ -247,4 +247,4 @@ see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
|
||||
measured campaigns
|
||||
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
|
||||
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — window discipline on live hosts
|
||||
|
||||
@@ -122,7 +122,7 @@ export HF_HOME=/mnt/entropy/huggingface
|
||||
|
||||
### Active Service Conflict
|
||||
|
||||
The retired `frank-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
||||
The retired `frank-secondary` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
||||
|
||||
```
|
||||
nvidia-smi # verify GPU is free
|
||||
|
||||
@@ -268,7 +268,7 @@ ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
|
||||
|
||||
**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
|
||||
llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
|
||||
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-small
|
||||
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-secondary
|
||||
was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
|
||||
counterpart here, and the two backends should not be reasoned about by analogy.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: Dankserver VRAM and RAM Growth Under Load
|
||||
description: Investigation of the recurring cuBLAS crash on frank-small. What grows, what does not, and what to do next.
|
||||
description: Investigation of the recurring cuBLAS crash on frank-secondary. What grows, what does not, and what to do next.
|
||||
tags:
|
||||
- dankserver
|
||||
- llamacpp
|
||||
@@ -18,7 +18,7 @@ timestamp: 2026-09-03
|
||||
# Dankserver VRAM and RAM Growth Under Load
|
||||
|
||||
This page records the investigation of the recurring cuBLAS crash on
|
||||
`frank-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
|
||||
`frank-secondary` (llama.cpp, RTX 3090 24 GiB). The service aborts
|
||||
with `CUDA error: an unsupported value or parameter was passed to the
|
||||
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
|
||||
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
|
||||
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
|
||||
|
||||
## Mitigation Options
|
||||
|
||||
The documentation for the benchmark ([[frank-small-benchmark]])
|
||||
The documentation for the benchmark ([[frank-secondary-benchmark]])
|
||||
prescribes the first option for this crash class. None of these change
|
||||
the model.
|
||||
|
||||
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-small-benchmark]]
|
||||
- [[frank-secondary-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[deployment]]
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-small.
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-secondary.
|
||||
tags:
|
||||
- llamacpp
|
||||
- cuda
|
||||
@@ -14,12 +14,12 @@ timestamp: 2026-08-24
|
||||
|
||||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
|
||||
> **Active 2026-08-25.** `frank-small` serves this model. It
|
||||
> **Active 2026-08-25.** `frank-secondary` serves this model. It
|
||||
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
||||
> The Q6_K file was downloaded from the model repository and validated with
|
||||
> `llama-gguf` before installation.
|
||||
|
||||
`frank-small` (port 7770) serves
|
||||
`frank-secondary` (port 7770) serves
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
||||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||||
attention architecture, abliterated (uncensored). The GGUF also contains a
|
||||
|
||||
@@ -21,7 +21,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.
|
||||
|---|---|---|---:|
|
||||
| Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-small` | 7771 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-secondary` | 7771 |
|
||||
|
||||
## Scoped workflow
|
||||
|
||||
@@ -53,13 +53,13 @@ on the selected host. The following files select the corresponding service:
|
||||
checkout, so any file there changes deployed state.
|
||||
- `scripts/frank-large.env`, `frank-large.service`, and
|
||||
`serve-frank-large.sh` select `frank-large` on Dankclaw.
|
||||
- `scripts/frank-small.env`, `frank-small.service`, and
|
||||
`serve-frank-small.sh` select `frank-small` on Dankclaw.
|
||||
- `scripts/frank-secondary.env`, `frank-secondary.service`, and
|
||||
`serve-frank-secondary.sh` select `frank-secondary` on Dankclaw.
|
||||
- Documentation, tests, deployment tooling, and unrelated container changes
|
||||
do not select any inference service.
|
||||
|
||||
The target-side `scripts/deploy-remote.sh` installs the selected service.
|
||||
`frank-large` and `frank-small` are **root units**: the handler relinks the
|
||||
`frank-large` and `frank-secondary` are **root units**: the handler relinks the
|
||||
repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
|
||||
enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
|
||||
handler retires its predecessors (the replica Quadlets and the retired llama
|
||||
@@ -67,7 +67,7 @@ root unit), stops the service's own unit, waits for the published port to
|
||||
clear, installs the `.container` and restarts the generated unit. Each
|
||||
handler then waits on that service's health endpoint —
|
||||
`http://127.0.0.1:7770/health` for frank-large and frank-medium,
|
||||
`http://127.0.0.1:7771/health` for frank-small. The wait budget is 20 minutes
|
||||
`http://127.0.0.1:7771/health` for frank-secondary. The wait budget is 20 minutes
|
||||
because cold model loads are slow. A failed handler does not advance its
|
||||
per-service state.
|
||||
|
||||
@@ -76,7 +76,7 @@ per-service state.
|
||||
The root units point into the stable target checkout:
|
||||
```text
|
||||
/home/dank/serverAppConfigs/scripts/frank-large.service
|
||||
/home/dank/serverAppConfigs/scripts/frank-small.service
|
||||
/home/dank/serverAppConfigs/scripts/frank-secondary.service
|
||||
```
|
||||
`frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
|
||||
(`frank-medium.container`) rather than a unit in `scripts/`.
|
||||
@@ -110,7 +110,7 @@ change in that service's inputs.
|
||||
|
||||
If the LLM handler fails after the target pull:
|
||||
|
||||
1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-small
|
||||
1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-secondary
|
||||
-n 200` (root units), or `journalctl --user -u frank-medium -n 200`
|
||||
(rootless Quadlet).
|
||||
2. Confirm the model files, the runtime binary and (for frank-medium) the
|
||||
@@ -120,13 +120,13 @@ If the LLM handler fails after the target pull:
|
||||
```bash
|
||||
./scripts/deploy.sh dankserver --service frank-medium
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
./scripts/deploy.sh dankclaw --service frank-small
|
||||
./scripts/deploy.sh dankclaw --service frank-secondary
|
||||
```
|
||||
4. Verify the matching health endpoint before declaring recovery.
|
||||
|
||||
Do not stop the other host's inference service during recovery. The two
|
||||
Dankclaw endpoints are independent processes on different devices: restarting
|
||||
frank-large does not disturb frank-small, and the reverse holds too — with one
|
||||
frank-large does not disturb frank-secondary, and the reverse holds too — with one
|
||||
exception worth knowing, that the deployer waits for port 7770 to be released
|
||||
before it starts the replacement container. For benchmark work, follow the
|
||||
separate stop/benchmark/restart procedure in [[Testing Policy]] and the
|
||||
|
||||
@@ -112,7 +112,7 @@ instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
|
||||
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
|
||||
mandatory.
|
||||
- Vision: OpenAI `image_url` content parts only (the fleet's standard
|
||||
contract; the legacy top-level `images` field is the frank-small trap).
|
||||
contract; the legacy top-level `images` field is the frank-secondary trap).
|
||||
≥256 px probe images or the model answers "video".
|
||||
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
|
||||
truncation. `max_tokens` INCLUDES thinking.
|
||||
@@ -181,7 +181,7 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
`maxTokens: 57344`, `input: [text, image]`,
|
||||
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
|
||||
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
|
||||
frank-small (the `compactionModel` steering is unchanged).
|
||||
frank-secondary (the `compactionModel` steering is unchanged).
|
||||
|
||||
## Rollback
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ confirmed by Frank before deploy:
|
||||
|
||||
1. **Text-only.** halogen loads the language tower only; README "honest
|
||||
limits": *"The vision encoder is ignored; image inputs are not supported."*
|
||||
The mmproj-era vision role moves wholly to frank-small.
|
||||
The mmproj-era vision role moves wholly to frank-secondary.
|
||||
2. **Censored base.** The checkpoint is a finetune of the censored
|
||||
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
|
||||
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
|
||||
@@ -83,7 +83,7 @@ bearing numbers:
|
||||
|thinking field|`reasoning_content` (same as before)|
|
||||
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|
||||
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|
||||
|image input|ignored/error — route to frank-small|
|
||||
|image input|ignored/error — route to frank-secondary|
|
||||
|
||||
## Deploy mechanics
|
||||
|
||||
@@ -114,7 +114,7 @@ bearing numbers:
|
||||
- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
|
||||
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
|
||||
the legacy `frank-large` container name by NAME (never `pkill -f
|
||||
llama-server` — that hits frank-small), retires the old quadlet if the 2026-09
|
||||
llama-server` — that hits frank-secondary), retires the old quadlet if the 2026-09
|
||||
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
|
||||
- **Bench law:** stop+mask the unit before any GPU experiment on this box
|
||||
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
|
||||
|
||||
@@ -148,7 +148,7 @@ in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
|
||||
|
||||
Sweep key for scratch servers is now the payload name:
|
||||
`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
|
||||
llama-server` (frank-small is also one).
|
||||
llama-server` (frank-secondary is also one).
|
||||
|
||||
## Operations
|
||||
|
||||
|
||||
@@ -162,7 +162,7 @@ Three traps, all hit for real:
|
||||
enough for the vision probe).
|
||||
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
|
||||
appears in the log under draft resets. Self-recovering; not an outage signal.
|
||||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
|
||||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-secondary is also a
|
||||
`llama-server`. The sweep key here is the payload name:
|
||||
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ timestamp: 2026-10-01
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
|
||||
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
|
||||
(17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
|
||||
(17.53 GiB, 4 shards) — the same checkpoint frank-secondary serves as an INT8
|
||||
W8A16 AutoRound conversion on the 3090, here unquantized.
|
||||
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
|
||||
are loaded into the draft pass).
|
||||
@@ -82,7 +82,7 @@ Qwen3.5-9B that preserves the hidden-state geometry — identical
|
||||
per-position acceptance is the direct measurement that the distillation
|
||||
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
|
||||
(verified in both the BF16 base and the INT8 conversion) — which is exactly
|
||||
why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||||
why frank-secondary's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||||
reads the head from the TARGET checkpoint, vLLM reads the draft from a
|
||||
separate path. That difference is the whole trick:
|
||||
|
||||
@@ -183,4 +183,4 @@ unit/env/serve trio and redeploying returns the 27B endpoint.
|
||||
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
|
||||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||||
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
|
||||
[[frank-small-sglang-mimo-v26]]
|
||||
[[frank-secondary-sglang-mimo-v26]]
|
||||
|
||||
@@ -211,7 +211,7 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
|
||||
|
||||
- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
|
||||
history
|
||||
- [[frank-small-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
|
||||
- [[frank-secondary-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
|
||||
at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
|
||||
difference: with no KV to attend over, per-pass overhead dominates and a
|
||||
bigger micro-batch wins; at 64k of resident KV it does not.
|
||||
|
||||
@@ -312,7 +312,7 @@ came up healthy and *text-only*: image requests answered 500 "image input is
|
||||
not supported — hint: … provide the mmproj" and the load log had no mmproj
|
||||
line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the
|
||||
model directory. The serve script now passes `--mmproj` explicitly (the
|
||||
frank-small pattern), refuses to start without the file when vision is on, and
|
||||
frank-secondary pattern), refuses to start without the file when vision is on, and
|
||||
the deploy handler's preflight checks for it. Verified end-to-end after the
|
||||
fix: a generated 64x32 test image was described from its pixels — with
|
||||
thinking on the model reasoned about "the three horizontal bars" (its reading
|
||||
@@ -724,7 +724,7 @@ fleet.
|
||||
2026-09-15, whose `--split-mode layer` measurements this service reuses
|
||||
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
|
||||
collective, and every software escape that is closed
|
||||
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
|
||||
- [[frank-secondary-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
|
||||
3090, MTP, the image path)
|
||||
- [[Dankserver System Specs]] — hardware inventory
|
||||
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
|
||||
|
||||
@@ -144,7 +144,7 @@ ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
|
||||
|
||||
First deploy (2026-09-15) also ran `retire_root_llm_small()` in
|
||||
`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
|
||||
disabled and moved to `~/.AGENTTRASH/20260915-frank-small-retire/host-units`
|
||||
disabled and moved to `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units`
|
||||
(the host's `systemctl disable` has been observed to delete unit files,
|
||||
2026-08-26 incident, so both outcomes are tolerated).
|
||||
|
||||
|
||||
@@ -55,9 +55,9 @@ image path is halogen's vision tower on `frank-large`.
|
||||
| consumer | setting | now |
|
||||
|---|---|---|
|
||||
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) |
|
||||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-small` |
|
||||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-secondary` |
|
||||
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` |
|
||||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-small` |
|
||||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-secondary` |
|
||||
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` |
|
||||
|
||||
The `flm (dankclaw-npu)` provider block was removed from the harness model
|
||||
|
||||
@@ -254,7 +254,7 @@ refetch.
|
||||
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
|
||||
b11030-mix binary family on gfx1151.
|
||||
- [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
|
||||
- [[frank-secondary-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
|
||||
3090; the only audio-capable endpoint.
|
||||
- The harness contract lives in `/home/blank/.omp/agent/models.yml`
|
||||
(`llamacpp (dankserver)` provider, `contextWindow 262144`,
|
||||
|
||||
@@ -55,7 +55,7 @@ would drift from the qualification — don't.
|
||||
## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
|
||||
|
||||
llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
|
||||
frank-small/large). The fork's `speculative_config` is a **single dict**
|
||||
frank-secondary/large). The fork's `speculative_config` is a **single dict**
|
||||
(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
|
||||
mutually exclusive — no chaining seam exists in the fork. What IS active:
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ The old scripts moved to `old/scripts/` in this repository.
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
| frank-secondary | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
|
||||
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
|
||||
tensor split, no NPU.
|
||||
@@ -40,7 +40,7 @@ tensor split, no NPU.
|
||||
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
|
||||
(48/64/24).
|
||||
|
||||
### frank-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
### frank-secondary — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
|
||||
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
|
||||
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
|
||||
@@ -103,10 +103,10 @@ Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
|
||||
6. Confirm VRAM headroom with `nvidia-smi`.
|
||||
|
||||
### Dankclaw (frank-small)
|
||||
### Dankclaw (frank-secondary)
|
||||
|
||||
1. Stop the old services: `sudo systemctl stop frank-large
|
||||
frank-small` (retired PP head and NPU endpoint).
|
||||
frank-secondary` (retired PP head and NPU endpoint).
|
||||
2. Install uv and create the venv:
|
||||
```
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
@@ -131,9 +131,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
vision encoder falls back to torch SDPA and engine init OOMs.
|
||||
5. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-small
|
||||
sudo systemctl restart frank-small
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-secondary
|
||||
sudo systemctl restart frank-secondary
|
||||
```
|
||||
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
|
||||
|
||||
@@ -159,7 +159,7 @@ on gfx1151; newer vLLM releases may already include it.
|
||||
|
||||
Three changes landed after the initial deployment:
|
||||
|
||||
### 1. Tool calling on frank-small (HTTP 400 fix)
|
||||
### 1. Tool calling on frank-secondary (HTTP 400 fix)
|
||||
|
||||
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
|
||||
unless the server enables tool calling. Fixes applied:
|
||||
@@ -184,7 +184,7 @@ model behavior, which motivated the swap.
|
||||
|
||||
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
|
||||
|
||||
`frank-small` now serves
|
||||
`frank-secondary` now serves
|
||||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
|
||||
vLLM-native expert layout). The Avesed compressed-tensors quant's
|
||||
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
|
||||
|
||||
+4
-4
@@ -1,7 +1,7 @@
|
||||
---
|
||||
timestamp: 2026-09-03
|
||||
type: reference
|
||||
title: frank-small Benchmark — ncmoe and Batch Tuning
|
||||
title: frank-secondary Benchmark — ncmoe and Batch Tuning
|
||||
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
|
||||
tags:
|
||||
- benchmark
|
||||
@@ -11,11 +11,11 @@ tags:
|
||||
- dankserver
|
||||
---
|
||||
|
||||
# frank-small Benchmark — ncmoe and Batch/Ubatch Tuning
|
||||
# frank-secondary Benchmark — ncmoe and Batch/Ubatch Tuning
|
||||
|
||||
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
|
||||
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
|
||||
`--n-cpu-moe` and `--ubatch-size` values for `frank-small`.
|
||||
`--n-cpu-moe` and `--ubatch-size` values for `frank-secondary`.
|
||||
|
||||
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
|
||||
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
|
||||
@@ -106,7 +106,7 @@ service runs 700416 cells (13,680 MiB).
|
||||
|
||||
## Chosen State
|
||||
|
||||
> **Live drift (2026-08-22 audit):** the deployed `frank-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
|
||||
> **Live drift (2026-08-22 audit):** the deployed `frank-secondary` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
|
||||
|
||||
Prefill speed is the priority, so the state keeps the top prefill tier and
|
||||
moves one expert layer to CPU for stability:
|
||||
+15
-15
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
|
||||
description: "The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
|
||||
description: "The active frank-secondary runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
@@ -14,25 +14,25 @@ tags:
|
||||
timestamp: 2026-09-17
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
|
||||
# frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
|
||||
|
||||
> **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]]
|
||||
> **ACTIVE since 2026-09-17.** Replaces [[frank-secondary-qwen38-gsq-rco]]
|
||||
> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
|
||||
> [[frank-small-vllm-gemma4]] (the same-day vLLM experiment, also the
|
||||
> rollback target). Same port (7771), same alias (`frank-small`), same
|
||||
> [[frank-secondary-vllm-gemma4]] (the same-day vLLM experiment, also the
|
||||
> rollback target). Same port (7771), same alias (`frank-secondary`), same
|
||||
> client contract surface.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
|
||||
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
|
||||
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
|
||||
collapses prefill ~12x on CUDA.
|
||||
- **Service shape:** root systemd unit `scripts/frank-small.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-small.sh` with tunables from
|
||||
`scripts/frank-small.env`.
|
||||
- **Service shape:** root systemd unit `scripts/frank-secondary.service`
|
||||
(`User=dank`), launched by `scripts/serve-frank-secondary.sh` with tunables from
|
||||
`scripts/frank-secondary.env`.
|
||||
|
||||
## Shape
|
||||
|
||||
@@ -90,7 +90,7 @@ fault).
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-small` (the `id` at `/v1/models`).
|
||||
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
|
||||
deepseek`, unlimited budget).
|
||||
- **Tools:** supported (`--jinja` is pinned even though this build defaults
|
||||
@@ -109,12 +109,12 @@ pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
|
||||
|
||||
## Rollback
|
||||
|
||||
Payload and runtime for [[frank-small-qwen38-gsq-rco]] (ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-small-vllm-gemma4]]
|
||||
Payload and runtime for [[frank-secondary-qwen38-gsq-rco]] (ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-secondary-vllm-gemma4]]
|
||||
is the same-day vLLM experiment (also a rollback target).
|
||||
|
||||
# Citations
|
||||
|
||||
- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
|
||||
- llama.cpp: https://github.com/ggml-org/llama.cpp
|
||||
- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
|
||||
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
|
||||
+16
-16
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
|
||||
description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
|
||||
description: "The retired 2026-10-02/03 frank-secondary runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
@@ -14,17 +14,17 @@ tags:
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
|
||||
# frank-secondary — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
|
||||
|
||||
> **RETIRED 2026-10-03** — superseded by
|
||||
> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
|
||||
> [[frank-secondary-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
|
||||
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
|
||||
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
|
||||
> decode is the measured price of the higher rung on this bandwidth-bound
|
||||
> card. This file remains the rollback reference and the owner of the
|
||||
> measured boot-pair table.
|
||||
|
||||
> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]]
|
||||
> Its own history: one quant step down from [[frank-secondary-llama-mimo-q6k]]
|
||||
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
|
||||
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
|
||||
> conversations of 200,704 tokens with the KV VALUES unquantized** —
|
||||
@@ -36,11 +36,11 @@ timestamp: 2026-10-03
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-secondary.sh`|
|
||||
|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
|
||||
|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
@@ -128,7 +128,7 @@ The per-slot window is the whole price, in both directions:
|
||||
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
|
||||
but refuses a request without `messages` in this build, so chat is the only
|
||||
vision path.
|
||||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
|
||||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-secondary.env` drops
|
||||
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
|
||||
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
|
||||
refusal, never a silent fallback to blindness.
|
||||
@@ -145,15 +145,15 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|
||||
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-secondary.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|
||||
|Q6_K, same runtime|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-secondary-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|
||||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SECONDARY_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-secondary-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
|
||||
- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script exits 1
|
||||
naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
+17
-17
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
|
||||
description: "The RETIRED frank-small runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
|
||||
description: "The RETIRED frank-secondary runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
@@ -12,10 +12,10 @@ tags:
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
# frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **RETIRED 2026-10-05 (evening).** Superseded by
|
||||
> [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B
|
||||
> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B
|
||||
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
|
||||
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
|
||||
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
|
||||
@@ -29,14 +29,14 @@ timestamp: 2026-10-03
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
|
||||
| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
@@ -67,7 +67,7 @@ compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
|
||||
were declined. Re-open only on evidence the Q6_K residual matters.
|
||||
|
||||
**Pool prices on this card come from boots, never formulas** — the measured
|
||||
boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
|
||||
boot-pair table lives in [[frank-secondary-llama-mimo-q5km]] (*Shape and KV
|
||||
contract*) and still prices every rollback shape.
|
||||
|
||||
## Compute-buffer geometry (2026-10-05)
|
||||
@@ -78,7 +78,7 @@ measured rather than assumed, because the predecessor shape recorded **"ubatch
|
||||
|
||||
**That failure was a different shape, and it did not reproduce here.** The
|
||||
service itself is the bench (one card holds one server, so no scratch port):
|
||||
the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
|
||||
the old rung was re-booted through a transient `FRANK_SECONDARY_ENV_FILE` drop-in
|
||||
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
|
||||
2,849 MiB free — which makes the ubatch the only variable in the comparison.
|
||||
|
||||
@@ -99,7 +99,7 @@ Three readings:
|
||||
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
|
||||
inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
|
||||
and the pool's 12,288-token (384 MiB) ladder in
|
||||
[[frank-small-llama-mimo-q5km]] is the other way to buy room back.
|
||||
[[frank-secondary-llama-mimo-q5km]] is the other way to buy room back.
|
||||
|
||||
**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
|
||||
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
|
||||
@@ -125,7 +125,7 @@ that is the pair the curve was measured at.
|
||||
|
||||
## Client contract
|
||||
|
||||
llama.cpp surface, live-verified on this model family: model id `frank-small`,
|
||||
llama.cpp surface, live-verified on this model family: model id `frank-secondary`,
|
||||
native tools via `--jinja` (`tool_calls` in the response), thinking in
|
||||
**`reasoning_content`**, vision via `image_url` content parts. Two vision
|
||||
traps, both measured here:
|
||||
@@ -147,13 +147,13 @@ so the harness compacts before llama.cpp answers a longer prompt with a 400.
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|
||||
|Q5_K_M at the same shape|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|
||||
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|
||||
|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*):
|
||||
Same decision tree as [[frank-secondary-llama-mimo-q5km]] (*When it misbehaves*):
|
||||
`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
|
||||
200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
|
||||
a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
|
||||
+9
-9
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||||
description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||||
description: "The retired 2026-10-01 frank-secondary runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
@@ -14,10 +14,10 @@ tags:
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
# frank-secondary — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
|
||||
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
|
||||
> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> [[frank-secondary-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||||
> target.
|
||||
>
|
||||
@@ -29,11 +29,11 @@ timestamp: 2026-10-01
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
|
||||
|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
@@ -100,7 +100,7 @@ use the parent of the 2026-10-01 night swap commit).
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
|
||||
- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script
|
||||
exits 1 naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
|
||||
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
+14
-14
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-secondary runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
@@ -15,7 +15,7 @@ tags:
|
||||
timestamp: 2026-10-06
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV
|
||||
# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A
|
||||
> 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit
|
||||
@@ -32,7 +32,7 @@ timestamp: 2026-10-06
|
||||
> full context (the 262,144 draft window), not the 3-token proposal
|
||||
> depth. The 2–3 GiB floor is knowingly violated, as ordered. This
|
||||
> supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K
|
||||
> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> shape ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
||||
> deployed.
|
||||
|
||||
@@ -41,14 +41,14 @@ timestamp: 2026-10-06
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
|
||||
| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
|
||||
|
||||
## Why this payload (2026-10-06 evening)
|
||||
|
||||
@@ -89,7 +89,7 @@ context (262,144 — the draft window, not the 3-token proposal depth), so
|
||||
it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB
|
||||
the one-slot era assumed. The lighter quant pays for it: **23,636 used /
|
||||
489 MiB free** (nvidia-smi `free` field) — the tightest fit in
|
||||
frank-small's history, the 2–3 GiB floor knowingly violated, and the
|
||||
frank-secondary's history, the 2–3 GiB floor knowingly violated, and the
|
||||
512×512 vision transient (34 MiB) still fits and recovers. Compute buffers
|
||||
allocate at graph capture during BOOT, so a miss is loud and fast — the
|
||||
failure mode is a crash loop, not silent corruption.
|
||||
@@ -225,7 +225,7 @@ recipes:
|
||||
`/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
|
||||
2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||||
the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||||
3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||||
@@ -243,10 +243,10 @@ recipes:
|
||||
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
|
||||
label-committed both sets at Frank's direction — 1919ad7 (frank-primary
|
||||
radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
|
||||
Deploy ran `deploy.sh dankclaw --service frank-small`; the service
|
||||
Deploy ran `deploy.sh dankclaw --service frank-secondary`; the service
|
||||
dispatch and health gate went green; the first e2e run failed ONE spec
|
||||
(12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
|
||||
unrelated to frank-small), green on rerun.
|
||||
unrelated to frank-secondary), green on rerun.
|
||||
- The unit's `Description=` (MiMo era) was stale at cutover time and was
|
||||
updated in-repo afterward — the relinked unit picks it up at the next
|
||||
deploy/restart, not retroactively.
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
|
||||
description: "The RETIRED frank-secondary runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-09-16
|
||||
---
|
||||
+16
-16
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
|
||||
title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- sglang
|
||||
- rtx3090
|
||||
@@ -15,10 +15,10 @@ tags:
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
|
||||
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
|
||||
> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
|
||||
> payload was staged to
|
||||
@@ -28,13 +28,13 @@ timestamp: 2026-10-01
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
|
||||
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
|
||||
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
|
||||
systemd unit `scripts/frank-small.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
|
||||
tunables from `scripts/frank-small.env`.
|
||||
systemd unit `scripts/frank-secondary.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with
|
||||
tunables from `scripts/frank-secondary.env`.
|
||||
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
|
||||
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
|
||||
|
||||
@@ -55,7 +55,7 @@ timestamp: 2026-10-01
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
|
||||
180,224, which 183,501 tokens already exceeds — see *What three slots
|
||||
cost* in [[frank-small-llama-mimo-q5km]].
|
||||
cost* in [[frank-secondary-llama-mimo-q5km]].
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
@@ -126,15 +126,15 @@ change lands.
|
||||
|
||||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||||
all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
|
||||
all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh`
|
||||
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
|
||||
cut over to llama.cpp + GGUF on 2026-10-01 night, and
|
||||
`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
|
||||
`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The
|
||||
serve script refuses to start with a missing shard.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-small` (the `id` at `/v1/models`).
|
||||
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
|
||||
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
|
||||
as llama.cpp; models.yml sets `reasoningContentField:
|
||||
@@ -166,8 +166,8 @@ serve script refuses to start with a missing shard.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
|
||||
[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at
|
||||
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
|
||||
Gemma4-12B QAT payload, was staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
|
||||
@@ -177,4 +177,4 @@ Gemma4-12B QAT payload, was staged to
|
||||
|
||||
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
- SGLang: https://github.com/sgl-project/sglang
|
||||
- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
|
||||
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
|
||||
+20
-20
@@ -1,10 +1,10 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||||
description: "The RETIRED frank-small runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-small-gemma4-12b]] on the same day."
|
||||
title: frank-secondary on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||||
description: "The RETIRED frank-secondary runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-secondary-gemma4-12b]] on the same day."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- vllm
|
||||
- rtx3090
|
||||
@@ -15,12 +15,12 @@ tags:
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
|
||||
# frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||||
# frank-secondary — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||||
|
||||
> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
|
||||
> back to [[frank-small-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
|
||||
> back to [[frank-secondary-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
|
||||
> the same day it landed — this page is the rollback target. Same port
|
||||
> (7771), same alias (`frank-small`), same client contract surface — the
|
||||
> (7771), same alias (`frank-secondary`), same client contract surface — the
|
||||
> runtime, the model payload, the drafter and the service shape (root
|
||||
> systemd unit → rootless Quadlet) all changed.
|
||||
|
||||
@@ -28,12 +28,12 @@ timestamp: 2026-09-19
|
||||
|
||||
| what | value |
|
||||
|---|---|
|
||||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d7…`) |
|
||||
| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (root systemd unit retired 2026-09-19) |
|
||||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-secondary` (`8dad46d7…`) |
|
||||
| unit | **rootless Quadlet** `dankclaw/frank-secondary/quadlet/frank-secondary.container` (root systemd unit retired 2026-09-19) |
|
||||
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
|
||||
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
|
||||
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
|
||||
| port | 7771 → 8000 (alias `frank-small`; `frank-large` keeps 7770) |
|
||||
| port | 7771 → 8000 (alias `frank-secondary`; `frank-large` keeps 7770) |
|
||||
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
|
||||
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
|
||||
|
||||
@@ -113,7 +113,7 @@ trade for the native w4a16 format + audio + CUDA graphs.)
|
||||
|
||||
## The image
|
||||
|
||||
`dankclaw/frank-small/Dockerfile` (repo) — 16 comment lines + `FROM
|
||||
`dankclaw/frank-secondary/Dockerfile` (repo) — 16 comment lines + `FROM
|
||||
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
|
||||
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
|
||||
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
|
||||
@@ -144,9 +144,9 @@ not a hardware mystery.
|
||||
|
||||
## Deployed config
|
||||
|
||||
Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`
|
||||
Repo truth: `scripts/frank-secondary.env` + `scripts/serve-frank-secondary.sh`
|
||||
(mounted into the container at `/scripts/`, which is the entrypoint) +
|
||||
`dankclaw/frank-small/quadlet/frank-small.container`. Key values:
|
||||
`dankclaw/frank-secondary/quadlet/frank-secondary.container`. Key values:
|
||||
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
|
||||
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
|
||||
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
|
||||
@@ -186,27 +186,27 @@ container while it loads.
|
||||
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
|
||||
must be absent. 8.6 is correct.
|
||||
- **Audio 400/500** — the container is running the *base* image, not the
|
||||
derived `:frank-small` (missing `av`). Check the deployed image digest
|
||||
derived `:frank-secondary` (missing `av`). Check the deployed image digest
|
||||
is `8dad46d7…`.
|
||||
- **CUDA not visible in container** — a `Device=` line or a lib mount
|
||||
dropped out of the unit after an edit; the unit is the source of truth,
|
||||
the host-side generated unit is not.
|
||||
- Logs: `journalctl --user -u frank-small` (rootless Quadlet; `podman
|
||||
- Logs: `journalctl --user -u frank-secondary` (rootless Quadlet; `podman
|
||||
logs` covers only the current `--replace --rm` instance).
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||||
[[frank-secondary-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||||
mmproj and MTP sidecar are still on the host under
|
||||
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
|
||||
`scripts/.AGENTTRASH/frank-small.service.llama.cpp`. Restoring it means
|
||||
`scripts/.AGENTTRASH/frank-secondary.service.llama.cpp`. Restoring it means
|
||||
reinstalling that unit and stopping the Quadlet (one owner per port).
|
||||
|
||||
## References
|
||||
|
||||
- `scripts/serve-frank-small.sh` — the actual exec line
|
||||
- `dankclaw/frank-small/Dockerfile` — the audio-derived image
|
||||
- `dankclaw/frank-small/quadlet/frank-small.container` — the unit
|
||||
- [[frank-small-gemma4-12b]] — RETIRED predecessor (rollback)
|
||||
- `scripts/serve-frank-secondary.sh` — the actual exec line
|
||||
- `dankclaw/frank-secondary/Dockerfile` — the audio-derived image
|
||||
- `dankclaw/frank-secondary/quadlet/frank-secondary.container` — the unit
|
||||
- [[frank-secondary-gemma4-12b]] — RETIRED predecessor (rollback)
|
||||
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
|
||||
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
|
||||
@@ -1,15 +0,0 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
|
||||
description: "The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-09-16
|
||||
---
|
||||
@@ -6,7 +6,7 @@ tags:
|
||||
- freetoken
|
||||
- benchmark
|
||||
- dankserver
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- qwen3.6
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
@@ -1,27 +1,27 @@
|
||||
---
|
||||
type: configuration
|
||||
title: FreeToken frank-small Runbook
|
||||
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090 era).
|
||||
title: FreeToken frank-secondary Runbook
|
||||
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-secondary on Dankserver (RTX 3090 era).
|
||||
tags:
|
||||
- freetoken
|
||||
- dankserver
|
||||
- frank-small
|
||||
- frank-secondary
|
||||
- qwen3.6
|
||||
- moe
|
||||
- fp8
|
||||
timestamp: 2026-08-30
|
||||
---
|
||||
|
||||
# FreeToken frank-small Runbook
|
||||
# FreeToken frank-secondary Runbook
|
||||
|
||||
> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
|
||||
> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
|
||||
> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
|
||||
> retirement record of the FreeToken/CUDA era; the unit was archived to
|
||||
> `~/.AGENTTRASH/20260915-frank-small-retire/host-units/` and the repo files
|
||||
> `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/` and the repo files
|
||||
> to `scripts/.AGENTTRASH/`.
|
||||
|
||||
The `frank-small` service on Dankserver is served by
|
||||
The `frank-secondary` service on Dankserver is served by
|
||||
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
|
||||
expert weights in host RAM with an LRU GPU cache. Model history on this
|
||||
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
|
||||
@@ -134,8 +134,8 @@ modelopt block-fp8 remains the only FP8 path that loads.
|
||||
|
||||
## Sampling
|
||||
|
||||
**frank-small samples by default, and our config never said so.**
|
||||
`scripts/serve-frank-small.sh` passes no sampling arguments at all — only
|
||||
**frank-secondary samples by default, and our config never said so.**
|
||||
`scripts/serve-frank-secondary.sh` passes no sampling arguments at all — only
|
||||
`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
|
||||
`--num-tokens` and `--max-running-requests`. The behaviour comes from the
|
||||
runtime's own default:
|
||||
@@ -160,7 +160,7 @@ would be byte-identical.
|
||||
|
||||
### This is the opposite of frank-large
|
||||
|
||||
| | frank-small (FreeToken) | frank-large (halogen) |
|
||||
| | frank-secondary (FreeToken) | frank-large (halogen) |
|
||||
|---|---|---|
|
||||
| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
|
||||
| Deterministic across runs | No | Yes |
|
||||
@@ -168,7 +168,7 @@ would be byte-identical.
|
||||
|
||||
A client that talks to both endpoints and sends no sampler gets
|
||||
non-deterministic answers from :7770 on Dankserver and reproducible ones from
|
||||
:7770 on Dankclaw. If a workflow needs determinism from frank-small it must
|
||||
:7770 on Dankclaw. If a workflow needs determinism from frank-secondary it must
|
||||
send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
`--sampling-defaults none` (framework defaults only).
|
||||
|
||||
@@ -188,9 +188,9 @@ send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-small.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-small.service` | Systemd unit |
|
||||
| `scripts/serve-frank-secondary.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-secondary.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-secondary.service` | Systemd unit |
|
||||
|
||||
## Limitations
|
||||
|
||||
@@ -209,7 +209,7 @@ send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
- **`model` field is required** in `/v1/chat/completions` bodies —
|
||||
FreeToken rejects requests without it with a 422 (`Field required`),
|
||||
unlike llama.cpp's server which defaults to the loaded model. Send
|
||||
`"model": "frank-small"`.
|
||||
`"model": "frank-secondary"`.
|
||||
- Load on this swap took well under the old llama.cpp numbers:
|
||||
weights → experts (parallel build) → CUDA graph capture → Uvicorn
|
||||
in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
|
||||
|
||||
@@ -23,7 +23,7 @@ timestamp: 2026-09-13
|
||||
> shape the fleet asked for: two 200k-token conversations rather than the
|
||||
> native-context pair. Deployed as a rootless Quadlet container,
|
||||
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
|
||||
> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
|
||||
> it does not touch now belongs to [[frank-secondary-qwen38-gsq-rco]].
|
||||
|
||||
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
|
||||
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
|
||||
@@ -98,9 +98,9 @@ upstream's defaults:
|
||||
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
|
||||
| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
|
||||
| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
|
||||
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
|
||||
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-secondary, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
|
||||
| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
|
||||
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
|
||||
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
|
||||
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
|
||||
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
|
||||
|
||||
@@ -374,7 +374,7 @@ tables above.
|
||||
|
||||
**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
|
||||
the tower is a 1.97 GiB charge against the same device budget as the pool and
|
||||
the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
|
||||
the arena, and the fleet's image path moved to [[frank-secondary-qwen38-gsq-rco]],
|
||||
which loads the BF16 projector alongside its own weights. This server refuses
|
||||
image content parts and says why, rather than ignoring them:
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-secondary on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
tags:
|
||||
- ikllama
|
||||
- llamacpp
|
||||
@@ -16,7 +16,7 @@ timestamp: 2026-08-26
|
||||
|
||||
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
||||
|
||||
> **Active 2026-08-26.** `frank-small` switched from upstream
|
||||
> **Active 2026-08-26.** `frank-secondary` switched from upstream
|
||||
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
||||
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
||||
> the MoE / CPU-offload runtime arguments.
|
||||
@@ -101,8 +101,8 @@ a hard compile error.
|
||||
|
||||
## Runtime configuration
|
||||
|
||||
Files: `scripts/serve-frank-small.sh`, `scripts/frank-small.env`,
|
||||
`scripts/frank-small.service` in the repo. The service just calls
|
||||
Files: `scripts/serve-frank-secondary.sh`, `scripts/frank-secondary.env`,
|
||||
`scripts/frank-secondary.service` in the repo. The service just calls
|
||||
the serve script; the env file holds all tunables.
|
||||
|
||||
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
|
||||
@@ -241,7 +241,7 @@ apply.
|
||||
The previous runtime `upstream llama.cpp b10121` and its binary
|
||||
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
||||
rollback to the old stack, change `LLAMA_SERVER=` in
|
||||
`scripts/frank-small.env` back to that path and
|
||||
`scripts/frank-secondary.env` back to that path and
|
||||
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
|
||||
spec / `--spec-type` flags. The model file is not changed either way.
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
|
||||
ctx, vision ON)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
|
||||
**llama.cpp CUDA (frank-secondary, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
|
||||
Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file
|
||||
IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's
|
||||
128k cap (the tightest fit in the endpoint's history — 489 MiB free),
|
||||
@@ -51,7 +51,7 @@ frank-large, port 7770]
|
||||
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
|
||||
2 x 200192 ctx, MTP always-on, image input]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
frank-secondary, port 7771]
|
||||
H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF
|
||||
2 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||||
E2 --> E --> F
|
||||
@@ -103,7 +103,7 @@ frank-small, port 7771]
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since
|
||||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||||
2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master
|
||||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||||
@@ -120,7 +120,7 @@ frank-small, port 7771]
|
||||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||||
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
|
||||
(77decb5..380aafa) but never deployed
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||||
@@ -129,7 +129,7 @@ frank-small, port 7771]
|
||||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||||
probe-size trap (>=256 px images or the model says "video") live inside
|
||||
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
- [[frank-secondary-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
@@ -265,10 +265,10 @@ frank-small, port 7771]
|
||||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||||
runtime it replaced
|
||||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||||
frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp
|
||||
frank-large: the FreeToken v0.1.2 frank-secondary runbook (batsclamp
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||||
[[frank-small-qwen38-gsq-rco]]
|
||||
[[frank-secondary-qwen38-gsq-rco]]
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
@@ -337,35 +337,35 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
- [[frank-secondary-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||||
[[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||||
`/home/dank/models/MiMo-Q6K/`
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-secondary-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv still on disk, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-secondary-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
f16 KV at 131k
|
||||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||||
@@ -396,9 +396,9 @@ frank-small, port 7771]
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-small, retired 2026-09-02)
|
||||
Dankserver (frank-secondary, retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||||
frank-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
frank-secondary (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||||
@@ -408,7 +408,7 @@ frank-small, port 7771]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||||
the flash-server v2 runtime since 2026-10-06, frank-small on the RTX 3090)
|
||||
the flash-server v2 runtime since 2026-10-06, frank-secondary on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
@@ -432,7 +432,7 @@ frank-small, port 7771]
|
||||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||||
notes and sizing (this architecture family is what the retired frank-small
|
||||
notes and sizing (this architecture family is what the retired frank-secondary
|
||||
served in the Huihui abliterated FP8 variant — see
|
||||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||||
that preceded it is archived at [[dealign-ornith-crack]])
|
||||
|
||||
@@ -204,4 +204,4 @@ clipping at depth) starts with sections 1–3 here before touching anything.
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
|
||||
- [[frank-large-benchmark]] and [[frank-small-benchmark]] — probe/ladder hygiene on live hosts
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — probe/ladder hygiene on live hosts
|
||||
|
||||
@@ -249,13 +249,13 @@ distinguishes a real split from a mislabelled sweep.
|
||||
|
||||
The fleet's answer is unchanged — **two independent endpoints**
|
||||
([[halogen-flash-server]] on the iGPU via halogen,
|
||||
[[frank-small-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
|
||||
[[frank-secondary-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
|
||||
claims from the first round of this experiment are now corrected:
|
||||
|
||||
1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
|
||||
tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode),
|
||||
and it costs the 3090 ~15 GiB — which this fleet spends on the vision endpoint
|
||||
(`frank-small` holds 22.4 of the card's 24.6 GiB). An operator who wants the
|
||||
(`frank-secondary` holds 22.4 of the card's 24.6 GiB). An operator who wants the
|
||||
7 % must give up the fleet's only image-capable endpoint.
|
||||
2. **Per-tensor placement across two devices is impossible while `-ot` is used**,
|
||||
for a scheduler-mechanical reason, regardless of driver or vendor mix.
|
||||
@@ -309,8 +309,8 @@ Operational notes for the next run:
|
||||
`GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
|
||||
pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
|
||||
host's co-tenants.
|
||||
- `systemctl mask` is **refused** for a relinked root unit: frank-small's
|
||||
`/etc/systemd/system/frank-small.service` is a symlink into the repo, so
|
||||
- `systemctl mask` is **refused** for a relinked root unit: frank-secondary's
|
||||
`/etc/systemd/system/frank-secondary.service` is a symlink into the repo, so
|
||||
`mask` exits with "File ... already exists". Stop it and rely on the absence
|
||||
of a restart trigger, or move the symlink aside.
|
||||
- Restarting frank-large needs the quadlet source back in
|
||||
|
||||
@@ -217,7 +217,7 @@ done
|
||||
```
|
||||
|
||||
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
|
||||
= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix
|
||||
= AI stack + `frank-large` :7770 + `frank-secondary` :7771 (RTX 3090 + Strix
|
||||
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
|
||||
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
|
||||
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
|
||||
|
||||
@@ -12,7 +12,7 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
`frank-secondary` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
Quadlet).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
@@ -103,7 +103,7 @@ the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
|
||||
Any scratch/experimental vLLM container on a host that also runs a production
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-secondary`) MUST stop the
|
||||
production unit first, and the window script must verify it. Two independent
|
||||
failure modes if skipped:
|
||||
|
||||
|
||||
@@ -64,7 +64,7 @@ done
|
||||
Then **read every hit**; do not count them. Most are benign (comments,
|
||||
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
|
||||
quadlet not a model) — but on Dankclaw this check found a live bug:
|
||||
`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
`serve-frank-secondary.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
payload retired a week earlier. The env file always overrode it, so it was
|
||||
invisible; a missing env file would have produced a "missing asset" naming a
|
||||
model that no longer exists.
|
||||
@@ -99,7 +99,7 @@ active).
|
||||
## 5. Verify, then report — and do not purge
|
||||
|
||||
```sh
|
||||
systemctl is-active frank-large frank-small
|
||||
systemctl is-active frank-large frank-secondary
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
|
||||
@@ -191,7 +191,7 @@ Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
|
||||
licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
|
||||
Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
|
||||
- **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the
|
||||
TTS engine shares the card with frank-small; move to the R9700s only if TTS
|
||||
TTS engine shares the card with frank-secondary; move to the R9700s only if TTS
|
||||
throughput demand grows.
|
||||
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
|
||||
Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
|
||||
|
||||
@@ -107,7 +107,7 @@ TTS), [Kokoro-82M](https://github.com/hexgrad/kokoro) (8,933, Apache-2.0, TTS),
|
||||
the `Comfy-Org` org. Already running on this fleet (Qwen-Image 2.1 GGUF) and
|
||||
wired as the Open WebUI image engine.
|
||||
- **llama.cpp** — the reference GGUF inference engine; `llama-server` exposes an
|
||||
OpenAI-compatible endpoint. Backbone of this fleet's frank-small/frank-large.
|
||||
OpenAI-compatible endpoint. Backbone of this fleet's frank-secondary/frank-large.
|
||||
- **vLLM** — high-throughput PagedAttention serving with an OpenAI-compatible
|
||||
API; the multi-GPU option (frank-medium runs vLLM tensor-parallel across two
|
||||
R9700s).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
description: "Consolidated research from the `frank-small` research subagents (2026-08-14) plus the pi agent knowledge base."
|
||||
description: "Consolidated research from the `frank-secondary` research subagents (2026-08-14) plus the pi agent knowledge base."
|
||||
tags:
|
||||
- research
|
||||
timestamp: 2026-09-03
|
||||
@@ -8,7 +8,7 @@ timestamp: 2026-09-03
|
||||
|
||||
# Service Research Brief
|
||||
|
||||
Consolidated research from the `frank-small` research subagents (2026-08-14) plus the pi agent knowledge base. This is the source material behind the per-service docs pages.
|
||||
Consolidated research from the `frank-secondary` research subagents (2026-08-14) plus the pi agent knowledge base. This is the source material behind the per-service docs pages.
|
||||
|
||||
## Method
|
||||
|
||||
|
||||
@@ -227,7 +227,7 @@
|
||||
|
||||
<div class="app">
|
||||
<div class="app-head"><h3>llama.cpp <span class="fit exc">EXCELLENT</span></h3><span class="tag"><span class="st">129,149★</span> · MIT · LLM runtime</span></div>
|
||||
<p>The reference GGUF inference engine; <code>llama-server</code> exposes an OpenAI-compatible endpoint. Backbone of this fleet's frank-small/frank-large endpoints. Runs on CPU, CUDA, ROCm, Vulkan, Metal.</p>
|
||||
<p>The reference GGUF inference engine; <code>llama-server</code> exposes an OpenAI-compatible endpoint. Backbone of this fleet's frank-secondary/frank-large endpoints. Runs on CPU, CUDA, ROCm, Vulkan, Metal.</p>
|
||||
</div>
|
||||
|
||||
<div class="app">
|
||||
|
||||
@@ -372,15 +372,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE
|
||||
frank-small runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE
|
||||
frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
|
||||
with MTP back on since 2026-10-06 evening**): llama.cpp master (CUDA
|
||||
sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
|
||||
(the MTP head in the file is drafted) on the RTX 3090 — TWO
|
||||
131,072-token conversations at Frank's 128k cap, the tightest fit in
|
||||
the endpoint's history (489 MiB measured free), q8_0 K + q8_0 V, vision
|
||||
through the publisher's own projector, ~63 t/s recipe decode with MTP
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
superseded by swift15): llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
|
||||
200,704 tokens each with fully unquantized f16 KV and mmproj vision;
|
||||
@@ -388,32 +388,32 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
|
||||
2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
|
||||
images answer "video")
|
||||
- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
|
||||
- [[frank-secondary-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target, payload on disk; owns the measured boot-pair table): same runtime
|
||||
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
|
||||
- [[frank-secondary-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
|
||||
Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
|
||||
[[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
|
||||
for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
|
||||
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
|
||||
SGLang in a self-contained python3.12 venv serving
|
||||
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
|
||||
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
|
||||
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
|
||||
2 x 262,144 slots, bf16 mamba states, eager prefill
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
|
||||
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
token pool, 2 concurrent full-context sessions, native text+image+audio
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
|
||||
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
|
||||
the prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-secondary (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
|
||||
- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||||
@@ -433,7 +433,7 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[frank-small-benchmark]] — the small-model benchmark suite
|
||||
- [[frank-secondary-benchmark]] — the small-model benchmark suite
|
||||
- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x
|
||||
R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the
|
||||
logical batch buys nothing measurable, decode never moves, and the ubatch's
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
|
||||
## 2026-10-06
|
||||
- **frank-small (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
|
||||
- **frank-secondary (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
|
||||
Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k
|
||||
contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb):
|
||||
22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's
|
||||
@@ -19,7 +19,7 @@
|
||||
uncensored model thinks more — 64 tokens is all `reasoning_content`).
|
||||
ngram-mod still skipped (recurrent-state rollback law on the qwen35
|
||||
hybrid). Runbook renamed + rewritten:
|
||||
`serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`.
|
||||
`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md`.
|
||||
- **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server
|
||||
0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
|
||||
commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
|
||||
@@ -120,12 +120,12 @@
|
||||
Live pass of the frankbook walk itself is deferred to the next deploy with
|
||||
an awake host.
|
||||
- **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
|
||||
left `frank_small_unit.bats` and the deployer's payload gate stale** —
|
||||
left `frank_secondary_unit.bats` and the deployer's payload gate stale** —
|
||||
the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
|
||||
`LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
|
||||
loads (the anti-drift seam doing its job). Both repointed to the
|
||||
Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
|
||||
- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win
|
||||
- **frank-secondary: compute-buffer pair doubled to 4096/2048, and the prefill win
|
||||
measured rather than assumed.** Frank asked for the batch and ubatch doubled.
|
||||
The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
|
||||
failed the scratch boot"**, and this shape carries 616 MiB less free space
|
||||
@@ -135,7 +135,7 @@
|
||||
`FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
|
||||
boot **byte-identically** (21,276 used / 2,849 free), which is what makes
|
||||
ubatch the only variable. Repo never saw a fake revert commit; scaffolding
|
||||
staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`.
|
||||
staged to `~/.AGENTTRASH/20261005-frank-secondary-ubatch-ab/`.
|
||||
- **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
|
||||
for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
|
||||
noise). The control that proves it is geometry and not drift: a 279-token
|
||||
@@ -158,7 +158,7 @@
|
||||
edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
|
||||
flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
|
||||
second burst and ~90 s). Docs:
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
|
||||
|
||||
## 2026-10-03
|
||||
- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
|
||||
@@ -504,8 +504,8 @@
|
||||
**sparse** file, 3.3 G apparent but 4 K real, so removing it frees
|
||||
nothing.
|
||||
- **A live reference check came first, and it earned its keep.**
|
||||
`serve-frank-small.sh`'s *fallback* `MODEL_DIR` still pointed at the
|
||||
retired GSQ-RCO payload frank-small served on 2026-09-16/17. The env
|
||||
`serve-frank-secondary.sh`'s *fallback* `MODEL_DIR` still pointed at the
|
||||
retired GSQ-RCO payload frank-secondary served on 2026-09-16/17. The env
|
||||
file has always overridden it, so the drift was invisible — but a
|
||||
missing or unreadable env file would have turned a config problem into a
|
||||
"missing asset" naming a model that no longer exists. Repointed at
|
||||
@@ -519,7 +519,7 @@
|
||||
- Pre-existing: **118 G of trash from 2026-09-21** was already staged
|
||||
(`20260921-frank-large-assembled` 88 G, `-redownload` 30 G) and never
|
||||
purged. Total now staged on this host: **627 G**.
|
||||
- Verified after the moves: frank-large and frank-small both `active`,
|
||||
- Verified after the moves: frank-large and frank-secondary both `active`,
|
||||
both `/health` 200, and all three live assets (Swift target, DFlash2
|
||||
drafter, Gemma4 target) still present.
|
||||
- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
|
||||
@@ -735,14 +735,14 @@
|
||||
|
||||
## 2026-09-25
|
||||
- `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
|
||||
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-small → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
|
||||
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-small = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
|
||||
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-secondary → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
|
||||
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-secondary = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
|
||||
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
|
||||
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).
|
||||
|
||||
## 2026-09-22
|
||||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-small-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B).
|
||||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-secondary-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-secondary (Gemma4-12B).
|
||||
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.
|
||||
|
||||
## 2026-09-21
|
||||
@@ -757,7 +757,7 @@
|
||||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
|
||||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
|
||||
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
|
||||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-secondary/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||||
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
|
||||
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
|
||||
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.
|
||||
@@ -766,7 +766,7 @@
|
||||
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
|
||||
- `/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` — New runbook.
|
||||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
|
||||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||||
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||||
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
|
||||
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.
|
||||
|
||||
@@ -775,9 +775,9 @@
|
||||
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.
|
||||
|
||||
## 2026-09-17
|
||||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||||
- `/scripts/frank-small.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||||
- `/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` — New ACTIVE runbook; `frank-small-qwen38-gsq-rco.md` retired as rollback target.
|
||||
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||||
- `/scripts/frank-secondary.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||||
- `/serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md` — New ACTIVE runbook; `frank-secondary-qwen38-gsq-rco.md` retired as rollback target.
|
||||
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
|
||||
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
|
||||
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.
|
||||
@@ -786,10 +786,10 @@
|
||||
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
|
||||
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
|
||||
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
|
||||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||||
- `/scripts/frank-small.env`, `/scripts/frank-small.service`, `/scripts/serve-frank-small.sh` — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-secondary); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||||
- `/scripts/frank-secondary.env`, `/scripts/frank-secondary.service`, `/scripts/serve-frank-secondary.sh` — New frank-secondary: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
|
||||
- `/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
|
||||
- `/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
|
||||
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
|
||||
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
|
||||
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.
|
||||
@@ -828,7 +828,7 @@
|
||||
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
|
||||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
|
||||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||||
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||||
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
|
||||
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
|
||||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
|
||||
@@ -840,7 +840,7 @@
|
||||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
|
||||
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
|
||||
- `/dankclaw/frank-large/quadlet/frank-large.container` — frank-large: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
|
||||
- `/scripts/frank-small.env` — frank-small (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||||
- `/scripts/frank-secondary.env` — frank-secondary (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||||
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
|
||||
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).
|
||||
|
||||
@@ -1076,8 +1076,8 @@ any repo service.
|
||||
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
|
||||
|
||||
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
|
||||
frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
|
||||
frank-secondary, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-secondary — SGLang reads the MTP head from the TARGET checkpoint
|
||||
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
|
||||
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
|
||||
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
|
||||
@@ -1147,12 +1147,12 @@ bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
|
||||
fetchers moved to `old/`. Runbook
|
||||
[[frank-large-rocmfpx-q8agent-mtp]].
|
||||
|
||||
## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
## 2026-10-02 — frank-secondary: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
|
||||
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
|
||||
with two contexts ("whatever context size fits the 3090"), then **three
|
||||
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
|
||||
the 3090", then **"frank-small should have vision"**.
|
||||
the 3090", then **"frank-secondary should have vision"**.
|
||||
|
||||
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
|
||||
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
|
||||
@@ -1190,7 +1190,7 @@ the 3090", then **"frank-small should have vision"**.
|
||||
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
|
||||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||||
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
|
||||
gufo DFlash2 drafter). Runbook [[frank-secondary-llama-mimo-q5km]].
|
||||
|
||||
## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
|
||||
|
||||
@@ -1260,7 +1260,7 @@ top of the proven-good base, root cause first.
|
||||
- **Harness contract followed the window:** `models.yml` `contextWindow`
|
||||
262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
|
||||
200,704). Side effect: frank-primary's compaction request now fits inside a
|
||||
frank-small slot again, which it had stopped doing when frank-small went to
|
||||
frank-secondary slot again, which it had stopped doing when frank-secondary went to
|
||||
3 x 180,224.
|
||||
- **Comment-only follow-up** `295d578` recorded the measured numbers and was
|
||||
deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
|
||||
|
||||
Reference in New Issue
Block a user