rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
This commit is contained in:
Frank committed
1 parent
c71a3066c0
commit
87721ab0fe
87 files changed
+460
-460
No files matched your search
@@ -2,7 +2,7 @@
|
||||
|
||||
|
||||
## 2026-10-06
|
||||
- **frank-small (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
|
||||
- **frank-secondary (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
|
||||
Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k
|
||||
contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb):
|
||||
22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's
|
||||
@@ -19,7 +19,7 @@
|
||||
uncensored model thinks more — 64 tokens is all `reasoning_content`).
|
||||
ngram-mod still skipped (recurrent-state rollback law on the qwen35
|
||||
hybrid). Runbook renamed + rewritten:
|
||||
`serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`.
|
||||
`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072.md`.
|
||||
- **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server
|
||||
0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
|
||||
commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
|
||||
@@ -120,12 +120,12 @@
|
||||
Live pass of the frankbook walk itself is deferred to the next deploy with
|
||||
an awake host.
|
||||
- **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
|
||||
left `frank_small_unit.bats` and the deployer's payload gate stale** —
|
||||
left `frank_secondary_unit.bats` and the deployer's payload gate stale** —
|
||||
the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
|
||||
`LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
|
||||
loads (the anti-drift seam doing its job). Both repointed to the
|
||||
Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
|
||||
- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win
|
||||
- **frank-secondary: compute-buffer pair doubled to 4096/2048, and the prefill win
|
||||
measured rather than assumed.** Frank asked for the batch and ubatch doubled.
|
||||
The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
|
||||
failed the scratch boot"**, and this shape carries 616 MiB less free space
|
||||
@@ -135,7 +135,7 @@
|
||||
`FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
|
||||
boot **byte-identically** (21,276 used / 2,849 free), which is what makes
|
||||
ubatch the only variable. Repo never saw a fake revert commit; scaffolding
|
||||
staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`.
|
||||
staged to `~/.AGENTTRASH/20261005-frank-secondary-ubatch-ab/`.
|
||||
- **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
|
||||
for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
|
||||
noise). The control that proves it is geometry and not drift: a 279-token
|
||||
@@ -158,7 +158,7 @@
|
||||
edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
|
||||
flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
|
||||
second burst and ~90 s). Docs:
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
|
||||
|
||||
## 2026-10-03
|
||||
- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
|
||||
@@ -504,8 +504,8 @@
|
||||
**sparse** file, 3.3 G apparent but 4 K real, so removing it frees
|
||||
nothing.
|
||||
- **A live reference check came first, and it earned its keep.**
|
||||
`serve-frank-small.sh`'s *fallback* `MODEL_DIR` still pointed at the
|
||||
retired GSQ-RCO payload frank-small served on 2026-09-16/17. The env
|
||||
`serve-frank-secondary.sh`'s *fallback* `MODEL_DIR` still pointed at the
|
||||
retired GSQ-RCO payload frank-secondary served on 2026-09-16/17. The env
|
||||
file has always overridden it, so the drift was invisible — but a
|
||||
missing or unreadable env file would have turned a config problem into a
|
||||
"missing asset" naming a model that no longer exists. Repointed at
|
||||
@@ -519,7 +519,7 @@
|
||||
- Pre-existing: **118 G of trash from 2026-09-21** was already staged
|
||||
(`20260921-frank-large-assembled` 88 G, `-redownload` 30 G) and never
|
||||
purged. Total now staged on this host: **627 G**.
|
||||
- Verified after the moves: frank-large and frank-small both `active`,
|
||||
- Verified after the moves: frank-large and frank-secondary both `active`,
|
||||
both `/health` 200, and all three live assets (Swift target, DFlash2
|
||||
drafter, Gemma4 target) still present.
|
||||
- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
|
||||
@@ -735,14 +735,14 @@
|
||||
|
||||
## 2026-09-25
|
||||
- `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
|
||||
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-small → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
|
||||
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-small = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
|
||||
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-secondary → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
|
||||
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-secondary = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
|
||||
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
|
||||
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).
|
||||
|
||||
## 2026-09-22
|
||||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-small-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B).
|
||||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-secondary-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-secondary (Gemma4-12B).
|
||||
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.
|
||||
|
||||
## 2026-09-21
|
||||
@@ -757,7 +757,7 @@
|
||||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
|
||||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
|
||||
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
|
||||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-secondary/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||||
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
|
||||
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
|
||||
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.
|
||||
@@ -766,7 +766,7 @@
|
||||
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
|
||||
- `/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` — New runbook.
|
||||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
|
||||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||||
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||||
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
|
||||
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.
|
||||
|
||||
@@ -775,9 +775,9 @@
|
||||
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.
|
||||
|
||||
## 2026-09-17
|
||||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||||
- `/scripts/frank-small.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||||
- `/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` — New ACTIVE runbook; `frank-small-qwen38-gsq-rco.md` retired as rollback target.
|
||||
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||||
- `/scripts/frank-secondary.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||||
- `/serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md` — New ACTIVE runbook; `frank-secondary-qwen38-gsq-rco.md` retired as rollback target.
|
||||
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
|
||||
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
|
||||
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.
|
||||
@@ -786,10 +786,10 @@
|
||||
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
|
||||
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
|
||||
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
|
||||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||||
- `/scripts/frank-small.env`, `/scripts/frank-small.service`, `/scripts/serve-frank-small.sh` — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-secondary); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||||
- `/scripts/frank-secondary.env`, `/scripts/frank-secondary.service`, `/scripts/serve-frank-secondary.sh` — New frank-secondary: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
|
||||
- `/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
|
||||
- `/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
|
||||
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
|
||||
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
|
||||
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.
|
||||
@@ -828,7 +828,7 @@
|
||||
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
|
||||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
|
||||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||||
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||||
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
|
||||
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
|
||||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
|
||||
@@ -840,7 +840,7 @@
|
||||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
|
||||
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
|
||||
- `/dankclaw/frank-large/quadlet/frank-large.container` — frank-large: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
|
||||
- `/scripts/frank-small.env` — frank-small (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||||
- `/scripts/frank-secondary.env` — frank-secondary (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||||
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
|
||||
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).
|
||||
|
||||
@@ -1076,8 +1076,8 @@ any repo service.
|
||||
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
|
||||
|
||||
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
|
||||
frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
|
||||
frank-secondary, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||||
not on frank-secondary — SGLang reads the MTP head from the TARGET checkpoint
|
||||
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
|
||||
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
|
||||
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
|
||||
@@ -1147,12 +1147,12 @@ bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
|
||||
fetchers moved to `old/`. Runbook
|
||||
[[frank-large-rocmfpx-q8agent-mtp]].
|
||||
|
||||
## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
## 2026-10-02 — frank-secondary: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
|
||||
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
|
||||
with two contexts ("whatever context size fits the 3090"), then **three
|
||||
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
|
||||
the 3090", then **"frank-small should have vision"**.
|
||||
the 3090", then **"frank-secondary should have vision"**.
|
||||
|
||||
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
|
||||
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
|
||||
@@ -1190,7 +1190,7 @@ the 3090", then **"frank-small should have vision"**.
|
||||
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
|
||||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||||
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
|
||||
gufo DFlash2 drafter). Runbook [[frank-secondary-llama-mimo-q5km]].
|
||||
|
||||
## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
|
||||
|
||||
@@ -1260,7 +1260,7 @@ top of the proven-good base, root cause first.
|
||||
- **Harness contract followed the window:** `models.yml` `contextWindow`
|
||||
262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
|
||||
200,704). Side effect: frank-primary's compaction request now fits inside a
|
||||
frank-small slot again, which it had stopped doing when frank-small went to
|
||||
frank-secondary slot again, which it had stopped doing when frank-secondary went to
|
||||
3 x 180,224.
|
||||
- **Comment-only follow-up** `295d578` recorded the measured numbers and was
|
||||
deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
|
||||
|
||||
Reference in new issue
Block a user