-**Frank-Large**: Sampler defaults overridden from halogen/greedy: `temp=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0`. Added `preserve_thinking=true` (retains thinking blocks across turns). Config: `agent/models.yml`. Deployed to `dank/command-center` (dankclaw). Live effect on next config read (no restart).
-**Frank-Small**: Uses checkpoint's fixed Qwen3.6 template. `deploy-plan.py` mapping is vestigial.
- Frank desktop: cursor size 36, capitaine-cursors theme, Authenik 2026.8. Authenik caches claims per session (1-24h TTL) → silent 403s after group add.
- Inference: `handy` app (`/usr/bin/handy`, hub op=start, application=/usr/bin/handy, pty=true, persist=true) with 69 catalog models. FLM NPU gemma4-it on port 7771 — PNG/JPEG only (no WebP), needs `thinkingLevelMap` in models.json, `.xclbin` symlinks under `~/.config/flm/xclbins/`.
-`sudo usermod -aG input blank` was required for `/dev/input` device access (660 root:input).
- Compositor AA in niri 26.04 is limited to corner rounding via `niri_rounding_alpha()`; blur/noise settings cause softness, not AA. Client-side text AA is not controlled by niri.
## Configuration & Repository
- **OMP Harness**: Repo at `~/.omp`. Cloned `serverAppConfig` to `~/Projects/serverAppConfig` (local singular vs remote plural `serverAppConfigs`). HTTP transport with embedded dank token (no SSH keys). HEAD: 701ab14 (main tracks origin/main).
1.**Noctalia theme change**: edit generated palette/theme source before hardcoding per-app colors; terminal text color must match accent in both palette and terminal override. `noctalia msg` for IPC; `git add && git commit && git push` in dotfiles repo.
3.**Ghostty AA fix**: replace `font-hinting = hinting` with `freetype-load-flags = hinting,light`. Reload via USR1: `pgrep ghostty | while read pid; do kill -USR1 "$pid"; done`.
4.**Handy restart**: `pkill -f handy`, then hub daemon start with pty=true, persist=true, ready timeout 10s.
5.**llama.cpp parallelism**: layer (default) = pipeline; tensor (experimental) = true tensor parallelism requiring flash attention + non-quantized KV; row (deprecated); none = single GPU. `--parallel` is serving-level concurrency, not model parallelism. No context/sequence parallelism mode.
-**Device Permissions**: `/dev/input/*` are `root:input` (660). User `blank` must be in `input` group (`sudo usermod -aG input blank`). Pitfall: `usermod` without sudo fails to lock `/etc/passwd`; `sudo` password entry in this CLI environment requires PTY.
-**Ghostty Reload**: `pgrep ghostty | while read pid; do /usr/bin/kill -USR1 "$pid"; done`.
## Pitfalls
-Portrait 180° rotation in niri is 90°↔270° swap, NOT adding 180°.
- Noctalia crispness: no single compositor-AA switch; scoped options only (hard corners; hard corners + no blur/noise; full crispness with opacity 1.0/blur 0/fontconfig AA off; terminal-only).
Context: OMP repo at ~/.omp with serverAppConfig (local) <-> serverAppConfigs (remote). Frank-large uses temp=1.0, preserve_thinking=true; frank-small uses Qwen3.6. FLM NPU (port 7771) needs .xclbin symlinks and WebP conversion. Authenik caches claims (1-24h), watch for post-group-add 403s. Handy daemon runs via hub (pty:true); ensure user 'blank' is in 'input' group for /dev/input access. Git LFS locks are advisory; use USR1 for Ghostty reload. Current shift: noctalia theme & font-hinting fixes.
Frank workstation (user blank) facts and reusable workflows: Noctalia/Niri DE config and display rotation (portrait 180° = 90°↔270° swap), Ghostty/Alacritty text-AA fixes, niri compositor-AA limits, handy/FLM inference constraints, llama.cpp split modes, and dotfile git workflow. Dotfiles repo is ~/.omp (Gitea dank/command-center). Current layout: DP-2@0 → HDMI-A-1@1200 → eDP-1@2400. Repository state wins over this memory.
Ghostty font AA is controlled by `freetype-load-flags`, NOT `font-hinting`.
## Fix
- Replace `font-hinting = hinting` with `freetype-load-flags = hinting,light` in `~/.config/ghostty/config`.
## Reload
- Send USR1 to all ghostty processes: `pgrep ghostty | while read pid; do kill -USR1 "$pid"; done`
## Related
- Alacritty text AA: `font.antialias=false` in its config.
- Noctalia crispness: niri compositor AA only affects corner rounding (`niri_rounding_alpha()`); text AA is client-side. No nearest-neighbor resample knob in this build.
- **layer** (default): Pipeline parallelism. Each GPU holds a contiguous layer slice. Token flows sequentially through GPUs. Minimizes inter-GPU data movement; tolerates slow interconnects (PCIe fine). Best for prefill and fitting large models. Weakness: needs many tokens to keep pipeline full, poor low-batch latency.
- **tensor** (experimental): True tensor parallelism. Splits each layer's weights AND KV cache across all GPUs via 'meta device', doing cross-GPU all-reduce per layer (NCCL). Minimizes latency, parallelizes any workload size. Heavily bottlenecked by interconnect speed (wants NVLink + NCCL). Best for fast token generation.
- KV cache must be non-quantized: f32/f16/bf16 only
- NOT implemented for: MoE/hybrid, Mamba/SSM, and several other architectures (hard-fails with SPLIT_MODE_TENSOR not implemented)
- --fit is disabled; must size --ctx-size / --parallel manually
## Separate axis (NOT model parallelism)
- **--parallel / -np**: Number of concurrent server slots (each with own KV cache). Serving-level concurrency/batch parallelism. On single GPU this is the main throughput knob.
- **CPU+GPU offload** (--n-gpu-layers, --n-cpu-moe): Hybrid inference when model exceeds VRAM. Device-distribution strategy, not 'parallelism' in the strict sense.
## What does NOT exist in llama.cpp
- No first-class context/sequence parallelism mode (unlike vLLM/TensorRT-LLM). If that term is applied to llama.cpp, it's loose third-party language.
## Hardware context (this environment)
- Single RTX 4070 Max-Q (~12 GB VRAM). Multi-GPU split modes do not apply. Relevant knobs: --parallel for concurrency, CPU+GPU offloading for models >12 GB.
-Two lessons from a bulk identity purge (2026-09-28, a sold workstation across serverAppConfigs + dank/command-center): 1) NEVER delegate mechanical prose/file rewrites to a subagent without a full diff audit. Observed failure modes in one run: list markers (- , * ) stripped so bullets stop rendering, numbered lists left with duplicate 3. entries, paths "corrected" to directories that do not exist (system/workstation/), and whole sentences deleted rather than reworded. Subagent edits are cheap to revert (git checkout the touched paths, excluding any files that carried pre-existing WIP) and expensive to trust. Reliable substitute: an exact-string, expected-count-asserted substitution script over a file list (abort on any count mismatch, write nothing for the failing entry), then print every + line and read it. 2) Classification beats blanket search when purging an identity from a config+docs tree. Three buckets: (a) narrative mentions - reword to a class ("the workstation"); (b) LOAD-BEARING mentions - keep, e.g. install.sh resolved packages/$HOST.pkgs from $(hostname), so the manifest FILENAME was the lookup key and the --host case list had to keep the hostname; (c) evidence - keep but de-identify (syslog excerpts keep timestamps and lines, only the host field is generalized). Hardware model strings stayed: subject matter, not identity. Frank's preferences: distill per-host doc directories into general pages rather than renaming them into a graveyard, and exclude auto-learned memory (agent/memories/**) from purges. Also worth knowing: OMP's user persona is ~/.omp/agent/APPEND_SYSTEM.md (not config.yml — that only holds personality: default); ~/.omp IS the live config checkout for the dank/command-center repo, so a branch switch swaps live config under the running agent, and a small live-config fix is best committed on the current branch with only its own file staged. _(context: 2026-09-28: Frank sold the 'the workstation' workstation. Purge run across two repos (serverAppConfigs, dank/command-center at ~/.omp). Trigger for the lesson: I delegated a 17-file mechanical text scrub to a subagent and waited on it; it returned files with dropped list markers, a duplicated numbered item, an invented system/workstation/ path, and a deleted fact sentence. Reverted and redid it as a coun)_
-DeepSeek-V4.1-Flash vs the box it must run on — durable sizing facts (verified 2026-09-28 from the HF API, not model cards): PAYLOAD. deepseek-ai/DeepSeek-V4.1-Flash = **475.27 GiB** across 48 safetensors shards. Of that, **189.13 GiB is Engram** — exactly two 94.56 GiB shards holding layers.{1,14}.engram.embed.weight (196B params, 4-gram lookup, 384,006,168 / 384,016,682 rows). Routed-expert params = 40 layers x 384 experts x 3 x 5120 x 2304 = **543.6 B**, shipped as expert_dtype: fp4 = 269 GiB. Dense ~9.5 GiB. Lucebox's ROCMFP23.gguf = 377.51 GiB (179 GiB experts at 2.5/3.5 bpw + 189 GiB Engram + dense). **The Engram tables are never mapped** (2 table(s) embedded in the GGUF (not mapped)), so they do not evict the expert page cache. THE DECODE WALL ON A 63.7 GiB / 2x-R9700 BOX. A token selects 6x40 = 240 routed experts at ~11.96 MiB each. At the residency a hot stack can hold (37 of 384 per layer = 9.6%) that is **~2.4 GB of expert weights crossing PCIe per token**; PCIe 5.0 x8 does ~55 GB/s real, giving a **~23 tok/s absolute ceiling** — below a 35 tok/s goal before any attention/Engram/dense work. Resident fraction is the whole game. DS41.md's own reference host had 61 GB RAM; ours has 246 GiB, which is the only untested lever. CONTEXT — the three 2026-09-28 candidates: DeepSeek-V4.1-Flash 552B/16B-active (40 layers, 384 top-6, CED 20enc+20dec, CSA2, FP4 KV 890 B/tok, 1M ctx); MiMo-V2.6-Flash-MOPD 309B/15B (48 layers, 39 SWA/9 GA, 256 top-8, **no shared expert**, 5-layer MTP); GLM-5.3-Flash 320B/18B (KDA+DSA hybrid, mHC). Of these only V4.1 is 2.5x too big for this box; MiMo and GLM are smaller but have **no AMD engine at all** (vLLM/SGLang are NVIDIA-first; radiance-vLLM is welded to qwen4_exp). AFFINITY (Codeberg StillDeadcode) PORT SCOPE, from the real safetensors indices of both checkpoints. Its config parser is data-driven and its **expert machinery ports unchanged** — expert tensor naming is IDENTICAL (layers.N.ffn.experts.E.w{1,2,3}.{weight,scale}; 15,360 _(context: Frontier-MoE feasibility study on dankserver 2026-09-28: sizing arithmetic for DeepSeek-V4.1-Flash / MiMo-V2.6-Flash / GLM-5.3-Flash, the Affinity-to-V4.1 port scope from real checkpoint indices, and three stale knowledge-base claims corrected.)_
-Retiring a host from Frank's fleet means a name purge across four surfaces, and the doctrine is "generalize, never delete the lesson": (1) serverAppConfigs/ repo — grep -rniI -c name . (also sweep the machine's LAN IP, NM connection names and wifi SSID, which carry identity even after the hostname goes), retitle per-host doc sections to the CLASS of thing they describe ("## the workstation host DNS" → "## Multi-homed client DNS"), keep the operational content, and update any in-page [[#Anchor]] references the rename breaks; verify with ./tests/run.sh (346 bats seams incl. docs link integrity). (2) ~/.omp/AGENTS.md machine-facts table. (3) ~/.omp/agent/APPEND_SYSTEM.md — this IS the agent persona; the "### Your identity" paragraph names the fleet and the host, and new sessions only pick up edits. (4) ~/.omp/agent/managed-skills/*/SKILL.md and agent/memories/*/. A purge log entry in the work log must name the box neither, or it reprints what was just removed. Leave git history, browser history, and OMP logs/http-400-requests/* (past session payloads) alone unless explicitly asked; a history rewrite needs force-push + re-clones on both servers. TWO operational gotchas: ~/.omp is a LIVE deployment checkout (branch switches swap config under the running agent), so isolate a config fix by staging one path, not by branch-switching — switching back reverts the file you just fixed. And a dirty tree full of peer/agent WIP means git add path, never git add -A. _(context: 2026-09-28, Frank reported the dev workstation (the workstation) sold and asked for a full purge, then "fix the agent persona" after I flagged it. Persona location and the live-checkout hazard were the non-obvious finds; the rename-don't-delete rule is the reusable part for future host retirements.)_
-Never carry line numbers across hashline edit calls: re-read the target range for a fresh tag before every subsequent edit, and never compute offsets from a botched apply's result. Multi-op edits resolve all PUT/CUT indices against the ORIGINAL numbering, and a post-failure diff view elides lines — so a second call built on those numbers duplicates text instead of replacing it. Same rule for the imperative edit tool: assert the whole replaced region in the anchor, re-read to confirm, and repair any duplicated lines immediately. (Context: a hostname purge on serverAppConfigs 2026-09-28 — a two-line edit left a 3-line remnant, and the follow-up PUT 29.=35/60.=63 then pasted 7 fresh lines beside 3 stale ones, duplicating the Proton-gate and deploy-flow paragraphs.) _(context: 2026-09-28 serverAppConfigs doc purge (5 files) — reproduced a line-number-drift failure in the hashline edit tool and repaired it.)_
-R9V radiance-vLLM (dankserver frank-primary): PLE mmap→HIP registration (54.4 GB) can exceed the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured 740 s, right after a bench teardown). Signature: worker log "TimeoutError: PLE offload worker did not become ready" ~16 min into load, then the PLE worker finishes registration AFTER the engine already tore down (BrokenPipeError on the late worker). WORST part: serve-frank-primary.sh exits 0 on docker wait, so systemd logs "Deactivated successfully" with NO error in the journal — a silent prod failure that reads like a clean stop. Fixed in production 2026-09-28: R9V_PLE_READY_TIMEOUT=1800 in scripts/frank-primary.env, forwarded by fork launch.sh 2de0357 (local-only). Any future R9V engine that waits on a registration/registration-like step: check for a hidden readiness timeout env in vendor/vllm/vllm/envs.py before debugging "clean" deactivations. _(context: frank-primary 2026-09-28: bench matrix E1/E3 init failures + a failed 09:46 production boot all traced to the 600 s PLE readiness timeout)_
-R9V radiance-vllm-qwen4exp (dankserver frank-primary, Q8_0) — verified 2026-09-28: (1) PLE registration (54.4 GB mmap - HIP) can exceed the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT (default, vendor/vllm/vllm/envs.py) under page-cache churn — measured 740 s right after a bench teardown. Signature: container dies ~19 min in, unit logs "Deactivated successfully" with ZERO error in journal (serve-frank-primary.sh exits 0 on docker wait), endpoint just never serves. Prod fix: R9V_PLE_READY_TIMEOUT=1800 in scripts/frank-primary.env, forwarded by launch.sh fork commit 2de0357 (local-only). (2) Bench matrix laws for the q8 shape: prefill chunk 4096 is the ceiling (16384 boots but 0.93 GiB VRAM free - requests hang/OOM; 8192 works: PP8K +25.9% but decode -8.5%); expert cache 80 slots is the POLICY max (doctor: "cache prepare backend requires capacities (48, 64, 80) with synchronous single-fill LRU" — async unsupported, 96/112 also fail expert-budget [108,198] rank maxima); MTP depth 6 is 3.7x SLOWER than depth 4 (454-489 vs 128-159 ms/step, tokens/step barely grows); unqualified shapes fail MID-REQUEST (health stays 200, stream dies). (3) Prefix caching (R9V_ENABLE_PREFIX_CACHING=1): zero VRAM cost (hits reuse freed KV blocks), cold path unchanged, reuse hits 32k 44.2-3.6-4.3 s (9x), 131k 439-6.9 s (63x); hit pattern rounds 1-2 MISS, round 3+ hit (identical-prompt and conversation-growth). Recommended for prod, pending Frank's go-ahead (q8 state opts out with =0; launcher default is ON). (4) deploy.sh refuses to run from non-main branches ("deploy.sh must run from the main branch") — ff main to the working branch and push before deploying. _(context: frank-primary bench matrix + 9h downtime incident + PLE timeout boot failure, dankserver 2026-09-28; repo commit f187165, runbook serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md "Prefill / decode bench matrix" section)_
-frank-primary Q8_0 cutover (2026-09-27, commits e5a06f5-era trio + docs): (1) The R9V launcher now carries an opt-in chat-template passthrough — R9V_CHAT_TEMPLATE_FILE mounts file:/opt/r9v/share/chat-template.jinja:ro and passes --chat-template (wins over the tokenizer dir's metadata/chat_template.jinja); R9V_CHAT_TEMPLATE_KWARGS maps to --default-chat-template-kwargs. Value with braces in the env file gets the single-quote law (bare double quotes are stripped by bash assignment parsing). (2) Behavioral template test: request reasoning_effort "high" — the embedded template 500s ("Unexpected reasoning effort"), the sharp template folds high→xhigh → 200. (3) Probe gotcha: bare docker on dankserver is NOT the shim (PATH-dependent) — docker logs/inspect silently return nothing for podman-rootless containers; use podman logs (as dank) or export PATH with .podman-shim first. A green health probe plus "no such object" from docker inspect is the shim-missing signature, NOT the false-green stray law. (4) vLLM MTP acceptance lives in APIServer logs: SpecDecoding metrics: Mean acceptance length: N (metrics.py:120, ~every 10 s); measured 2.88–4.50 live. (5) systemd unmask returned success but LEFT the runtime mask symlink (/run/systemd/system/unit - /dev/null) in place on the relinked unit — verify with ls -la /run/systemd/system/unit after unmask, remove by mv if still present. (6) vLLM engine logs do NOT reach docker/podman logs when the launcher runs the engine detached inside the container; APIServer lines do — grep those for spec metrics. _(context: frank-primary q8 cutover on dankserver 2026-09-27; launcher commit 30d4a32 (fork local), repo commit + docs pushed)_
-Why Q8_0 decoded ~11 tok/s vs uncensored IQ4_XS 59–96 tok/s on radiance-vllm dual-R9700 (verified 2026-09-27, not guessed): decode on this runtime is expert-bandwidth-bound; every token activates the same top-8×48-block expert set, so per-token expert BYTES decide speed. (1) Q8_0 doubles bytes/expert (1.0625 vs ~0.5625 B/param; 92.39→175.30 GiB payload). (2) Frozen-config budgets then HALVE, because the tiered residency budget is bytes-driven over fixed ~32 GiB cards: R9V_MAX_EFFECTIVE_EXPERTS_PER_RANK 222,428→108,198, R9V_TIERED_EXPERT_CACHE_SLOTS 160→80, R9V_MIN_FREE_VRAM_GIB_BY_RANK 1.5→3.0, PLE table 28.8 GB (residency ssd, 16 GiB pinned reserve)→54.4 GB (residency pinned). (3) Live uncensored residency that buys 59–96 tok/s: rank0 hot=4.36/cold_UVA=36.03/dynamic_cache=10.91 GiB/155 slots; rank1 hot=16.18/cold_UVA=4.24 GiB (~31 GiB expert working set in VRAM). Q8_0 cannot afford that shape. (4) The 2× bytes became 5–9× slowdown via the residency cliff: below the hot-set size the LRU hit rate collapses non-linearly and misses stream from pinned host RAM over PCIe (same law as the x4-pair experiment: residency fraction is the whole game). (5) Probe caveat: the 11.0 tok/s figure was ~the 3rd request of the container's life (80-slot LRU cold, reasoning tokens + TTFT included, single 256-tok shot) — a floor, not a steady-state number; measuring true steady state needs a bench window: stop prod, relaunch from /home/dank/r9v-state/q8xl/START-COMMAND.txt, warm the cache, run the cutover battery harness. Diagnostic template for any future quant comparison on this fork: diff the two state-dirs' setup.json config blobs (the R9V_* keys above are the ones that matter) and grep 'Tiered GGUF experts ready' from each runtime's logs for hot/cold_UVA/dynamic_cache per rank. Evidence: fork docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md, vault log.md 928fd81. _(context: Frank asked "why is decode so much slower?" after the Q8_0 qualification window on dankserver (2026-09-27). Answered from frozen-config diffs (r9v-state/q8xl vs uncensored setup.json) + live 'Tiered GGUF experts ready' log lines; no bench run needed to establish the mechanism.)_
- Dankserver frank-primary experiment-window restore posture (proven 2026-09-27): (1) systemctl mask REFUSES a unit whose /etc/systemd/system file is the deployer's relink symlink into the repo checkout ("File already exists and is a symlink") — use systemctl mask --runtime unit instead: it shadows via /run/systemd/system (verified /run/systemd/system/frank-primary.service → /dev/null), leaves the relink untouched, and SELF-CLEARS at reboot, which is exactly the wanted window semantics (reboot returns the repo-configured llama.cpp). (2) Restore a frozen launch.sh-based vLLM deployment by dumping its state-dir setup.json config to an env file with export K=V lines, overriding the deltas (R9V_HOST_PORT, R9V_CONTAINER_NAME, R9V_HOST_BIND=0.0.0.0 — the base profile default is loopback-only, which breaks models.yml's LAN target), then R9V_PROFILE=base profile.env R9V_CONFIG_FILE=env file bash scripts/launch.sh — launch.sh's doctor (PASS=24 FAIL=0) then binds exactly like the incumbent. (3) vLLM containers hang on SIGTERM (60 s timeout → SIGKILL needed, same law as llama.cpp's SIGTERM cleanup hang). (4) launch.sh refuses to run when a stale exited container holds the name — docker rm it first. Evidence: health 200 in ~60 s (warm page cache), LAN-path completion, qualification record radiance-vllm-qwen4exp docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md (commit 50a79ec), vault log.md 928fd81. _(context: Q8_0 bring-up qualification window on dankserver (2026-09-27): q8 served + battery on :7773, then uncensored UD-IQ4_XS restored to the frank-primary :7770 slot; frank-primary.service masked for the window.)_
- radiance-vllm-qwen4exp Q8_0 bring-up on dankserver (2026-09-27, fork commits 363d606+2fe6dbb, container r9v-qwen38-q8, port 7773): (1) The plugin's fused-HC uniformity check assumes the PUBLISHED IQ4 layout — injection shard (blk.N.hc_attn_inject → attn_hyper_connection.block_inject_weight) is F32 while down shard is Q8_0. Unsloth's Q8_0 conversion quantizes BOTH shards → ValueError "Detected some but not all shards of ...input_mix_weight_down_block_inject are quantized" at WorkerProc init (~112 s). Fix: dev-overlay qwen4_exp.py (env R9V_DEV_QWEN4_EXP_ADAPTER_PY, mount block in launch.sh) extends dequantize_qwen4_exp_packed_hc_down to dequantize a quantized inject shard in-stream + marks the module unquantized; numerics identical to file conversion, F32 path unchanged. (2) runtime.json env.always force-injects R9V_FULL_MUTABLE_CACHE=1, whose baseline check demands the QUALIFIED 160-slot uncensored geometry; q8 profile runs 80 slots → launch.sh gained a passthrough (profile-set R9V_FULL_MUTABLE_CACHE appended AFTER runtime-overlay envs, docker last-wins); q8 profile sets 0 → plain tiered path. (3) GOTCHAS: setup_profile.py start uses the FROZEN setup.json config (+os.environ), NOT profile.env — profile.env edits are inert at start time; pass R9V_PROFILE=q8 profile.env explicitly (launch.sh line 6 sources it). launch.sh sources the DEFAULT (dual-r9700) profile when R9V_PROFILE is unset. A stale exited container makes launch.sh exit 1 "Container already exists" — docker rm -f first. Without the podman-shim PATH, real docker answers "No such image". Tolerance note: Q8_0 roundtrip error ~0.03/step (gguf.dequantize of quantize'd (N,K) returns (N,K); quantize returns (N,K+20)). Profile: dual-r9700-mtp4-q8, PLE pinned 54.4 GB (preflight 108.8 required vs 140.9 available GiB), model load 21.84 GiB/158 s per worker, compile then serve ~16 min total cold. Verified: /health 200, /v1/models qwen3.8-flash-next, completions return content + reasoning_content (17×23=391). _(context: Qwen3.8-Flash-Next Q8_0 GGUF launch failure root-cause session on dank@192.168.0.235, fork repo /home/dank/GitPackages/radiance-vllm-qwen4exp, state /home/dank/r9v-state/q8xl/)_
-Verified model-path facts for the dankserver 2xR9700 fleet (2026-09-27, primary-source checked — supersedes any scout claims): (1) DeepSeek V4.1-Flash (released 2026-09-10): 552B backbone, 8B active prefill / 16B active decode, CED (Causal Encoder-Decoder) arch, 1M ctx, MIT, multimodal. NOT the April preview (DeepSeek-V4-Flash 284B/13B, ggml-org/llama.cpp#22319 — early vLLM/SGLang successes were Hopper/Blackwell only). BEST path = Lucebox (github.com/Luce-Org/lucebox): gfx1201 is a first-class tested platform (ROCm 7.2), payload Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF live on HF (24.9k downloads), heterogeneous R9700+Strix Halo measured 86 tok/s decode / 788 tok/s prefill @2K, and the fleet already ran this stack (frank-primary-lucebox-ds4 runbook) until the 2026-09-26 Qwen cutover — retired for the swap, not for failure. llama.cpp mainline CED support: unverified as of late Sept 2026; vLLM/R9V fork: qwen4exp-only, not a path. (2) GLM-5.3-Flash (zai-org): 320B/18B active, KDA+DSA hybrid attention + mHC, 1M ctx (default — hard-hung a 128 GB box once, always bind -c), native FP8 328 GB, MIT. llama.cpp arch glm5next lives in unslothai/llama.cpp branch glm5next/upstream = ggml-org PR #27754, NOT merged to mainline as of 2026-08-30 (check PR state before any build). Measured: 9.31 tok/s decode on Strix Halo UD-IQ1_S 93.1 GB, RADV Vulkan, no MTP (MTP landed d07e71ede 2026-08-30, unbenched); -fa off is MANDATORY (garbage output otherwise). Quants: unsloth ladder IQ1_S 93.1 GB (70.9% top-1) to UD-Q4_K_XL 199.7 GB (92.2%); GSQ-RCO 3.5-bit 137.07 GB (MMLU-Pro -1.40 pp vs Q8_0 ref 61.95%). On discrete R9700s GLM's ~7.9 GiB/token expert volume rides the exps=CPU law to ~4-5 tok/s [ESTIMATE] — #2 for dankserver, but a strong frank-large (Strix Halo) candidate. (3) Process law: LLM research-scout drafts carried invented citations twice in a row (fake repos, wrong PRs, impossible versions like "vLLM v0.21" in a 0.26 era); before any report ships, verify every load-bearing URL di _(context: Session 2026-09-27: research scout (R9700ModelResearch) returned two consecutive drafts with fabricated/incorrect sources ("MiaAI-Lab", "freenode", wrong PR, vLLM v0.21 claim); direct primary-source verification (GitHub issue 22319, HF modelcards, julianmb/halo-glm53flash receipts, pfeifferj GSQ-RCO card) superseded it. HTML report delivered at /tmp/r9700-ds41-glm53-report.html (volatile path — f)_
-R9V qwen4exp Q8_0+MTP4 profile bring-up (2026-09-27, dankserver): all artifacts staged and validated, bring-up blocked ONLY on GPU headroom (production uncensored holds 29.9/30.8 GiB per R9700; rank1 has 1.59 GiB free vs 3 GiB requested — a second instance cannot coexist; qualification needs a stop window for frank-primary). Staged state: model dir /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V (6 sha256-verified shards, 188.2 GB); PLE derived /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/r9v-data/per_layer_token_embd.iq4_nl.bin = 54,400,261,120 B sha256 91743e33b6de2272b50a6269a6fe692f56244946fdecf19ca1eba0ba640b303f; placement manifest validated (hot 28/198, rank0 static 4.56 GB / rank1 17.37 GB vs proven budgets 4.68/17.41 GB, MAX_EFFECTIVE=108,198); profile.env + bootstrap.json wrapper written; setup command ready: ./r9v setup qwen38-q8-mtp4 --state-dir /home/dank/r9v-state/q8xl --model-dir /home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V --image sha256:2dac17a215fb5b0e3461e4c3e36a2981eec8ac3d6021e73183d247e819740c03 --local-image --reuse-from /home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS --accept-model-license. Bring-up bugs fixed along the way (durable): (1) GGUF expert tensor names split into 4 parts ("blk.0.ffn_up_exps.weight" - parts[0]="blk", layer=parts[1]) — the 3-part assumption crashed with int(''); (2) expert_budget.expert_memory(data, slots, cache_ranks, asynchronous=False, *, catalog=None, runtime=None) — runtime/catalog are KEYWORD-ONLY; (3) bootstrap.json wrapper canonical schema = exactly {schema, id, model_package, hardware, artifact_path, artifact_sha256, scope} — hardware "dual-r9700-32gb-asymmetric", scope "Unranked routing bootstrap only; does not certify fit or headroom", and wrapper id MUST equal the id declared in profile.json's placement descriptor or setup rejects with "does not match profile placement"; (4) nohup X & inside an ssh command races ssh teardown and can kill the script before it writes a byte — run remote long scri _(context: Frank ordered "modify the q4k xl profile for q8 with mtp" + "ensure ple always stays in ram, not disk" for the R9V fork on dankserver; pipeline authored and run to completion while production serves uncensored on :7770; scratch bring-up awaits a GPU window.)_
-R9700 (gfx1201) model research — VERIFIED facts (2026-09-27, all URL-checked directly, NOT from the scout whose citations were partly fabricated): (1) DeepSeek V4.1-Flash released 2026-09-10: 552B backbone params, 8B active prefill / 16B active decode, Causal Encoder-Decoder (CED) arch, 1M ctx, multimodal, MIT (HF deepseek-ai/DeepSeek-V4.1-Flash model card). The April V4-Flash preview was 284B/13B (issue ggml-org/llama.cpp#22319, opened 2026-04-24; preview weights are FP4+FP8 mixed precision, community GGUF conversion initially produced garbage, dequant+requant IQ4 ~153 GB). (2) Lucebox = first-class R9700 path: Luce-Org/lucebox (2,883 stars) lists RDNA4 gfx1201 + ROCm 7.2 as a tested platform; heterogeneous "R9700 + Strix Halo" DeepSeek V4 measured 86 tok/s decode / 788 tok/s prefill@2K; Qwen3.8-27B UD-IQ4_XS + DFlash2 on one R9700 = 208.1 tok/s; HF org carries DeepSeek-V4.1-Flash-ROCMFP23-GGUF (the exact family frank-primary served 2026-09-26 per serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md) and DeepSeek-V4-Flash-0731-ROCmFP3. (3) GLM-5.3-Flash (zai-org): 320B MoE / 18B active, hybrid KDA+DSA attention + mHC, 1M ctx default, native FP8 checkpoint 328.3 GB. llama.cpp arch = glm5next, support lives in unslothai/llama.cpp branch glm5next/upstream = ggml-org PR #27754, NOT merged into mainline as of 2026-08-30; quant-side PR #27773 cited by the GSQ-RCO card. MTP/NextN landed upstream d07e71ede 2026-08-30 (glm5next.cpp:846 implements DECODER_MTP; earlier stub asserted at glm5next.cpp:690). -fa off is MANDATORY (flash-attn breaks MLA correctness → garbage); default 1,048,576 ctx must be explicitly bounded (hard-hung a 128 GB Strix Halo once). Only measured receipt: Strix Halo gfx1151 RADV Vulkan, UD-IQ1_S 93.1 GB → 9.31 tok/s decode / 19.7 prefill, 16 threads, no MTP (julianmb/halo-glm53flash, 2026-08-28). Quant ladder: unsloth UD-IQ1_S 93.1 GB (70.9% top-1) → UD-Q4_K_XL 199.7 GB (92.2%); GSQ-RCO (pfeifferj, IST-DASLab methods; taurusduan = mirror): 3.5-bit 13 _(context: Research turn 2026-09-27: scout output was partly fabricated (invented sources, wrong vLLM version); direct URL verification produced the reliable evidence set below. The HTML report on R9700 paths for DeepSeek 4.1 + GLM 5.3 Flash is still pending (bg_31 pipeline check in flight).)_
-frank-primary on dankserver (192.168.0.235:7770) is now the R9V radiance-vLLM fork (supersedes the 2026-09-26 llama.cpp Q8_0 entry): drwolfen/radiance-vllm-qwen4exp, docker-pinned image bundle r9v-qwen38-flash-next-wmma@sha256:2dac17a2 (loaded from a containerd/OCI docker-save — podman keys images by config digest but keeps the manifest digest in RepoDigests), serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + blockfp8 MTP4 depth 4 + F16 mmproj, TP=2 tiered experts (UVA pinned host copy 43 GiB rank-0 + 16 GiB PLE pinned reserve; PLE residency ssd-mmap; budgets 146.25/78.75 GiB by device), CED on, MAX_MODEL_LEN=196608, tool parser qwen3_coder, reasoning channel reasoning. Production shape: repo-tracked ROOT unit scripts/frank-primary.service (User=dank, LimitMEMLOCK=infinity — but measured fact: the full tiered load reached healthy under a PLAIN dank session, so the pin path does not hard-require it on this host) + serve-frank-primary.sh which layers the on-host 104-key setup state (/home/dank/r9v-state/uncensored/setup.json — the tuned shape lives ON-HOST by design, repo env carries only bind/port/alias/container overrides) under scripts/frank-primary.env, execs the fork's OWN scripts/launch.sh through the docker→podman shim, then docker waits — launch.sh --detach alone would make a crashed engine a green unit. Docker→podman shim laws (in .podman-shim/docker, PATH-prepend, logs every call): (1) containerd-store ID law (inspect sha256:X answers X from RepoDigests; bare sha256:X run refs rewritten to repo@sha256:X); (2) LogConfig law (podman inspect answers LogConfig.Config:null where docker returns the log-opts map — the R9V doctor CRASHES on None.get; shim records run --log-opt pairs and normalizes inspect output); (3) docker info law (SecurityOptions/ServerVersion 27.5.1/DockerRootDir). R9V tooling gotchas: ./r9v start reads state from --state-dir (or default ~/.local/state/r9v/profiles/id) — passing --state-dir to setup but not to star _(context: frank-primary R9V radiance cutover on dankserver 2026-09-27, commits 095128b/7562e41/ca6ba6d; scratch-start debugging (state-dir mismatch, LogConfig crash, flag-consumer bug) all fixed in-shim or by rerun; production deploy green with MainPID check; e2e 32/33 with sleeping FrankBook the only failure.)_
-frank-primary (dankserver :7770, Qwen3.8-Flash-Next Q8_0 MoE): measured 2026-09-26 with AMD IBS on the live server that per-BLOCK expert read volume is FLAT (0.93–1.08× mean across all 40 CPU-resident blocks) — MoE top-8 routing fires every block every token, so "most active layers" do not exist at -ot granularity; ANY 8 GPU-resident blocks are equivalent and the only lever is COUNT (already at the 5/3 VRAM ceiling). Real skew is per-EXPERT (top 10% of 48×512 (block,expert) cells carry 40.5% of samples) — below tensor-override reach. Instrument: scripts/frank-primary-expert-activity.py (commit f41e038) — perf IBS op sampling + /proc/pid/maps attribution + GGUF header parsing. Critical implementation law: derive GGUF tensor extents from CONSECUTIVE header offsets, never from a quant type-size table — a wrong per-element size silently mis-buckets each tensor's tail into junk (first run lost 59% of expert samples to a 'gap' bucket with zero warning). Runbook: serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md "Which blocks, measured". _(context: Frank asked whether a tensor-override regex could target the most active layers of frank-primary's experts; measured live instead of guessing. Runbook updated + script committed f41e038; no config change (block choice proven immaterial).)_
-llama.cpp b11030 --override-tensor laws (proven live on frank-primary, dankserver, 2026-09-26): (1) The loader applies the FIRST matching pattern per tensor — a broad pattern like exps=CPU placed before block-specific GPU patterns SILENTLY swallows them (zero warnings in the log; buffers stay byte-identical to no-override). Specific patterns must precede the broad sweep. (2) Repeated --override-tensor flags log DEPRECATED ("only last value will be used" — misleading; actually all apply with first-match-wins); the canonical form is ONE flag with the full comma-joined list. (3) GPU-target overrides ARE legal under a layer split when the tensor sits on the card that holds that block's layers (both ROCm devices are in the main graph's backend list) — the x4-pair "=CPU only" wall applies to draft-graph borrows and non-primary devices, not main-graph co-located blocks. (4) Granularity: the whole per-block expert tensor (512 experts share one tensor, 2.42-2.49 GiB at Q8_0) is the atomic placement unit; there is no per-expert routing knob. (5) Env-file pitfall: the value with parentheses MUST be single-quoted in the env file — source parses values, unquoted ( is a bash syntax error. (6) Probe procedure: FRANK_PRIMARY_ENV_FILE=/tmp/probe.env + scratch PORT, warm loads ~10-30 s; verify placement via load_tensors: ROCmN model buffer size lines (exact arithmetic: base + N×2550 MiB) and headroom via rocm-smi bytes; total card = 34,208,743,424 B (31.86 GiB, not 32). ssh+zsh layers mangle sed backslash escapes and treat bare === as =cmd expansion — use heredocs. _(context: frank-primary 4-slot/V-q8/block-fill reshape on dankserver 2026-09-26, commits de85654/d317d6b/d67768e)_
-llama.cpp --override-tensor (b11030 unsloth build, verified 2026-09-26 on frank-primary): the loader applies the FIRST matching pattern per tensor — a broad exps=CPU listed before block-specific GPU patterns silently swallows them (all experts stayed CPU, zero warnings anywhere in the log; only the per-card "model buffer size" load-log lines revealed it). Order: specific patterns first, broad sweeps last. Repeated --override-tensor flags also log DEPRECATED (its "only last value will be used" text is misleading — matching is first-match-wins across the joined list); pass ONE comma-joined flag. Placement verification is buffer-size arithmetic, never absence-of-errors. Related measured fact: --cache-type-v q8_0 does NOT apply to the shared-MTP drafter's KV (stays f16 — draft pool = f16 law at the target pool size). Fleet workflow corollary: on the zsh-login hosts, write regex-bearing remote env files via heredoc cat file (sed piped through ssh+zsh layers eats backslash escapes), never use bare === as an echo marker (zsh =cmd expansion aborts the command), and quote globs inside remote grep patterns. _(context: frank-primary 4-slot/V-q8/hot-block reshape on dankserver 2026-09-26; commits de85654 (shape) + e330d56 (runbook law); the broad-first probe silently no-op'd the GPU block placement)_
-Dankserver fleet delta (2026-09-26 cutover, supersedes the frank-medium entry): frank-medium (vLLM/radiance/Swift-Qwen3.8-27B-PARO-MXFP6) is RETIRED — the active Dankserver endpoint is now **frank-primary**: unsloth llama.cpp b11030 prebuilt (root systemd unit, enabled+active), unsloth/Qwen3.8-Flash-Next-GGUF Q8_0 (175.3 GiB, 6 split parts, mmap — never --no-mmap) with shared-Q8_0 MTP drafter (depth 2) + ngram-mod, BF16 mmproj, 262,144 ctx × 1 slot, port 7770 @ 192.168.0.235, alias frank-primary, -ot exps=CPU,per_layer_token_embd.weight=CPU (experts+PLE computed in host RAM), -ts 1,1 across both R9700s, **SPEC_DRAFT_DEVICE=ROCm1** (last-layers card — ROCm0 crash-loops with the ggml-backend.cpp:941 borrowed output.weight abort). Measured: 11.8 t/s decode, 0.86 tool-task draft acceptance, ~7 s warm load, 9.4 GiB/card (29%/36% of 32 GiB), 29,696 B/token ctx-scaled KV law. KV is f16 (no --cache-type flags). Rollback: lucebox quadlet + 378 GiB payload retained on host (~/.AGENTTRASH/20260926-llamacpp-cutover/). Runbook: serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md. All three endpoints now export /metrics (frank-large was silently 501 since its own deploy; fixed 82ff9dd). Repo HEAD de4d227. _(context: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 cutover on dankserver 2026-09-26 (commits 8500600, a9b94a7, 5d632bb, 82ff9dd, de4d227))_
-llama.cpp b11030 (unsloth prebuilt family): (1) /metrics is DISABLED by default — without --metrics it answers 501, and Prometheus marks the target down while /health looks fine; both dankclaw frank-large and the new frank-primary had this silently (fixed 2026-09-26, commit 82ff9dd). (2) A spec-draft deployment that borrows target tensors (shared MTP head) must place the drafter on the card holding the target's LAST layers (-ts 1,1 → output.weight lands on ROCm1): with tensor overrides set, n_copies==1 and a draft context elsewhere aborts at load with ggml-backend.cpp:941 "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run the operation (NONE)" — crash-looped 4× at ~12 s until SPEC_DRAFT_DEVICE=ROCm1. (3) llama-server's SIGTERM cleanup can HANG after spec-decode service (logs "cleaning up before exit", never exits) — a timeout 300 probe survived its own kill signal and later blocked a deploy's port gate; kill -9 is the reliable stop for stray instances. (4) A deployer whose health check is a bare curl against the port cannot tell WHO answers — a stray process holding the port produces a FALSE GREEN deploy (seen 2026-09-26 when the deploy's wait_for_health was satisfied by a leftover probe while the real unit was stopped); after any aborted deploy, confirm the responder is the unit's own MainPID before trusting the health gate. _(context: frank-primary llama.cpp cutover on dankserver 2026-09-26; commits 5d632bb (draft device), 82ff9dd (metrics + -lv 4), de4d227 (KV law))_
-When debugging a crash-looping LLM service by running the serve script manually with a scratch env file (e.g. FRANK_PRIMARY_ENV_FILE=/tmp/probe.env), the probe binds the service port and can satisfy a subsequent deploy's health gate: deploy.sh exited green with "frank-primary healthy after ~104s" while systemctl showed inactive — the health curl hit my probe process, not the unit. Rule: before redeploying after a manual probe, kill the probe (pgrep llama-server on the host) and let the deployer's own unit answer; treat a green deploy that ran concurrently with a manual probe as unverified. Related: a wait_port_free failure on a lone LAST-ACK tail (dead client socket, e.g. 192.168.0.60 not ACKing FIN) is not a held listener — it often clears on its own in ~90 s; sudo ss -K sport = :port errors with RTNETLINK Invalid argument when only the tail remains, but by the next run the port is free — just rerun the deploy rather than editing the gate. _(context: frank-primary deploy 2026-09-26: LAST-ACK tail aborted deploy run 1; probe-A held :7770 while deploy run 2 went falsely green; unit was actually stopped)_
-On a llama.cpp layer-split (-ts) deployment that carries --override-tensor flags AND a borrowed-head speculative drafter (draft-mtp shared head, e.g. Qwen3.8-Flash-Next's shared MTP GGUF), the drafter MUST sit on the card where the main model's output head landed — the last-layers card under a balanced -ts. Reason: -ot sets has_tensor_overrides(), which pins n_copies==1 in the draft's backend scheduler; a borrowed output.weight pre-allocated in card B's buffer aborts the draft graph with ggml-backend.cpp:941 "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run the operation (NONE)" when the draft context is on card A. Signature: SIGABRT ~12 s into load (right after "loading draft model"), crash-looping every restart. Verified live 2026-09-26 on frank-primary (dankserver 2x R9700, -ts 1,1, -ot exps=CPU,per_layer_token_embd.weight=CPU): SPEC_DRAFT_DEVICE=ROCm0 abort-looped 4x, ROCm1 loaded and served. This extends the fleet's x4-pair law (overrides ⇒ n_copies==1) into spec-decode territory, and matches the frank-medium layersplit precedent (SPEC_DRAFT_DEVICE=ROCm1) — check the repo's retired-era env files for precedent before pinning a device. _(context: frank-primary cutover 2026-09-26: deploy attempt 1 crash-looped; fixed by 5d632bb "pin the drafter to ROCm1")_
-GGUF pre-deploy verification (Dank fleet, frank-primary 2026-09-26): (1) A GGUF's KV metadata + tensor-info table sit at the file START, so a partially-downloaded split part already exposes its tensor names — scripts/dump-gguf.py (repo-tracked, dependency-free) can validate -ot override patterns mid-fetch, before the payload completes. This confirmed exps=CPU matches all of blk.N.ffn_{up,gate,down}_exps.weight on Qwen3.8-Flash-Next. (2) Parser gotchas: GGUF v3 string/array lengths are u64; KV value-type enum is 8=string, 9=array; nested array element types include enum 5 (i32) — handle unknown array types with a size-table skip ({0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:0,9:0,10:8,11:8,12:8} bytes/elem), don't raise. (3) The unsloth Qwen3.8-Flash-Next embedded tokenizer.chat_template (dumped from part 1's KV) natively folds high→xhigh before the xhigh/medium/low gate (only minimal/max raise) AND natively consumes preserve_thinking — so frank-primary needs NO external --chat-template-file; the models.yml reasoningEffortMap (minimal→low, max→xhigh) exists only for the two OMP-only levels. Pre-reading the embedded template from the GGUF before deploy prevents an entire deploy-cycle loss on a 500-ing template rejection. _(context: frank-primary cutover 2026-09-26: verified -ot tensor patterns and chat-template effort gate from the partially-fetched payload before deployment)_
-frank-medium (dankserver :7770) now serves hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, W4A8, 25.16 GiB) on image localhost/radiance-vllm:0.13.0-e060895 (PR #51) — the repo quadlet (dankserver/frank-medium/quadlet/frank-medium.container) is the source of truth; the AGENTS.md prose (MXFP4 / radiance-vllm 0.9.3) is STALE. Boot-crash lesson (2026-09-26): vLLM's startup check requires free VRAM = gpu_memory_utilization x 31.86 GiB per card, and free-at-check includes the worker's own HIP context plus the amdgpu per-boot baseline (~690 MiB total), which drifts tens of MiB between boots — so GPU_UTIL=0.98 (threshold 31.22 GiB) deterministically crash-looped a boot that measured 31.21 GiB free (ValueError "Free memory on device cuda:N ... less than desired GPU memory utilization"), identical numbers on every restart. Fixed by GPU_UTIL=0.97 (threshold 31.06 GiB, ~160 MiB headroom, cost ~5k tokens on the auto-sized ~486k-token KV pool, commit c5b55e1). If frank-medium crash-loops after a reboot with that ValueError, lower GPU_UTIL further rather than hunting for a GPU tenant — the GPUs are otherwise idle.
-serverAppConfigs e2e gate: the admin crawl test (tests/e2e/specs/11-admin.spec.ts:24) walks every admin open link, including Prowlarr at 192.168.0.112:8989 (the FrankBook-Pro Mac mini). When the mini is asleep (ARP STALE, 100% ping loss; it's a Mac mini with NO remote WoL/power-on path) that test fails with net::ERR_CONNECTION_TIMED_OUT — a pre-existing environmental failure, not a regression. For changes that don't touch the mini, deploy with --skip-e2e (justified) and note the dependency. The user crawl (12-crawl.spec.ts) is also slow (~1.2m) and can pass on retry. Every non-Mac-mini e2e test passes while the mini is down.
- Fleet model input modalities: models.yml (/home/blank/.omp/agent/models.yml) is the authoritative source, NOT the AGENTS.md prose (partly stale — it claims "no audio anywhere in the fleet"). Ground truth: frank-large = text only (no vision projector); frank-medium = text+image (Gemma4 vision path, verified on held-out images); frank-small= text+image+audio. Audio nuance: frank-small's models.yml entry declares only input: [text, image] because OMP 18.0.6's schema rejects 'audio' (a failed schema check drops the WHOLE models file), but the endpoint genuinely accepts input_audio parts via the Gemma4 projector's audio encoder (verified 2026-09-17 deploy — a WAV transcribes verbatim). No fleet model does video. _(context: serverAppConfig 2026-09-26: adding per-model input modalities to the homepage LLM cards)_
- serverAppConfig homepage (shared/homepage/): the three LLM status cards use a structured details object in services.json (fields role, spec, inputs[], use) instead of the plain description string the other ~32 cards use. app.js descHTML(svc) renders details as tiered divs — .desc-role (bold, --head pale green #aaffcc), .desc-spec (--dim), .desc-inputs (--input #00ff9d spring green, 'Inputs:' prefix, modalities joined with ' · '), .desc-use (--dim, 'Use:' prefix) — and falls back to esc(svc.description) when details is absent. Tiers are deliberately staggered light-dark-light-dark (pale role / dim spec / bright inputs / dim use). The /docs model guide (docs.html #models) mirrors it with Model/Capacity/Inputs/Best-for/Not-for bullets and was the source of a stale-info bug (it had drifted to retired model names/sizes — verify against models.yml/AGENTS.md before trusting it). Deploy: ./scripts/deploy.sh dankclaw --service homepage --skip-e2e (ships to /srv/homepage; e2e skipped because admin crawl test 11-admin.spec.ts:24 times out on the asleep Mac mini 192.168.0.112, which has no remote wake). Verify with bats tests/homepage_*.bats tests/docs_completeness.bats + a local browser render (python3 -m http.server + a scratch status.json, since file:// fetch is CORS-blocked; move the scratch status.json to .AGENTTRASH, never rm). _(context: serverAppConfig 2026-09-26: reworked the LLM model cards (structured role/spec/inputs/use, neutral roles, per-model input modalities, staggered green tones))_
-frank-small (Dankclaw :7771, Gemma4-12B QAT) DOES accept audio input (input_audio WAV transcribes verbatim, verified at the 2026-09-17 deploy) via the Gemma4 projector's audio encoder — even though the harness models.yml only declares input: [text, image] for it. The reason is a harness schema limit, not a model limit: OMP 18.0.6's models.yml schema only accepts input: "text"|"image", and a failed schema check drops the WHOLE models file, so audio is deliberately omitted from the list while the endpoint still accepts audio parts. Consequence: the AGENTS.md prose "There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11" is STALE — frank-small is the fleet's only audio-capable endpoint. When routing an audio task or fact-checking the docs, trust the models.yml comment + the Gemma4 projector over the AGENTS.md one-liner. (frank-large = text only, no vision projector; frank-medium = text+image via the Gemma4 merge path; none do video.)
-Fleet model input modalities (authoritative source: /home/blank/.omp/agent/models.yml input: field): frank-large = text only (no vision projector); frank-medium = text + image (Gemma4 vision path, verified on held-out images); frank-small = text + image + audio. IMPORTANT correction: frank-small's audio is NOT in models.yml's input list because OMP 18.0.6's schema only accepts text|image (a failed schema check drops the WHOLE models file), but the Gemma4-12B-QAT projector's audio encoder DOES accept input_audio parts — verified at the 2026-09-17 deploy (an input_audio WAV transcribed verbatim). So the AGENTS.md line "There is no audio input anywhere in the fleet" is STALE/wrong for frank-small: audio left with the FLM NPU endpoint for the other models, but the Gemma4 swap brought it back on frank-small specifically. _(context: serverAppConfigs 2026-09-26: adding per-model input modalities to the homepage LLM cards; cross-checked models.yml vs the stale AGENTS.md audio claim)_
-serverAppConfigs homepage LLM cards: the three LLM status cards (frank-large/medium/small) in shared/homepage/services.json carry a details object {role, spec, inputs, use} rendered by app.js descHTML() as a 4-tier hierarchy — bold pale-green .desc-role / bright-green .desc-spec / bright-green .desc-inputs ("Inputs: a · b") / dim .desc-use ("Use: …") — falling back to the plain description string for all other cards. The desc-* CSS classes live in style.css and each needs a bare-mode (html[data-bare=1]) reset. Deploy with ./scripts/deploy.sh dankclaw --service homepage (the deploy_homepage handler copies the 6 assets to /srv/homepage, regenerates deployed.js, restarts the status timer). The /docs model guide is the #models section of shared/homepage/docs.html and must be kept in sync with the cards — it once drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8 ~31 GiB) and had to be corrected to Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8 19.4 GiB. _(context: serverAppConfigs 2026-09-26: reworked the homepage LLM model cards into structured role/spec/inputs/use lines)_
-serverAppConfigs homepage (shared/homepage/): the LLM status cards use a structured details object in services.json — {role, spec, use} — rendered by app.js descHTML() as a bold pale-green role line, a bright-green spec line, and a dim "Use:" line (.desc-role/.desc-spec/.desc-use in style.css, plus a bare-mode reset). descHTML() falls back to the plain description string for any card lacking details (the other ~32 services). So: structured card = add a details object; plain card = add description. The e2e (10-homepage/11-admin) asserts only on .panel .name and a.open, NOT .desc content, and homepage_config.bats' schema test checks only required fields (name/category/section/probe) — so details is a safe optional extra field. Gotcha: the /docs model guide (docs.html) is a SEPARATE hand-maintained copy of the model facts and had drifted to retired hardware (it listed frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k and frank-medium as FP8 ~31 GiB, both stale); re-verify it against AGENTS.md ground truth whenever the models change. _(context: serverAppConfigs 2026-09-25: reworked the three LLM homepage cards from a single dense description string to a structured role/spec/use shape (details object + app.js descHTML + new CSS), neutralized the role labels, and fixed the drifted /docs model guide.)_
-Jellyfin 12.1 removed the legacy POST /Users/Auth/Password route (and its LegacyEmbyRouteRewriteMiddleware). Probing that dead route returns a SILENT 401 with ZERO plugin/UserManager log lines even at Debug level — it reads like an LDAP/auth failure but is just a dead route. The modern password-auth endpoint is POST /Users/AuthenticateByName, which REQUIRES the header Authorization: MediaBrowser Client="...", DeviceId="...", Device="...", Version="..." (no header → 401). The LDAP-Auth plugin runs on this modern path. So a "Jellyfin LDAP 401 after upgrade" that shows no plugin log is almost always a test hitting the removed legacy route, not a broken integration — verify with the modern endpoint + MediaBrowser header before touching plugin config. (Dankserver Jellyfin 12.1.0, 2026-09-25; alex authenticated HTTP 200 via AuthenticateByName with full plugin flow: service bind → search → ldapUid → user bind → token.) _(context: Dankserver Jellyfin 12.1.0 LDAP-Auth 401 diagnosis — the 401 was a test artifact from probing the removed legacy /Users/Auth/Password route; the web UI's modern /Users/AuthenticateByName endpoint works fine.)_
-Deploying a quadlet image change on the Dank fleet (dankclaw 192.168.0.143 / dankserver 192.168.0.235): pulling the repo does NOT update the running container, because the installed unit in ~/.config/containers/systemd/name.container is GENERATED state, not the repo file. A plain systemctl --user restart name restarts from the STALE installed unit (old image) — I hit this live (image IDs didn't change after a bare restart). To deploy a pinned-image bump: commit+push → git pull --ff-only on host → cp host/svc/quadlet/name.container ~/.config/containers/systemd/ → systemctl --user daemon-reload → systemctl --user restart name → verify podman inspect name --format {{.ImageID}} matches the freshly-pulled image. For local-build images (caddy/alloy/loki/comfyui, tagged :latest) no unit reinstall is needed (same tag), BUT deploy-remote.sh's build_compose_image gates on a cfhash label (sha256 of the Containerfile) and SKIPS the rebuild when the Containerfile is unchanged — so a base-image refresh needs a manual podman build --no-cache -t image -f containerfile . first. Gotcha: the hermes gateway container is named systemd-hermes, not hermes (no ContainerName= in its quadlet), so podman inspect hermes fails — use podman ps --filter name=hermes. anubis is pinned at v1.27.0 (latest stable; only pre-releases beyond). _(context: serverAppConfigs 2026-09-25: fleet-wide container update (pinned bumps + local-build rebuilds) across dankclaw + dankserver)_
-Updating a container image on the Dank fleet (dankclaw/dankserver, rootless quadlet) — a naive git pull + systemctl --user restart silently keeps the OLD image, for two distinct reasons: (1) QUADLET INSTALLED-UNIT STALENESS: the repo quadlet file (e.g. dankserver/plex/quadlet/plex.container) is the source of truth, but the INSTALLED unit at ~/.config/containers/systemd/name.container is generated state that git pull does NOT touch. Bumping Image= in the repo, pulling, and restarting restarts the container from the STALE installed unit (old image). Fix: cp repo/host/svc/quadlet/svc.container ~/.config/containers/systemd/ + systemctl --user daemon-reload + systemctl --user restart svc. Verify by comparing podman inspect c --format {{.ImageID}} against the freshly-pulled image ID — if they differ, the unit was stale. (2) LOCAL-BUILD CFHASH GATE: the deploy functions (deploy_caddy, build_compose_image in scripts/deploy-remote.sh) rebuild only when the Containerfile's sha256 (the cfhash image label) changed. A base-image bump (caddy:2, grafana/loki:latest, grafana/alloy:latest, the rocm/vllm comfyui base) leaves the Containerfile byte-identical, so the gate SKIPS the rebuild and the old base stays. To force a refresh: podman build --no-cache -t image -f containerfile . from the repo root (caddy=franksdank/caddy-porkbun:latest, alloy/loki=franksdank/{alloy,loki}:latest — loki needs --format docker for its HEALTHCHECK, comfyui=comfyui-rocm:gfx1151). hermes (image localhost/hermes-agent:latest, Pull=never): podman pull docker.io/nousresearch/hermes-agent:latest + podman tag ... localhost/hermes-agent:latest + restart the trio (hermes, hermes-dashboard, hermes-webui). Gotcha: the hermes GATEWAY container is named systemd-hermes (not hermes), so podman inspect hermes fails. anubis is pinned at v1.27.0 in deploy-remote.sh (latest stable; GHCR tag-list API is capped at 1000 unsorted tags — use the GitHub releases API to find the latest). Excluded from "update to latest": frank- _(context: serverAppConfigs 2026-09-25 fleet-wide container update: pinned bumps (authentik 2026.8.3, tautulli ls245, calibre-web ls402, plex 1.43.4) + local rebuilds (caddy, alloy, loki, comfyui-rocm, hermes x3). Hit the installed-unit staleness trap on dankserver (restarted with old image IDs until the units were re-copied + daemon-reloaded).)_
- Dankclaw guard-chained vhost 400 diagnosis (Anubis → Authentik embedded outpost, e.g. marreta.franksdank.cloud): a 400 where the APP's own logs show NO user request (only periodic health probes) is the embedded outpost's handle_auth_callback returning 400 with the warn "auth callback without a valid session cookie" (src/outpost/proxy/application/handlers/mod.rs). The outpost sets a session cookie on the external host during auth_start; the callback must carry it back. If the browser (observed: Firefox) doesn't send it — private/incognito window, cookie blocking (ETP Strict / extension), or stale site data — the callback 400s and the app never sees the request. The cookie name is 'authentik_proxy_' + first 8 hex of SHA256(provider.client_id) (compute: hashlib.sha256(client_id).hexdigest()[:8]); it is SameSite=Lax, Secure, HttpOnly, host-only when provider.cookie_domain is empty — and the callback is same-site (auth.→vhost., same registrable domain) so Lax allows it, meaning the server is fine. Confirm server health by running the single-vhost e2e spec (clean Chromium drives the full chain; a pass means the cause is browser-side). Two server-side gotchas that ALSO produce the same 400: provider.access_token_validity unset/None → session_max_age()=0 → zero-max-age cookie the browser discards; and cookie_domain set to a domain that doesn't match the callback host. ProxyProvider is at authentik.providers.proxy.models; on this fleet the marreta provider is pk 13 named 'marreta-proxy' (NOT 'marreta'). The outpost runs inside the authentik-server container (logs via journalctl --user -u authentik-server), reached as remote = the Anubis container IP. _(context: serverAppConfigs 2026-09-25: user reported "400 from marreta"; traced to the embedded outpost's handle_auth_callback missing-session-cookie 400, verified server healthy via e2e spec 16-marreta (3/3 pass), root cause = Firefox not sending the authentik_proxy_6536c831 session cookie on the callback.)_
-In authentik (2026.8.0, Dankclaw), self-serve password change is a PROMPT FIELD, not a stage. The UserWriteStage writes whatever the preceding PromptStage collected: for key password it calls user.set_password(value) (stages/user_write/stage.py:116). So the fix is a password field on the default-user-settings prompt (field_key='password', type='password', required=False) — NOT a PasswordStage. A PasswordStage is an AUTHENTICATION stage: its validate_password runs authenticate() against the backends to check the CURRENT password; it never sets a new one. (A PasswordStage was added then removed for exactly this reason on 2026-09-25.) Two gotchas: (1) a blank password field must be guarded — set_password('') leaves the account with no password, so bind an ExpressionPolicy to the prompt via PromptStageValidationPolicy that denies when prompt_data['password']=='' (allow when set/absent); (2) a prompt field is a Prompt row linked through PromptStageField(prompt_stage, prompt) — the order comes from Prompt.order, and there is no PromptField model. Verify the write path by driving UserWriteStageView.update_user on a throwaway user (it only modifies in-memory; call user.save() after, as dispatch() does). The self-serve flow /if/flow/default-user-settings-flow/ is open to ANY authenticated user (its authorization policy only gates email/name/username changes, not password); the admin Directory→Users screen is separate and admin-only.
-authentik 2026.x default-user-settings-flow (the self-serve account page, https://auth.domain/if/flow/default-user-settings-flow/) ships with ONLY a PromptStage + a UserWriteStage (name/email/locale) — NO PasswordStage, so users cannot change their password. To add it: create a PasswordStage (backend authentik.core.auth.InbuiltBackend, the "standard password" backend) and bind it to the flow. Two ORM gotchas (verified on dankclaw authentik-server 2026.8.0): (1) Flow.stages.add(stage) FAILS with "Can't bulk create a multi-table inherited model" because the through model FlowStageBinding is multi-table-inherited — instead create the row directly: FlowStageBinding.objects.create(target=flow, stage=ps, order=N) where N is max(existing orders)+1; (2) PasswordStage.objects.create(name=...) alone hits a NOT NULL violation on the backends ArrayField — you must pass backends=[...]. Valid backend values come from authentik.stages.password.__init__: BACKEND_INBUILT="authentik.core.auth.InbuiltBackend", BACKEND_LDAP, BACKEND_APP_PASSWORD (TokenBackend), BACKEND_KERBEROS. Flow execution order is the FlowStageBinding.order field, NOT the M2M insertion order (flow.stages.all() returns a different order — don't trust it). To verify a stage isin the executed plan WITHOUT a browser: FlowPlanner(flow).plan(request) (request.user set) → plan.bindings lists the stages in order; FlowPlanner.__init__ takes only the flow, user/request go to .plan(request). Note: a headless browser will NOT render authentik flow stages (the Lit app renders via XHR after the shell, leaving only "Powered by authentik") — verify via FlowPlanner, not the DOM.
-Running a single Playwright spec via the fleet e2e runner requires the file path to come BEFORE --project: tests/e2e/run.sh specs/18-openwebui-sso.spec.ts --project standard. The reverse order (--project standard specs/...) fails with Project(s) "specs/..." not found because playwright parses the positional file path as a project name. Spec 18 is in the "standard" project (testMatch /(10|12|13|16|17|18)-.*\.spec\.ts/), which depends on the "setup" project (00-auth.setup.ts) that logs in and writes .auth/qa-e2e.json + .auth/qa-e2e-admin.json. The runner needs tests/e2e/.env.local (git-ignored) present and auto-installs chromium. This lets you verify one SSO/guard spec against the live fleet without running the whole (slow, Mac-mini-dependent) suite. _(context: serverAppConfigs 2026-09-24: ran 18-openwebui-sso.spec.ts in isolation to verify the SSO-only flip against the live fleet)_
-Flipping OpenWebUI (dankclaw, ai.franksdank.cloud) to SSO-only requires changing BOTH the quadlet env AND the per-key config DB, because the config DB (webui.db config table) wins over env vars. Concretely: (1) add Environment=ENABLE_LOGIN_FORM=false and Environment=ENABLE_PASSWORD_AUTH=false to dankclaw/open-webui/quadlet/open-webui.container; (2) set ui.enable_login_form to false in the config table on the live data dir (/home/dank/Containers/open-webui/data/webui.db). If you only set the env, the DB override ui.enable_login_form=true silently keeps the password form showing. ui.enable_signup was already false. Verified 2026-09-24: the login view at /auth renders zero inputs and only the "Continue with authentik" button, and the OIDC round-trip lands signed in. To restore the password form: set ui.enable_login_form back to true (Admin Panel or DB) and drop the two env vars. Note the dankclaw host has NO sqlite3 CLI — use python3 -c "import sqlite3; ...". _(context: serverAppConfigs 2026-09-24: OpenWebUI SSO-only flip (commit 33cdeca) + config DB override flip)_
- OpenWebUI (dankclaw, ai.franksdank.cloud, quadlet on bridge 10.89.3.0/24, published 3005) can do authentik OIDC SSO per the official integration docs, with these fleet-specific adaptations: (1) OpenWebUI's container cannot reach authentik-server:9000 by name (isolated bridge), but CAN reach https://auth.franksdank.cloud — Caddy publishes 0.0.0.0:443 and netavark DNATs published ports from other containers — so OPENID_PROVIDER_URL uses the public URL; (2) authentik runs 2026.8.0 (≥2026.5), so the provider needs the new typed Strict Authorization redirect URI (https://ai.franksdank.cloud/oauth/oidc/callback), not the old plain list; (3) OAUTH_CLIENT_SECRET must not enter the repo quadlet — use EnvironmentFile= pointing at a git-ignored host env file (Caddy-env pattern); (4) OpenWebUI persists config in /app/backend/data fill-missing-only, so on the existing data dir the OAuth settings must be set via Admin Panel → Settings, env vars only seed a fresh data dir; (5) rollout: enable SSO first, verify, then flip ENABLE_LOGIN_FORM/ENABLE_PASSWORD_AUTH=false — flipping first locks you out of the admin panel if the wiring is wrong. _(context: serverAppConfigs 2026-09-24: answering whether the goauthentik.io OpenWebUI integration docs apply to the fleet setup)_
-When a subagent is told to write its report to local://name.md, the file lands in the GLOBAL dir /home/blank/.omp/local/name.md — NOT in the session-local dir (/home/blank/.omp/agent/sessions/session/local/). In the 2026-09-22 serverAppConfigs session, three subagent reports written to local:// all resolved to the global dir; a first read attempt at the session-local path failed and needed a find /home/blank/.omp -name file to locate them. Durable rule: read subagent local:// outputs from /home/blank/.omp/local/ directly; if not found there, find under /home/blank/.omp. _(context: serverAppConfigs session 2026-09-22: subagent outputs (appdocs-ai, research-job-search, research-popular-ai-apps) all landed in the global dir, not the session-local one)_
-ComfyUI v0.37.0 (released 2026-09-21, tag v0.37.0) ships NATIVE Qwen-Image 2.1 support (CORE-423, PR #16400 — comfy/ldm/qwen_image21/model.py, model detection + supported-models registration). The Dankclaw ComfyUI image (comfyui-rocm:gfx1151, built from dankclaw/Containerfile.comfyui) currently bakes in CUSTOM GGUF patches (apply_qwen21_patch.py: VAE 4D→5D expansion, in_channels shape[0] read, QwenImage21 arch registration) for the Qwen-Image 2.1 GGUF setup. On the next image rebuild, check whether the native 0.37.0 path makes those patches redundant BEFORE re-applying them. Also: the repo now lives in the Comfy-Org org (comfyanonymous/ComfyUI redirects). _(context: serverAppConfigs vault fact-check 2026-09-22, AppDocsAI upstream research)_
-frank-medium image rebuilds on dankserver: build.sh invokes docker build (rootful docker), but the quadlet runs under ROOTLESS podman — a freshly built image is NOT visible to podman. Bridge it: docker save img:tag | podman load (lands as docker.io/name:tag), then podman tag name:tag localhost/name:tag because rootless quadlets cannot resolve registry-prefixed refs. The 2026-09-22 MXFP6 image was built this way as ggz14/vllm-radiance-mxfp4:0.13.0-e060895 → localhost/radiance-vllm:0.13.0-e060895. The /patches clone must be at the same commit as the image (e060895) so the entrypoint re-applies the matching patches.
-Diagnosing frank-medium (radiance) "restart loop" false alarms: the journal's repeating lines every ~10 s — [radiance.gemm]/[radiance.paroquant]/[radiance.mxfp4]/[radiance.paroquant_mxfp6] registered plus a NOOP file already applied line — are NOT a crash loop. They are the entrypoint's ~20 sequential python3 patch_*.py invocations, each re-importing sitecustomize (which prints the radiance banners) and printing its own NOOP line in FORWARD order, then exec vllm. Proof it's not looping: the container reaches Application startup complete and podman inspect shows Started=t Restarts=0 with a stable Started timestamp. A real crash loop is the RuntimeError: Engine core initialization failed / ValueError: Free memory … less than desired GPU memory utilization traceback (a real prior failure: the 0.99 GPU_UTIL crash). Distinguishing: grep for Traceback|ValueError|Engine core initialization failed (real failure) vs the registration/NOOP cadence (normal slow patch phase, ~3 min).
-HF download verification gotcha: the resolve endpoint's X-Linked-ETag differs between authenticated (Bearer token) and unauthenticated requests for the SAME LFS file. The actual downloaded file's sha256 matches the UNAUTHENTICATED etag. So scripts/hf-manifest.py (which does Bearer-token HEAD) and hf-fetch.sh's sha256 check produce a systematic false-positive "mismatch" for this repo (e.g. hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6). This is HF API behavior, not a script bug or a stale manifest. To verify a download, compare local sha256 against an UNAUTHENTICATED curl -sI https://huggingface.co/repo/resolve/main/file X-Linked-ETag (strip CRLF with tr -d '\r'). The 2026-09-22 MXFP6 download was verified correct this way (PASS=19 FAIL=0).
-frank-medium (dankserver, dual R9700 TP=2): vLLM's startup memory check (gpu_worker.py request_memory) runs BEFORE weight load and requires gpu_memory_utilization × total free-at-device-init. On the 32 GB R9700 the reported total is 31.86 GiB but only 31.23 GiB is free at device-init — the 0.63 GiB gap is the ROCm/HIP context + NCCL + radiance's custom all-reduce buffers (84 MB fine-grained AR + 44 MB AR_QUANT). So the max that starts is 0.98 (0.98×31.86=31.22 31.23); 0.99 (31.54) fails with "ValueError: Free memory on device cuda:0 (31.23/31.86 GiB) on startup is less than desired GPU memory utilization" and crash-loops. 0.98 is the ceiling AND the prior production value. The repo fit-gate test does NOT model this device-init baseline (it only checks GPU_UTIL×total vs weights+KV), so it wrongly passed 0.99. If startup failures recur on reboot the lever is a further drop (0.975).
-The integration of ComfyUI as the image generation engine in OpenWebUI is fully verified. The workflow is set, the engine is active, and the 'red cube' prompt successfully generated the expected visual output.
-OpenWebUI successfully generates images using the ComfyUI engine and the Qwen-Image 2.1 workflow. The 'red cube' prompt produced a 1024x1024 image with a red center, confirming the end-to-end integration is complete.
- The 4 missing models for ComfyUI have been downloaded to the correct directories in the dankclaw host. This completes the model download task for the Qwen-Image 2.1 and Qwen3VL models.
-Testing the /api/v1/generate endpoint with the 'comfyui' model to verify if it's the correct endpoint for producing images in the current OpenWebUI version.
-Confirmed that setting 'image_generation.engine' to 'comfyui' and 'image_generation.model' to 'comfyui' correctly wires the two. The workflow is the most important part, defined as a JSON string in the SQLite config.
-Final verification of OpenWebUI image generation with ComfyUI as the engine using the 'red cube' workflow. The endpoint is confirmed to be responsive once the 'image_generation.model' is explicitly set to 'comfyui'.
-The ComfyUI integration is verified by successfully setting the workflow and engine in the OpenWebUI config. The next step is to confirm the visual output via a real request to the /api/v1/generate (or similar) endpoint to ensure the 'Method Not Allowed' error was resolved.
-Setting 'image_generation.model' to 'comfyui' ensures that even if 'image_generation.engine' is 'comfyui', the model picker and internal logic have an explicit model identifier to refer to, preventing potential ambiguity in the SQLite config.
-Successfully integrated ComfyUI as the image generation engine for OpenWebUI. The workflow for a "red cube on a white background" was successfully loaded via the SQLite config table. The engine was set to 'comfyui' and the base URL was set to 'http://host.containers.internal:8188'.
-OpenWebUI's /api/images/generate returned 'Method Not Allowed', suggesting the endpoint might be /api/v1/generate or requires a different structure. The 'image_generation.model' was empty, which might also affect the resolution.
-The 4 missing models for ComfyUI were: qwen_image_2.1_bf16.safetensors, qwen_image_2.1_int8_convrot.safetensors, qwen3vl_8b_bf16.safetensors, and qwen3vl_8b_w4a8.safetensors. These were identified as the missing pieces for the Qwen-Image 2.1 and Qwen3VL models on Dankclaw.
-When 'image_generation.engine' is set to 'comfyui', the 'image_generation.model' key can be left empty (""), but 'image_generation.comfyui.workflow' must be a JSON string containing the full workflow dictionary. In the current setup, 'image_generation.comfyui.base_url' should point to 'http://host.containers.internal:8188'.
-ComfyUI Qwen-Image 2.1 (qwen_image21) setup gotchas, verified 2026-09-21: (1) The VAE MUST be qwen_image_2.1_vae_bf16.safetensors (z_dim=64, 5D video VAE with time_conv). The Comfy-Org qwen_image_vae.safetensors (z_dim=16) is INCOMPATIBLE — the model's QwenImage21 latent format has latent_channels=64, so a z_dim=16 VAE causes "expected input to have 16 channels, but got 64" in VAEDecode. Both are video VAEs (5D convs); the 4D→5D expansion patch in comfy/sd.py is needed for both. (2) The model detection in_channels must come from img_in.weight.shape[0] (=64, the latent input dim), NOT shape[1] (=4096, the hidden dim). The GGUF stores the weight transposed relative to PyTorch's [out_features, in_features] convention — the model definition has in_channels=64, inner_dim=4096, so the GGUF tensor [64, 4096] is [in, out]. (3) Three patches are required for an unknown GGUF arch: general.architecture set in the GGUF file, the arch string in IMG_ARCH_LIST (loader.py line 12), and a detection block in model_detection.py. All three are baked into the image via apply_qwen21_patch.py run by comfyui-start.sh. _(context: ComfyUI Qwen-Image 2.1 Q8 GGUF setup on Dankclaw, 2026-09-21. After fixing the VAE 4D→5D patch and the in_channels shape, the remaining failure was using the wrong VAE file (Comfy-Org z_dim=16 vs repo z_dim=64).)_
-frank-large (dankclaw :7770, Strix Halo gfx1151 iGPU) was cut 2026-09-21 to stock llama.cpp (unsloth b11030 prebuilt, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.4 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) + q8_0 KV + 262144 ctx / 1 slot + text-only + Qwen-Sharp v22.4.0 template. Verified live: health ok, answer "Paris" + reasoning_content, prompt 177 tokens (template wired, NOT the 73,600-token peg-native fallback), DFlash2 draft acceptance 0.85 (short easy prompt) / 0.519 (mean len 3.08, longer varied "explain a socket" prompt), warm decode ~17 t/s, GTT 57 GiB of 114 GiB. Durable facts: (1) unsloth b11030 does NOT support fp8 KV — allowed types are f32,f16,bf16,q8_0,q4_0,q4_1,iq4_nl,q5_0,q5_1; q8_0 is the highest-precision 8-bit option (~8 GiB at 262k). (2) --ngl (double-dash) is INVALID in b11030 — use --n-gpu-layers or -ngl. (3) curl -C - resume corrupted a 31.5 GiB HF download (213 MB oversized); fresh re-download (no resume) is the safe fix. (4) The 8-bit UD-Q8_K_XL decodes ~17 t/s vs the prior 4-bit Kairic Edge IU4 at ~22 t/s — the ~23% larger weight size (31.4 vs 26 GiB) explains the gap, it is not a regression. (5) DFlash2 draft acceptance is text-variety-dependent: 0.85 on a trivial one-word answer, 0.519 on a varied explanatory prompt — do not read a single-request acceptance as the steady-state. _(context: serverAppConfigs frank-large swap to stock llama.cpp UD-Q8_K_XL + DFlash2, 2026-09-21, commits 7689092 + 600812d)_
-ComfyUI Qwen-Image 2.1 GGUF setup (dankclaw, 2026-09-21): Three non-obvious issues required patches baked into the image via a startup wrapper (apply_qwen21_patch.py). (1) VAE/model latent channel mismatch: the model's QwenImage21 latent format has latent_channels=64, but the Comfy-Org VAE (qwen_image_vae.safetensors) is a video VAE with z_dim=16 — the repo's VAE (qwen_image_2.1_vae_bf16.safetensors) has z_dim=64 and is the correct one. Both are video VAEs (5D convs, time_conv); only the repo's matches the model's latent channel count. (2) in_channels shape: GGUF stores linear layer weights as [in_features, out_features], not [out_features, in_features] like PyTorch — model detection must read shape[0], not shape[1]. (3) 4D→5D VAE expansion: the model produces 4D image latents but the video VAE expects 5D; fix is elif self.latent_dim == 3 and samples_in.ndim == 4: samples_in = samples_in.unsqueeze(2) in comfy/sd.py decode(). Also: Q8_0 quantized 1D tensors have raw data shape ≠ logical tensor shape; dequantize all 1D tensors unconditionally in the GGUF node loader.
-llama.cpp's --chat-template-kwargs alone does NOT load a custom template — you must also pass --chat-template-file (or --chat-template) for the binary to use it. Without --chat-template-file, the binary falls back to the built-in template (logged as "Chat format: peg-native"), which can produce a wildly different prompt size than the custom template. On the Kairic Edge build (ciru-ai/ROCmFPX, gfx1151), the built-in peg-native template produced a ~73,600-token prompt for a ~250-token request; the Qwen-Sharp v22.4.0 template (wired via --chat-template-file) produces a 62-175-token prompt for the same request. The "Chat format: peg-native" log line is the format-type label, NOT the template indicator — the custom template is active even when that line appears. The serve script must pass --chat-template-file explicitly in its ARGS array; setting LLAMA_ARG_CHAT_TEMPLATE_FILE as an env var before invoking the script is insufficient because the script constructs the binary command line explicitly and does not forward that env var.
-In serverAppConfigs, adding a new docs runbook (e.g. serverAppConfigs-docs/LLM/foo.md) requires linking it in TWO indexes or the "docs index links every docs page" bats test (tests/docs_completeness.bats:24) fails: (1) the section index (serverAppConfigs-docs/LLM/index.md), and (2) the TOP-LEVEL serverAppConfigs-docs/index.md — the test greps ONLY the top-level index for [[basename]]. Adding the link to just the LLM index is a false sense of completion; the test still fails. When a runtime is swapped same-day (old ACTIVE → new ACTIVE), also mark the old runbook RETIRED in both indexes and add it to the AGENTS.md retired-runbook table so the chain stays documented. _(context: serverAppConfigs, 2026-09-21: created frank-large-kairic-edge-qwen38-27b.md runbook, added it to the LLM index, but tests/run.sh failed on "docs index links every docs page" because the top-level index.md also needed the link.)_
- frank-large (dankclaw :7770, Strix Halo iGPU gfx1151) was swapped 2026-09-21 from the unsloth llama.cpp prebuilt (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE, 87 GiB) to the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B DENSE, IU4 W4A4). Key facts: (1) DFlash is NOT available for this model — the Kairic Edge runtime's spec-type constants are DRAFT_EAGLE, DRAFT_MTP, DRAFT_SIMPLE, NGRAM_* (no DRAFT_DFLASH); the spec method is native MTP depth 4 + n-gram drafting (24/64/64). (2) The fast-greedy path rejects sampling, penalties, probabilities, grammar-constrained tool calls, logit bias, LoRA and a reasoning budget; the harness needs all three, so COMPATIBILITY_MODE=1 (9.7% slower per the build guide). (3) The payload is 4 files (15.5 GiB GGUF + 7.99 GiB FFN + 1.88 GiB GDN + 0.70 GiB GDN-Output .pfs sidecars), sha256-verified. (4) The runtime is built from source (scripts/build-kairic-edge-gfx1151.sh), TheRock 7.15.0a20260718 toolchain, CK fdf4bb7f + gfx1151-iu4 patch. (5) Reasoning is enabled (auto + deepseek format, thoughts land in reasoning_content). (6) The Qwen3.8 template gates reasoning_effort to {xhigh, medium, low}; the harness folds levels via models.yml reasoningEffortMap. (7) The 27B dense model fits the ~114 GiB GTT aperture easily (vs the 125B MoE's 87 GiB). Rollback: the unsloth prebuilt is still on disk at /home/dank/GitPackages/unsloth-llama-b11030/ and the model at /home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/.
-vllm-radiance 0.9.3 (stilldeadcode/vllm-radiance + ggz14/radiance-vllm-mxfp4) is RDNA-4 (gfx1201) end to end — NOT portable to Strix Halo (gfx1151 / RDNA 3.5) as published. Confirmed live (frank-medium image torch arch list = ['gfx1201']; no radiance image on the Strix Halo host — the only gfx1151 image there is the stock ComfyUI base, a different vLLM 0.16.0) and by primary source (Dockerfile ARG GFX_ARCH=gfx1201; README "Compiled for gfx1201 only, won't run on other GPUs. Two GPUs (TP=2) is the only configuration tested"). THE HARD BLOCKER is the MXFP4 W4A8 GEMM: it is built on __builtin_amdgcn_wmma_f32_16x16x16_fp8_fp8_w32_gfx12 (fp8 WMMA), which clang gates on gfx12-insts (RDNA 4); there is NO fp8 WMMA for gfx11-insts (RDNA 3.5). The kernel's own comment: "The only way to the 2x is v_wmma_f32_16x16x16_fp8_fp8 by hand." So recompiling it for gfx1151 won't work — you'd dequantize weights to f16/int8 and use the f16/int8 WMMA, losing the exact fp8 speedup that makes MXFP4 fast. Component status for gfx1151: (a) image gfx1201-only; (b) aiter unified-attention Triton path DOES support gfx1151 (explicit tuning: waves_per_eu=8 if arch=="gfx1151", "tuned for the Qwen3-Omni"; gfx1151 is in RDNA_ARCHS) but fp8 ATTENTION is excluded (FP8_ARCHS={gfx942,gfx950,gfx1200,gfx1201} omits gfx1151) → bf16 attention; (c) GDN fused kernels use bf16/int8 WMMA (have gfx11 forms) → port with a source change; (d) R4D one-shot allreduce is a multi-GPU collective, moot at TP=1 (single Strix Halo). To port: gfx1151 image rebuild (build is parameterized by GFX_ARCH, possible in principle, never published) + patch aiter's gate (radiance enables it for on_gfx12x() only) + bf16 KV/attn + rebuild GDN/R4D kernels with gfx11 WMMA + replace the W4A8 GEMM (lose the 2x). "Work well" is separate from "run at all": even ported, the W4A8 GEMM + attention are bandwidth/compute-bound and Strix Halo's LPDDR5X is far slower than the R9700's HBM3 (RDNA 3.5 is a gen behind RDNA 4); Strix Halo's advantage is m_(context: serverAppConfigs research 2026-09-21: "does radiance-vllm optimizations also work well for strix halo, or just r9700s?" — live probes (torch arch list, aiter JIT, image inventory) + librarian primary-source research (StillDeadcode/vllm-radiance Dockerfile/README, ROCm/aiter v0.1.17, StillDeadcode/libr4d, ggz14/radiance-vllm-mxfp4, clang BuiltinsAMDGPU.def).)_
-OpenWebUI (dankclaw) LLM wiring — three non-obvious facts, verified 2026-09-21: (1) The model picker is built from the model TABLE in webui.db ONLY (Models.search_models, filtered on base_model_id != None), NOT live-merged with connection-discovered models — a fresh data volume shows whatever rows are seeded, and connection discovery alone will not populate the picker. (2) llama.cpp serves /v1/models in Ollama shape ({"models":[{"name":...}]}); OpenWebUI's extract_data() only reads the OpenAI {"data":[{"id":...}]} shape, so connection discovery SILENTLY misses llama.cpp endpoints (frank-large :7770, frank-small :7771) → 500/404 'Model not found'. Fix: set model_ids in each connection's api_config (when non-empty, OpenWebUI builds the model list directly and never fetches /v1/models); also set base_model_id = the model name on each model row so the picker filter passes. vLLM (frank-medium) serves the OpenAI shape and would discover fine. (3) The open-webui container sits on a netavark BRIDGE; it can hairpin to a DIFFERENT host's LAN IP (dankserver 192.168.0.235) but NOT to its own host's non-published ports via the host LAN IP (192.168.0.143:7770/7771 → connection refused). host.containers.internal (169.254.1.2) resolves on the bridge and reaches both on-host llama.cpp services — this CORRECTS the stale OpenWebUI.md gotcha that claimed host.containers.internal is 'slirp4netns-only and does not resolve on a bridge' (verified end-to-end: all 3 models returned completions). The model table + connection configs are DB-only (not git-managed); a fresh volume loses them. Hermes surfaces: web UI client = hermes.franksdank.cloud (:8787), agent management dashboard = hermesadmin.franksdank.cloud (:9119); Pi-hole (replaced the retired dnsmasq quadlet 2026-08-24) admin = pihole.franksdank.cloud (:8083, 403 basicauth counts as up).
-serverAppConfig homepage (www.franksdank.cloud, /admin/) — how to verify a card/link change is actually live: (1) Panels are rendered CLIENT-SIDE by shared/homepage/app.js from services.json + status.json; the raw HTML (fetched via curl) is just the shell, so grepping the fetched HTML for a link's href gives a false negative. Verify via the DEPLOYED /srv/homepage/services.json on the serving host (Dankclaw) and/or the e2e admin-link test (tests/e2e/specs/11-admin.spec.ts "admin open links reach their apps"), not by grepping HTML. (2) scripts/homepage-status.py probe() treats ANY HTTP response — including 403 — as "up" (it only returns "down" on a connection/URLError). So a basicauth-gated service (Pi-hole admin at 192.168.0.143:8083 → 403) probes up fine; no need to find a 200 path. (3) A probe of 127.0.0.1:X is relative to the HOST RUNNING THE CHECKER. Running homepage-status.py on the dev box (the workstation) gives a false "down" for host-local probes like hermes-webui 127.0.0.1:8787; scp the script+config to the serving host (Dankclaw) and run it there. (4) Pi-hole FTL (a dnsmasq fork) retired the standalone dnsmasq quadlet on 2026-08-24 — the homepage had carried the stale "dnsmasq" card (probe-only, no link) until it was replaced with a linked Pi-hole card (2026-09-21, commit af36595). Hermes has two surfaces: the web UI client (hermes.franksdank.cloud :8787) and a separate admin dashboard (hermesadmin.franksdank.cloud :9119).
- frank-medium (dankserver :7770, vllm-radiance TP=2, 2x R9700) was cut on 2026-09-21 to the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, quark/W4A8, 19.37 GiB) with UNQUANTIZED bf16 KV (--kv-cache-dtype auto), replacing the brief FP8/fp8-KV shape (orcarouter/...FP8, 09-20→09-21). Three durable facts: (1) CAUSAL: the endpoint hardcoded --kv-cache-dtype fp8 on the COMMON exec path (outside the IS_MXFP4 quant branch) in every commit since the MXFP4 era, so BOTH FP8 and MXFP4 ran fp8 KV identically — fp8 KV therefore CANNOT explain the FP8-vs-MXFP4 throughput gap (92.4 vs 100.0 t/s); that gap was weight precision. fp8 KV stacked on MXFP4's already-W4A8 weights compounds attention-path precision loss → a QUALITY (not throughput) hypothesis. KV_DTYPE is now a shared quadlet tunable (entrypoint default fp8); quadlet sets auto. (2) VLM: the MXFP4 checkpoint DOES serve vision despite config.json language_model_only=true — the Qwen3_5ForConditionalGeneration arch carries the vision tower through the Gemma4 merge path and it is NOT MXFP4-quantized, so vLLM serves image input (verified: solid red 48x48 PNG named "Red"; vLLM would 400 if unsupported). The old quadlet "TEXT ONLY...400 here" comment was wrong; harness models.yml keeps frank-medium input:[text,image]. RULE: don't trust language_model_only=true as evidence vLLM rejects images — probe with a distinctive-color image. (3) POOL: f16 KV = exactly 2x the bytes of fp8 KV per token → pool halves. MXFP4+fp8 was 943,581 tokens; MXFP4+bf16 auto-sized to 486,464 (Maximum concurrency for 262,144: 1). MXFP4's freed weights (19.4 vs FP8's 30.9 GiB) keep the full native 262,144 window fitting (MAXLEN stays 262144); FP8+bf16 would only reach ~192,000 so it needs MAXLEN=192000. Rollback to FP8: SNAP + SPEC + KV_DTYPE=fp8 (+ MAXLEN=192000 only if KV stays bf16). _(context: serverAppConfigs frank-medium MXFP4 + unquantized bf16 KV deploy 2026-09-21, commits a45f396 + d8dbd4f; user hypothesis that fp8 KV was hurting MXFP4 quality.)_
-frank-medium (dankserver :7770, vllm-radiance) KV-cache dtype is a SINGLE shared flag on the common exec line of radiance-entrypoint.sh — OUTSIDE the IS_MXFP4 quantization branch — so it applies to BOTH the FP8 and MXFP4 checkpoints, not per-path. It was --kv-cache-dtype fp8 for both (confirmed in commits 6b647c1 MXFP4-era and d389d09 FP8-swap). Per-card VRAM budget (31.9 GiB R9700 @ 0.98 util): ~15.29 GiB FP8 weights + ~1.7 GiB HIP graphs leaves ~8.61 GiB for KV = 389,624 tokens at fp8, ~194,800 at f16/bf16 (exactly 2x bytes). So unquantized (auto/bf16) KV CANNOT hold the native 262,144 window — the max window drops to ~192k (set MAXLEN=192000). KV_MEM=0 auto-sizes the pool to fill remaining VRAM; the pool halves on fp8-f16. The fit-budget gate is tests/frank_medium_unit.bats "the KV pool holds a full window" (asserts the maxlen+KV_DTYPE shape when KV_MEM=0). If KV_DTYPE is a shared quadlet tunable, the MXFP4 rollback inherits it — harmless there (MXFP4 weights are smaller, 19.4 GB, so f16 KV + 192k fits with headroom), but no longer byte-for-byte the old MXFP4 shape (fp8 KV + 262144). _(context: frank-medium KV-quantization work 2026-09-21: user asked to turn off KV quant (fp8-auto/bf16); discovered the dtype flag is shared across both quant paths and that f16 KV halves the auto-sized pool below the 262k window.)_
-vLLM logs a non-default args: {...} line at boot (api_utils.py) — the AUTHORITATIVE set of every arg that differs from vLLM's internal defaults. Everything NOT in that dict is at vLLM's stock default. This is the right way to answer "what is vllm endpoint running with / what's still default" (better than diffing the cmdline against remembered defaults). Parsing gotchas: (1) it's Python repr (single quotes), not JSON → ast.literal_eval, not json.loads; (2) it contains enum reprs like DynamicShapesType.BACKED: 'backed' that break literal_eval → re.sub(r"[^]*", "'ENUM'", s) first; (3) the line is long and display-truncates at ~768 chars → save to a file and parse it, don't eyeball. Live argv: podman exec c cat /proc/1/cmdline | tr '\0' ' ' (tr runs host-side; a /proc/... redirect inside podman exec reads the host /proc and returns empty). Runtime-adjusted "defaults" (auto-sized KV pool, mamba block size 16→1664, max_num_scheduled_tokens, CUDA-graph capture) come from the boot log, not the CLI. Rootless quadlet: podman logs = current instance only; journalctl --user -u c spans recreations. _(context: Tabulating frank-medium (dankserver :7770, vllm-radiance 0.9.3 / vLLM v0.27.1) runtime args; the non-default args line gave the authoritative 24-override list; the enum repr and 768-char truncation both broke naive parsing.)_
-OpenWebUI (container open-webui on dankclaw, DB /app/backend/data/webui.db): the model picker (/api/models/list → Models.search_models) reads the model table ONLY — an enabled connection does NOT auto-appear in the picker; a model must be a row in the model table. LLM connections live in the config table as position-indexed parallel arrays openai.api_base_urls / openai.api_configs (+ openai.api_keys, openai.enable); Ollama in ollama.base_urls/ollama.enable. RAG: rag.embedding_engine == "" (empty string) means a local SentenceTransformer loaded from rag.embedding_model. To run backend python: podman exec -w /app/backend open-webui python — working dir MUST be /app/backend, the package is open_webui (underscore, not openwebui), and you MUST set os.environ["WEBUI_SECRET_KEY"] BEFORE importing any open_webui module (import-time hard requirement). Model rows are pydantic ModelModel objects (attribute access m.id/m.base_model_id, not .get()). llama.cpp ignores the model field in chat requests, so a stale/renamed model name still routes to the loaded model.
-SearXNG (Dankclaw) agent access + limiter, verified in the live :latest image 2026-09-21: (1) Agents/OpenWebUI do NOT go through the Anubis→Authentik guard chain — that chain sits only on the public vhost search.franksdank.cloud and is browser-only (a plain HTTP client gets Anubis's "not a bot" PoW HTML, not JSON). Agents dial the internal host-published port http://192.168.0.143:8081 (gluetun's published 8081:8080; SearXNG shares gluetun's netns), which bypasses Caddy entirely. SearXNG's own SEARXNG_BASE_URL is that internal port. (2) The JSON rate cap is API_MAX=4 req/IP/hour for any non-HTML format — a BARE PYTHON CONSTANT at searx/botdetection/ip_limit.py:82, no config knob in limiter.toml or settings.yml. (3) pass_ip is checked FIRST in searx/limiter.py and short-circuits the whole botdetection chain (return None before user-agent/header/ip_limit), so any pass_ip source is fully exempt from the cap. pass_ip covers 127.0.0.1, 192.168.0.0/24 (LAN), 10.88/10.89.0.0/16 (all rootless podman bridges) — so every on-host agent and LAN client is exempt (verified: 8-req burst from inside open-webui = 8x200, no 429). To exempt a new source, widen pass_ip (config-native); patching API_MAX or server.limiter:false are the only other (fragile/nuclear) options. (4) OpenWebUI web search is wired to SearXNG via per-key SQLite config (webui.db, web.search.*): engine=searxng, searxng_query_url=http://192.168.0.143:8081/search, enable=true. Quadlet env vars (ENABLE_WEB_SEARCH, WEB_SEARCH_ENGINE, SEARXNG_QUERY_URL, SEARXNG_LANGUAGE, WEB_SEARCH_RESULT_COUNT) SEED that table on first start (fill-missing only); afterwards the DB is authoritative and Admin-UI edits persist there. Now git-managed in dankclaw/open-webui/quadlet/open-webui.container + documented in OpenWebUI.md. _(context: Answered "can agents access SearXNG behind Authentik / can we disable the limiter cap" and wired OpenWebUI web search, serverAppConfig repo, commit d63acd4)_
-Agents/services (OpenWebUI backend, LLM agents) reach SearXNG via the INTERNAL port http://192.168.0.143:8081 — gluetun's published 8081:8080, SearXNG shares gluetun's netns — NOT the guarded public vhost search.franksdank.cloud (a plain HTTP client hitting that gets Anubis's "not a bot" PoW HTML, not JSON). SearXNG's SEARXNG_BASE_URL is already that internal port and formats includes json. Limiter (shared/searxng/limiter.toml): the JSON cap is a HARDCODED 4 requests/IP/hour in Python; the ONLY bypass is a pass_ip source (127.0.0.1, ::1, 192.168.0.0/24, 10.88.0.0/16, 10.89.0.0/16). Rootless podman strips the real client IP on published-port hits, so the source is the bridge IP (10.88/10.89) → pass_ip → UNRESTRICTED. Verified live from inside the open-webui container: 8-request JSON burst all 200 (no 429), egress IP seen by SearXNG = 10.89.3.2. So on-host/LAN agents are rate-cap-exempt; to wire OpenWebUI web search just point it at http://192.168.0.143:8081. For an OFF-HOST agent that needs a DNS name: add an unguarded search-api vhost straight to :8081 (exempt-by-construction, same pattern as Gitea/OpenWebUI), optionally bearer/IP-gated — NOT an Authentik service token, since the guard chain is proxy-provider inline flow with no forward_auth/token gate.
-When trimming a CPU-side memory cap on a llama.cpp LLM host (e.g. --cache-ram), verify the effect with Pss_Anon from /proc/PID/smaps_rollup, NOT RSS. Pss_Anon is the process's proportional share of anonymous memory — where the cache actually lives — so its delta matches the expected reduction. Measured on frank-small (dankclaw 3090) cutting --cache-ram 8192→2048: Pss_Anon dropped 8.97→3.03 GiB (~5.9 GiB ≈ the 6 GiB delta), but RSS dropped only ~4.3 GiB (9.75→5.48). RSS is misleading because it counts shared + file-backed pages and a CUDA context re-maps file-backed pages on restart, so RSS can drop LESS than the memory actually freed (here Pss_Anon dropped MORE than RSS). Cross-check with host free -h available as the real-world benefit (6.1→13 GiB). Related facts: --cache-ram (env LLAMA_ARG_CACHE_RAM) build default is 8192 MiB; -1 = no limit, 0 = disable. Gemma4-12B is NOT a DeltaNet model, so the repo's "12.8 GiB CACHE_RAM floor" rule (which protects Gated-DeltaNet state) does not apply to it. _(context: frank-small --cache-ram 8192→2048 on dankclaw, 2026-09-21. Verify via Pss_Anon delta (matched the 6 GiB cap reduction) while RSS under-reported the drop; host available doubled.)_
-Homepage stale-cache trap (serverAppConfig fleet, 2026-09-21): deploys overwrite /srv/homepage in place and Caddy file_server sends no revalidation headers, so a user's browser keeps serving pre-deploy CSS/JS/HTML — the new sectioned board rendered VERTICAL for Frank while a fresh profile proved the live page was correct. Fix (commit 8ea6486): the @home handler on the post-auth :8017 listener in dankclaw/Caddyfile sends header Cache-Control "no-cache" (ETag round-trip, cheap 304); pinned in tests/homepage_caddy.bats. RULE: for any user-facing static surface deployed in place, verify revalidation headers exist or users see stale assets. Live-page verification trick that worked well: reuse the e2e Playwright storage state (tests/e2e/.auth/qa-e2e.json, user; qa-e2e-admin.json, admin) — map its cookies and page.setCookie(...) in a headless browser, then navigate to the guarded vhost (bypasses the Anubis+Authentik chain); measure layout with getBoundingClientRect and read response headers with fetch(url, {cache:'no-store'}).
-Frank's homepage (www.franksdank.cloud) layout and copy preferences, set 2026-09-21: (1) Sections render as horizontal ROWS of panels (auto-fill grid, as many per row as the viewport allows) with sections stacking vertically — "render the sections horizontally" means panels-in-a-row, NOT side-by-side section columns (that was the wrong read and got reverted). (2) User page section order: Web Apps, Media Library Apps, AI — the AI section sits at the BOTTOM. Admin page order: Media Library Apps, Observability, AI, Infrastructure. (3) AI model card + /docs#models descriptions are MODEL-ONLY: hosted model, size, context/parallel capacity, brief what-it's-for. "My users don't care about the infrastructure" — no runtimes, GPUs, tensor parallelism, or draft heads in user-facing copy. Sizes: frank-large 87.3 GiB UD-IQ4_XS, frank-medium ~31 GiB FP8, frank-small 7.4 GiB Q4_K_M. Pinned in tests/homepage_pages.bats (row layout + AI-at-bottom order greps).
-frank-large (dankclaw, Strix Halo APU, unsloth llama.cpp b11030 gfx1151, Qwen3.8-Flash-Next UD-IQ4_XS 87 GiB 3-way split, -c 200000 -np 1 -ngl 999) — where the memory goes, measured 2026-09-21. REAL iGPU memory = /sys/class/drm/card*/device/mem_info_gtt_used (~70 GiB of a 113 GiB GTT aperture); rocm-smi VRAM is MISLEADING on this APU (reports the 2 GiB carveout, ~172 MiB used). Process RSS (~28-30 GiB, mostly anon) UNDERCOUNTS total LPDDR because GTT allocations are separate from the process page tables; system free showed ~115/121 GiB used. Of the 87 GiB model: ~70 GiB (MoE experts + attention + MTP) is RESIDENT in GTT (MoE must keep all experts in memory); ~17 GiB (the PLE n-gram embedding tables, blk.N.ple_*.weight) is STREAMED from NVMe on demand via --lazy-mode auto (the DEFAULT — the serve script never overrides it). --lazy-mode and -ngl 999 COMPOSE, not conflict: -ngl picks the compute device, lazy-mode controls data residency for per-layer embedding tensors 4 GiB. Proof of streaming = disk I/O during inference: watch /proc/PID/io read_bytes — a NEW prompt raises it ~1-15 MiB, a REPEAT prompt raises it 0. KV cache is only ~4.6 GiB at 200k ctx because Qwen3.8-Flash-Next is a hybrid SSM model: block_count=48, full_attention_interval=4 (only 12 full-attention layers, 36 Gated DeltaNet with fixed per-slot state) + aggressive GQA (head_count=24, head_count_kv=2, key/value_length=256, f16 KV = 2 heads × 512 × 2 B × 12 layers × 200192 tok). So the memory wall is the MoE experts, NOT the KV: shrinking -c saves little, -ot exps=CPU is the real footprint lever. To read GGUF arch values: llama-gguf part1.gguf r lists KEYS only (no values) — parse the metadata with a Python GGUF reader (value types 4=u32,5=i32,6=f32,8=str,9=array,10=u64,11=i64; part 1 has n_tensors=0, it is all metadata). _(context: frank-large cutover follow-up Q&A on dankclaw: user asked whether the PLE/engrams stream from disk, whether lazy-mode conflicts with -ngl 999, and how large the KV cache is. Answered by measuring mem_info_gtt_used, free -h, /proc/PID/smaps_rollup, /proc/PID/io across new-vs-repeat prompts, and parsing the GGUF metadata.)_
- frank-large (dankclaw, Strix Halo APU iGPU, unsloth-llama b11030 serving Qwen3.8-Flash-Next UD-IQ4_XS): the PLE n-gram embedding table ("engrams") is LAZY-STREAMED from NVMe by default, not resident in LPDDR. Mechanism: llama.cpp --lazy-mode defaults to auto = on for per-layer-embedding tensors 4 GiB; the serve script (scripts/serve-frank-large.sh) does NOT override it, so it runs at that default. The PLE is exactly that shape: blk.N.ple_{key,value,norm_*,conv1d}.weight across 48 blocks, each head a ~20M-row table (qwen4exp.ple.head_vocab_sizes = [20000003, ...]). Consequences: model RSS ~28 GiB (not the 87 GiB file), Pss_File ~204 MiB, fast load (~34s), fits in shared LPDDR; cost is a little NVMe I/O per NEW n-gram, and the page cache keeps repeated/similar content at 0 disk cost. To make it fully resident: --lazy-mode off (pushes whole table into LPDDR, slower load, no gain here). DIAGNOSTIC METHOD (reusable, to prove lazy tensor streaming on any llama.cpp endpoint): run a throwaway server on a scratch port with the prod flags, then measure /proc/PID/io read_bytes (actual disk I/O, NOT rchar) before/after a completion. Signature = content-dependent disk reads: a NEW prompt reads ~1-15 MiB from disk (PLE rows for new n-grams), a REPEATED prompt reads 0 MiB (rows now cached). Cross-check with /proc/PID/smaps_rollup: Pss_File tiny vs large model = table not resident. Always kill the throwaway and sudo systemctl start frank-large (root unit) after; verify :7770 health + a completion. _(context: frank-large PLE lazy-streaming investigation, 2026-09-21, serverAppConfigs. User asked "is it set to stream the engrams from disk instead of memory?")_
-"Change not visible on the live homepage" diagnosis (serverAppConfig fleet): deploys overwrite /srv/homepage in place, so a user's browser can keep serving pre-deploy CSS/JS/HTML from cache — new HTML (e.g. class="sections") running against stale CSS silently falls back to the old layout, and the user reports "it's still vertical" while the server is correct. Procedure: (1) grep the deployed file on the host (server-side truth); (2) prove it in a FRESH headless profile by reusing the e2e SSO storageState — tests/e2e/.auth/qa-e2e.json (user) / qa-e2e-admin.json (admin) are regenerated by every e2e run; load them with page.setCookie(...cookies) (map expires0 only) then navigate to https://www.franksdank.cloud/ — no need to drive the Anubis/Authentik login UI. Measure getBoundingClientRect/computed styles, screenshot. Permanent fix landed 2026-09-21 (commit 8ea6486): the @home handler on the post-auth :8017 Caddy listener sends Cache-Control: no-cache (ETag revalidation, cheap 304), so any deploy is visible on the next navigation; tests/homepage_caddy.bats pins the header. Also learned the same session: "render the sections horizontally" from Frank meant panels in horizontal ROWS per section (sections stack vertically), NOT sections side by side — ambiguous layout asks get a screenshot before shipping. _(context: Homepage sectioning session 2026-09-21: user reported the horizontal section layout was "still vertical"; live file was correct, browser cache was the cause; verified via fresh-profile drive with e2e storageState cookies; added no-cache header to the Caddyfile @home handler.)_
-Serving a concatenated split GGUF is a trap: concatenating the three -0000N-of-00003.gguf parts yields a byte-correct single file, but part 1's header still carries ggml.file_count=3, so llama.cpp's loader "advertises itself as a split" and dies with llama_model_load: error loading model: invalid split file name (the unit crash-loops, restart counter climbs, yet an older orphan llama-server may still hold the port and answer /health — so the endpoint looks up while the unit is down). The fix (peer commit d6f9b85): point -m at PART 1 and let llama.cpp load the split natively (parts 2-3 auto-discovered in the same directory) — do not concatenate for serving. This bit frank-large on 2026-09-21 after a re-download. Diagnosis shortcut: ls -la the payload dir and compare the concatenated file's byte size to the sum of the parts (a mismatch like exactly 1e9 bytes short = truncated concat), and read the full journalctl -u svc line for the "invalid split file name" path.
-When a user reports a homepage/static-site change "isn't showing" after a green deploy, suspect browser cache BEFORE doubting the deploy: deploys overwrite /srv/homepage in place, and with no revalidation header the browser keeps serving the pre-deploy CSS/JS/HTML (new HTML ran against old CSS and silently fell back to the old layout). The fix (commit 8ea6486): the @home handler on Caddy's post-auth :8017 listener sends Cache-Control: no-cache (ETag round-trip, cheap 304), pinned by tests/homepage_caddy.bats. To PROVE what the live page actually renders — independent of the user's cached tab — drive it in a FRESH headless profile using the e2e SSO cookies: load tests/e2e/.auth/qa-e2e.json (user) or qa-e2e-admin.json (admin) via page.setCookie(...cookies), goto https://www.franksdank.cloud/, waitForFunction on '#grid .panel', then read getBoundingClientRect of '#grid .section' (equal y = horizontal, distinct x = columns) or getComputedStyle. The e2e .auth files are regenerated by every deploy's e2e gate, so they're current. The user still needs one manual hard-refresh of their already-open tab (stale CSS is in memory until reload).
-Serving a llama.cpp model that ships as split parts (HF caps uploads at 50 GB, so any 50 GB GGUF is multi-part): do NOT concatenate the parts into a single file to serve. Part 1's GGUF header carries ggml.file_count=N, so a byte-correct concatenation still advertises itself as an N-way split, and the loader rejects a non-split filename with "invalid split file name" (fails in ~50 ms, before reading any weights — reads like a corrupt file but the bytes are fine). Fix: point -m at part 1 (name-00001-of-NNNNN.gguf) and let llama.cpp load the split natively; parts 2..N are auto-discovered in the same directory. The split parts ARE the canonical payload. Caught 2026-09-21 on frank-large (unsloth Qwen3.8-Flash-Next UD-IQ4_XS, 3-way split, 87.3 GiB) — the assembled file crash-looped the systemd unit ~110 times before the diagnosis. Generalizes to every multi-part GGUF. _(context: frank-large cutover to unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on dankclaw, 2026-09-21; serverAppConfigs repo, runbook serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md)_
-serverAppConfig multi-agent repo, two more peer-collision facts (2026-09-21): (1) A peer's git add can sweep YOUR in-flight uncommitted files into THEIR commit (my homepage layout changes landed inside the peer's frank-large fix commit d6f9b85) — after any surprise "nothing to commit, working tree clean", check git log --oneline -5 + git log -S your-string before assuming your work vanished (stash was the first suspect from an earlier lesson; a peer commit is the other). (2) A service-scoped deploy (deploy.sh host --service X) leaves other changed services PENDING on the remote: a later local --dry-run prints "changed services: (none; the remote will check pending service state)" even though /srv assets are stale — run the real deploy and the remote --pending runner dispatches the leftover services. Verify staleness by grepping /srv/svc assets on the host, not by the local plan. _(context: Horizontal homepage section deploy, 2026-09-21: peer committed frank-large fix sweeping my 4 in-flight files; peer's scoped frank-large deploy left homepage pending; my full deploy dispatched it (e2e 32/32).)_
-llama.cpp: concatenating a split GGUF's parts into a single .gguf file does NOT produce a loadable single-file model — part 1's header carries ggml.file_count=3, so the byte-correct concatenation still advertises itself as a split and the loader fails with "invalid split file name: path" (it demands a -NNNNN-of-NNNNN filename). The fix (frank-large, 2026-09-21, commit d6f9b85): point -m at part 1 of the split; llama.cpp auto-discovers parts 2-3 in the same directory and loads the split natively. serve-frank-large.sh and the deploy-remote.sh LLM_LARGE_MODEL gate both follow this. Don't re-attempt the concat-and-rename shape. _(context: frank-large crash-loop diagnosis on dankclaw: unit failed with "invalid split file name" on the 93.68 GB concatenated GGUF; peer fixed by native split load; endpoint active/healthy after.)_
-Peer-agent coordination on blankpad: hub list can show ZERO peers even when another omp process is running in the same repo directory (separate broker instances are not visible to each other; ps -eo pid,args + /proc/pid/cwd is how you find the peer's session). When a live-fleet anomaly appears that belongs to a peer's in-flight deploy (e.g. frank-large crash-loop + orphan llama-server during a homepage deploy), do NOT fix it unilaterally — verify your own deploy only dispatched its planned services, gather the evidence (journal, process age, file sizes), and hand it to the user as a relayable handoff. Seen 2026-09-21: homepage deploy was clean while a peer's frank-large work left the unit crash-looping ("invalid split file name") with an orphan llama-server holding :7770; the assembled GGUF was exactly 1,000,000,000 bytes short of the parts sum (truncated concat). _(context: Homepage deploy vs peer frank-large deploy, 2026-09-21)_
-Homepage registry contract (2026-09-21, commit 384747a): every shared/homepage/services.json entry now carries a section field that MUST be in app.js's SECTION_ORDER for its category (user: AI, Web Apps, Media Library Apps; admin: Media Library Apps, Observability, AI, Infrastructure) — a section not in SECTION_ORDER is SILENTLY dropped from the page (pinned by tests/homepage_config.bats). wip is no longer a boolean: it is false | "open" | "closed", and homepage-status.py passes it through as state "wip-open"/"wip-closed" (badges "WIP: OPEN" amber / "WIP: CLOSED" dim). Panels render under section headings via shared renderPanel()/renderSection() components in app.js (both pages share one app.js, so that IS the shared-component mechanism). Adding a homepage service = one JSON entry with the right section; no deploy-script change (deploy copies a fixed file list). _(context: Homepage sectioning deploy, serverAppConfig repo, 2026-09-21)_
-Fleet security-posture audit (franksdank.cloud), traps + 2026-09-21 baseline: (1) WAN port scans MUST run from gluetun's ProtonVPN egress on dankclaw (ssh dankclaw 'podman exec gluetun ...', verify external IP via api.ipify.org first) — LAN→WAN is hairpin NAT and lies. (2) TLS-port gotcha: plain-HTTP wget probes against TLS ports (443, Plex PMS 32400) are FALSE NEGATIVES — re-probe with wget -qO- https://IP/ (the gluetun container's wget may lack --secure; the https:// URL suffices). (3) Calibre on dankserver is a HOST GUI APP, not a container service — only Calibre-Web (:8083, books. vhost) is on the wire; the optional calibre-server (:8080) is not running. AGENTS.md's stack list ("Calibre, Calibre-Web") misleads; don't audit or chase Calibre as a service. If calibre-server is ever started: host Calibre 9.2.1 9.9.0 (content-server file-read fix) and use --enable-auth. (4) 2026-09-21 verified baseline: WAN has ONLY 80+443 open (Caddy edge); public names = www, search, ai, gitea, plex, books, jelly, auth (IdP); 9 vhosts serve the Anubis challenge (www, search, grafana, prometheus, loki, seerr, marreta, romm, files); all exempt vhosts behaviorally gated (OpenWebUI /api/v1/users/me 401, Gitea 401 + repos private, *arr API 401, qB 403, pihole basicauth 401); LLM endpoints (143:7770, 143:7771, 235:7770) accept requests with a BOGUS Bearer token (auth entirely absent — LAN trust model); Grafana admin:admin confirmed live but loopback-only (127.0.0.1:3002); plex.franksdank.casa public record was STALE (73.59.75.210 = previous Comcast IP, dead on 80/443 — works on LAN only because Pi-hole shadows it); Plex vhost is WAN-broken (302 to LAN origin 192.168.0.235:32400); :8017 carries vestigial *arr handlers (exempt vhosts never enter the chain). A managed skill fleet-security-posture-audit exists from an earlier audit — enhance it with these traps rather than duplicating.
-Fleet security posture as audited 2026-09-21 (all behavioral): the WAN (71.227.227.172) exposes ONLY ports 80+443 (Caddy edge); 8 public vhosts — www/search behind the Anubis→Authentik guard chain, ai/gitea/plex/books/jelly exempt with verified own auth (401s/login redirects), plus auth. (IdP, public by design). Every other vhost (grafana, prometheus, loki, seerr, marreta, romm, files, pihole, *arr, hermes, hermesadmin) is LAN-only by DNS absence. Known residuals: (1) all three LLM endpoints bind 0.0.0.0:7770/7771 with NO auth — LAN trust model, the first thing to gate if untrusted LAN devices appear; (2) Grafana still answers basic auth admin:admin on loopback 127.0.0.1:3002 (200 with creds, 401 without) — loopback-only and behind the chain via vhost; (3) plex.franksdank.casa is a STALE public record at 73.59.75.210 (the previous Comcast WAN IP — current is 71.227.227.172), so the plain-HTTP legacy name is dead from the WAN; (4) plex.franksdank.cloud is WAN-broken functionally because Plex Web 302-redirects to its LAN origin 192.168.0.112… correction: 192.168.0.235:32400, unreachable off-LAN; (5) the :8017 post-auth listener carries vestigial @prowlarr/@radarr/@lidarr/@readarr/@qbittorrent handlers that nothing routes to (exempt vhosts never enter the chain). Probe traps: plain-HTTP wget against TLS ports (443, 32400) is a false negative — re-probe with https://; busybox wget has no --secure flag and fails silently inside && chains; the authentik flow page is React so grep for 'password' finds nothing — a 200 on /if/flow/... is the proof. _(context: Security posture audit of the franksdank.cloud fleet, 2026-09-21, from blankpad)_
-qBittorrent WebUI on FrankBook-Pro (http://192.168.0.112:8080, published on the gluetun unit): username admin; password = QBIT_WEBUI_PASSWORD in /home/core/Containers/qbittorrent/env on the Mac — a 20-char random string generated once at provisioning by the deploy handler (scripts/deploy-remote.sh VM_SEED: openssl rand -base64 18 | tr -d '/+=' | head -c 20), machine-local per the repo secret rules, with a PBKDF2-HMAC-SHA512/100k hash of the same value embedded in qBittorrent.conf. Access gap (verified 2026-09-21): blankpad (current dev box) has NO SSH alias or key for core@192.168.0.112 — the frankbook-pro alias lived on the retired the workstation dev box; neither jump host (dankclaw, dankserver) has the Mac's key; blankpad has no sshpass/expect. Bootstrap a key (ssh-[REDACTED] skill) or use the Mac's password before reading the env file. _(context: Frank asked for the qBittorrent WebUI credentials; repo + skill pointed to the machine-local env file, but no SSH path from blankpad existed (4 probes: direct, alias, both jump hosts))_
-deploy.sh's [3/3] Playwright e2e gate can outlast a 300 s bash timeout and get killed mid-suite, but the deploy itself is already complete: the service was restarted, health-checked, and its STATE revision recorded BEFORE the gate starts. When the gate is cut off, do NOT re-run deploy.sh (it would re-dispatch) — run ./tests/e2e/run.sh standalone to finish the gate; the repo's deploy-completeness rule (e2e green) is satisfied by that standalone run. Flaky tests (fail once, pass on retry) do not fail the suite. Seen 2026-09-21 on the frank-medium chat-template deploy: gate died at 14/32, standalone re-run gave 31 passed + 1 flaky (12-crawl). _(context: frank-medium Qwen-Sharp chat template deploy, 2026-09-21, serverAppConfig repo)_
-In the serverAppConfig multi-agent repo, when expected WIP vanishes from a clean tree (files missing, git status empty), the first place to look is git stash list — peer agents stash in-flight work with descriptive messages (e.g. "peer WIP: frank-medium qwen-sharp template (stashed for prometheus deploy)") to land their own commits. Verify contents with git stash show --include-untracked --stat stash@{N} (untracked files live in the stash's third parent, stash@{N}^3) and git stash pop on the clean tree. Seen 2026-09-21: the Qwen-Sharp template + quadlet flip had been stashed by a peer for a prometheus deploy and looked lost until the stash list was checked. _(context: frank-medium Qwen-Sharp chat template deploy, 2026-09-21, serverAppConfig repo)_
-On the frank-medium FP8 Qwen3.8-27B checkpoint (dankserver, vllm-radiance TP=2), the z-lab DFlash2 EXTERNAL drafter beats the model's own MTP self-speculation on DECODE (geomean 100.0 vs 85.9 t/s, burst-4 287.1 vs 179.7 t/s) despite a LOWER mean accept length (4.24/7 vs 5.66/8). The trap: accept length is a PROXY, not the metric. Decode throughput = (accept+1) / (draft time + verify time). The DFlash2 drafter is a small separate model that steps fast; the MTP head is the target's own layer applied autoregressively N times, so its draft is slower per step. The faster step more than makes up the lower accept length. MTP still wins on TTFT (0.06 vs 0.19 s) and prefill (1096.9 vs 923.0 t/s); the drafter costs 2.1 GiB VRAM (KV pool 384,238/1.47x vs MTP's 461,540/1.76x). RULE: when choosing between MTP self-speculation and an external drafter, measure decode throughput on the SAME target — never compare accept lengths across different targets/quantizations (that was the apples-to-oranges error: MTP-on-FP8 5.66/8 vs DFlash2-on-MXFP4 ~4.1). FP8 + DFlash2 (100.0 t/s) also beats MXFP4 + DFlash2 (92.4 t/s) — the higher-precision target verifies more of the drafter's proposals. DFlash2 is the deployed shape (SPEC_METHOD=dflash, SPEC=7). _(context: frank-medium DFlash2 vs MTP A/B test on the FP8 checkpoint, 2026-09-20. User: "so, you're lying to me then... test it" — the prior MTP-vs-DFlash2 comparison was across different targets/quantizations.)_
-In vLLM/radiance speculative decoding, mean accept length is a PROXY, not the metric — a small separate drafter (DFlash2) can beat the target's own autoregressive MTP head on decode throughput DESPITE a LOWER accept length, because the separate drafter steps faster than the target's own layer applied autoregressively K times. Throughput = (accept+1)/(draft time + verify time); the verify (one target forward pass) is shared, so the draft time is the differentiator. Measured on the FP8 Qwen3.8-27B target (frank-medium, dankserver, 2026-09-20): DFlash2 (4.24/7 accept, 100.0 t/s decode, 287.1 t/s burst-4) beat MTP self-speculation (5.66/8 accept, 85.9 t/s, 179.7 t/s); MTP wins on TTFT (0.06 vs 0.19 s) and prefill. The DFlash2 drafter is quantization-agnostic (proposes, target verifies) and works with any Qwen3.8-27B target; it costs 2.1 GiB VRAM (KV pool 1.47x vs MTP's 1.76x). FP8 + DFlash2 (100.0 t/s) also beats MXFP4 + DFlash2 (92.4 t/s) — a higher-precision target verifies more of the drafter's proposals. Rule: when A/B-ing spec-decode variants, measure decode t/s (the bench-conversation.py battery), never accept length alone; flip the quadlet SPEC_METHOD, deploy, read the boot-log SpecDecoding metrics, run the battery, keep the winner. _(context: frank-medium DFlash2 vs MTP A/B on the FP8 checkpoint, 2026-09-20; commit f998a2d. Prior turn I wrongly claimed MTP was the better choice based on accept length (5.66 vs 4.1) without testing DFlash2 on the FP8 target; the user called it out ("you're lying to me then... test it").)_
-frank-medium (dankserver :7770, vllm-radiance TP=2) model swaps are ONE quadlet edit + deploy, because the entrypoint (dankserver/frank-medium/radiance-entrypoint.sh) is QUANTIZATION-AWARE: it reads quantization_config.quant_method from the checkpoint's config.json and branches. fp8 - plain path (no MXFP4 patches, --quantization fp8, ROCM_AITER_UNIFIED_ATTN attention, MTP self-speculation SPEC_METHOD=mtp SPEC=8 — the model's own single-layer MTP head, no external drafter). mxfp4/quark/empty - patch path (R4D attention if R4D_ATTN=1, DFlash2 external drafter SPEC_METHOD=dflash SPEC=7, applies radiance-vllm-mxfp4 patches every start). A swap edits exactly 4 env vars in the quadlet: SNAP=/models/name, SPEC_METHOD, SPEC, KV_MEM (0=auto-size). The deploy handler (deploy-remote.sh deploy_llm_medium) reads SNAP/SPEC_METHOD from the quadlet and gates on the right assets (drafter only for dflash, libr4d only for mxfp4/quark). KV pool auto-sizing is quantization-dependent: bigger weights leave less VRAM (FP8 30.9GB - 461,540 tokens 1.76x; MXFP4 19.4GB - 943,581 tokens 3.59x on the same 2x R9700). One-command swap: scripts/swap-frank-medium.sh --repo o/r --target /home/dank/models/name --generate-manifest --apply --deploy. Full process: serverAppConfigs-docs/LLM/frank-medium-model-swap.md. _(context: frank-medium MXFP4-FP8 swap 2026-09-20; the entrypoint was made quantization-aware so future swaps are a quadlet edit, not an entrypoint rewrite)_
-Generating a HuggingFace fetch manifest (filename/size/sha256 per file) from the repo API: the GET https://huggingface.co/api/models/owner/repo response's siblings[] carries ONLY rfilename — no size, no lfs hash. So the size and LFS sha256 must come from a HEAD on the resolve endpoint https://huggingface.co/owner/repo/resolve/main/file: size = Content-Length header, LFS sha256 = X-Linked-ETag header (64-hex; non-LFS files like config.json/[REDACTED] have no 64-hex etag, so leave the hash blank and verify size only). Token via Authorization: Bearer header. This is what scripts/hf-manifest.py does (dankserver frank-medium model-swap toolchain, 2026-09-20). A naive siblings[].size reads 0 for everything. _(context: frank-medium MXFP4-FP8 swap streamlining; first hf-manifest.py version printed size 0 for all files because it read siblings[].size)_
-When probing a reasoning-on LLM endpoint for a capability (vision, tools), a small max_tokens budget (e.g. 32) can be fully consumed by the thinking block, leaving content null/empty — which reads as "the capability is broken" when it is not. Re-probe with headroom (200+) before concluding. Seen 2026-09-20 on frank-medium (vLLM): a 32-token vision probe returned content None; the same image at 200 tokens answered "The image is white."
-When running a Python heredoc over ssh host 'python3 - PY ... PY', the REMOTE shell (bash) expands $var inside the heredoc body before Python sees it — an unquoted PY delimiter with a $job in the code silently becomes replace("", ...) (inserts the replacement between every character) and produces misleading results (here: 21/21 "HTTP 400 parse error" from a query with frank-medium spliced into every letter of rate(...)). Use a quoted delimiter 'PY' (or escape the $) whenever the script body contains shell-expandable $ sequences. Diagnose by printing repr() of the first extracted string — the splicing is visible immediately.
-On dankclaw, the Grafana quadlet container is published to 127.0.0.1:3002, NOT 3000 — port 3000 is a different service that answers 404/"Not found." to every path (this masqueraded as "dashboard not provisioned" for a day; the file was in the container and the provider was active the whole time). Grafana 13.2.0 has no GF_SECURITY_ADMIN_PASSWORD set, so basic auth is admin:admin. Verify any dashboard via curl -u admin:admin http://127.0.0.1:3002/api/dashboards/uid/uid and /api/search?type=dash-db.
-Two prefix-cache semantics traps measured 2026-09-20 on the Dank fleet: (1) llama.cpp's llamacpp:prompt_tokens_total EXCLUDES cached tokens (its own HELP line: "Number of prompt tokens processed, excluding cached tokens"), while llamacpp:prompt_[REDACTED] counts reused tokens — so the reuse FRACTION is cached/(cached+prompt), and the naive cached/prompt ratio is unbounded (read 77 on a fully-warm frank-large, 2.98 on frank-small). (2) vLLM (radiance 0.9.3, Qwen3.8 hybrid GDN/mamba) prefix caching: metric names are vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total, both token-counted (NOT 'reused'); the attention block size is padded to the mamba page size (1,648 tokens here, boot log "Setting attention block size to 1648 tokens"), and reuse happens at that granularity — an identical 13,021-token prompt went 2.30s cold → 0.68s warm with 9,888 tokens (76%) reused, while an identical 2,521-token prompt (one full block) registered +0 hits. So shared prefixes under ~2 blocks (~3.3k tokens) re-prefill in full. The API returns prompt_tokens_details: null in this build, so the counters are the only instrument. Prefix caching in mamba 'align' mode is flagged experimental at boot ("Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental."). _(context: frank-medium radiance TP=2 cutover monitoring work, 2026-09-20, serverAppConfigs)_
-When parallel doc-writing agents edit an OKF/Obsidian vault (serverAppConfigs-docs), the recurring damage is STRUCTURAL, not factual, and no existing gate catches it: (1) prose prepended ABOVE the --- frontmatter opener (so the frontmatter silently stops being frontmatter), (2) whole body sections duplicated wholesale — an agent appended its rewrite instead of editing in place, leaving two ## Architecture/## Why TP=2 cannot run here blocks plus a stray second ---, (3) a literal model tool-call token ("|channel}tool_call||tool_callcall:read{i:) pasted into the body, (4) line-by-line bullets emitted with a leading - on EVERY wrapped line (- - , - 2026-09-20 ...) and stray + diff-artifact prefixes, (5) an agent line-range replacement that dropped a sentence tail, leaving a dangling fragment. check-vault-links.py reports 0 unresolved links the whole way through because links resolved while structure rotted. Remedy that worked: git checkout HEAD -- file to restore the destroyed file (move the agent version to .AGENTTRASH/date-topic/ first so nothing is lost), then re-apply the intended flips by hand; salvage genuinely unique sections with CUT N.=M @reg + PUT K @reg before deleting duplicate ranges. VERIFY AFTER EVERY AGENT DOC PASS: frontmatter --- is line 1 and appears exactly twice, headings are unique (grep -n '^## ' | sort -k2 | uniq -f1 -d), no +- /- - /literal token artifacts, and the link checker plus the docs-completeness seam both run. Also: agents keep writing after you think they are done — a completed job re-wrote a file I had already fixed, so re-check wc -l/structure after the last job settles. _(context: 2026-09-20 frank-medium radiance TP=2 cutover in serverAppConfigs: two doc agents (EvidenceAndLog, RunbookSwap) left these wounds across serverAppConfigs-docs/LLM/index.md (restored from HEAD twice), frank-medium-radiance-mxfp4.md (deduped 827-549 lines), frank-medium-llamacpp-dflash2.md, index.md, log.md.)_
-The 3090 on dankclaw is an EVGA FTW3 Ultra V2 (PCI 10de:3982); its RGB is driven by an iCX3 lighting MCU on the GPU's OWN nvidia i2c bus — i2c-1 (NVIDIA port 1), address 0x2D. The nvidia i2c driver exposes ports 1,5,6,7,8 as /dev/i2c-1..5 but only i2c-1 carries a live slave. Protocol (raw I2C_RDWR, NOT the SMBus ioctls): write reg 0xB2 04 C6 EB EA 15 (enable, required once first), then reg 0xC0 mode 09 m0 m1 m2 m3 .. .. .. .. sync where m0..m3 are the four zone modes and MODE_OFF=0x00 (OFF = 09 00 00 00 00 FF FF FF FF 00), then reg 0x90 04 9E EB 00 90 to commit to EEPROM. Read the mode back by writing reg 0xC0 and reading 16 bytes — bytes 1..4 are the zone modes (0x00=OFF), bytes 6..9 are per-zone LED counts. The EEPROM save persists across a full reboot with NO Linux-side daemon — verified 2026-09-19: after sudo reboot the mode read back 09 00 00 00 00 1c 04 05 3c 00 … (all four zones OFF). Works headless (no display engine / X needed). /dev/i2c-* are root:i2c 660, so run the tool with sudo (dank has passwordless sudo). _(context: dankclaw 3090 RGB turned off + EEPROM-saved, then reboot-verified 2026-09-19)_
-serverAppConfigs authentik user provisioning (five accounts kirby/ry/nic/debo/freya pk 17-21 created 2026-10-09; law now in vault commit 65dec6a): (1) After ak shell provisioning with the human-user template (groups Non-Admin Users + jellyfin_users + openwebui_users), OpenWebUI mints each account on FIRST SSO login with role pending (ui.default_user_role=pending in webui.db config table) — they log in fine, then cannot chat / picker is empty until UPDATE "user" SET role='user' WHERE email=…; the row does not exist before first login, so the flip is a first-login-sighting follow-up, not a provisioning step. No per-user model grant needed — the picker model carries the global access_grant row (model, frank-secondary, user, *, read). Written up in serverAppConfigs-docs/Dankclaw/OpenWebUI.md (SSO section). (2) The proxysession claim purge is only due when a group change follows an existing login; users created with their groups already in place have zero rows — verify with a per-user census instead of purging blindly. (3) The authentik-postgres container's DB role/db are authentik/authentik (POSTGRES_USER env), NOT postgres: podman exec authentik-postgres psql -U authentik -d authentik -tAc "select session_data-'claims'-'preferred_username', count(*) from authentik_providers_proxy_proxysession group by 1". (4) User.app_entitlements() requires the Application argument (Application.objects.get(slug="open-webui")) — bare call raises TypeError. Verification bar for a provision: authenticate(request=None, …) accepts the password and rejects a wrong one (the exact PasswordStage path) plus the open-webui entitlement resolves. _(context: serverAppConfigs 2026-10-09: provisioned five authentik human accounts on Dankclaw; vault pages Infrastructure/Authentik.md, Dankclaw/OpenWebUI.md, log.md updated (commit 65dec6a), bats suite 366 green.)_
-radiance arch-capability audit (dankserver, pin 1.0.13, verified 2026-10-08): the container ships 6 arch plugins in /opt/radiance/share/radiance/architectures/ — qwen4exp_fp8, qwen35moe_fp8, qwen35_fp8, qwen35_bf16, llama_fp8, llama_dense_fp8. Capability check per plugin: grep -ac RADIANCE_CALIB / grep -ac gram / grep -ac mtp / grep -ac ngram on the .so (qwen35moe_fp8: calib=2 grams=2 mtp=4 ngram=0; gptq=0 in ALL plugins — GPTQ lives in rad-convert, not the plugin). Introspection pitfalls: the digest-pinned name (stilldeadcode/radiance@sha256:4c3f48…) does NOT resolve locally (podman tries a registry pull and fails); use the local tag docker.io/stilldeadcode/radiance:latest — image lives in ROOTLESS storage as dank (frank-primary runs rootless under the root unit; sudo podman ps is empty). Image has no ls/strings; podman exec frank-primary ls fails; correct form: podman run --rm --entrypoint sh docker.io/stilldeadcode/radiance:latest -c '…'. Verdict recorded: Qwen3.6-35B-A3B (qwen3_5_moe_text, 40 layers, 256 experts top-8, MTP×1) is plugin-supported; w4dn8 recipe maps minus PLE/HC terms; ~20 GiB w4+i8 = fully VRAM-resident on ONE 32 GiB R9700 (no TP=2/host-pool/disk-tier/MNBT-2048 law); convert ~1–1.5 h. Full audit in serverAppConfigs-docs/LLM/Quantization Production.md. _(context: serverAppConfigs 2026-10-08: feasibility audit of porting the frank-archon w4dn8 radiance recipe to Qwen3.6-35B-A3B; KB page Quantization Production.md carries the full write-up.)_
-Quantization production research (2026-10-08, full detail in repo page serverAppConfigs-docs/LLM/Quantization Production.md, linked from both vault indexes): (1) Law — production is hardware-agnostic (imatrix/GPTQ/ablation/pruning are torch-CPU work); the card only picks what it can DECODE. (2) Frank's framing: the 3090 is the ONLY unlocked fleet device — llama.cpp there is a choice; sm_86 decodes GGUF K/I, GPTQ/AWQ-Marlin INT4, SmoothQuant INT8, weight-only FP8 (FP8 W8A8 compute is Ada/Hopper SM≥89 only; NVFP4 Blackwell; 3090 has NO FP8/FP4 silicon), EXL3 (ExLlamaV3, convert.py runs on-card). The 3090 is the natural quant R&D bench (produce on host, decode+score vs fleet KLD gates). (3) REAP (Cerebras, arXiv 2510.13999, github CerebrasResearch/reap) expert saliency = router gate-values × activation energy — the INVERSE of our derive-protected-experts.py energy-share law; REAPing Flash-Next experts 25–50% re-enters the radiance pipeline at calibration, zero new tooling. (4) HauhauCS K_P "Perfect" = imatrix on abliterated weights + per-model per-tensor mixed-type profile via llama-quantize --tensor-type (plain GGUF, no fork; Q8_K_P census: Q8_K+F32+F16); reproducible with binaries built by scripts/build-llama-rdna4.sh. (5) Heretic (p-e-w/heretic, AGPLv3) = automated abliteration (refusal-direction ablation + layer dropping, refusals-vs-KL tuned), NOT a quantizer; abliteration (Arditi 2406.11717) is rank-1 fp-editing. Order law: surgery → calibrate → quantize; keep refusal tokens out of calibration. (6) .hgn production door is CLOSED — no public quantizer exists in peonist-ai repos; self-quant on Strix runs through GGUF runtimes. _(context: serverAppConfigs research session 2026-10-08; scouts + first-hand verification of REAP/Heretic/abliteration/HauhauCS K_P and per-runtime quant matrices; KB page Quantization Production.md committed to main.)_
-serverAppConfigs: (1) The quantization-production research (2026-10-08) lives in serverAppConfigs-docs/LLM/Quantization Production.md — per-card playbooks (3090 imatrix/llama-quantize; R9700 radiance rad-convert pipeline is fully self-owned; Strix .hgn production is CLOSED, no public quantizer) and the verified prior-art families: REAP (Cerebras expert pruning — saliency = our derive-protected-experts.py energy-share statistic, inverse polarity; REAPing Flash-Next's 49,152 experts 25-50% drops the w4 resident set to ~58-87 GiB and kills the disk tier), Heretic (p-e-w/heretic = automated abliteration + layer dropping, NOT a quantizer), abliteration order law ablate→calibrate→quantize, HauhauCS K_P = plain GGUF with imatrix + per-tensor mixed-type profile (llama-quantize --tensor-type), no fork needed. (2) Docs seam law: tests/docs_completeness.bats requires EVERY page under serverAppConfigs-docs/ to be linked from the ROOT index.md (serverAppConfigs-docs/index.md), not just the section index (LLM/index.md) — a new vault page linked only in LLM/index.md fails seam 2. _(context: Quantization Production research turn, 2026-10-08; suite caught the root-index link requirement)_
-Dankclaw homepage (serverAppConfig shared/homepage) durable facts from 2026-10-08 wave 8: (1) Smoke-server trap escalation: when a localhost http.server smoke check serves stale content, the authoritative diagnostic is for p in $(pgrep -f "http.server 8099"); do readlink /proc/$p/cwd; done — a listener from an OLD staged tree can survive and serve while your new files sit untouched elsewhere; a ss | grep || (start) guard silently skips the restart when the stale listener is found. Kill the exact pid, restart from the fresh tree, re-curl for a new-file marker. (2) Warning UI convention (Frank-approved): all privacy/uncensored warnings are native details disclosure chips ("⚠ privacy note ▸/▾"), collapsed by default, pinned to the box bottom via margin-top:auto in a flex-column panel/card; the clickable-panel open handler MUST return on ev.target.closest('a, .svc-warn') or the chip tap navigates to the app. (3) The docs page is served at path /docs/ — a bare /docs.html URL hits the SPA fallback and renders the homepage shell. (4) frank-archon runs the fleet's OWN payload: homepage card + docs Model bullet link https://huggingface.co/frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 (published on HF, verified via API); spec = in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next, ~114 GiB, three submissions (--max-num-seqs 3). The other two cards' HF links are their publishers' upstreams (accurate — verbatim artifacts). (5) OpenWebUI 0.6 API-key flow for docs/screenshots: avatar (bottom-left) → Settings → Account → API keys → Show → copy icon (⋯ rotates); no "Create new secret key" button exists. _(context: serverAppConfig homepage wave 8: bottom-pinned disclosure chips, live OWU api-keys screenshot, archon HF publication link; smoke verification against staged trees on the dev box.)_
-HF publication completed 2026-10-08: (1) Model repo frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 — public, 15 files, .rad = qwen4exp-swift15-w4dn8-i8-hc8m.rad 122,071,973,888 B sha256 09dd9306e859… uploaded via hf_xet at ~46 MB/s in ~29 min from dankserver (systemd-run --user --collect transient unit + .rc sentinel). (2) Dataset frank8465/swift15-gptq-w4-calib-corpus — public, CC BY-SA 4.0, anonymized calibration corpus (calib.jsonl 1,294 prompts, kld.jsonl 197, manifest, make-corpus.py, scrub-corpus.py); original unpublished hashes calib edb9b745…, kld 6e7df2f5…; cross-linked with model card (§2/§4 links, 4 refs). Scrub law: fleet slices are agent-session transcripts — scrub substring-class (not word-boundary): dankserver/dankclaw/frankbook/franksdank (bare brand = container image namespace franksdank/*), franksdank\.(cloud|casa), /home/dank, emails; corpus carries NO real credentials (92 FRANK_SUDO_PW hits = variable refs; private-key hits = bats template headers; pw-assign hits = upstream test fixtures). HF card validator law: custom licenses must be license: other + license_name + license_link (raw SPDX-unknown value = ValueError at upload_file, AFTER sha gates). hub 1.28 API: upload_file/create_commit kwargs keyword-only; HfApi has no rename_file/hf_hub_url/commit_api attrs; whoami hides canWrite for legacy tokens (probe = create_repo exist_ok=True). Secret storage pattern: ~/.config/secrets/name 600 + .zshrc loader export, token mirrored to ~/.cache/huggingface/token; write token installed on dev box + dankserver.
-Dankclaw homepage/docs (serverAppConfigs, deployed 2026-10-08): (1) Frank's warning doctrine — hover tooltips are DEAD (unreachable on touch); warnings render as native details disclosure chips ("⚠ privacy note ▸") collapsed by default, pinned to the box bottom via margin-top:auto in flex-column .panel/.mi-card; the clickable-panel open handler must ignore .svc-warn taps (chip taps bubble → navigation bug, fixed). Same chips on docs.html (privacy + uncensored-model). (2) Live-verification trap: https://www.franksdank.cloud/docs.html is swallowed by the SPA fallback and serves the homepage shell — the docs page lives at /docs/. (3) Served model ids on the API layer are STILL frank-primary/frank-large/frank-secondary (/v1/models live-verified) — the frank-archon/dragoon/zealot rename is harness+homepage+docs only; a served-model-name flip (needs inference restart + OpenWebUI picker ACL re-arm) is offered to Frank, not yet approved. (4) OWU 0.6 API-keys screenshot flow (headless, SSO profile, QA E2E Bot account): click img[src*="profile/image"] → menu Settings → tab Account → button Show → scroll the settings dialog (largest scrollHeight element) to bottom; the screenshot({path}) option did not write a file — grab the returned omp-sshots webp artifact and ffmpeg-convert to PNG. _(context: serverAppConfigs homepage wave-8b: disclosure chips, docs page mobile overhaul, live OWU screenshot, 2026-10-08)_
-HF corpus publication staged 2026-10-08 (NOT yet published — needs Frank's explicit go): dataset repo id frank8465/swift15-gptq-w4-calib-corpus, uploader /home/dank/hf-swift15-corpus/upload-corpus.py (dry-run default, --go arms; radcorpus venv), package = README.md datasheet (CC BY-SA 4.0 for the wikitext share-alike; tiny-codes permissive; fleet slice ours), manifest.json (paths replaced with source dataset names), anonymized calib.jsonl (39,236,524 B, sha256 3e79837f…) + kld.jsonl (5,702,825 B, sha256 b5b70ae9…), scripts/make-corpus.py (literals anonymized) + scripts/scrub-corpus.py. The scrub is deterministic (13 rules: LAN→10.x, dankserver/dankclaw/frankbook→server-a/server-b/client-c, franksdank.cloud|.casa + bare brand→example, /home/dank→/home/user, emails→user@example.com, FRANK_SUDO_PW→DEPLOY_SUDO_PW); audited-clean findings: corpus carries NO real credentials (92 FRANK_SUDO_PW hits = variable refs; private-key hits = bats template headers; password hits = upstream fixtures), the September HF-token leak did NOT reach the corpus; residual probes after scrub are self-replacement/encyclopedic-content noise. Originals stay private, receipted in the datasheet: calib sha256 edb9b7456dbeb8d7def21380778988201aead1cf3f073d3266aaa1447c437603, kld 6e7df2f562d712e04a669220a0c76788688aca5548867596b4e070902937b750 — published corpus is method-reproducible, deliberately NOT byte-identical to the shipped .rad's calibration input. Model repo frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 was created public and its 13 small files uploaded; the 113.7 GiB .rad streams via hf-upload.service (log /home/dank/logs/hf-upload.log, rc sentinel /home/dank/logs/hf-upload.rc). Pending after stream: repo census, dataset --go, model-card §4 dataset link, runbook/AGENTS link. Homepage commit c4dca68 (AI guide link) undeployed — Frank's uncommitted docs.html + openwebui-api-keys.png keep the tree dirty and deploy.sh refuses dirty trees. _(context: serverAppConfigs 2026-10-08: model+corpus publication campaign state at end of session; dataset upload awaits user go.)_
- Dank fleet secret layout (2026-10-08 cutover): all shell secrets live in ~/.config/secrets/ (dir 700, files 600) on BOTH the dev box (blank) and dankserver (dank) — kagi_api_key (dev), hf_token (both; the write-scope fine-grained token is the canonical value, Frank's copy named hf_token_write on dev). The .zshrc files carry ONLY loaders of the form [ -f ~/.config/secrets/X ] && export X="$(cat ~/.config/secrets/X)" — NEVER literal secrets in rc files (the pre-cutover literals on dev .zshrc:3 Kagi and dankserver .zshrc:40 HF were extracted; dankserver keeps .zshrc.bak-20261008). ~/.cache/huggingface/token (600) is kept in sync on dankserver — it is what huggingface_hub and the upload scripts read. Drop-in procedure for a new HF token, on dankserver: read -rs KEY; printf '%s' "$KEY" | tee ~/.config/secrets/hf_token ~/.cache/huggingface/token; chmod 600 both files — value never in chat/argv/history. HF card-metadata law: license: must be a recognized SPDX id — custom licenses need the trio license: other + license_name: + license_link: (the swift-open-license-1.0 upload died on this; the ukisai base card uses the trio). _(context: serverAppConfigs 2026-10-08: HF publication campaign; token-scope 403 led to rc-file secret audit and remediation on both hosts.)_
-Frank's secret-storage convention (established 2026-10-08, both dev box + dankserver): secrets live in ~/.config/secrets/ (dir 700, files 600; HF write token as hf_token_write, canonical copy as hf_token synced to ~/.cache/huggingface/token); .zshrc carries ONLY loaders like export HF_TOKEN="$(cat ~/.config/secrets/hf_token 2/dev/null)" — never literal secrets in rc files (a literal HF token sat in dankserver .zshrc:40 since 2026-08-18 and a literal Kagi key in dev .zshrc:3; both defused this day, .zshrc.bak-20261008 kept). New tokens are dropped in by Frank himself with read -rs — value never touches chat, argv, or history. Hub API facts: huggingface_hub 1.28 whoami is uninformative for fine-grained tokens (type access_token, canWrite/scopes absent) — the decisive write-scope probe is create_repo(real-target, exist_ok=True) (403 = no write; success proves scope and pre-creates); create_repo returns bare RepoUrl with no .private attr. Corpus audit (swift15 calib, /home/dank/corpora/out): calib.jsonl 1,294 prompts 39.5 MB = tiny-codes 45% (permissive/MIT) + wikitext-103 30% (CC BY-SA 4.0 share-alike on derived data, attribution; does not reach model weights) + fleet 25%; fleet+KLD slices are agent session transcripts with pervasive infra fingerprints (47,969 LAN-IP hits, 663 franksdank, 13,644 /home/dank; zero credential VALUES incl. FRANK_SUDO_PW) — scrub vs byte-faithful is Frank's pending ruling; recommended route: scrubbed corpus + scrub script + original's sha256 receipt.
-Dank fleet secret convention (est. 2026-10-08): literal API keys/tokens never live in rc files. Canonical storage = ~/.config/secrets/name (dir 700, file 600) on each host; .zshrc lines are loaders (export KEY="$(cat ~/.config/secrets/key)"). Fixed on 2026-10-08: dev box .zshrc:3 KAGI_API_KEY literal and dankserver .zshrc:40 HF_TOKEN literal (agent-added 2026-08-18) extracted to secret files + loaders; dankserver kept ~/.zshrc.bak-20261008. HF token canonical file ~/.cache/huggingface/token (600) is kept in sync from the secret file and is what the upload scripts read. HF token state 2026-10-08: frank8465's cached token is a legacy read-scoped access_token — repo create fails with 403 "don't have the rights to create a model under the namespace"; Frank is minting a fine-grained WRITE token (needs repos:write + datasets:write); drop-in = read -rs KEY; printf '%s' "$KEY" | tee ~/.config/secrets/hf_token ~/.cache/huggingface/token; chmod 600 … run by Frank himself on dankserver (token must never enter chat/argv/history). Payload corpus audit 2026-10-08: calib/kld corpora contain ZERO literal secrets (all 92 FRANK_SUDO_PW hits are variable references); fingerprints present are /home/dank paths, dankclaw/dankserver names, LAN IPs (publishing the corpus needs a scrub pass). HF model-card payload (README/scripts/artifacts) verified clean: no credentials, no IPs/hostnames.
-Swift1.5 GPTQ-w4 campaign corpus (dankserver /home/dank/corpora/out/, manifest.json seed 20261006): calib.jsonl = 1,294 prompts, mix 45 % code from nampdn-ai/tiny-codes (permissive, attribution) / 30 % prose from Salesforce/wikitext-103 (**CC BY-SA 4.0 — any published derivative dataset must be CC BY-SA + attribution**) / 25 % tools harvested from serverAppConfigs (own content); kld.jsonl = 197 prompts from local repo material only, pools disjoint from calib. Frank approved publishing the corpus as an HF dataset (45 MB; verdict: closes the reproducibility loop StillDeadcode left open — their card ships the recipe with $CALIB unpublished). Gates before publishing: scrub fleet infra fingerprints from harvested slices; license manifest per slice; fine-grained token must carry dataset-write. Token law (measured 2026-10-08): the cached dankserver token is READ-scoped — hf whoami succeeds but create_repo 403s "no rights to create under frank8465"; an unauthenticated API 404 on a repo path is indistinguishable from private (HF returns 401 "Invalid username or password" for both); authed 404 = repo truly absent. Frank must mint the write token himself (write via read -rs on dankserver, never chat/argv). HF upload campaign staging: frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700, payload 122,071,973,888 B sha256 09dd9306…, staging /home/dank/hf-swift15-upload, uploader upload-to-hf.py (dry-run default, --go), launch as transient unit hf-upload.service.
-On the Dank fleet hosts (systemd-logind, KillMode=control-group), setsid nohup CMD & launched from ssh host '…' is SIGKILLed silently ~seconds after the ssh command returns — logind deactivates session-N.scope and kills the whole cgroup; setsid/nohup do not escape the cgroup, and there is NO journal/OOM trace of the death. The reliable detached-launch pattern is ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); systemd-run --user --collect --unit=NAME -p SendSIGKILL=no -p StandardOutput=append:/path.log -p StandardError=append:/path.log sh -c "CMD; echo \$? /path.rc"' — a transient user unit outside the session scope (works because Linger=yes), with the rc sentinel written by the unit itself. Diagnose by: ps -p PID (gone), log file 0 bytes (buffered stdout lost on SIGKILL; python needs -u), and journalctl --since … | grep "session-.*Deactivated" timestamps matching the death window. Hit 2026-10-08 launching the HF 113.7 GiB upload from dankserver.
-Radiance Swift1.5 quant baselines (verified 2026-10-08 from StillDeadcode/swift1.5-qwen3.8-next-flash-fp8-iq4r-moe card + HF API): engine author's file = 122,013,646,848 B; recipe core identical to frank's (gptq table=w4nl group=64 scale=fp8_e4m3 scale2_value=2^-13 transform=fwht128 cd=3; int8 trunk; fp8 HC; PLE fixed scale 1.99317932128906e-4) — differences: 1M vs 10M calibration, 16 protected experts total vs 24/layer (1,152). THEIR FILE WAS NEVER MEASURED against own bf16; card quotes base-model recipe numbers: mean KL 0.0789 all-tokens / 0.0239 text-only, top-1 92.85/94.57 %, and the two-bf16-implementations reference floor is 0.0399. Frank's measured file: KL 0.01314 / top-1 97.92 % — beats the bf16-implementation noise floor ~3x; 97.9 % top-1 ≈ Q6_K-band agreement on the GGUF ladder (no public GGUF ladder exists for this arch, KL is the honest yardstick). Use these as standing comparison baselines for future Swift1.5 quant campaigns. Model-card doctrine Frank approved for HF publications: hard-first ordering (technical content leads, a bold "In plain English:" paragraph CLOSES each section); Randall-Munroe voice (dry concrete humor, no marketing, no AI-isms); include a "Relationship to the engine author's release" section with the measured table; state receipts as exact bytes. _(context: HF publication of frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 (upload launched 2026-10-08); comparison research against the radiance author's own quant of the same checkpoint.)_
-Comparison facts vs StillDeadcode/swift1.5-qwen3.8-next-flash-fp8-iq4r-moe (the radiance author's own quant of our base checkpoint ukisai/Swift1.5, fetched 2026-10-08 via HF API): their .rad = 122,013,646,848 B (ours 122,071,973,888 → +55.6 MiB / +0.048 %). Expert GPTQ rule is byte-identical to our staged recipe (gptq table=w4nl group=64 scale=fp8_e4m3 scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3), as are int8 trunk group-128, fp8 e4m3 HC, PLE fp8 fixed scale 1.99317932128906e-4, and the inert u2 mtp.draft_head rule. Differences: their calibration 1M tokens (ours 10M); their protected set = 16 experts TOTAL (base-model rule transplanted; card example layer4/expert108 = 15.8 % energy from 0.18 % routing) vs our 24/layer = 1,152 (energy ≥3 % AND ≥3× routing, parity-padded). Their card states the file "was not measured against its own bf16 weights" — quoted KL 0.0789 all-tokens / 0.0239 text-only, top-1 92.85/94.57 % are the recipe-on-base numbers, with their two-bf16-implementations reference row at 0.0399 mean KL. Ours measured on-file: 0.01314 / 97.92 % → ~3× better than their bf16-implementation noise floor. Frank's framing (adopted in card §2 "Relationship to the engine author's release"): 93–94 % top-1 ≈ Q4-class, 97.9 % ≈ Q6_K band; "the recipe is theirs, the extra numbers are ours."
-Frank's homepage doctrine updates (2026-10-08, eighth wave — supersedes the octagon-cut corner rule): (1) NO clip-path chamfers on bordered boxes — clip-path slices the 1px border and the corners "feel like gaps"; use border-radius (panels/mi-card 6px, model-info 8px, badge 3px, buttons/callouts 4px). (2) Section-titlediamond must be a fixed-width inline glyph (inline-block, width 18px, margin-left -24px to eat the 24px padding), never absolutely positioned (drifts/wraps). (3) No hover-only content: privacy notes (.svc-warn/.mi-warn) render FULL text inline — title-attribute tooltips are unreachable on touch; the "hover to read" pattern is retired. (4) Closed WIP items grey out via :has(.badge.wip-closed) — dim #5d87a8 name/desc, muted border, art grayscale(0.75) brightness(0.7); WIP: DOWN keeps thered fault treatment (down=fault, closed=status). (5) Homepage data facts: services.json "wip" is a STRING ("down"/"closed"), not boolean — wip badges come from services.json, status.json only drives non-wip probes; the Dragoon tagline is ONE spliced file sfx/dragoon-tagline.ogg (loudnorm -16 LUFS/TP -1.5 per track, silence-trimmed, 350 ms beat, limiter; the two SC2 source oggs trashed). Voice playback proves via tab.requests() 206 + mi-speaking class (detached new Audio, never in DOM).
-Frank's doctrine for public-facing docs (model cards, READMEs): every section must open with a bold-lead "**In plain English:**" paragraph aimed at a non-technical-but-educated reader, and the overall voice must read like Randall Munroe — short declaratives, concrete analogies (cartridge vs DVD, laminating the pages you touch daily), jokes only in the margins, no AI cadence (no delve/seamless/moreover, no bullet-spam, no bold-everything). Technical facts, numbers, and verbatim receipts stay untouched under the plain layer. Applied to the HF card staged at ~/Projects/hf-swift15-upload/README.md (final, synced to dankserver /home/dank/hf-swift15-upload/, gates green: sha256 MATCH on the 122,071,973,888 B .rad = 09dd9306e859f5343d9cb77ee6736636ab9df50f3ed2153350dfbb35be5e6948; publication awaits Frank's --go of upload-to-hf.py). Card prose deliberately omits the w4dn8 relic lore and the mnbt-3072 gate; staged scripts keep their verbatim fleet names as provenance receipts — renaming staged copies would falsify the chain.
-HF publication-card laws (frank8465 Swift1.5 w4 campaign, 2026-10-08): (1) The published .rad filename is a creation-time choice (Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700.rad) — no re-convert needed to rename for readers; the old internal name survives only in the .rad's baked storage.filename metadata (engine ignores it; visible in rad-info artifacts) and in staged toolchain scripts, which stay verbatim fleet receipts — renaming staged copies would forge provenance. Final card carries ZERO relic-name prose except the convert script filename; Frank's doctrine: reader-facing files get clean names, provenance receipts stay untouched. (2) Card style preference (Frank, final review): drop engine-shape bookkeeping readers won't care about (dual-gate MNBT comparison rows), EXPAND the interesting engineering (protected-bf16-expert law: energy-share ≥3% AND energy/routing ≥3×, even parity padding per rank half, lands 24/512 per layer = 1,152 experts = 2.72 GiB). (3) Preview trap: Frank screenshots Chrome with DevTools device-emulation ON — the "fullPage" badge clips tall pages at the emulated viewport height and looks like truncated content; check the served HTML tail first, then say Ctrl+Shift+M. (4) Line-numbered PUT edits on a script shifted by earlier inserts silently CLOBBER the line at the old number (killed two module constants in upload-to-hf.py; caught by dry-run, NameError after the sha gate) — re-view the region in the returned anchored view after every multi-edit sequence, and grep-verify constants before shipping. _(context: Swift1.5 GPTQ-w4 HF publication staging, 2026-10-08: card finalized, dry-run gates green, awaiting Frank's --go)_
-dankserver deployed payload receipt: /home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad is 122,071,973,888 B with sha256 09dd9306e859f5343d9cb77ee6736636ab9df50f3ed2153350dfbb35be5e6948 (computed 2026-10-08 with sha256sum on the host, verified twice by the upload script's gate). scripts/frank-primary.env's FRANK_PRIMARY_SHA256 pins only the fp8-native rollback file, not this w4dn8 one — this is the only recorded receipt for the active payload. HF publication staged as frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 via /home/dank/hf-swift15-upload/upload-to-hf.py (dry-run default, --go arms it). _(context: serverAppConfigs 2026-10-08: HF publication staging for the deployed frank-primary w4dn8 payload; upload pending Frank's --go. Staging at ~/Projects/hf-swift15-upload (dev) + /home/dank/hf-swift15-upload (dankserver).)_
-HF Hub facts for frank8465 (free plan), verified 2026-10-08 via whoami API + hub docs: (1) Free private repos are hard-capped at 100 GB total file storage (all private repos combined); exceeding it disables private repos at renewal, public repos unaffected. (2) Public repo storage on the free tier is best-effort with NO published limit — precedent: a public model repo with a 121.97 GiB single file lives on the free tier. (3) hf auth whoami --output-format json on dankserver's token shows name/fullname/orgs[]/plan (plan None = free) — and access tokens CANNOT list/write datasets (write op → 403 "Invalid credentials in Authorization header"; read → 401), so create_repo must use repo_type="model". (4) NEVER pass a Bearer token in a curl command line — argv is world-readable in /proc/pid/cmdline (ps/pgrep leak it); use curl -K mode-600 file with header = "Authorization: Bearer …". (5) Gated bases (ukisai/Swift1.5) are gated at the CONVERSATIONAL layer only: base weights, tokenizer, AND the MTP tensors ship openly; the ablation variant Quat3rnion/halogen-…-v2-abliterated is ungated. (6) The fleet radiance w4 payload: /home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad, 122,071,973,888 B, sha256 09dd9306e859f5343d9cb77ee6736636ab9df50f3ed2153350dfbb35be5e6948, KLD mean 0.01314 / top-1 97.92 % / ppl ratio 1.0052 vs BF16. Upload staging: /home/dank/hf-swift15-upload/ + upload-to-hf.py (dry-run default, --go arms, radcorpus venv with hf_xet). _(context: Swift1.5 GPTQ-w4 .rad publication campaign on frank8465, October 2026)_
-radiance container introspection: the image entrypoint is radiance, so podman run ... image /opt/radiance/bin/rad-info FILE fails with "E unknown option" — the path is parsed as a radiance flag. Correct form: podman run --rm --entrypoint /opt/radiance/bin/rad-info -v /home/dank/models:/models:ro image /models/rad/FILE.rad. rad-info prints: RAD2 header (bytes, tensor/plane counts), a 181-key metadata block (n_layers, n_expert, full_attention_interval, n_ctx_train...), and an encoding census with per-encoding tensor counts, sizes, and % share — the authoritative receipt for a .rad (gptq params incl. calib path, group, table, transform; rtn scale values; cast bf16 lines). No strings(1) in the image; dump to a temp file and grep it — zsh over ssh mangles sed -n "X,Yp" with $( ) substitutions. _(context: serverAppConfigs 2026-10-08: model stat sheet for frank-primary's qwen4exp-swift15-w4dn8-i8-hc8m.rad came from rad-info census (gptq w4 59.17 GiB/52.1%, PLE fp8 47.68 GiB/41.9%, i8 trunk 3.41 GiB, bf16 2.72 GiB).)_
-radiance swift15 w4 cutover laws (Dankserver frank-primary, live 2026-10-08, main a30ac94): (1) w4 expert tables stream from the file tier IN PLACE only at --max-num-batched-tokens ≤ 2048; at MNBT 32768 the engine demands a widened form and the disk tier refuses ("DID NOT FIT ... stored form is not the container's") — a KLD/eval gate must run at an MNBT the w4 file can serve, and chunk size is softmax-invariant (two PASSes 0.0146 vs 0.0131 mean-KL at MNBT 3072/2048 bound the float-noise band at ~0.0015). (2) The radiance harness contract survives model swaps via --served-model-name "$FRANK_PRIMARY_SERVED_NAME" (=frank-primary): /v1/models id is independent of the .rad filename — models.yml needs no edit on payload swaps. (3) frank-primary prod shape at cutover: fp8 KV + spec depth 2 + seqs 3 + vram pair 26071/3543 (KLD gate scored at fp8 KV, gate and boot agree); decode ~190 t/s live (512-token streamed probe, 30.8 gate). The int8+spec3 campaign arm (136 t/s @44.5k) is NOT what shipped — verify env against the deployed container argv (podman inspect ... --format {{join .Config.Cmd " "}}, rootless as dank) before writing docs. (4) radiance /v1/completions SSE: guard payload=="[DONE]" before json.loads; there is no bare /completion (404). (5) deploy.sh ff-pull on dankserver fails on untracked campaign files (autoresearch-frank-primary.sh) that the merge wants to track — move to ~/.AGENTTRASH first. (6) kld-gate-swift15.sh edits nearly clobbered set -euo pipefail/PIN= by a comment PUT: never place body PUTs over script prologue lines. _(context: serverAppConfigs swift15-radiance-w4dn8 campaign completion: final convert → KLD gate (KL 0.0131, top-1 97.92%) → deploy → live verification, 2026-10-08.)_
-Dank fleet (serverAppConfigs, 2026-10-08): frank-primary Swift1.5-GPTQ-w4 campaign gates ALL PASS — KLD mean KL 0.0131 / top-1 97.92 % (two shapes: MNBT 3072 → 0.0146, MNBT 2048 → 0.0131; the Δ≈0.0015 is the float-order noise band of chunk size, so KLD verdicts are chunk-size invariant), throughput 136 t/s median @44.5k vs 30.8 bar, fleet probes green, 363 bats green on merged main (02fa1e2+). Radiance laws added: (1) w4 expert tables stream from the file tier IN PLACE only at MNBT ≤ 2048 — at MNBT 32768 the engine demands a widened form and the disk tier refuses ("DID NOT FIT … stored form is not the container's"); any offline mode (KLD scoring, calibration) on a disk-tier w4 model must run MNBT ≤ 2048 with a big host pool (40960). (2) A KLD gate must mirror the reference RECORDING shape (tp, pool, MNBT, kv dtype, deterministic) — shape deltas ride on the KL as noise. Watcher lesson: a log-path watcher racing a log truncation of a relaunched job can match the PREVIOUS run's verdict — anchor patterns (^KLD GATE PASS$) and grep for verdict lines the final writer emits, not substrings the launcher echoes. _(context: swift15-radiance-w4dn8 cutover on dankserver; merge to main 02fa1e2, deploy pending at capture time)_
-Radiance w4 disk-tier laws (measured on dankserver swift15 campaign, 2026-10-08): (1) GPTQ w4 expert tables stream from the file tier IN PLACE only at --max-num-batched-tokens = 2048; at larger MNBT the engine demands a widened form and the placement step dies with "DID NOT FIT ... stored form is not the container's (a relayout, a widening or a shard)" — production MNBT 2048 is a kernel law, not a tuning choice. (2) A KLD-scoring boot must mirror the reference RECORDING shape (same pin, tp, kv dtype, headroom, pool, MNBT) because chunk size changes float order that rides on the KL floor; the reference recording ran MNBT 32768 (bf16 model, no kernel constraint), so candidate scoring at 2048 accepts float-order noise as documented deviation. (3) host-pool sized for a bf16 model (12 GiB) under-serves the w4 candidate — the w4 gate_up slab needs the production pool (40960). (4) The calibration tap lives in /opt/radiance/share/radiance/architectures/qwen4exp_fp8.so, not the radiance binary — grep -ac the .so before declaring a flag missing (the image has no strings(1)). _(context: serverAppConfigs swift15-radiance-w4dn8 cutover: KLD gate boot failed twice (pool 12288 DID NOT FIT at MNBT 32768) before shape was matched to recording + w4 kernel law; scripts/kld-gate-swift15.sh encodes the final shape.)_
- Frank's homepage aesthetic doctrine (2026-10-08, after three theme waves): StarCraft Protoss console = flat deep-navy (#050d16) panels with octagon-cut corners on crisp 1px steel borders, warp-blue/psychic-cyan accents used sparingly, Share Tech Mono body + Rajdhani 600 headings (Google Fonts, linked in all three HTML pages: index/admin/docs), colored app icons. His verdict on the maximalist wave: "too busy — clean up and simplify/solidify" — so: no scanlines, no lattice backdrop, no ambient text glows, no inner frame lines, no blinking cursors, no pulsing states; motion is reserved for state only (speaking-voice pulse). Default to minimal on this page; maximal effects get rejected. Docs-page rules from the same session: family docs order = getting-in → troubleshooting → media → AI sections last; the *arr stack is admin-only and never mentioned in family docs; each fleet model carries an HF source link (services.json model.hf + docs Model: bullets). Smoke-test pitfalls on this repo: (1) a python http.server left on port 8099 from an earlier wave serves the OLD tree — pkill -f "http.server 8099" before re-serving a fresh copy; (2) the shared headless browser disk-caches the pages — every verification URL needs a fresh ?v=timestamp query, and style.css/app.js are cached too (the bust propagates via the page URL). Deploy flow unchanged: tests/run.sh (the frank_primary_unit MAX_SEQS failure is Frank's campaign, hands-off) → deploy.sh dankclaw --service homepage → e2e gate.
-Dankclaw homepage (serverAppConfig shared/homepage) aesthetic direction from Frank, 2026-10-08: StarCraft Protoss console — deep space navy #050d16 bg with warp-blue #3fa9ff/psychic-cyan #7fe7ff accents; amber CRT palette explicitly ABANDONED ("abandon amber"). Cornering: MAXIMAL — octagon-cut (all four corners chamfered 18–24px) on panels/cards/badges/buttons with a second inset octagonal frame line (::before double-bevel). Fonts: body "Share Tech Mono", headings "Rajdhani 600" (Google Fonts); VT323 retired ("more legible yet retro-clear terminal font… Think Starcraft"). Tiles must carry pictograms, NOT letter monograms — initials-on-hex icons read as "reverted" and get rejected (app-*.svg: book/GameBoy/folder/crumbling-wall). Two verification gotchas learned this session: (1) a reused python http.server smoke port serves browser disk-cache — always append ?v=epoch cache-bust when re-verifying, and restart the server from the fresh tree (a second python3 -m http.server on a busy port silently fails; the OLD tree keeps serving). (2) deploy.sh homepage runs the Playwright e2e gate ([3/3]); the known pre-existing failure frank_primary_unit.bats MAX_SEQS is Frank's radiance-campaign hands-off zone — never fix it. _(context: Waves 4–6 of the homepage theme work on 2026-10-08: Protoss theme, HF links, navy recolor; live census verified at commit 90f6785 with e2e 34/34.)_
-Verifying static frontend changes via a scratch python3 -m http.server + headless browser on the Dank fleet has two traps that fake the result: (1) a second python3 -m http.server same-port launch silently fails when the port is occupied — the OLD server keeps serving the OLD tree copy, so the browser census reports stale state and a fresh copy looks broken (observed: fresh docs.html served 0 HF links while curl on the same URL proved the new file was live); (2) the headless browser disk-caches heuristic-cached responses (http.server sends no Cache-Control), so re-navigating to the same URL after re-syncing files returns the cached page. Protocol: kill any prior server on the port (pkill -f "http.server port") before starting the fresh one, verify the server actually serves the new tree with curl | grep -c new-marker, and cache-bust browser navigation with ?v=timestamp. Also: bash tool calls with async + name + timeout can fail with "Service mode does not accept async or timeout"; plain async works. _(context: 2026-10-08 serverAppConfig homepage waves: stale smoke5 server on :8099 + browser cache produced two false-negative verification rounds before curl proved the tree was fresh.)_
-Dankclaw homepage client-side identity gating: Caddy's post-auth listener (http://:8017, homepage host block) can echo the Authentik outpost's injected identity headers via a templated respond — handle /whoami { templates; header Content-Type application/json; respond "{\"user\":\"{header.X-Authentik-Username}\",\"groups\":\"{header.X-Authentik-Groups}\"}" }. Spoof-proof because the proxy outpost strips client-supplied X-Authentik-* before injecting. app.js shows the admin button only when groups match homepage-admins; the /admin* 403 regex gate stays the real enforcement. Sequencing law learned the hard way: deploy caddy (whoami) BEFORE homepage — a homepage deploy whose e2e gate runs before /whoami exists fails the admin-visibility spec. Homepage card assets: every app card renders art (services.json field, hex-clipped .app-art); brand marks fetched from cdn.simpleicons.org/slug/ffb000 (dataset is lean — many slugs 404: openwebui, seerr, romm, seerr, loki, valkey, *arr beyond sonarr/radarr, tautulli, comfyui), improvised amber-hex monogram SVGs for the rest; official logos: openwebui.com/logo.png, seerr.dev/os_logo_filled.svg. Model cards render in registry order (zealot→dragoon→archon pinned in homepage_config.bats); e2e panel census counts #grid .panel .name, #grid .mi-name because model cards fuse into one panel. _(context: serverAppConfig homepage waves 2026-10-08 (commits through the icons/clickable/header wave); e2e gate 33/33 after caddy-before-homepage deploy ordering.)_
-Dankclaw homepage admin identity-gate pattern (deployed 2026-10-08, serverAppConfigs): to gate a client-side control to authentik group members, add a Caddy handle /whoami { templates; header Content-Type application/json; respond "{...X-Authentik-Username/X-Authentik-Groups...}" } on the post-auth :8017 listener — the Authentik proxy outpost injects X-Authentik-* headers on its upstream request AND strips client-supplied copies, so echoing them back is spoof-proof. JS fetches /whoami and shows the control only when groups match (homepage-admins). The real gate stays server-side (the /admin* Caddy 403 on the same group header); the echo only controls what is shown. Sequencing law learned the hard way: deploy the caddy change (whoami endpoint) BEFORE the frontend that consumes it — the homepage deploy's e2e gate fired in the window where the new page expected /whoami and caddy didn't serve it yet, failing the admin-visibility spec. Related homepage conventions (all pinned in tests/homepage_config.bats): services.json registry order IS the card render order (model cards zealot→dragoon→archon); every app card carries hex-clipped art via an art field (brand marks from cdn.simpleicons.org/slug/ffb000, amber-tinted SVGs; improvised monogram SVGs = hex polygon + amber initials for apps with no mark — most *arr/wiki brands are absent from the CDN's dataset, check cdn 404s, don't assume slugs); card-level warn field renders as a red hover-label ("⚠ … — hover to read" with title tooltip); the common uncensored legal warning text (adult content, harmful/hateful/illegal, "sensitive offensive cybersecurity topics", "don't ruin this server for everybody") is shared byte-identical between dragoon and zealot. _(context: serverAppConfigs homepage wave 2026-10-08 (commit c1e192d): /whoami gate, app icons, warning unification; documented in serverAppConfigs-docs/Infrastructure/Homepage.md wave entries)_
-Radiance (StillDeadcode) qwen4exp w4-GPTQ laws, measured 2026-10-07/08 on dankserver (Swift1.5 campaign): (1) ONE ENCODING PER EXPERT TABLE: rad_arch_declare rejects splitting ffn_gate_up_exps (w4) from ffn_down_exps (block-fp8) — "one table of experts takes one encoding, and only a whole expert kept plain bf16 is served beside it"; the PROTECTED section may hold only whole bf16 experts. (2) GPTQ w4 gate_up experts CANNOT be disk-tier streamed — stored form (u4[1x1] codes + fp8/f32 scale planes) is not the served form (dequantized tables); boot fails "DID NOT FIT ... store is not the container's"; they must fit the host pool (measured 11.72 GiB/rank for 48×512 at hidden 2560). (3) The activation arena scales HARD with --max-num-batched-tokens on w4 geometry: MNBT 32768 → 33.3 GiB arena on a 31.86 GiB card (262.5 MiB per 256 tokens); prefill-only jobs (KLD) run fine at MNBT 3072. (4) --kld-ref takes NO --kld-corpus (engine rejects the pair: tokens come from the reference); --kld-out FILE writes the JSON report + FILE.rows. (5) KLD scoring of a TP=2 container can place rank 1 on the host backend when the placement plan lets card 0 absorb the resident set (GPU[1] flat at 241 MiB while GPU[0] holds 25.88 GiB) — mathematically valid (reference-verified host kernels), and at pool 40960 it scored at 22.6 pos/s vs the disk-tier record's 3.3 pos/s. (6) rad-convert GPTQ on this box: ~6 min/layer × 49 layers (~4 h CPU, load 8), fully production-compatible (no GPU). _(context: serverAppConfigs swift15-radiance-w4dn8 campaign; candidate qwen4exp-swift15-w4dn8-i8-hc8m.rad 116,416 MiB)_
-Dank-fleet family-facing model names (2026-10-08, Frank's StarCraft II Protoss theme): frank-large → frank-dragoon, frank-primary → frank-archon, frank-secondary → frank-zealot. These are DISPLAY names on the homepage and user docs only — endpoint services, OpenWebUI model ids, and API model ids remain frank-large/frank-primary/frank-secondary; agent configs (opencode/Pi) keep using frank-secondary. Supervision tiers (Frank's classification): frank-dragoon and frank-archon are safe for autonomous low-supervision use; frank-zealot (Qwen3.5-9B-Uncensored) requires tight human supervision. frank-zealot is the recommended compaction model for agent harnesses (compaction severely slows the bigger models far longer). Frank's rule: legal-consequence warnings for uncensored/abliterated models render as mouseover hover labels (title tooltips), NOT as always-visible wall text; other descriptive additions display inline. Homepage model cards fused into a single "Model Information" component (app.js renderModelInfo, .mi-* classes) — the e2e user-homepage census counts #grid .panel .name, #grid .mi-name because of this. A wip:"down" state renders the red "WIP: DOWN" badge (Seerr, Calibre-Web, RomM, Jellyfin are WIP: DOWN, Open links removed). _(context: serverAppConfig homepage Protoss wave 2026-10-08: /sfx/ ogg voices + unit art added; @data allowlist, deploy_homepage sfx block, caddy bats pin, e2e census fix (fused .mi-name cards) all required for a green deploy.)_
-NEVER edit a bash script on disk while bash is executing it: bash reads scripts lazily by byte offset, so mid-run edits shift offsets and corrupt the remaining execution. On the 2026-10-07 Swift15 convert this made the plan "hot-patch the running convert script" impossible — the fix went to git (canonical copy) and the host file was left untouched until the run finished. Related corollary that bit the same night: a scp round-trip during an emergency edit silently dropped cleanup lines from scripts/convert-swift15-w4dn8.sh — after scp'ing a host file back as canonical, diff the whole file, not just the edited lines. _(context: serverAppConfigs swift15-radiance campaign: convert running from host copy; kld.jsonl cleanup lines vanished through the scp-back path; verified via grep on the running host copy (landmine disarmed by accident).)_
-Radiance qwen4exp_fp8 plugin law (measured at rad-convert declare, 2026-10-07, Swift1.5 campaign): ONE ENCODING PER EXPERT TABLE per layer — "one table of experts takes one encoding, and only a whole expert kept plain bf16 is served beside it" (rad_arch_declare). A recipe may NOT split ffn_gate_up_exps (w4 GPTQ) from ffn_down_exps (block-fp8): the serve kernel is a single w4nl64a8h table for the whole expert set; the only legal beside-table content is whole experts cast dtype=bf16 (the PROTECTED section). Related: the ukisai/Swift1.5-Qwen3.8-Flash-Next "BF16" checkpoint is pure BF16 (1655 BF16 + 3 I64 PLE-index tensors, HF tensor names — no fp8 storage); rad-convert maps HF names to blk.N.ffn_*_exps.M itself. rad-convert's positional input must be the in-container path INCLUDING subdirs (basename-stripping the mount prefix breaks nested checkpoint dirs like /models/ukisai/name). GPTQ receipt line to watch in convert logs: "libquant: GPTQ quantises 'tensor' against N calibration file(s), K=cols". _(context: serverAppConfigs swift15-radiance-w4dn8 campaign, dankserver: the designed w4dn8 split tier was vetoed at rad-convert declare; recipe reduced to all-experts GPTQ w4 + 24 whole-bf16 protected experts (commit 73c37f4).)_
- Dankclaw homepage family model names (2026-10-08, Frank's StarCraft Protoss theme): frank-large → frank-dragoon, frank-primary → frank-archon, frank-secondary → frank-zealot. Endpoint/API/OpenWebUI ids are UNCHANGED (frank-large/frank-primary/frank-secondary remain the ids agents and API configs must use). Frank's supervision classification: dragoon and archon are safe for autonomous low-supervision use; zealot requires tight human supervision. Compaction model for agent harnesses: frank-zealot (compaction jobs severely slow the bigger models). Frank's presentation preference: legal/uncensored warnings must be mouseover tooltip labels (⚠ hover-to-read), not visible wall text; other descriptive notes stay plainly visible. Media apps Seerr, Calibre-Web, RomM, Jellyfin are WIP: DOWN for family users (a third wip state "down" exists in services.json/badge CSS); Copyparty is the access point for all media apps' content including the WIP ones; Gitea and Plex use logins separate from the Authentik SSO; SearXNG is obfuscated-not-anonymous (upstream API keys tie to Frank's identity). Homepage assets: new top-level files (e.g. sfx/) must be added to the Caddy @data path allowlist AND the deploy_homepage copy list — same law as img/. _(context: serverAppConfig homepage Protoss rename wave, commit 203b0be; component = Model Information fused panel in shared/homepage/app.js with /sfx/ voices fetched from starcraft.fandom.com (MediaWiki API: action=parse wikitext, {{sm2|File.ogg}} quotes, imageinfo URLs for unit art).)_
-Dankclaw homepage live verification (serverAppConfigs): (1) The post-auth Caddy listener (http://:8017, @home handler) serves files ONLY through the @data path allowlist (@data path /services.json /status.json /app.js /style.css /deployed.js /img/* in dankclaw/Caddyfile); anything unmatched falls to the catch-all and is answered with index.html as 200 text/html — a PNG arriving as text/html passes every status-code and file-listing check, so verify assets by content-type/bytes, never by HTTP status. New top-level homepage assets MUST be added to @data (pinned in tests/homepage_caddy.bats). (2) Live-check guarded pages with the eval browser runtime: browser.open({url, name}) (single options object; same for browser.close({name})); browser.tabs() lists {name,url,title}; tab.evaluate(string) runs JS in the page and returns a string — inside it, escaping: the eval script is built in a JS template literal, so use '\\n' (a raw \n interpolates and breaks page-code string literals); tab.url() returns an object, not a string. The shared headless browser keeps the qa-e2e SSO session, so https://www.franksdank.cloud/docs renders through Anubis+Authentik without manual login (the naked domain 308s to www and STRIPS the path — always probe www directly). Fetch-in-page probe: fetch('/img/x.png').then(r=r.status+' '+r.headers.get('content-type')+' '+blob.size) proved the @data bug; [...document.images].map(i=i.complete && i.naturalWidth0) proves rendering. _(context: 2026-10-08 homepage amber-theme session: @data allowlist bug (img/* served as index.html), eval-browser live verification of /docs and AI cards through the guard chain)_
-Running-bash-script and scp round-trip discipline (serverAppConfigs campaign ops): (1) NEVER edit a bash script on the host while it is executing — bash reads script files lazily by byte offset, so edits shift offsets and corrupt the remaining execution. Fix committed copies, sync after the job ends. (2) After scp-based host edits, the HOST file is the ground truth for any running job (e.g. rad-convert loaded the recipe at launch); before git merges on the host worktree, diff each locally-modified/untracked file against the branch tip and commit the host copy as canonical when it is what the job loaded — a partial/stale copy can silently get committed (a committed recipe lost 10 of 24 protected experts this way; the running convert proved it). (3) The convert-script cleanup section (rm corpus files) was silently lost during a python line-rewrite round-trip: after scripted edits, grep for the intended deletions/markers to prove they survived. (4) A truncated ls | head -N listing caused a false "corpus deleted" alarm — when a critical file is absent from a truncated listing, stat the file directly beforedeclaring loss. _(context: swift15-radiance-w4dn8 campaign 2026-10-07/08 on dankserver: KLD gate preparation between rad-convert stages)_
-frank-large (Dankclaw halogen-flash-server) 400-zone law (2026-10-07/08): the harness clamps max_tokens = min(maxTokens, contextWindow − chars/4 est − 4096), and the est UNDER-counts the rendered prompt by ~3 % on code/agent traffic (measured twice: est 152,938 → server 157,388; est 158,906 → server 165,301). Compaction fires only at 85 % of the harness contextWindow, so window == engine ceiling made every request in the est ~146k–184k zone a guaranteed 400 ("max_tokens N does not fit: prompt is X tokens and the context is 200192"). Fix contract (deployed via scripts/frank-large.env): HALOGEN_CTX=262144 (native ceiling), HALOGEN_KV_POOL_POSITIONS=524288 (2×ctx, 14.0 GiB), models.yml frank-large contextWindow=217088 (= ceiling − 45k estimate headroom), maxTokens=45056; worst-case request ≈ 219k ≤ 262,144. Compaction input rides on frank-secondary: ≈ 170k ≤ its 200,704 slot. The general rule: a harness contextWindow must sit BELOW the engine ceiling by (maxTokens + reserve + est-error margin), never equal it. A "context is 200192" 400 after the env change means the OLD boot is still serving — deploy is pending, not the fix failing. _(context: serverAppConfigs frank-large token-limit fix session 2026-10-08: env+bats+AGENTS.md+runbook edited, frank_large_unit.bats 20/20 green, full suite + deploy dankclaw --service frank-large pending at hand-off. Supersedes the older "clamp max_tokens to 57344" note.)_
-Dank fleet + omp harness token-fit law (diagnosed 2026-10-07, frank-large halogen 400s): the engine fit check is prompt_tokens + max_tokens ≤ engine window (halogen HALOGEN_CTX, llama.cpp slot ctx — a 400, never truncation), while the omp harness clamps max_tokens = min(maxTokens, contextWindow − chars/4_est − 4096) and only compacts at compaction.thresholdPercent (85 % in config.yml; reserveTokens-16384/70 % comments in models.yml/AGENTS.md were STALE). chars/4 under-counts the rendered prompt by ~3 % on code/agent traffic (measured: est 152,938 vs server 157,388 = 4,450 the 4,096 margin). Consequence: if harness contextWindow == engine ceiling, every request above ~4096/0.03 ≈ 137-146k est (65-85 % of a 200k window) is a guaranteed 400 — the zone between where the estimate error exceeds 4096 and the 85 % compaction threshold. FIX shape: engine ceiling − harness contextWindow must exceed maxTokens + ~10 % of contextWindow (safety for est error). Deployed fix for frank-large: HALOGEN_CTX 200192→262144 (native), HALOGEN_KV_POOL_POSITIONS=2×ctx=524288 (~28 KiB/pos = 14.0 GiB), models.yml contextWindow 217088 (212×1024) keeping maxTokens 45056 → compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k target), worst-case request ≈ 219k ≤ 262,144 with 42k slack, and the compaction input (threshold − keepRecentTokens 20,000 ≈ 170k) still fits frank-secondary's 200,704 compaction slot. Frank explicitly demanded reaching 180k tokens on frank-large; he considers prior config "misunderstood" — check these three numbers (CTX, contextWindow, thresholdPercent) together, never one alone. _(context: serverAppConfig frank-large halogen-flash-server 400 "max_tokens does not fit" campaign, 2026-10-07; artifacts: scripts/frank-large.env, tests/frank_large_unit.bats fit-law test, models.yml frank-large block)_
- OpenWebUI 0.6.x (open-webui on Dankclaw): external OpenAI-connection models are invisible to non-admin users unless a workspace row EXISTS in the model table (singular) with id == upstream model id, base_model_id NULL, is_active 1, PLUS an access_grant row (resource_type='model', resource_id=model id, principal_type='user', principal_id='*', permission='read'). The /api/models aggregation (utils/models.py get_filtered_models) DROPS models with no info for role=user; the workspace row supplies the info; the admin bypasses the filter, so admin UI looks fine while all family users see "No results found". An alias row (id != base_model_id) breaks chat completion ("Model not found") — id must equal the upstream id. API keys for regular users require user.permissions.features.api_keys=true in the config table (default false); the UI path is User menu → Settings → Account → API keys → Show → Create new secret key (0.6 layout, not the old SettingsAPI Keys tab). All three endpoints: DB at /home/dank/Containers/open-webui/data/webui.db (host), mount /app/backend/data; no sqlite3 CLI on host — use host python3; live process caches model lists ~1 s (MODELS_CACHE_TTL). The 2026-10-07 18:35 picker rebuild (frank-secondary swap) dropped the grants — fixed by inserting the row+grant pair.
-Dankclaw Authentik (auth.franksdank.cloud) human roster as of 2026-10-08: akadmin (pk 4, superuser, frank8465@protonmail.com), admin (pk 5, admin@franksdank.cloud, homepage-admins only — legacy setup-era account, deactivation pending Frank's call), alex (pk 15), ridley (pk 13), kassie (pk 14). Non-human: AnonymousUser, qa-e2e, qa-e2e-admin, svc-jellyfin-ldap, ak-outpost-* (path goauthentik.io/outposts). The human-user template (documented in serverAppConfigs-docs/Infrastructure/Authentik.md §User provisioning playbook) = username + user@franksdank.cloud + display name + is_active + set_password + groups {Non-Admin Users, jellyfin_users, openwebui_users}; EXCLUDE authentik Admins / homepage-admins / *admins. Audit a user row BEFORE deleting: MFA device tables (authentik_stages_authenticator_{totp,webauthn,static,email,duo,sms}*device), authentik_core_token, app entitlements, authentik_providers_proxy_proxysession (stale claim cache). Event history survives a user delete (authentik_events_event "user" is denormalized JSONB, filter with "user"-'pk'='N'). Column names in the authentik-postgres DB differ from the ORM: certificatekeypair has no user_id, authentik_core_user has no source_id, events column is quoted "user" (jsonb) — select * / information_schema first, never guess. Passwords travel only on the ak shell stdin (heredoc over ssh), never argv/ps. _(context: 2026-10-08 session on Dankclaw: created kassie (pk 14), deleted pk 7 and recreated alex (pk 15) into the template, audited via ak shell + direct psql against authentik-postgres. Roster verified live with User.objects.exclude(path__startswith="goauthentik.io").)_
-Dank fleet operational constraint (from Frank, 2026-10-07, verbatim intensity intact): "DO NOT FUCKING TOUCH ANYTHING BELONGING TO FRANK-PRIMARY" while dankserver's radiance box is running a multi-day quantization job (the Swift15-radiance KLD/calibration campaign). Practical meaning for an agent: do not restart, redeploy, benchmark, mask, flag-tune, or commit-and-push changes to scripts/frank-primary.env / frank-primary.service / serve-frank-primary.sh, and do not run the fleet-wide deploy.sh dankserver drain (it walks every service revision marker and would fire the frank-primary restart). Test failures that belong to frank-primary (e.g. tests/frank_primary_unit.bats asserting FRANK_PRIMARY_MAX_SEQS=2 while the env says 3) are NOT yours to fix during the window — commit only the files in your own task's blast radius; a green-suite gate yields to the campaign. Second signal from the same window: when Frank reports an endpoint healthy and says "no need to verify", stop benchmarking — record the evidence you already observed (boot ledger, e2e gate) and name the evidence gap explicitly instead of probing further. _(context: serverAppConfigs session 2026-10-07: frank-secondary (Dankclaw) swap work was blocked by a pre-existing frank-primary test failure; asking about resolving it drew the hands-off instruction. Deployed frank-secondary at ec6184d/5278f5b with frank-primary untouched.)_
-Dank fleet (serverAppConfigs): every LLM serve script is a DIRECT systemd ExecStart (frank-primary/large/secondary), so rewriting one with a file-writing tool that drops #!/usr/bin/env bash makes the unit boot-loop with status=203/EXEC, Mem peak: ~1.9M, Failed at step EXEC spawning /home/dank/.../serve-svc.sh in the journal, and nvidia-smi flat at 1 MiB (the GPU is never touched). Diagnostic: 203/EXEC + zero GPU use = the script never ran; check head -1 and the exec bit (git ls-files -s shows the tracked mode; git preserves it, so a non-exec mode is committed drift). systemctl status's unit Description shows the LAST-deployed era's text, not the repo's — a stale Description is cosmetic until the next relink_unit (don't restart a serving endpoint for it). Related llama.cpp fact: the log line n_ctx_seq (N) n_ctx_train (M) is informational when N M (context below the trained window is safe); it is only alarming when a drafter runs ABOVE its own n_ctx_train. _(context: 2026-10-07 frank-secondary Qwen3.5-9B swap on Dankclaw: my rewrite of scripts/serve-frank-secondary.sh lost the shebang; the unit restart-looped 14 times (10 s apart) before the deploy's health gate; fix commit ec6184d. Boot ledger after the fix: 21,436 MiB used / 2,689 free, KV 12,544 MiB f16 K + f16 V.)_
-llama.cpp master (build 5fc4f3c) attention_k_eq_v does NOT shrink the KV cache — the cache bills the K+V pair TWICE: 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at q8_0/q8_0. Verified byte-exactly on Gemma4-26B-A4B (frank-secondary, dankclaw 3090, 2026-10-07) against the engine's own verbosity-4 llama_kv_cache lines: full cache = n_full_layers × n_head_kv × head_dim × rate × ctx = 5 × 2 × 256 × 6.125 × 200,704 = 3,001.25 MiB ✓; SWA cache = n_swa_layers × window × n_head_kv_swa × head_dim × rate = 25 × 1024 × 8 × 256 × 6.125 = 306.25 MiB ✓. Vcur = Kcur is a compute-graph mapping only. Also: read n_head_kv per layer (it is an ARRAY — gemma4 has 8 KV heads on sliding layers, 2 on full-attention layers) and sliding_window_pattern from the load log at verbosity ≥3, never from the HF config's single-value fields. Lesson: earlier fleet notes claimed the pool "bills K-only" (2.05 GiB at 200,704) — that figure came from arithmetic, not from the engine; a rate law must be fitted to the engine's own printed buffer sizes (-v/--log-verbosity 4 makes them appear in the journal). _(context: serverAppConfigs frank-secondary Gemma4 swap 2026-10-07: the f16-K raise (+2,161 MiB KV) booted at 21,868 used / 2,257 free because the ubatch 2048→1024 cut returned ~616 MiB of compute buffer; decode measured 139.9 t/s temp 0. Runbook: serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md)_
-frank-secondary (Dankclaw RTX 3090, :7771) runs Gemma4-26B-A4B QAT Uncensored Q4_K_M (HauhauCS, 16,780,192,888 B) since 2026-10-07, ONE 200,704-token slot (--parallel 1), f16 K + q8_0 V, MTP OFF. Measured VRAM (nvidia-smi at boot, the referee, not arithmetic): MTP OFF = 21,890 MiB used / 2,235 free; MTP ON (separate gemma4-assistant sidecar mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB, loaded via --spec-draft-model + --spec-type draft-mtp) = 23,806 / 319 free. So the drafter costs ~1.9 GiB. Gemma4 Q8 with 2×200,704 is IMPOSSIBLE on the 24 GB card (weights alone ~26 GiB). Measured engine facts, not estimates: the assistant warns "n_ctx_seq (200704) n_ctx_train (131072)" (draft ctx is built at the target's full window, 53 % past the assistant's trained window) and declares shared-KV layers (main-model layers 0-3 sharing with 28/29); the main model's n_ctx_train is 262,144, so 200,704 is inside its trained window. With MTP off the main boot prints neither warning. Q4_K_M 1×200,704 with the MTP assistant is knife-edge; MTP off leaves ~2.2 GiB margin, which is what funds raising K to f16 (+2.05 GiB: 5 full-attention layers × 200,704 tok × 8 heads × 256 dim × 2 B). All three fleet endpoints (frank-primary, frank-large, frank-secondary) now use frank-secondary as compactionModel; compaction input fits its 200,704 window (~133k worst case) — a single --parallel 1 slot means compaction requests serialize behind live generation. _(context: serverAppConfig main branch, 2026-10-07 frank-secondary swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS (2×114,688) to Gemma4-26B-A4B Q4_K_M; VRAM numbers read from live nvidia-smi boots, warnings from journalctl -u frank-secondary.)_
-Gemma4-26B-A4B Q4_K_M on the RTX 3090 (frank-secondary, dankclaw), MEASURED boot 2026-10-07 — supersedes the same-morning head-math estimate that said 1×200,704 does NOT fit: Q4_K_M 1×200,704 ctx with q8_0 K+V BOOTS with margin: 21,890 MiB used / 2,235 MiB free. With MTP assistant loaded it is 23,806 / 319 (drafter costs ~1.9 GiB — its KV is built against the target's full window). Key sizing facts: KV scales ONLY with the 5 full-attention layers of 30 (25 sliding-window layers at window 1024 cost a fixed 1024-token ring each), 8 KV heads × 256 dim; attention_k_eq_v=True makes Vcur=Kcur in the compute graph (gemma4.cpp: absent wv → Vcur = Kcur) and in llama.cpp build 5fc4f3c the KV cache honors k_eq_v (pool measured ~2 GiB at 200,704, matching K-only accounting), so bill K-only, not 2×. llama.cpp master build at commit 5fc4f3c accepts cache types: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 — NO fp8/f8 (user asked for "FP8 KV" → q8_0 is the FP8 class and is what ships). MTP for Gemma4 is a SEPARATE gguf (arch gemma4-assistant) loaded via --spec-draft-model + --spec-type draft-mtp; disabling = SPEC_TYPE=none which must strip BOTH flags (flags move as a set). The endpoint serves ONE 200,704 slot, MTP OFF, vision via publisher mmproj. _(context: serverAppConfig frank-secondary swap 2026-10-07: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP Q4_K_M replaced Swift-1.5-27B at 2×114,688. Verified via llama-server argv, /v1/models (n_ctx 200704, ftype Q4_K_M), nvidia-smi. Corrects the earlier estimate memory entry that claimed 1×200,704 needed ~29 GiB.)_
-llama.cpp build (commit 5fc4f3c, 2026-10-01) KV cache types: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1. NO true fp8 KV dtype — q8_0 is the "FP8 class" (1.0625 B/elem, 8.5 effective bits). The grep -ac arch on the llama-server binary returns 0 for ALL arch names (including qwen3_5 which it's actively serving) — the arch strings are stored in a packed/constructed form that grep can't catch; use source registration (llama-arch.cpp) to verify arch support, not binary grep.
-Gemma4-26B-A4B (google/gemma-4-26B-A4B-it) VRAM fit on the RTX 3090 (24,125 MiB), computed 2026-10-07 for the frank-secondary replacement question: Q4_K_M 2×200,704 does NOT fit — estimated ~33 GiB. Breakdown: weights 17,958 MiB (Q4_K_M 16,018 + mmproj-BF16 1,700 with compute buffers + MTP assistant 240) + KV pool 7,864 MiB (401,408 tok × 5 full-attn layers × 8 KV heads × 256 dim × 2 [K+V] × 1 B q8_0) + MTP draft KV ~1,600 MiB + compute/runtime overhead ~5,800 MiB. The swing factor: Gemma4 is MoE hybrid, 30 layers = 25 sliding-window (fixed 1024 ring) + 5 full-attention (layers [5,11,17,23,29]), so KV scales only with the 5 full layers, not 30. CRITICAL non-obvious build fact: in llama.cpp master build 5fc4f3c, attention_k_eq_v: True makes Vcur=Kcur in the compute graph (gemma4.cpp: Vcur = wv ? ... : Kcur), but the KV CACHE still allocates full 2× (K+V) — there is no k_only flag, so k_eq_v does NOT halve the pool. Feasible shapes: Q4_K_M 1×64k (~24.4 GiB, razor-thin) or Q3_K_M 1×128k (~24.1 GiB, needs a Q3_K_M download); dropping MTP saves ~1.6 GiB. Gemma4 MTP is a SEPARATE file (mtp-gemma-4-26B-A4B-it.gguf, arch "gemma4-assistant") loaded via --spec-type draft-mtp, unlike Swift's in-file MTP head. Diagnostic lesson: grep -ac arch llama-server returns 0 for EVERY arch name (including qwen3_5 which it's actively serving) because the packed binary doesn't expose arch names as plain strings — do NOT use grep to check arch support; check the source tree (src/models/gemma4.cpp exists at the build commit) instead. _(context: serverAppConfigs, frank-secondary (llama.cpp on RTX 3090, Dankclaw) — evaluating HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP as a replacement; build /home/dank/GitPackages/llama-master/build-cuda/bin/llama-server at commit 5fc4f3c.)_
- Two durable lessons from the Swift15 campaign (2026-10-07, dankserver): (1) The script-swap law applies to plain scp too: overwriting a bash script a running process owns shifts its lazy-read byte offset — when the long child (podman run | tee) exits, bash resumes mid-token and dies rc=127 ("--security-opt: command not found") AFTER the real work succeeded. Recovery pattern: gate on the engine's own receipt (e.g. kld.json "rows": 50235 + file count + size) and resume the pipeline with a SKIP flag (kld-then-calib.sh SKIP_KLD=1); never restart the long job over a wrapper death. (2) serverAppConfigs docs seam "docs index links every docs page" (tests/docs_completeness.bats) requires every new page under serverAppConfigs-docs/ to be linked from the TOP-LEVEL serverAppConfigs-docs/index.md — a link in the sub-index (LLM/index.md) is not enough; add the top-level entry in the same commit as the page. _(context: serverAppConfig swift15-radiance-w4dn8 branch: KLD recording wrapper killed by mid-run scp of commit 45de2fd (rc=127, record intact); docs seam 135 failed until the new runbook was linked from the top-level index.)_
-Overwriting a running bash script (scp/git checkout) shifts bash's lazy-read byte offset: the running script resumes mid-lineat the next read boundary and executes garbage — symptom is a bogus "line N: token: command not found" and exit 127 AFTER the real work finished. Recovery pattern: judge the ARTIFACT, not the wrapper — e.g. radiance KLD record was complete (kld.json rows: 50235) despite the wrapper's rc=127; add a resume flag (SKIP_KLD=1) that gates on the artifact receipt (rows count, file count, size floor) and skips straight to the next phase. Prevention: campaign-named script files launched once per phase; never scp a script a running process owns; sync edits BEFORE launch. (serverAppConfigs 2026-10-07: mid-run scp of a perf edit killed the 4-hour KLD wrapper; KLD reference 24 GiB/50,235 rows survived.) _(context: Third fleet occurrence of the bash script-swap hazard (after the 2026-10-01 autoresearch collision); this one added the artifact-receipt resume pattern.)_
-Radiance (StillDeadcode) VRAM ledger laws on dankserver R9700 (32 GiB card), measured 2026-10-06 bf16 disk-tier boots: the weights slab DERIVES as card − headroom − vram-kv-mib; the engine refuses (DID NOT FIT) and never shrinks for you. The disk tier has a hard weights floor of 9.57 GiB (4.21 GiB of two layer buffers + 5.36 GiB fixed weights) — so on a 32 GiB card with ~18 GiB fixed per-rank claims (QSA/DeltaNet/arena), --vram-kv-mib is capped near 3.5 GiB; 3072 MiB is the safe value, 4608 starves the disk tier. The ratio-derived pool (default EVC split) came up SHORT by 217 MiB at 32×16k fp8-KV slots — for offline (KLD/calib) shapes set --vram-kv-mib explicitly. Disk-tier prefill rate is STEP-SIZE-driven, not slot-count: MNBT 32768 warms to ~1,360 tok/s, MNBT 65536 ≈ 930 (vendor); slot count only queues pushes. Also: a foreground-podman systemd unit (frank-primary pattern) shows systemctl is-active=failed after every intentional stop (main process exits 143 on SIGTERM) — cosmetic, not a crash; check ExecStop status + journal, not is-active. _(context: serverAppConfig swift15-radiance-w4dn8 campaign 2026-10-06: three boot failures on the bf16 KLD/calib shape cost two relaunch cycles; the slab formula + disk-tier floor explain all three.)_
-Radiance offline-run (KLD record / calibration serve) sizing law on dankserver, measured 2026-10-06 with the bf16 qwen4exp container: the VRAM weights slab DERIVES as card − headroom − KV, and --weights-disk-tier REFUSES under ~9.57 GiB (4.21 GiB layer buffers + 5.36 GiB fixed) — so on a 32 GiB card, explicit --vram-kv-mib must stay ≤ ~3.5 GiB for offline arms (3072 works with --max-num-seqs 4 at 16k fp8); the ratio-derived default came up SHORT by 217 MiB at 32×16k and the engine never shrinks for you. Recording is prefill/disk-bound, so fewer resident slots only queue pushes. Also: the image ENTRYPOINT is already radiance (passing "radiance" as argv → "E unknown option 'radiance'" rc=2), and the image has no ls — exec probes use sh. _(context: serverAppConfig swift15-radiance-w4dn8 campaign 2026-10-06: three failed KLD boots (entrypoint arg, KV SHORT, weights-slab starvation) before the offline shape booted; codified in scripts/run-swift15-bf16.sh comments.)_
-Radiance (StillDeadcode) engine-feature probes MUST grep the architecture plugins, not just the binaries: model-specific features (calibration tap, KLD, QSA) live in /opt/radiance/share/radiance/architectures/arch.so (e.g. qwen4exp_fp8.so carries RADIANCE_CALIB_DIR/RADIANCE_CALIB_TOKENS on 1.2.0; the 1.0.13 plugin has RADIANCE_CALIB_DIR only) and quantizers in share/radiance/quantizers/libquant.so (gptq has calib=/damp/act_order/cd on BOTH 1.0.13 and 1.2.0). Grepping only bin/radiance produced a false "1.0.13 has no calibration tap" verdict. ALWAYS run a positive control (grep -c a string known present, e.g. kld-record) in the same probe; the image has NO strings(1) — use grep -ac on binaries. The radiance image ENTRYPOINT isalready radiance: passing "radiance" as argv to podman run yields "E unknown option 'radiance'" rc=2. _(context: serverAppConfig swift15-radiance-w4dn8 campaign 2026-10-06: GPTQ Hessian pipeline version-blocker investigation on dankserver; two false negatives (missing strings tool; grepping the wrong file) cost a pin re-evaluation.)_
-omp harness max_tokens clamp (pi-ai simple-options.js): max_tokens = min(models.yml maxTokens, contextWindow - estimateContextTokens - 4096). The 4096 CONTEXT_SAFETY_TOKENS is hardcoded in node_modules. estimateContextTokens = last assistant usage.totalTokens (actual from server) + Math.ceil(chars/4) for trailing messages after the last assistant turn. The chars/4 heuristic underestimates code-heavy content by ~2.8%, so in coding-agent sessions with tool results the estimate can be 4000+ tokens low, exceeding the 4096 safety buffer. Engines differ in how they handle over-cap max_tokens: Halogen (flash-server) returns 400 (max_tokens N does not fit: prompt is P and the context is C, leaving room for R), llama.cpp silently clamps. Fix: lower maxTokens in models.yml so the harness clamp stays under the engine's actual room even with estimation error. Halogen's max_tokens INCLUDES thinking tokens (no separate thinking budget field sent on the OpenAI path — thinkingTokenBudgetField is undefined for Halogen).
-OMP harness max_tokens clamp (pi-ai/dist/api/simple-options.js): max_tokens = min(maxTokens, contextWindow − estimateContextTokens − 4096). estimateContextTokens = last assistant usage.totalTokens + Math.ceil(chars/4) for trailing messages (pi-ai/dist/utils/estimate.js). The chars/4 heuristic underestimates code-heavy content by ~2.8 %; CONTEXT_SAFETY_TOKENS = 4096 is hardcoded in node_modules (cannot be patched via config). Compaction threshold is contextWindow − reserveTokens (default 16,384 in config.yml), NOT the stale thresholdPercent: 70 comment in models.yml. Fix pattern for 400 "max_tokens N does not fit": lower maxTokens in models.yml to ≤ contextWindow − compactionThreshold − worstCaseEstimateError so max_tokens = maxTokens (fixed) rather than available (variable). Applied 2026-10-06: frank-large 57,344→45,056, frank-primary 36,864→16,384. _(context: serverAppConfig, 2026-10-06: diagnosed 400 "max_tokens 50013 does not fit" from frank-large; traced through pi-ai simple-options.js clamp + estimate.js chars/4 heuristic; fixed by lowering maxTokens in models.yml)_
-Approved design (grilling session 2026-10-06, Frank's answers; execution pending — start with the purge + download + recipe file, service live). Goal: convert ukisai/Swift1.5-Qwen3.8-Flash-Next (335.3 GiB BF16) and serve it on frank-primary (radiance, 2×R9700 TP=2, dankserver:7770). Decisions: in-place swap of frank-primary (FP8-native .rad kept as rollback); engine pinned to the radiance 1.2.0 digest; keep the vendored Froggeric v22.5 template (verified: it handles vision + exposes tool_call_format) and sampler 1.0/0.95/20; base recipe = qwen4exp-w5dn64-i8-hc8m with the expert rules replaced by **down = rtn codes=fp8_e4m3 block=128x128 scale=bf16 and gate_up = gptq table=w4nl group=64 … cd=3 calib=$CALIB** (Frank's "down w8, rest w4"); everything else verbatim (i8 trunk, hc/mtp fp8 rows, ple_ngram fixed scale 1.99317932128906e-4, u2 draft head, QSA indexer + norms untouched). Expected container ≈130–140 GiB, experts ≈38 GiB/rank. Ten protected bf16 experts recomputed for Swift from per-expert down-Hessian energy + routing share from the engine's expert-selection metrics (even count per rank half). Quality gate: text+assistant mean KL ≤ 0.03 and top-1 ≥ 95 % (KLD corpus 1.8M tokens, source-labelled, rendered with Froggeric, disjoint from calibration); throughput gate decode ≥ 31.72 t/s × 0.97 and prefill ≥ −20 % vs the replicated baseline; on quality failure escalate gate_up w4→w5nl (pre-authorized, Grams reusable). Calibration corpus 10M agent-shaped tokens (45 % code / 30 % prose / 25 % tool JSON). Shape carried over: TP=2, wht6, seqs 3, maxlen 200000, MNBT exactly 2048 (MTP resonance law), KV fp8, spec 2, PLE in RAM, host pool 40960, with an in-window EVC bracket {0.82, 0.90, 0.95} because the smaller expert mass changes the residency optimum. Drain the whole campaign (multi-day, no clock). Disk: purge .AGENTTRASH glm-purge (172 GB) + v4flash-abandoned (42 GB) + models/stage-qwen4exp-fp8 (211 GB) + models/Qwen3.8-Flash-Next-Uncensored-FP8 (173 GB) + models/Qwen/Q _(context: serverAppConfig, frank-primary (radiance, dankserver). Grilling session produced a 25-decision design tree; Frank accepted every recommendation except Q4 (kept Froggeric over Swift's own template) and Q9/Q16 (chose the down=fp8/gate_up=w4 hybrid over the recommended all-fp8 experts). Confirmation to execute was pending when the session context ended; nothing on disk had been changed yet (working t)_
-radiance (StillDeadcode) precision facts, verified in source at /tmp/rad-src (Codeberg main, 1.2.0): (1) There is NO servable 6-bit tier. codes= accepts fp6_e2m3/fp6_e3m2/u6/i6 and table= = {nf4, iq4nl, w4nl, w5nl, binary, ternary}, but the qwen4exp MoE block serves exactly kServed[8] = {fp8a8, w4a8, w4a8h, bf16, w4nla8h, w4nl64a8h, w4nl32a8h, w5nl64a8h} (arch/common/rad_block_moe_fp8.h:457); dense serves fp8a8/i8a8/fp8a16/mxfp4a8/bf16 + w2 draft head. "A container can hold only what some kernel reads" — rad-convert REFUSES an encoding no kernel reads (docs/TOOLS.md:184), so a q6 recipe fails at convert, not at boot. (2) Published Flash-Next quality ladder (docs/MOE-W4.md:24-32, mean KL / top-1, all tokens): w4nl64a8h served 0.0789/92.85%, w5dn64 0.0717/93.34%, bf16 floor 0.0399/95.13%; text+assistant scope w5dn64 = 0.0204/95.10%. Any KL gate ≤0.05 on the all-token scope is unreachable by every quantized tier. Author's lever (MOE-W4:68): 5 bits on layers 16-31 both projections, ~0.066 expected. Heavier expert tiers cost PREFILL (w5dn64: decode +4%, prefill −19%). (3) Calibration: the ENGINE records — serve with RADIANCE_CALIB_DIR [+ RADIANCE_CALIB_TOKENS, default 65536], drive prefill, it writes gram.rrank.weight stem.bin once. A BF16 Flash-Next (335 GiB) serves on two cards via --weights-disk-tier and records ~930 tok/s (10M in 3 h) — documented verbatim (MOE-W4:244-253, GUIDE:529). down gets a per-expert Hessian; gate_up is pooled per layer. GPTQ adds "tens of minutes" per 48-layer container + 1.3 GB factor cache. (4) Scoring: radiance --model BF16.rad --kld-record DIR --kld-corpus C.jsonl once (exact wire + bf16 KV), then --kld-ref DIR --kld-out REPORT.json per candidate; report is per source, so turn-type scopes come from labelling the corpus's source field; corpus must be disjoint from the calibration corpus; speculation/prefix-cache off in that mode, MNBT ≤ ~8192 (logits buffer 2 GiB/rank per 4096 tokens). (5) ukisai/Swift1.5-Qwen3.8-Flash-Next: 335.3 GiB BF16 _(context: Grilling session 2026-10-06 on serverAppConfig: designing a rad-convert recipe for ukisai/Swift1.5-Qwen3.8-Flash-Next on frank-primary (radiance, 2×R9700 TP=2). Facts gathered from a shallow clone of Codeberg StillDeadcode/radiance + HF API/safetensors-header probes; a scout subagent died on frank-secondary's 131k slot overflow (exceed_context_size_error) so the facts were collected inline.)_
-radiance (StillDeadcode) serve-side precision surface, verified from source (clone of main, 2026-10-06) — the quantizer can WRITE more widths than any kernel can RUN, and rad-convert enforces it: "A container can hold only what some kernel reads. The converter checks every planned encoding against the layout hooks of the kernels declare resolved and refuses one nothing accepts" (docs/TOOLS.md:184-186). Consequences: (1) NO 6-bit tier exists — codes= accepts fp6_e2m3/fp6_e3m2/u6/i6 (libquant/lq_quant.h:15-17) and RAD_FP6E2M3 is a real ABI dtype (abi/rad_types.h:186), but libr4d has zero fp6 GEMM rows (lq_common.h:105 calls the fp6 grids conversion-only) and table= codebooks stop at five bits: nf4, iq4nl, w4nl, w5nl, binary, ternary (lq_quant.h:19) → a q6/fp6 recipe FAILS at rad-convert, not at boot. (2) The qwen4exp MoE block serves exactly 8 formats, arch/common/rad_block_moe_fp8.h:457: kServed = fp8a8, w4a8, w4a8h, bf16, w4nla8h, w4nl64a8h, w4nl32a8h, w5nl64a8h — anything else is RAD_E_DTYPE (error names the list, :671); dtype is chosen PER PROJECTION PER LAYER (:396), so fp8 down + w5nl gate_up in one container is legal. (3) Dense/trunk GEMMs serve bf16, fp8a8, fp8a16 (m1/mt), i8a8, w4a16/w4a8(+tiled/prefill/asym), w8a16/w8a8, mxfp4a8, and w2a8 for the draft head (libr4d/r4d.h:430-670, r4d_moe.hip:2393-2527). (4) Calibration has no recorder tool: the ENGINE records — serve with RADIANCE_CALIB_DIR (+ RADIANCE_CALIB_TOKENS, default 65536), drive prefill traffic, it writes per-layer Grams and prints "radiance: calibration wrote layer Hessians after N tokens to dir"; calib= is accepted by gptq/awq/autoround/paroquant, unset $CALIB is refused, a weight with no Gram is quantised uncalibrated and says so on stderr (docs/MOE-W4.md:229-271, TOOLS.md:158). (5) A BF16 Qwen3.8-Flash-Next checkpoint (335 GiB) can be SERVED on two gfx1201 cards via --weights-disk-tier (~28 s a step, ~930 tok/s recorded, 10M tokens ≈ 3 h) precisely to record bf16-domain Hessians for a GPTQ recip _(context: serverAppConfig, 2026-10-06: grilling session for converting ukisai/Swift1.5-Qwen3.8-Flash-Next to radiance with a qwen4exp recipe; asked to "bump q4/q5 to q6/fp6" and the kernel library proved that impossible. Facts from /tmp/rad-src (clone of Codeberg StillDeadcode/radiance main) corroborated by a scout run.)_
-Radiance (StillDeadcode) precision facts — primary-source research 2026-10-06, not yet in local docs. PROJECT MAP: codeberg.org/StillDeadcode/radiance = the standalone C++/HIP engine (what frank-primary runs; latest 1.2.0, 2026-10-06); codeberg.org/StillDeadcode/vllm-radiance = DEPRECATED vLLM wrapper ("I ditched vLLM… it is faster, easier to work on, starts faster and has less cpu overhead"); codeberg.org/StillDeadcode/libr4d = shared kernel library, entry points named for exact geometry and rejecting mismatch (attn_decode_h256_gqa6_fp8kv, gdn_chunk_scan_k128_v128_c64_bf16, ar_oneshot_2rank_exact). PRECISION LADDER: block-FP8 (E4M3 codes + bf16 block scales) is the native tuned path — the "preshuffled block-FP8 dispatcher"; replacing it with generic upstream AITER linear is 9–11 % slower in matched controls, and every upstream-qualified model is FP8 (Qwen3.8-27B-FP8, Qwen3.6-27B-FP8, Qwen3.6-35B-A3B-FP8, Gemma-4-31B-it-FP8). MXFP4/W4A8 (AMD Quark weight path) stores packed OCP group-32 MXFP4 weights "and dynamically quantizes activations to FP8 E4M3 so the kernels use RDNA4's native FP8 WMMA path" — compute is FP8 regardless of weight packing. bf16 = KV (auto), skinny GEMMs, GDN (gdn_chunk_scan_…_bf16). INT4/INT8 = draft head (decoder projections "packed to signed symmetric int4… 4.25 bits per weight", int8 kernel above 16 rows). HARDWARE "WHY": RDNA4 WMMA is integer-biased — "gfx1201's f16 matrix instruction is half the rate of its int8 one and a quarter of its int4 one" ⇒ int4 int8 f16; FP16 saturates the 128-bit memory interface natively while FP8/INT8 need a fused double-K WMMA to reach 128-bit loads (AMD GPUOpen WMMA guide part 2). Sources: codeberg StillDeadcode/radiance + /libr4d + /vllm-radiance READMEs, github.com/magiccodingman/vllm-radiance fork README, gpuopen.com/learn/wmma-guide-amd-rdna-4-gpus-part-2. _(context: Online research for the radiance precision question (serverAppConfig, 2026-10-06); findings are pending a docs fold-in and feed the Swift1.5-Qwen3.8-Flash-Next rad-convert recipe design.)_
-Radiance (StillDeadcode) tensor-precision picture — verified 2026-10-06 from primary sources. Project map: codeberg.org/StillDeadcode/radiance = the STANDALONE engine (C++/HIP, no Python) that frank-primary runs; codeberg.org/StillDeadcode/vllm-radiance = DEPRECATED vLLM wrapper ("I ditched vLLM… standalone is faster, starts faster, less CPU overhead"); codeberg.org/StillDeadcode/libr4d = shared kernel library, entry points named for exact geometry (e.g. attn_decode_h256_gqa6_fp8kv, gdn_chunk_scan_k128_v128_c64_bf16, ar_oneshot_2rank_exact) and it rejects a shape mismatch rather than running. Precision ladder: (1) block-FP8 (E4M3 codes + bf16 per-block scales) is the NATIVE tuned GEMM path = the "preshuffled block-FP8 dispatcher"; replacing it with generic upstream AITER linear = 9-11% slower in matched controls; all upstream-qualified models are FP8 (Qwen3.8-27B-FP8, Qwen3.6-27B-FP8, Qwen3.6-35B-A3B-FP8, Gemma-4-31B-it-FP8). (2) Compute is FP8 WMMA on RDNA4's native matrix core — even the MXFP4/W4A8 (AMD Quark) weight path DYNAMICALLY quantizes activations to FP8 E4M3 to hit that native FP8 WMMA path (so compute is FP8 regardless of weight packing; MXFP4 = packed OCP group-32 weights). (3) bf16 = KV cache (the auto/bf16 option; fp8 is the qualified mandatory default), skinny GEMMs (split-K for small projections), and GDN (gdn_chunk_scan…_bf16, an fp16 matrix-core WMMA gated-delta-net path). (4) INT4/INT8 = draft head (drafter decoder projections "packed to signed symmetric int4… 4.25 bits per weight", int8 kernel above 16 rows). HARDWARE "why" (RDNA4 WMMA, gpuopen.com/learn/wmma-guide-amd-rdna-4-gpus-part-2): the matrix core is integer-biased — "f16 matrix instruction is HALF the rate of int8 and a QUARTER of int4", so hardware ranking int4 int8 f16; FP16 saturates the 128-bit memory interface natively but FP8/INT8 only do 64-bit loads, so the tuned GEMM kernels FUSE two WMMA instructions (double-K / K-dimension) to enable 128-bit vector loads and hit peak bandwidt _(context: serverAppConfigs LLM fleet — answering "what tensor precisions does the radiance runtime like"; online research against StillDeadcode Codeberg repos + AMD GPUOpen WMMA guide.)_
-radiance runtime (StillDeadcode C++/HIP engine, libr4d kernels for gfx1201) tensor-precision support: FIRST-CLASS/native = grouped/block-FP8 (E4M3 codes + bf16 per-block scale planes) — the ACTIVE frank-primary serves the official Qwen3.8-Flash-Next-FP8 natively with no requantisation (the qwen4exp_fp8 plugin is built around it). The vllm-radiance wrapper line (frank-medium, frank-primary R9V) extends the same libr4d kernels to a quant ladder: MXFP4/MXFP6 (ParoQuant W4A8, e4m3fn scale, group_size=128, krot=8), Q8_0, IQ4_XS. KV cache: fp8 native (7.32 KiB/token on frank-primary) or bf16 unquantized (--kv-cache-dtype auto). Scales: bf16 (block-FP8 planes, PLE one-element scale). Speculation draft heads: blockfp8 (MTP/DFlash2). TP wire: wht6 = lossy FP8-class fused one-shot all-reduce (vs --tp-wire exact bit-exact). The distinction that matters: "radiance" (pure engine, no Python, native block-FP8) vs "vllm-radiance" (vLLM front end over the same kernels, the quant ladder). Sources: serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md, frank-medium-radiance-mxfp4.md, frank-primary-r9v-radiance-qwen38-q8.md, RDNA4 Container Serving.md §1. _(context: Answered "what tensor precisions does radiance runtime like?" from the serverAppConfig LLM runbooks.)_
-serverAppConfigs shared-checkout diagnostic (proven 2026-10-06): to determine the TRUE position of a dirty shared checkout, do NOT trust the abbreviated git reflog tail — it can show commits (e.g. c80c250) as recent HEAD@{N} that are actually ORPHANED (the branch was reset away from them; they're reachable only via reflog). Verify with three agreeing sources: git rev-parse HEAD, git for-each-ref refs/heads/branch, and git log --oneline (tip). In the 2026-10-06 case, the reflog tail showed ...81e6122 - 95923d3 - df6b5bb - c80c250 commit as the last entry, but HEAD+branch+log all sat at 81e6122 — the three bench commits were orphaned after an autoresearch sweep (final commit c80c250 concluded "KEEPING Frank's config", so the branch was reset back to the harness commit 81e6122). Also: an UNSTAGED, uncommitted deletion of a tracked file that is present on the target branch is LOSSLESS to git restore -- file (it's pure working-tree state, in no commit); confirm with git ls-files --stage file (tracked?) + git ls-tree branch -- file (present on target?). This is the safe "clean up the working tree" move before git checkout branch. _(context: Cleaning up the serverAppConfigs shared checkout and returning to main (2026-10-06): resolved an apparent reflog/log discrepancy on the autoresearch/...frank-pri-20261006 branch, restored an unstaged deletion of autoresearch.sh, and switched to main.)_
-Ops laws confirmed live 2026-10-06 on dankserver: (1) Fleet containers are ROOTLESS under user dank — sudo podman ps/logs queries the empty ROOT store and shows NOTHING; production appeared "gone" while radiance was serving fine. Always query podman ps / podman logs AS dank without sudo; verify liveness via ss -ltnp (radiance pid on :7770) or /health. (2) frank-primary logging trap CONFIRMED: scripts/serve-frank-primary.sh runs podman run -d ... "$LOG", so /home/dank/logs/frank_primary.log holds 65 bytes (one container-ID hex line) and the boot ledger lives ONLY in podman logs frank-primary (rootless, as dank). Fix = foreground podman run under Type=simple (pattern proven on frank-large, commit ba67ca5); queued as post-segment maintenance commit (serve script is off-limits inside the autoresearch segment). (3) The radiance engine logs NO per-request draft acceptance server-side; timings.draft_n/draft_n_accepted are returned to the CLIENT only — any real-traffic spec-depth A/B needs a client-side collector. _(context: 2026-10-06 dankserver: production health checks during the autoresearch segment; root-store vs rootless-store confusion cost several turns; frank_primary.log inspected directly)_
- frank-primary (radiance, 2xR9700 TP=2) decode-sweep results 2026-10-06 (21 runs): final config = Frank's original spec=3/seqs=2/maxlen=200k (commit c80c250); baseline 31.72 t/s at 43.6k depth replicated 31.72/31.74. Noise floor ±3% decode (identical config drew 33.06/32.03/30.90 — single-run deltas 3% are noise; NEVER keep on 3%); prefill noise ±2%. MNBT RESONANCE LAW: with MTP on, MNBT=2048 is a sharp resonance point, NOT an interior optimum — 2560 collapses decode to 2.6–7.7 t/s (acceptance stays NORMAL 0.82, verification stalls ~130ms/step, output truncates 213/256; reproduced on clean boot), 1536 truncates 57/256 + prefill −22%, 4096 won't boot at seqs=3. MNBT must stay exactly 2048 at this shape; mechanism unresolved (kernel boundary × 3 slots × MTP verify). HOST_POOL FLOOR LAW: 36864 MiB crashes boot; 40960 is the load-bearing floor of expert page-staging (49152 boots flat — not a slack knob). Real regressions only: spec=4 (−23%, acceptance 0.50), spec=0 no-MTP (−7%, MTP genuinely pays), EVC 0.84 (−8.9%). Flag surface 100% bracket-mapped; spec 1–3 × seqs 2–3 statistically tied; depth 3 kept on acceptance tiebreak (0.831 vs 0.755) — the 256-token greedy probe cannot express it; real-traffic A/B is the only separating instrument. _(context: frank-primary radiance autoresearch segment 2026-10-06 (21 bench runs, branch autoresearch/...frank-pri-20261006, final commit c80c250); harness: autoresearch.sh + scripts/autoresearch-frank-primary.sh)_
-frank-primary (radiance, dankserver) tuning laws from the 2026-10-06 autoresearch window (branch autoresearch/iteratively-improve-the-performance-of-frank-pri-20261006, final config commit c80c250): (1) MNBT RESONANCE LAW — at MTP-on shapes, --max-num-batched-tokens=2048 is a sharp resonance point, NOT an interior optimum: 2560 collapses decode to 2.6-7.7 t/s (reproduced on a clean boot; draft acceptance stays normal at 0.82 while verify stalls ~130ms/step; generation truncated 213/256), 1536 truncates at 57/256 with prefill -22%, 4096 won't boot at seqs=3. MNBT must stay exactly 2048; re-testing is forbidden, mechanism unresolved (kernel boundary interaction between chunk size, slots, MTP verify budget). (2) HOST_POOL_MIB 40960 is the LOAD-BEARING FLOOR of the expert page-staging reservation — 36864 crashes boot, 49152 boots flat; it is not a slack knob. (3) MEASUREMENT LAW — decode noise floor is ±3% on identical config (33.06/32.03/30.90 on one config), prefill ±2%; single-run deltas 3% are noise, replicate before keeping any change. (4) Decode surface is bracket-complete: spec depth 1-3 statistically tied (0 and 4 are real regressions: 29.57 and 24.57), seqs 2-3 tied, expert-vs-cache 0.82 bracket-confirmed (0.78 crash / 0.84 -8.9%), maxlen 262144 steals resident-weights VRAM (-2.5%), GPU-headroom 48 and host-pool 49152 flat. Tie-break that kept Frank's original spec=3/seqs=2: draft acceptance 0.831 vs 0.755 favors real multi-turn traffic; the 256-token greedy probe cannot express it. The separating instrument for depth 2 vs 3 would be real-traffic per-request draft acceptance from the engine ledger (needs production window at depth 2, Frank's sign-off)._(context: serverAppConfigs autoresearch segment 2026-10-06 (frank-primary decode tuning, 21 runs): sweep closed at a tie with Frank's original config kept; MNBT resonance and host-pool floor laws measured; noise-floor law established by identical-config replication.)_
-frank-primary radiance tuning window (2026-10-06, 20 autoresearch runs): (1) MNBT RESONANCE LAW — at MTP-on/seqs=3, --max-num-batched-tokens is a sharp resonance point at 2048, NOT a smooth curve: 2560 collapses decode to 2.6-7.7 t/s with normal draft acceptance (0.82) and verify stalling ~130ms/step, generation truncating at 213/256; 1536 truncates at 57/256 and loses 22% prefill; 4096 fails boot at seqs=3 (booted at seqs=2, only -2.6% there). MNBT must stay =2048 at this shape; mechanism unresolved. (2) HOST_POOL_MIB 40960 is the load-bearing FLOOR, not slack: 36864 fails boot, 49152 boots but is decode-flat. (3) Noise floor law: decode is ~±3% run-to-run on IDENTICAL config (33.06/32.03/30.90 measured), prefill ~±2% — single-run deltas under those bounds are not evidence. (4) Viable envelope at seqs=3: EXPERT_VS_CACHE 0.82 (0.78 crash / 0.84 -9%), MNBT 2048, TP_WIRE wht6 (bf16 crashes), HOST_POOL 40960. Spec depth 1-3 statistically tied (0 and 4 real regressions: 29.57 / 24.57). Endpoint was already well-tuned; no flag clears noise. Scratch-bench hazard: if a prior bench is abandoned mid-run, the next scratch boot may race the leftover engine — verify the previous arbench container is gone and prod restored before trusting a collapse measurement (one 92% collapse was initially suspect for this; MNBT=2560 collapse then reproduced clean-boot, confirming it real). _(context: Autoresearch segment on serverAppConfig, frank-primary (radiance, Qwen3.8-Flash-Next-FP8, 2xR9700 TP=2, seqs=3, maxlen=200k, spec=2): harness scripts/autoresearch-frank-primary.sh boots a scratch radiance container per run; 20/30 runs consumed, flag surface 100% bracket-mapped, no keep beyond noise. Frank's pending decision: keep spec=2/seqs=3 vs revert to his spec=3/seqs=2.)_
-Single-run A/B on the radiance /v1/completions timings (256-token greedy decode, 43.6k-token prompt, fresh scratch container per boot on dankserver) has a true noise floor of ~±3% decode t/s, NOT the ±0.3% one might assume from short-interval probes. Identical config measured 33.06 / 32.03 / 30.90 t/s. Consequence: never keep a tuning change on a 3% single-run delta; re-run promising keeps once before believing them (and flag the first lucky draw as suspect), and bracket both sides of a tuned value before trusting an optimum. Boot-failure arms (health timeout ~720s, ~12 min) are informative walls, not wasted runs. Real, outside-noise effects at frank-primary: SPEC_TOKENS=4 (-23%, acceptance 0.83-0.50), SPEC_TOKENS=0 (-7%, MTP always pays), EXPERT_VS_CACHE=0.84 (-9%), KV shape seqs=4/maxlen=100k (-11%), and three boot crashes that define the seqs=3-viable envelope (EXPERT_VS_CACHE 0.78, MNBT 4096, TP_WIRE bf16 all crash at seqs=3; the wht6/2048/0.82 trio is what makes it fit). _(context: Autoresearch session on serverAppConfig (frank-primary radiance decode tuning, 2026-10-06): 16-run ladder over one flag at a time; runs 39/40/49 measured the SAME committed config (spec=2, seqs=3, maxlen=200k) at 33.06 / 32.03 / 30.90 t/s, exposing the true run-to-run spread; run #39 had been kept as "+4.2%" and had to be flagged suspect after the replication run exposed the draw.)_
- frank-primary (radiance, TP=2 Qwen3.8-Flash-Next-FP8) decode tuning laws, measured 2026-10-06 on a deterministic 43.6k-token greedy probe (scripts/autoresearch-frank-primary.sh, decode_tps = timings.predicted_per_second, noise ~0.3%): (1) MTP depth is UNIMODAL at 2: depth1 31.89, depth2 32.40, depth3 31.72, depth4 24.57 t/s (acceptance collapses 0.83-0.50 at depth 4). SPEC_TOKENS=2 is the production value. (2) MAX_SEQS 2-3 (+2.0%, co-scheduling headroom; single-stream probe sees it via scheduler shape) wins; 4 regresses (-3.1%). (3) The KV-shape surface is peaked at seqs=3/maxlen=200000: every window mutation regresses depth decode (native 262144: -2.5%; 100000x4: -10.6%) because the KV pool trades against resident-weights VRAM at expert_tiered. (4) EXPERT_VS_CACHE=0.82 is bracket-confirmed optimal: 0.78 fails boot at seqs=3 (KV unallocatable), 0.84 costs -8.9%, 0.87 known collapse. (5) At seqs=3 the VRAM envelope is tight: MNBT 4096 and TP_WIRE bf16 both FAIL BOOT (720s health timeout) — wht6 + MNBT 2048 are what make the seqs=3 shape fit. (6) GPU_HEADROOM 48 and HOST_POOL 49152 are decode-neutral (the pinned-pool streaming is not the single-stream bottleneck). Net kept gain: 31.72 - 33.06 t/s (+4.2%), committed as spec=2 + seqs=3. _(context: Autoresearch session frank-primary-decode-perf, runs #35-#48, autoresearch branch, harness stop-prod/boot-scratch-arbench/bench/restore on dankserver)_
-serverAppConfigs deployment-completeness policy: a behavior-affecting deploy needs the bats suite (./tests/run.sh) AND the Playwright e2e gate (tests/e2e/run.sh). deploy.sh normally runs the e2e gate as its [3/3] step, but its whole-tree clean-tree gate (scripts/deploy.sh lines ~116-120: git status --porcelain non-empty → FAIL) is blocked by ANY dirty file, including a parallel writer's unrelated WIP. When that happens, run the two gates standalone — ./tests/run.sh for the seams and ./tests/e2e/run.sh for the live-fleet e2e (it needs tests/e2e/.env.local credentials, drives the live fleet, and does NOT require a clean tree). This satisfies the deployment-completeness requirement without deploying; the actual service dispatch can be done manually (git push + remote ff-pull + systemctl start) when it only touches files the writer isn't touching. Proven 2026-10-06 on the frank-primary rad-tune/MNBT window: writer's dirty frank-large WIP blocked deploy.sh, so I ran tests/run.sh (all green) + tests/e2e/run.sh (33 passed) directly and restored frank-primary by manual push+pull+start.
-radiance (StillDeadcode) rad-tune on this fleet: the kernel-selection cache is written to $RADIANCE_HOME/tune/machine.tune, and $RADIANCE_HOME defaults to /opt/radiance — which lives INSIDE the image, so a --rm container's cache dies with the container. To persist it you must mount a host dir over it (-v hostdir:/opt/radiance/share/radiance/tune) in BOTH the rad-tune run AND the serve run, and pre-create the host dir (the engine's create is NOT recursive). Machine key on this box: gfx1201-x3-AMD_Radeon_Graphics (3 devices: 2x gfx1201 R9700 + 1x gfx1036 iGPU). Declare at the shapes you serve: --tp 2 (a world-size-1 declare measures a program a TP=2 serve never runs) and --num-speculative-tokens 3 (puts the drafter's kernels in the measured graph; default is the container's own depth). On this build the two attention kernels (attn_paged, attn_paged_gate_quant) are SKIPPED by rad-tune ("no shape recipe; see tools/opshapes.h") — only the 36 gemm_nt (kernel,shape) pairs get tuned, so the cache covers GEMM tile selection, not attention. rad-tune is ~110-150s for this 173 GiB model at TP=2. The radiance image has NO bash and NO env/ls — only sh; invoke it as podman run --entrypoint sh image -c "cmd" (the ---before-args form breaks). _(context: frank-primary radiance cutover bench window 2026-10-06: rad-tune --tp 2 --num-speculative-tokens 3 on qwen4exp-fp8-native.rad; cache path + mount law + attention-skip observed live)_
-Bonsai-2-27B swap on frank-small: scouted + smoke-booted 2026-10-06, then rolled back to Swift RCO-IQ3_XXS by Frank's order BEFORE any repo edit (tree stayed at bb3532d). Payload parked on disk for a retry. (1) RUNTIME LOCK: stock llama.cpp CANNOT load PQ2_0/PTQ1_0 ternary files (unknown types; Q2_0 loads silently-garbage — no Hadamard activation runtime). Required runtime = PrismML-Eng/llama.cpp fork (prism branch). Deployed prebuilt: release prism-b10754-2459f68 asset bin-linux-cuda-12.4-x64 (matches Dankclaw nvcc 12.4; CUDA 13.3 builds documented crash-prone; tarball sha256 80185ce29bed2456f099903a41fe20452121af0c2da6be84315c9145e46e3ad2), extracted FLAT — binary at /home/dank/GitPackages/llama-prism/llama-server, NO bin/ sublevel (a bin/llama-server path fails). Post-2026-09-23 releases carry the AVX-512 load-segfault fix #245 (Dankclaw Strix Halo = Zen 5 is the affected class; a pre-b10735 release would segfault). (2) PAYLOAD: dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF Bonsai-2-27B-PQ2_0-CRACK.gguf = 7,206,168,928 B, sha256 5b24ea3eebc3e0bccd05fb474eb88b10c57699d71a5db2f29485e3789a70d55d; PrismML ternary {−1,0,+1} g128 in a Hadamard-rotated basis at 2.13 bpw of Qwen3.8-27B, abliterated at weight level (0/320 HarmBench-320 refusals vs base 299/320; MMLU −0.62 pp); 64-block qwen3_5 hybrid — NO MTP head in the file (draft-mtp refused) and ngram-* spec types have NO effect on this model (fork issue #203) ⇒ spec=none is the only contract; mmproj ships in the BASE repo prism-ml/Ternary-Bonsai-2-27B-gguf (mmproj-Q8_0 629,246,976 B, sha256 6807ede61d570bb86ba34b756a0fa109edc33668604de867c6ea6d8f1d631903; BF16 reference pack 931,145,856 B) at /home/dank/models/Bonsai2-CRACK/. (3) SMOKE MEASUREMENTS (2×131072 q8_0 KV, ubatch 512, SPEC none): boot ~2.5 s, VRAM 17,006 MiB used / 7,570 free (widest fit the endpoint has ever had — a third slot plausibly fits), n_slots=2 confirmed, vision projector loads. (4) FORK QUIRKS for the eventual swap: reasoning_effort "high" → HTTP 50 _(context: 2026-10-06 frank-small Bonsai-2-27B session: Hub scouted (both repos), all three artifacts fetched+sha-verified on Dankclaw, smoke boot on :7771 verified 17.0/7.6 GiB, then Frank ordered rollback to the prior Swift endpoint; payloads and fork binary remain on disk with no repo changes.)_
-frank-small RCO-IQ3_XXS two-slot MTP-on pricing (2026-10-06): the MTP draft ctx measured ~2.4 GiB, not the ~0.4 GiB estimated from proposal depth — the draft context is built against the target's FULL n_ctx (262,144), per the existing "drafter's KV scales with the target's context" law, so price MTP draft ctx from a boot pair, never from the draft depth. Fit: 23,636 used / 489 MiB free (tightest frank-small fit in history; the 2–3 GiB floor knowingly violated by Frank's explicit order). Two companion facts for this payload/build: (1) the UNCENSORED model thinks more than censored Swift 1.5 — a 64-token vision probe came back all reasoning_content with empty content (finish_reason length); the probe needs a 128-token budget. (2) The current llama.cpp master /completion timings object exposes draft_n, draft_n_accepted, predicted_per_second — NOT n_prompt/n_text/t_prompt/t_text — so decode rates come from predicted_per_second and acceptance from draft_n_accepted/draft_n (measured 49.8–62.4% on this quant). _(context: frank-small 2026-10-06 RCO-IQ3_XXS + draft-mtp cutover (commits da36fa3/bb3532d): live probes on the deployed two-slot shape; the pre-boot VRAM estimate was off by 6x on the draft ctx)_
-HuggingFace repos on Xet storage (newer repos like ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF) expose NO LFS sha256 in the tree or blob APIs (the blob API returns a 404-ish "Sorry, we can't find the page" JSON for LFS files and resolve headers carry no X-Linked-Entity-Sha256). Verification for such repos must be: byte count against the Hub's declared size + local sha256sum after the first verified fetch, with the computed sha pinned in the download script (state in the script header that the sha was computed locally, not taken from the Hub). Proven on the 2026-10-06 frank-small retarget (both files matched Hub byte counts exactly; shas pinned in scripts/download-frank-small.sh).
-radiance checkpoint reader FILE-SHADOW law: the reader opens every file the safetensors index names, in SORTED file-name order, and emplaces EVERY tensor each file physically holds into one table, first-wins (checkpoint.cpp add_shard, std::unordered_map::emplace never overwrites) — the index only enumerates files, it does NOT scope which tensors count. Consequence: a replacement file whose tensors share a name with tensors physically present in an earlier-sorting file is shadowed (its data never read; the plan reads the older file's tensor under that name). Fix: name the replacement so it sorts FIRST ("00-ple.merged.safetensors" beats "model-00001-of-00131.safetensors"), then patch the index value. Companion engine fact: theradiance engine's directory create is NOT recursive — "cannot create /data/kvcache/..." ENOENT left the SSD prefix tier off until the serve script mkdir -p'd the full path; pre-creating the host dir activated the tier on the running container without a repo deploy (volume exposes it immediately; only the boot-time check is one-shot). _(context: frank-primary radiance cutover, 2026-10-06 live bring-up; root-caused from /tmp/radiance-src checkpoint.cpp add_shard + shard_list after the plan-only run read a raw fp8 shard as the PLE codes)_
-Fleet serve-script trap: podman run -d ... "$LOG" captures ONLY the container ID — the engine's stdout/stderr go to podman's internal store, so any runbook claiming "the boot ledger lives in $LOG" is reading a one-line hex file. Diagnosis signature: a ledger grep that returns empty, or a log file containing a single 64-hex string. Fix = run podman run FOREGROUND (Type=simple unit keeps the podman client as main process; podman run returns the container's exit code so Restart=always/ExecStop semantics are unchanged, and the client forwards SIGTERM). Before grepping any fleet service's log file, verify it is not this shape — frank-primary is suspect (its serve script uses podman run -d; its memory note claims logs land in /home/dank/logs/frank_primary.log, unverified against the -d trap as of 2026-10-06). _(context: frank-large tuning session 2026-10-06: the runbook's "engine stdout goes to /home/dank/logs/frank_large.log" had been false since the 2026-10-05 cutover — podman run -d ... "$LOG" captured only the container ID hash; the boot ledger existed only in podman logs frank-large-halogen. Fixed by foreground podman run (ba67ca5); AGENTS.md and the runbook now document it.)_
-frank-large (halogen 0.1.4, Dankclaw iGPU) decode-at-depth laws, measured 2026-10-06 on real traffic + controlled A/B: (1) Decode collapses linearly with context: 30-50 t/s shallow → 5.5 @46k → 3.3-3.8 @80k; round-time slope ~8 µs/token ⇒ KV pool reads ~8 GB/s effective (with verify-position factor ~28 GB/s) vs weights streaming ~180-240 GB/s. Not an FA tile artifact — HALOGEN_FA_TQ=256 A/B'd same-prompt at 80k: decode ±0.3%, prefill −9.5% on this topology (the knob helps only the 64GB-carve crash topology of upstream issue #12). (2) The engine binary advertises 40+ UNDOCUMENTED HALOGEN_* kernel levers (FA_*/DN_*/MATMUL_*/ATTN_SD family) — enumerable read-only via grep -aoE "HALOGEN_[A-Z0-9_]+" /usr/local/bin/halogen inside the image with --entrypoint sh (no GPU touched; strings is NOT in the image, use grep -a). (3) Same-prompt depth A/B method: stop+mask prod, boot probe containers on 127.0.0.1:17770, prompt cache ON (8192 MB) so the second rep per arm skips the ~178s 80k prefill; force 128-token ESSAY instructions or greedy answers in 2 tokens and the decode measurement is void; read timings.predicted_per_second/draft_n_accepted. (4) Applied mitigation: harness roles slow+advisor (the deep-conversation roles) rerouted from frank-large to frank-primary in ~/.omp/agent/config.yml — frank-primary's fp8 KV is 7.32 KiB/token vs halogen's 64 (8.7x less KV read at depth). frank-primary's depth-decode rate is NOT yet measured on record — first deep session there should record timings. (5) Boot ledger WARNING "HALOGEN_CACHE_MB smaller than ONE full-context entry (18.4 GB at kMaxCtx=262144)" is a FALSE POSITIVE when slot_ctx native: entries size to their prompt (~70 KB/token); at slot 200,700 the deepest possible entry is ~14 GB, inside a 16384 MB pin. _(context: frank-large halogen tuning session 2026-10-06: prompt-cache pin (16384 MB) deployed and verified (16x warm prefill); decode-at-depth collapse investigated — FA_TQ A/B in a stop window rejected it (±0.3% decode, −9.5% prefill on Dankclaw); slow/advisor roles rerouted to frank-primary (config.yml, operator home). Commits ba67ca5, d39424f, fadad18.)_
- radiance rad-convert from an HF safetensors checkpoint has FOUR non-obvious traps, each costing a full verify+declare cycle: (1) The reader's tensor table is FILE-DERIVED, not index-driven: checkpoint.cpp add_shard opens every index-named file in SORTED order and emplaces EVERY tensor each file physically holds, first-wins (std::unordered_map::emplace never overwrites). An HF checkpoint whose shards physically carry tensors under the SAME name a later replacement file uses will shadow it — e.g. the official Qwen3.8-Flash-Next-FP8 shards carry ...ple.ple_embedding.ngram_embedding.shard_0.weight ([2500012,160] fp8) which shadows a merged table of the identical name. Fix: name the replacement file so it sorts FIRST ("00-ple.merged.safetensors" beats "model-00001-of-00131.safetensors"). (2) scale_name() appends "_scale_inv" to the TENSOR NAME verbatim (checkpoint.cpp:521): a stacked tensor named ...experts.gate_up_proj (no .weight suffix) has its scale at ...experts.gate_up_proj_scale_inv — NOT ...gate_up_proj.weight_scale_inv. (3) Safetensors index.json/config.jsonin a staging dir are usually RELATIVE SYMLINKS to the raw tree; open(path,"w") writes THROUGH them and corrupts the raw checkpoint (verify then fails 143/144 on size). Always temp+os.replace for atomic swap — it replaces the symlink itself, leaving the target pristine. (4) numpy views over mmap: np.frombuffer keeps an exported pointer; mmap.close() raises "BufferError: cannot close exported pointers exist" (Python 3.14+) unless the view is del'd first. Also: rad-convert --plan-only prices the DATA planes; the written container carries a metadata tail on top (172.76 GiB plan - 177,109 MiB actual), so size floor gates must be set from the real file, not the plan. _(context: frank-primary radiance native-FP8 cutover, 2026-10-05/06; four convert blockers root-caused against /tmp/radiance-src (v1.0.7); all fixes landed in scripts/stack-qwen38-official-fp8.py + scripts/build-frank-primary-rad.sh)_
- frank-large (halogen 0.1.4, Dankclaw iGPU) runtime facts, measured 2026-10-06 on real traffic + probes: (1) DECODE-AT-DEPTH COLLAPSE IS STRUCTURAL — 30-50 t/s shallow (1k, commit 4-6.3/round) → 5.5 @ 46k → 3.3-3.8 @ 80k; round-time slope ~8 µs/token of context = the KV path (16 full-attention layers, 64 KiB/token d2 KV) reads at ~8 GB/s effective while weights stream 180-240 GB/s; commit also falls to 2.5-3 at depth. No SUPPORTED flag touches decode (docs/FLAGS.md: W4A4/W4A4_EXCL are prefill-only) and 0.1.4 is the latest release; the publisher's tg bench NEVER measured depth (tools/halogen-bench.py tg rows run over the ten SHORT eval prompts), so README's "decode barely degrades at depth" is unexercised. (2) PROMPT CACHE was auto-OFF (engine floor = 2 × full-context entry = 36.9 GB vs 35.7 free); pinned HALOGEN_CACHE_MB=16384 (one full-slot entry ≈ 14 GB at ~70 KB/token); probe-proven cold 27.0 s → warm 1.7 s prefill at 17.3k (cache_n 16,384, prompt_n 946). Boot WARNING "smaller than ONE full-context entry (18.4 GB at kMaxCtx=262144)" is a FALSE POSITIVE vs our 200,700 slot — documented as expected in the runbook checklist. (3) SERVE LOG FIX: podman run -d LOG captures only the container ID — the engine ledger existed nowhere on disk; foregroundpodman run makes the ledger land in /home/dank/logs/frank_large.log with unchanged exit-code/ExecStop semantics; after a manual stop the unit reads "failed" (cosmetic). (4) UNDOCUMENTED LEVERS: the binary reads 40+ HALOGEN_* kernel-selection vars; enumerate with grep -aoE "HALOGEN_[A-Z0-9_]+" /usr/local/bin/halogen | sort -u (image has NO strings binary). Public issue #12 (64GB-carve box): default k_attn_fa2 crashes at PREFILL, HALOGEN_ATTN_FA=2 + HALOGEN_FA_TQ=256 restores full prefill speed — their decode was FINE, so FA_TQ's decode-at-depth relevance on Dankclaw (healthy prefill, so not in the crash class) is UNPROVEN; A/B window (default vs FA_TQ=256, 80k filler, 2×128-token decode, timings-read) was Frank-approved and i _(context: frank-large halogen tuning session 2026-10-06: cache pin shipped (ba67ca5, probe-verified 16x warm prefill), decode-at-depth diagnosed structural, FA_TQ A/B window approved by Frank and launched (arm 0 default vs arm 1 HALOGEN_FA_TQ=256); routing fallback mapped (slow/advisor → frank-primary). Commits: ba67ca5 (cache+foreground log), d39424f (docs+depth law).)_
-podman log-capture trap (found 2026-10-06 on frank-large, likely affects frank-primary too): podman run -d ... "$LOG" 2&1 captures ONLY the container ID hash — the engine's stdout/stderr go to podman's internal log store, so the documented "boot ledger lives in /home/dank/logs/svc.log" is false and every runbook grep of that file reads a one-line hex string. Diagnosis: the log file contains exactly one 64-hex-char line. Fix that preserves systemd semantics (container exit code = service exit, ExecStop=podman stop still works, SIGTERM forwarded to the engine): run podman run FOREGROUND (drop -d, drop exec podman wait), keeping the "$LOG" redirect. Do not reach for --log-driver hacks (k8s-file JSON-wraps lines and races the shell redirect). Evidence + fix live in scripts/serve-frank-large.sh (commit ba67ca5); check frank-primary's serve script for the same gap before trusting its log file. _(context: frank-large tuning session 2026-10-05/06 (halogen on Dankclaw): runbook claimed engine ledger lands in /home/dank/logs/frank_large.log, but the file held one container-ID line; foreground-run fix shipped and verified (ledger grep now works).)_
-Radiance (v1.0.7) convert/payload laws, proven on the frank-primary native-FP8 cutover 2026-10-05/06: (1) READER TABLE SHADOWING: Checkpoint::add_shard emplaces EVERY tensor each opened file physically holds, first-wins (checkpoint.cpp), and shard_list opens index-named files in SORTED name order — so a raw shard whose tensor name collides with a staging-file tensor shadows it if the raw file sorts first. Fix pattern: prefix the staging file so it sorts FIRST ("00-…"), letting its tensor win the emplace race. This bit when the plugin asked for …ple.ple_embedding.ngram_embedding.shard_0.weight and got the raw fp8 shard [2500012,160] instead of the merged bf16 table. (2) scale_name() = tensor name + "_scale_inv" VERBATIM (checkpoint.cpp:521) — stacked tensors (no .weight suffix) take …gate_up_proj_scale_inv, NOT …gate_up_proj.weight_scale_inv. (3) Staging symlinks raw files; ANY non-atomic rewrite of a staged json (open(path,"w")) writes THROUGH the symlink into the raw checkpoint — use temp + os.replace, or copy index/config into the stage instead of symlinking. Measured: raw index.json restamped with the stacked layout, raw verify then failed 143/144. (4) Python 3.14 BufferError "cannot close exported pointers exist": numpy frombuffer views keep mmap exports alive — del the view before mmap.close(). (5) Plan-only measures the .rad at the checkpoint's own fp8 mass (official Qwen FP8 131 shards = 172.76 GiB / 185.5 GB; the ~161 GiB estimate was wrong) — size any floor gate from the plan line, not estimates. (6) The plugin's bf16 branch (table_bf16 = plain-bf16 one-plane encoding) vs fp8 branch (codes + ONE-element scale plane) is decided at declare from the checkpoint form; a bf16 checkpoint + ONE rtn fixed-scale rule reproduces the publisher's dense→fp8 recipe path with exact code round-trip. (7) Debug hygiene: bg watchers that tail a sorted error list clip it — the tail of a sorted list is NOT the failure scope; fetch the full list before counting failures. All of t _(context: frank-primary radiance native-FP8 TP=2 cutover on dankserver; four failed convert attempts (BufferError crash, raw-index symlink corruption, stacked scale names, PLE table shadowing) each root-caused in /tmp/radiance-src before the clean plan-only at 172.76 GiB)_
-Radiance rad-convert pipeline laws (proven live on dankserver 2026-10-05/06, frank-primary native-FP8 build; all three plan blockers root-caused from source, not guessed): (1) FILE-TABLE SHADOW: Checkpoint::add_shard (core/format/checkpoint.cpp) emplaces EVERY tensor each opened file physically holds into one name-tensor table, FIRST-WINS; shard_list opens files in SORTED name order. A merged/replacement tensor sharing a name with a tensor physically present in another index-named file is shadowed by whichever file sorts first. Symptom: declare sees the checkpoint form of a DIFFERENT tensor (e.g. merged PLE bf16 [320001536,160] vs "codes ... is [2500012,160]" = one raw fp8 shard inside model-000NN). Fix: name the new file so it sorts FIRST ("00-ple.merged.safetensors") — index values + script constants must follow. Corollary: the index's weight_map decides which files OPEN, not which tensors EXIST; 127 raw shard entries removed from the index still shadow via file contents. (2) SCALE-NAME LAW: checkpoint.cpp scale_name() = tensor name + "_scale_inv" VERBATIM. Stacked expert tensors have no .weight segment, so the scale plane is ...experts.gate_up_proj_scale_inv (NOT ...experts.gate_up_proj.weight_scale_inv). Wrong name = "FP8 with no scale plane beside it" on EVERY sliced expert (~100k errors). Also: a tail-N watcher artifact shows only the sorted TAIL of an error list — never size an error set from a tail (21 lines shown were the tail of ~100k). (3) NEVER write a staged file via open(path,"w") when staging uses symlinks: staging symlinks raw files into the stage dir, and the rewrite goes THROUGH the link into the raw tree (mtime proves it; raw verify then fails 143/144). Law: temp file + os.replace (replaces the link itself), and COPY (never symlink) any file the reshape step rewrites. Committed as fb1a4fd. (4) Python 3.14 mmap: np.frombuffer(mm,...) views keep the mmap exported; explicit mm.close() raises "BufferError: cannot close exported pointers exist". Law: d _(context: Session 2026-10-06: frank-primary radiance native-FP8 cutover on dankserver; three successive --plan-only failures (100,352 expert scale-name errors; symlink-through raw-index corruption found by the raw verify 143/144; PLE codes shadowed to one raw shard), each root-caused in /tmp/radiance-src source (checkpoint.cpp add_shard/shard_list/scale_name, rad_arch.h map_concat_shards, rad_block_ple.h fp)_
-Two durable notes from the 2026-10-06 frank-small two-slot reshape: (1) For ANY llama.cpp what-if ("can we use X flag / drop a knob"), grep the SERVING BINARY's --help over ssh FIRST — verified live on Dankclaw's llama-master build before promising flags: the spec-type enum is draft-simple/draft-eagle3/draft-mtp/draft-dflash/draft-dspark/ngram-simple/ngram-map-k/ngram-map-k4v/ngram-mod (ngram family has its own --spec-ngram-mod-n-match etc.), and there is NO checkpoint knob anywhere — "limit checkpoints" in llama.cpp maps to dropping the draft context (draftless ngram types need no ctx_dft) and the internal recurrent-state rollback checkpoints carry no user flag. Answering from memory of other builds would have invented a knob. (2) omp bash-tool quirk: backgrounding a job with async:true requires timeout=null and name=null — passing a timeout alongside async errors out with "Service mode does not accept async or timeout"; the shape that works is {command, i, cwd, pty:false, async:true, timeout:null, name:null}, result then arrives as bg_N. _(context: 2026-10-06 frank-small two-slot no-spec reshape (commits 812d9eb, 5e16654): priced the what-if via the serving binary's --help over ssh before answering, and hit a bash-tool arg rejection when backgrounding the suite)_
-Staged-tree rewrites MUST use temp+os.replace, never open(path, "w"): a staging tree that symlinks raw inputs lets a rewrite follow the link and corrupt the pristine source (measured 2026-10-05/06 on frank-primary radiance reshape: stack script rewrote stage's model.safetensors.index.json THROUGH the relative symlink, restamping the raw index with the stacked layout — 605 KB smaller, "ple.merged.safetensors" FileNotFoundError in the raw tree, raw verify failed 143/144). os.replace(tmp, path) swaps the symlink itself for a real file; the raw tree stays untouched. Corollaries: (1) a copy step beats a symlink for any staged file a later step rewrites — copy index.json/config.json, symlink only the big immutable shards; (2) recovery WITHOUT a 217 GB re-stack: safetensors headers can be patched in place — parse hlen u64 + JSON, rename keys, re-serialize, pad with b" " to the EXACT old hlen (json.loads tolerates trailing spaces; data_offsets unchanged), rewrite bytes 8..8+hlen; (3) disk-wall arithmetic: mv-to-trash is a rename and frees NO bytes, so a full re-stage only fits if free space ≥ old payload + new payload; 209 GB free vs ~217 GB stage meant the surgical header patch was the only path without a purge approval; (4) HF index.json sanity checks: total_size lives under metadata, not top-level (KeyError trap), and "listing says" size mismatches on ONE file with 143/144 passing = corruption/restamp, not a Hub revision. _(context: frank-primary radiance native-FP8 reshape, 2026-10-05/06: run 3 rewrote the staged index through a relative symlink into the raw checkpoint; overnight verify failed 143/144. Fixed in commits 272acb4/fb1a4fd (scripts/stack-qwen38-official-fp8.py temp+os.replace; scripts/build-frank-primary-rad.sh copies index.json instead of symlinking); recovered without re-stacking by patching 49 stacked headers )_
-ComfyUI 0.37 /prompt API requires the node graph wrapped as {"prompt": {...}} — a bare graph body (the pre-0.3.x format) returns {"error": {"type": "no_prompt"}} while curl still reports full Content-Length sent. Verify renders via /history/prompt_id → status.status_str=="success", outputs in /home/dank/comfyui-data/output/. Also note: this repo's parallel sessions run broad git add commits — untracked new files can be swept into a concurrent session's commit (my ComfyUI workflow JSONs landed in frank-small commit baf33d0); after committing, verify git show --stat HEAD actually contains YOUR files before assuming. _(context: ComfyUI Qwen-Image workflow goal session 2026-10-05 (Dankclaw, port 8188, ComfyUI 0.37.0): two repair cycles to find the API wrap; parallel session baf33d0 swept the untracked JSONs.)_
-Radiance rad-convert checkpoint laws (frank-primary native-FP8 build, 2026-10-05): (1) SCALE NAME LAW: checkpoint.cpp scale_name() appends "_scale_inv" to the tensor name VERBATIM — a STACKED expert tensor (…mlp.experts.gate_up_proj, no .weight suffix) has its scale plane at …mlp.experts.gate_up_proj_scale_inv, NOT …gate_up_proj.weight_scale_inv; the per-expert convention (experts.e.gate_proj.weight → .weight_scale_inv) does not carry over. Slice law: both codes and scale are sliced at the same sub-tensor index (checkpoint.cpp:579-589), so the stacked scale plane must be [E, r/128, c/128]. (2) Python 3.14 mmap law: np.frombuffer(mm,…) arrays hold exported buffer views — sm.close() raises BufferError: cannot close exported pointers exist unless the view is del'd before close; the loop's last view is always alive at close. (3) WATCHER LAW: tail -N of a convert log is NOT the error count — errors emit sorted lexicographically so the tail shows the sort's last names (.9/.80-.99), making a full 100k-error wall look like 21 strays; always grep -c '^E ' the whole log before classifying a failure. All three already fixed in scripts/stack-qwen38-official-fp8.py (commit 272acb4). _(context: frank-primary radiance native-FP8 cutover, 2026-10-05: first plan-only run failed with ~100k expert-name errors; root cause = stacked scale plane named with .weight; second failure = BufferError crash in PLE merge; misread tail of sorted error list as the full set before reading scale_name() in /tmp/radiance-src/core/format/checkpoint.cpp:579.)_
-Two lessons from the 2026-10-05 Swift-1.5 frank-small cutover (shared checkout, live parallel writer): (1) A background suite run (bg_3) reported docs-seam test 135 failing while the LIVE parallel docs writer was editing the very index it tested — the writer's 23:25 commit fixed the seam between bg_3's snapshot and my recheck. Before chasing a seam failure that "shouldn't exist," stat the suspect files' mtimes against the suite start time and re-run the seam alone; a shared-checkout bg result is a snapshot of a moving tree, not ground truth. (2) PUT N: inserts after line N — if line N is a WRAPPED bullet/paragraph (continuation lines below), the insert lands mid-paragraph and the original continuation lines orphan below the new block. Anchor gap-inserts on the LAST line of a logical block, never its first, and re-read the whole region after any gap insert; this cost three repair edits on the runbook Trail section (bogus pasted fragment + orphaned continuation). _(context: 2026-10-05 Swift-1.5 IQ3_S frank-small cutover session; docs seam race with a live parallel session, and a PUT N gap-insert that landed mid-sentence inside a wrapped bullet)_
-radiance/qwen4exp payload laws (2026-10-05, frank-primary native-FP8 cutover): (1) The shipped radiance qwen4exp_fp8 plugin maps ONLY the STACKED expert layout (mlp.experts.gate_up_proj [E,2F,H] + _scale_inv plane at the same index, gate rows first) while the official Qwen FP8 HF repo ships per-expert unfused tensors (experts.e.{gate,up,down}_proj.weight + weight_scale_inv each) — rad-convert refuses with 100,352 "needs checkpoint tensor ... not in stage" errors; 100,352 = 49 stacks (48 trunk + mtp.layers.0) x 512 experts x 4 tensors. Fix = staging byte-stack (pure concat, no requant). (2) The PLE table ships as 128 SCALELESS fp8 shards + ONE bf16 scalar scale (1.9931793212890625e-04); the plugin's fp8 branch declares the scale view as {1} — per-shard-blocked scales can never fit, and rtn on raw scaleless codes overflows (v/1.99e-4 e4m3's 448). Fix = staging decodes to one bf16 shard (exact), then ONE --quant rule 'blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=the checkpoint's own scale' re-encodes at that same scale — provable exact round-trip (e4m3 grid 2^-3 vs bf16 rounding perturbation 2^-9), verified empirically for 12 representative codes + the 448 clamp. container PLE stays fp8 (47.7 GiB pinned, unchanged ledger). (3) The checkpoint name map gotchas: the shard regex captures pre WITHOUT the '.ple.' segment — scale and merged names must re-add it ({pre}.ple.ple_embedding.ngram_embedding.weight_scale). (4) TTM on the current dankserver kernel exposes /sys/module/ttm/parameters/pages_limit ONLY (no pages_max — the two-knob story is stale); sysfs raise to 52,428,800 pages = 200 GiB works at runtime; persistence needs ttm.pages_limit=52428800 on the kernel cmdline at next reboot; a serve-script preflight must SKIP absent knobs, not read them as 0. (5) Python 3.14 (current CachyOS) + numpy over mmap: np.frombuffer(mmap,...) arrays EXPORT the buffer — mmap.close() then raises 'BufferError: cannot close exported pointers exist' _(context: frank-primary radiance cutover session 2026-10-05: rad-convert plan-only triage (100,353 errors), staging reshape script written/debugged (name bugs, byte math, BufferError), one --quant PLE rule, TTM raise, commits 1919ad7+baf33d0 pushed)_
-Radiance qwen4exp FP8 cutover reshape laws (2026-10-05, proven in source + on-host): (1) The shipped qwen4exp_fp8 plugin maps the STACKED expert layout ONLY (mlp.experts.gate_up_proj [E,2F,H] + mlp.experts.down_proj [E,H,F] with _weight_scale_inv planes at the same index = name+"_scale_inv"); the official Qwen3.8-Flash-Next-FP8 repo ships per-expert unfused tensors — the fix is staging-side byte-stacking (no requant; gate rows first inside each expert's slab), done by scripts/stack-qwen38-official-fp8.py with per-file sampled byte-identity asserts. (2) The PLE table's plugin fp8 branch wants codes beside a ONE-ELEMENT bf16 scale plane (decl_view {1} hardcode); a per-shard-blocked scale can NEVER fit it, and rtn on raw scaleless codes overflows (v/1.99e-4 e4m3 max 448) — the fix: staging decodes the 128 scaleless fp8 shards to ONE bf16 shard with the checkpoint's own scale (1.9931793212890625e-04), then ONE --quant rule blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=s re-encodes at that same s — an exact round-trip (code grid 2^-3 vs bf16 perturbation 2^-9); the engine's bf16-vs-fp8 branch keys off the RESOLVED encoding, so the rule lands it in the fp8 branch. (3) TTM on current CachyOS kernels exposes ONLY /sys/module/ttm/parameters/pages_limit (NO pages_max); a preflight that reads an absent knob as 0 refuses a healthy cap — raise pages_limit (52,428,800 pages = 200 GiB done 2026-10-05; sysfs write doesn't survive reboot, cmdline ttm.pages_limit= is the persistence). (4) safetensors reshape scripts: compute index total_size from declared tensor bytes, never from a writer's return value (a header-only return silently undercounts ~116 GB). Repository of record: scripts/stack-qwen38-official-fp8.py docstring, scripts/build-frank-primary-rad.sh header, serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md. _(context: frank-primary radiance cutover session 2026-10-05: convert blocked with 100,353 plan errors (100,352 expert-plane + 1 PLE); three source-reading sessions on /tmp/radiance-src (rad_block_moe_fp8.h, rad_block_ple.h, qwen4exp_fp8.cpp, TOOLS.md --quant grammar, lq_grid.cpp fixed rule) resolved it into two staging transforms + one recipe rule; round-trip exactness verified on-host for 12 representative)_
-radiance qwen4exp_fp8 plugin vs official Qwen/Qwen3.8-Flash-Next-FP8 repo — LAYOUT LAWS (2026-10-05 cutover): the shipped plugin maps (1) STACKED experts — per MoE layer ONE tensor pair mlp.experts.gate_up_proj [E,2F,H] + mlp.experts.down_proj [E,H,F] with _scale_inv planes at the same dim-0 index, names under model.language_model.layers.N. and mtp.layers.0.; the official repo ships per-expert UNFUSED (experts.e.{gate,up,down}_proj.weight, 6 tensors/expert) → rad-convert --plan-only fails 100,352 errors (49×512×4); (2) PLE table fp8 branch wants codes + a ONE-ELEMENT bf16 scale plane (view {1}), but the repo ships 128 SCALELESS fp8 shards + one bf16 scalar (1.9931793212890625e-04, = the publisher recipe's own scale_value in TOOLS.md). FIX (scripts/stack-qwen38-official-fp8.py, wired into build-frank-primary-rad.sh): stage-time reshape, both transforms value-preserving — experts byte-stacked (fp8 codes moved, never decoded; gate rows first; fused scale = gate scale ++ up scale per expert), PLE decoded e4m3→bf16 with its own scale (LUT decode exact; RNE pack), then ONE --quant rule blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=the checkpoint's own scale re-encodes at that scale = EXACT code round-trip (e4m3 grid 2^-3, bf16 rounding perturbation 2^-9 — cannot cross a midpoint; verified empirically incl. the 448 clamp). rtn on RAW scaleless codes would instead overflow (v/s ≈ ×5017 e4m3 448 max). Also: rtn re-encoding must never be applied to the scaleless form; a per-shard-blocked scale can NEVER fit the plugin's {1} view; rtn on scaleless-fp8 input destroys values. Pitfalls hit: name regex consumed the .ple. segment (scale + merged names must rebuild …layers.1.ple.ple_embedding.ngram_embedding.*); bf16 byte widths need explicit ×2 (element counts ≠ bytes); write_safetensors must return data bytes (header-only return mis-states logs and index total_size). TTM LAW (dankserver kernel): only /sys/module/ttm/parameters/pages_limit ex _(context: frank-primary radiance native-FP8 cutover session 2026-10-05: convert blocker triage (100,353 plan-only errors → 2-class fix), staging transform authored + round-trip verified on host, TTM cap raised pre-cutover, incumbent FreeToken stopped at Frank's order. Runbook + script docstring carry the full argument.)_
-Converting the official Qwen FP8 checkpoint for the shipped radiance qwen4exp plugin (frank-primary cutover 2026-10-05): the plugin maps (a) STACKED experts — mlp.experts.gate_up_proj [E,2F,H] + down_proj [E,H,F] with _scale_inv planes at the same dim-0 index — so the official repo's per-expert unfused tensors must be BYTE-STACKED in staging (gate rows first inside each gate_up slab; pure byte concat, no requant); 100,352 plan errors = 49 stacks (48 trunk + mtp.layers.0) × 512 experts × 4 tensors (weight+scale each) — the arithmetic confirms which names are missing. (b) The PLE table's fp8 branch wants codes beside a ONE-ELEMENT bf16 scale plane (decl_view {1}); a per-shard-blocked scale can never fit that view AND rtn on the raw SCALELESS fp8 codes overflows (v/1.99e-4 ≫ e4m3's 448). The working transform: staging decodes the 128 scaleless shards with the checkpoint's OWN bf16 scale (0.00019931793212890625 — matches the publisher recipe's 1.99317932128906e-4 in TOOLS.md:141 exactly) into one bf16 shard_0 (config split_ngram_parts: 1), and the convert's single --quant rule blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=s re-encodes at that same scale — an EXACT round-trip (nearest-code distance 2⁻³ vs bf16 rounding perturbation 2⁻⁹; empirically verified for 12 representative codes + the 448 clamp). Helper facts: e4m3fn decode is a 256-entry LUT (0x38=1.0, 0x7E=448, 0x7F=NaN only), f32→bf16 RNE = (bits+0x7FFF+((bits16)&1))16; np.argmin over a LUT with NaN entries returns the NaN index (mask with where(isfinite,…,inf) in tests). TTM on dankserver kernel 7.2.3: ONLY /sys/module/ttm/parameters/pages_limit exists (no pages_max — "both knobs" is stale); raised to 52,428,800 pages = 200 GiB pre-cutover (sysfs write, survives until reboot; persistence = ttm.pages_limit=52428800 kernel cmdline). Any cap function must SKIP an absent knob, not read 0. Also: dankserver host has numpy 2.3.5; the dev box does not. _(context: frank-primary radiance native-FP8 TP=2 cutover session, 2026-10-05; scripts/stack-qwen38-official-fp8.py, runbook serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md)_
-frank-primary radiance cutover, convert-blocker fix (2026-10-05): the official Qwen3.8-Flash-Next-FP8 checkpoint cannot rad-convert directly because the shipped qwen4exp plugin maps ONLY the stacked expert layout (mlp.experts.gate_up_proj [512,1280,2560] + _scale_inv at the same index; error-count arithmetic 49 stacks × 512 experts × 4 tensors = the 100,352 errors) and a PLE fp8 branch whose scale view is hard-coded {1}. Fix (all value-preserving, scripts/stack-qwen38-official-fp8.py + ONE --quant rule): (1) byte-stack per-expert gate/up/down + scales along dim 0 — no requantisation; (2) decode the 128 SCALELESS PLE fp8 shards with the checkpoint's OWN bf16 scale into one bf16 shard_0 (e4m3 values and the scale are exact in bf16), then blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=ckpt scale re-encodes at that same scale — nearest-code distance 2^-3 vs the bf16 perturbation 2^-9, so every original code is restored exactly and the one-element scale plane lands as the plugin declares (rtn on the RAW scaleless codes would overflow: v/1.99e-4 exceeds e4m3's 448). Empirically verified on the host for representative codes incl. the 448 clamp. Safetensors writing: 8B LE u64 header length + JSON header, data_offsets relative to data start, pad header with spaces to 8-byte alignment; bf16 = RNE pack of f32 via (bits + 0x7FFF + ((bits16)&1)) 16. Dev box has NO numpy — test numpy-dependent host scripts on dankserver (numpy 2.3.5). Also: hashline PUT/CUT with remembered line numbers across MULTIPLE edits in one file silently corrupts (a mis-aimed PUT N.=M ate main-body lines and a later insert landed mid-function) — after any multi-edit sequence, py_compile BEFORE running; recover by re-reading the touched region and restoring eaten lines first. TTM on dankserver: /sys/module/ttm/parameters/pages_limit exists, NO pages_max; raised to 52,428,800 pages (200 GiB) pre-cutover; sysfs does not survive reboot (cmdline ttm.pages_limit=52428800 pe _(context: frank-primary radiance cutover 2026-10-05: stacking/PLE reshape + convert-blocker triage + TTM raise; runbook frank-primary-radiance-qwen38-native-fp8.md)_
-TTM pinned-page cap on the CachyOS fleet hosts (verified 2026-10-05 on dankserver, kernel 7.2.3): only /sys/module/ttm/parameters/pages_limit exists — there is NO pages_max, so any preflight that mins over both knobs must SKIP an absent knob, not read it as 0 (a 0 makes a healthy cap refuse; fixed in scripts/serve-frank-primary.sh ttm_cap_mib). Pre-cutover cap was 32,252,917 pages = 123 GiB, BELOW frank-primary's 134 GiB pinned ledger. Raising is a runtime sysfs write (echo 52428800 | sudo tee ...pages_limit = 200 GiB) — allocates nothing, harmless, no GPU experiment rules apply; verified read-back. The write does NOT survive reboot: persist via kernel cmdline ttm.pages_limit=52428800 at the next scheduled reboot. Required pages = MiB × 256 (4 KiB pages); the serve gate is FRANK_PRIMARY_TTM_MIN_MIB=140000. _(context: radiance cutover pre-work for frank-primary (dankserver), 2026-10-05: TTM preflight gate design + live sysfs raise before the pinned-RAM engine boot.)_
-Converting a scaleless-FP8 artifact into a container that hard-codes a one-element scale plane (radiance plugin law, e.g. rad_block_ple.h wants fp8 codes + a single bf16 scale): NEVER rtn-re-encode the raw codes at the fixed scale — v/1.99e-4 overflows e4m3's 448 max and the codes become garbage. The exact recipe: (1) staging DECODES the scaleless shards with the artifact's OWN bf16 scale into a bf16 table (e4m3 decode is exact; the product may round to bf16 by 2^-9), then (2) the convert's rtn rule re-encodes at that same scale — nearest-code distance is 2^-3 while the bf16 perturbation is 2^-9, so every original code is restored EXACTLY and the one-element scale plane is produced. Round-trip proven empirically for representative codes incl. the 448 clamp. Companion facts: a per-shard-blocked scale can NEVER fit a {1} decl_view (plugin hard-codes extents); safetensors bf16 = 2 B/element (byte-stacking code that forgets the ×2 produces ValueError/buffer-shortage crashes, and argmin over an e4m3 LUT must mask NaN or it returns the NaN entry 0x7F). Implemented in scripts/stack-qwen38-official-fp8.py + one --quant rule in scripts/build-frank-primary-rad.sh; full argument in the script docstring and serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md. _(context: radiance native-FP8 TP=2 cutover for frank-primary (dankserver), 2026-10-05: convert of Qwen/Qwen3.8-Flash-Next-FP8 blocked by 100,353 plan errors (100,352 stacked-expert + 1 PLE scale-plane); derived the decode-reencode round-trip from rad_block_ple.h/rad_block_moe_fp8.h/lq_grid.cpp source tracing and verified it on-host before committing to a 102 GB staging write.)_
-TWO lessons from the 2026-10-05 frank-large halogen cutover: (1) ROGUE/PARALLEL-SESSION RECOVERY RECIPE (this checkout is shared): a second omp session force-committed OVER this session's uncommitted work (serve script reverted to the pre-fix --ulimit nofile version, AGENTS.md lessons deleted, 413 restored over 400), then push created divergence mid-deploy. Recovery that worked: git log --graph --all to map the two lineages (rogue 3ee88d7/77decb5 Ornith commits + my be840b4/b0b840b above shared base bdbe814), git diff origin/main HEAD --stat to classify every diverging file as MINE (keep) vs ROGUE (drop), git rebase origin/main (clean replay of my 2 commits), then re-apply the specific reverted hunks. The seam is the tripwire that catches this class: tests/frank_small_unit.bats failed 7 tests because the rogue swap commit (Ornith payload) never touched the deploy handler + seam twins — stale seams DETECT half-finished swaps. Also: my own edit-tool tags went stale after the clobber — trust grep, not remembered line numbers, and re-read every region before editing. (2) halogen (frank-large) contract facts verified LIVE 2026-10-05 (engine 0.1.4, health reports version.api==version.engine==0.1.4): boot ledger prints "kv pool: 1 slot(s) x 200700 tokens = 13.15 GB" + dtype census (bf16 2.96 GB, q4c 9.93 GB, fp8r 10.63 GB, i4l 12.35 GB — p1w4d-d2). Prompt cache AUTO SIZES ITSELF OFF on this box ("35.7 GB free after mapped checkpoint ... less than the 36.9 GB floor (2 x 18.4 GB full-context entry)") — expected, not a fault. Drafter verified from the RESPONSE timings object (llama-server shape): draft_n/draft_n_accepted counts at 39-42 t/s greedy — DFlash2 active on the response, not just /health's drafter_default. Tool round-trip REAL: get_weather - {"city":"Seattle"} as an OpenAI tool_calls array (wire format qwen-xml inside the engine). Contract: over-cap max_tokens AND prompt-over-slot AND response_format are ALL 400s naming the limit (README-verified; my earlier 413 cla _(context: frank-large halogen cutover session 2026-10-05; discovered when a parallel omp session force-committed over uncommitted work mid-deploy)_
-Parallel-session docs writer vs the deploy tree gate: deploy.sh refuses on ANY uncommitted change INCLUDING untracked files (whole-tree guard, scripts/deploy.sh ~line 117), and live docs writers (Frank's Obsidian vault work or another agent) constantly produce untracked serverAppConfigs-docs/LLM/*.md files. Stashing a LIVE writer's strays is a data-loss race: 2026-10-05 a stash -u of 6 stray docs was followed within seconds by 3 MORE untracked files appearing — the writer was active; had the stash held, their in-flight files would have vanished from under them. Reversed with an immediate git stash pop (safe: stash snapshots, nothing lost). Working protocol: (1) check writer liveness first — mtimes within ~10 min of now means LIVE, do not stash; (2) the labeled-stash convention ("parallel-session topic WIP - stashed for reason") applies only to quiet strays; (3) if the writer is live, the deploy is genuinely blocked on Frank's coordination — say exactly which files block the whole-tree gate and ask him to commit/close the writer; do NOT commit another session's WIP yourself, do NOT stash twice in a race, and do NOT bypass the gate. _(context: 2026-10-05 frank-small Ornith deploy: 364/366 tests green, swap pushed, download verified — deploy blocked solely by a live docs writer's strays at the whole-tree gate; one stash/pop round-trip proved the race is real before it cost anything.)_
-Recipe: read a remote HF GGUF's metadata without downloading it — HTTP range request + struct parse. curl -fsSL -r 0-8388607 -o head.bin https://huggingface.co/repo/resolve/main/file.gguf, then parse: magic 4B, version u32, tensor_count u64, kv_count u64, then KV pairs. Two parser traps cost three failed attempts (2026-10-05, Ornith-1.5-9B sizing): (1) initialize offset to 4 AFTER the magic assert — treating offset 0 as the version field reads the magic itself as u32 (0x46554747 = "GGUF") and every downstream field shifts; (2) string arrays must be fully consumed (n × (u64 len + bytes)) — early-returning "str[n]" without consuming leaves offsets inside array data and later keys misread as unknown types. The tokenizer vocab array (~150k strings) sits BETWEEN the general.* keys and the architecture keys (qwen35.block_count etc.), so 64 KiB is NOT enough — fetch ~8 MiB. Yields arch name, block_count, head counts, key/value lengths, full_attention_interval (→ KV rate = full_attn_layers × kv_heads × klen × 2 B × 2), nextn_predict_layers, expert_count (absent = dense), native context — enough to size VRAM before committing a payload swap. Note: an existing managed skill gguf-loader-compatibility-check may be the right home for this; verify its content before minting a duplicate. _(context: 2026-10-05 frank-small→Ornith-1.5-9B swap: needed Ornith's architecture (9 full-attn layers, 36 KiB/token f16 KV) to size the shape; the range-read settled it in two cheap requests against the live Hub file, no download required.)_
-Frank's standing working rules for skill management (established 2026-10-05): (1) Do NOT use subagents for reading/analysis/distillation work — he explicitly ordered solo work ("Without subagents… read through each skill", later "NO. SUBAGENTS."); parallel fanout is not authorized by default even when tools suggest it. (2) The knowledgebase (serverAppConfigs-docs) is the system of record for all fleet knowledge — managed skills are NOT durable homes for lessons; when skills duplicate or extend runbooks, distill their content into vault pages and delete the skills, staging deletions in ~/.AGENTTRASH first. His default is "no reason to keep these skills". _(context: 2026-10-05 managed-skills audit session: Frank twice ordered "Without subagents" and then "NO. SUBAGENTS." when the agent planned task-fanout for reading 173 skills and distilling them; and he ruled that none of the fleet skills are worth keeping as skills — their content belongs in serverAppConfigs-docs.)_
-serverAppConfig multi-session triage rule (proven 2026-10-05, three events in one day): when ./tests/run.sh goes red and the failing seams point at files you did not touch, FIRST split the breakage by git state: (a) COMMITTED-but-broken states from a parallel session (swap commits that forgot their seam/deployer twins) → repair them yourself in clearly-labeled follow-up commits; that is what kept the suite green twice. (b) UNCOMMITTED working-tree state (M/untracked files owned by a LIVE concurrent session — check git status --porcelain and untracked ?? entries against HEAD, which may not have moved) → do NOT edit those files; the other session is mid-flight and will reconcile env+seam+docs in its own commit; editing its dirty tree risks colliding with an active writer. Verify your own pushed state instead (rev-list HEAD...origin/main, targeted seams on your files) and report the named dirty files as the red-suite owner. Signature of case (b): HEAD identical across two suite runs yet different failures, because bats seams read the working tree, not the commit. _(context: 2026-10-05 goal session: frankbook e2e split (9c50821/2930431/1c725ea/5573ca1) interleaved with a live parallel session committing Ornith-swap breakage and later holding the tree dirty (M frank-small.env, M frank_small_unit.bats, ?? GGUF Compatibility.md).)_
-Wrapper-script law (serverAppConfigs tests/e2e/run.sh): when converting a pure passthrough exec tool "$@" into "default argument set + passthrough", the default set must be APPENDED to "$@", not substituted for it — exec npx playwright test "${PROJECT_ARGS[@]}" silently DROPS every other arg (--list, --grep, file filters), so run.sh --list ran the entire 34-test suite instead of listing. The regression is only visible by diffing observed behavior against requested semantics; the seam (bash -n) cannot see it. Correct shape: build PROJECT_ARGS only when no user --project is present, then exec npx playwright test "$@" "${PROJECT_ARGS[@]}". Caught 2026-10-05 because the --list proof printed execution checkmarks instead of a listing; verify arg-forwarding wrappers by exercising EVERY documented passthrough arg, not just the happy path. _(context: 2026-10-05 frankbook e2e split; regression introduced by my own default-project-set rewrite and caught during verification (commit 1c725ea).)_
-frank-small seam congruence rule (serverAppConfigs repo): ANY change to scripts/frank-small.env has TWO consumers that must flip in the same change: (1) scripts/deploy-remote.sh's LLM_SMALL_MODEL_FILE/LLM_SMALL_MMPROJ defaults (the payload-existence gate — test 196 greps for the exact env path inside the deployer default), and (2) tests/frank_small_unit.bats (TARGET_STUB/MM_STUB fixture names, the MODEL_FILE env assertion, and the KV-contract test pinning --cache-type-k/--cache-type-v values from the env). On 2026-10-05 two consecutive commits broke it: the Ornith-1.5-9B Q8_0 payload swap (77decb5) left both consumers on MiMo-Q6K paths, and the KV reprice (1862dc3, f16 V→q8_0 V) left the KV test asserting the old unquantized contract — its commit message even claimed the seam was 'already retargeted in 8f53df7' when 8f53df7 covered only payload paths. Symptom each time: full ./tests/run.sh red in frank_small_unit.bats only ('missing /tmp/.../mmproj-...' or argv-value mismatch). Fix = grep the env for the new values and update all three spots. _(context: 2026-10-05 goal session (frankbook e2e split) blocked twice by frank-small seam staleness from concurrent sessions; fixed via 8f53df7-equivalent edits and 5573ca1.)_
-Wrapper scripts that inject a default argument set must keep "$@" in the final exec — dropping it silently converts user flags into no-ops. Measured 2026-10-05 in serverAppConfig tests/e2e/run.sh: the first default-project-set rewrite exec'd only the injected --project ... args, so ./run.sh --list silently RAN the full 34-test suite (~79 s) instead of listing — the mislabeled output then nearly passed off a full run as a listing (and ran a test against a host that was assumed asleep). Correct shape: build PROJECT_ARGS only when the user gave no --project, then exec npx playwright test "$@" "${PROJECT_ARGS[@]}" — user args first, defaults appended, explicit --project still takes over. Corollary on verification hygiene: condition-dependent pass claims ("green with the host asleep") require the condition observed at run time (ping/curl), not assumed — the Mac was awake during that run, so only the mechanism (default set contains no frankbook project, proven by --list composition) carried the claim. _(context: serverAppConfigs e2e frankbook split session 2026-10-05: run.sh default project set (frankbook excluded, E2E_INCLUDE_FRANKBOOK=1 opt-in) introduced by commit 1c725ea after the passthrough regression was caught by an E2E_INCLUDE_FRANKBOOK=1 --list invocation that ran instead of listing.)_
Fleet consists of dankserver (2x R9700) and dankclaw (Strix Halo + 3090). LLM units: frank-medium (Qwen3.8-27B, layer-split 35:30, 16GB cache), frank-large (Halogen, 14.4GB KV pool), and frank-small (Gemma4-12B vLLM). Deployment is Quadlet-based with a path-aware planning pipeline. Key technical constraints include RCCL atomic routing, Halogen memory pinning, and the OMP session persistence model.
Current fleet is stable but marginal on `dankserver` RAM (4600 MT/s). `frank-large` (Halogen) and `frank-primary` (VLLM) are the main inference providers. Watch for `max_tokens` 400 errors on large prompts and orphaned `llama-server` processes after benching. `frank-medium` requires 16384 MiB `CACHE_RAM` for hybrid DeltaNet states. Quadlet units ensure boot persistence. Use `ak shell` for Authentik user management. Distill skills to vaults to reduce context bloat (~38k tokens overhead).
Transitioned frank-primary to Qwen3.8-Flash-Next-GGUF at Q_8. Configured with Q_8 drafter, vision enabled, and layered speculative decoding using mtp and ngram-mod.
Deployed prefix caching and integrated autoresearch branch updates (PLE timeout 1800s, bench matrix runbook, index/doc-link fixes) for the frank-primary model on R9V hardware.
Purged all documentation mentions of 'Dankware' (retired workstation) from both `~/.omp` and `serverAppConfigs`. Functional artifacts `packages/dankware.pkgs` and `scripts/install.sh` were preserved as the name is load-bearing for host lookups.
Deep research on DeepSeek 4.1 Flash quantizations for the current hardware (2x R9700 gfx1201 + 246 GiB DDR5 RAM). Identified `vcruz305 Q2_K` as the optimal fit for the backbone (~191 GiB) and `GSQ-RCO 3.0-bit` as the high-quality alternative. Key bottleneck is RAM bandwidth (42 GB/s), limiting decode to ~6-11 tok/s. Next step: Build native HIP version from the fork.
Autoresearch harness for frank-primary (vLLM on Dankserver 2×R9700 TP=2 port 7770) speed optimization. Harness: dev-box orchestrator → SSH → frank-primary-window.sh on Dankserver. Discipline: stop prod unit, boot candidate AS service (same script/port, no scratch ports), in-container measure.py, EXIT trap unconditional restore+health verify. Boot cost ~7min. Staging dir /home/dank/.autoresearch-frank-primary/ holds candidate env/serve/measure/run.json. AGENTS.md has contradiction between topology table (no MTP, 2×262144, bf16 KV) and detail section (MTP4, 200k×3, fp8 KV) — authoritative source is actual live scripts.
Determined log access paths for frank-primary on dankserver: engine logs via podman logs (container-lifetime only) or file log at /home/dank/logs/frank_primary.log; journalctl -u frank-primary only has script-level lines since container runs detached.
Comprehensive survey of tcclaviger vLLM fork logging levers on frank-primary: standard vLLM env vars (VLLM_LOGGING_LEVEL, VLLM_LOG_STATS_INTERVAL, VLLM_LOGGING_CONFIG_PATH, VLLM_DEBUG_WORKSPACE, VLLM_TRACE_FUNCTION, AMD_LOG_LEVEL) are the only surface; no R4D_*/CLAV_* debug switches exist in r4d.so/parohip/fp8hip. Log file trap: `podman run -d > "$LOG"` captures only container ID, not engine output.
Swap frank-large to Uncensored Q6_0_ROCMFPX_AGENT (23.15 GiB, ftype 114) from kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF; extend frank-small context to 200700/stream; download MiMo Q8_0 to host.
ASUS ProArt B850-Creator WIFI NEO: BIOS reset via CLRTC header short (pins 1-2, 5-10s) or F5 Load Optimized Defaults; max DDR5 spec is 8400+ MT/s (not marketing 9000); 4-DIMM derates below printed ceiling; MT/s = MHz×2; user hit 4600 MHz (9200 MT/s) before failure — well above spec; DIMMs do not need removal for CMOS clear; CMOS clear does not affect module SPD.
Fleet token-throughput benchmark across three llama.cpp endpoints (frank-primary, frank-large, frank-small). Discovered `bc` is not installed on the host. All endpoints healthy (200 OK, <10ms health checks). Sequential 256-token completions completed in ~19.5s total wall time across all three. Recent project work: Q6_K quantization cutover, arena/KV shape tuning.
User requested a 3000-word creative story about a Soviet pizza analog. Assistant incorrectly evaluated the html-change-report skill as a trigger for creative writing, then was interrupted before completing the story. No output was delivered.
Frank (Chief Designer) operates a multi-model local AI serving stack (frank-primary/large/small) on dankserver; frank-primary is an MoE Flash-Next tower on R9700 DDR5 with MNBT 2048 and 7,762-slot arena, frank-large is a 27B MoE on iGPU with MTP4, frank-small is a 9B hybrid on 3090. Token generation throughput test confirmed ~38 t/s on frank-primary with ~1.8s TTFT.
Added Authentik user `ridley` (ridley@franksdank.cloud) with user-level-only access via Django ORM in `ak shell` on dankclaw. Key workflow: SSH to dankclaw, run `ak shell`, create User with set_password(), assign groups (Non-Admin Users, jellyfin_users, openwebui_users). Verified auth via authenticate() ORM call. Browser SPA interaction is unreliable for Authentik flows—use ORM verification instead. Group membership is the sole gate for app access in this deployment.
Diagnosed frank-primary (tcclaviger vLLM fork, Qwen3.8-Flash-Next FP8, 2×R9700, Dankserver) repetition/corruption after ~40k tokens. Top hypotheses: fp8 KV precision degradation at long context, hybrid model (DeltaNet/Gated Attention) recurrent state not correctly checkpointed at preemption/recompute boundaries, and prefix-caching eviction triggering non-identical state recompute. Also recovered frank-large from a rogue-session git clobber (nofile ulimit regression, half-finished Ornith swap) and verified endpoint contract.
Diagnosed frank-primary ~40k-token output corruption (repetition/garbage) on dankserver. Exhaustive software hypothesis sweep (fp8 KV degradation, prefix caching eviction, expert offload LRU, RoPE, chunked prefill boundaries, CUDA graph, block table overflow) all failed to reproduce or fix. Root cause: marginal DDR5 at 4800 MT/s on 4 DIMMs silently corrupting data; fix was dropping to 4600 MT/s. Key methodological lesson: consumer non-ECC RAM produces no MCE for silent flips, short pattern checks miss intermittent corruption, and software reverts tested under the same marginal hardware are confounded.
Ornith-1.5-9B-Q8_0 swap on dankserver (AMD/ROCm) replacing MiMo: f16 KV, ubatch 1024, single 262,144 slot. Deploy blocked by tree-clean gate due to parallel docs writer.
Context bloat diagnosis and skill-library distillation: measured context components (AGENTS.md, memory_summary.md, 175 managed-skills descriptions = major bloat source), distilled skills into vault reference pages on Gitea, staged deletion to ~/.AGENTTRASH/, purged after explicit user approval.
Replaced frank-primary from tcclaviger vLLM fork to radiance (StillDeadcode v1.0.7) runtime serving Qwen/Qwen3.8-Flash-Next-FP8 at native FP8 (no requantization) with TP=2 on 2x R9700, MTP depth 3, expert_tiered placement, and fp8 KV cache. Native FP8 checkpoint at its own precision requires no quantization step or container.
Attempted frank-small swap to ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF IQ3_S with MTP+vision+FP8 KV on RTX 3090; rollback ordered mid-flight. Final live: Swift RCO-IQ3_XXS + draft-mtp 2×131072. Bonsai2-CRACK + llama-prism parked on disk, not deployed.
Fixed halogen 400 max_tokens-does-not-fit error by raising HALOGEN_CTX from 200192 to 262144 (native Qwen3.8-Flash-Next window) on dankclaw frank-large service, enabling 180k+ token sessions. Harness contextWindow set to 217088 = 262144 - 45056. KV pool expanded to 524288 positions (13.3 GiB).
The standard 'bench stop-and-mask' protocol. A session that benches owes the fleet a health 200 before it ends. Safe order: 1. Confirm GPUs idle. 2. systemctl start. 3. Health 200. 4. Verify geometry (/proc/MainPID/cmdline). 5. Behavioral proof (completion).
4x DIMMs at 4800 MT/s caused silent corruption at ~40k tokens. Fix: Drop to 4600 MT/s. Consumer non-ECC RAM raises no MCE for silent flips. Short in-band checks miss intermittent corruption. Software reverts tested under same marginal hardware are confounded.
Core stall: With `HALOGEN_FLASH_PIN_TRUNK=1`, co-tenant can stall engine for minutes. Last resort: `HALOGEN_FLASH_PIN_TRUNK=0`. First request after restart may be silent ~45-60s. `frank-large``maxTokens` must be <=65536. Prompt + max_tokens must fit KV pool or queue ~3600s then 503s.
Rootless Podman `restart: unless-stopped` only survives crashes. Bare containers vanish on reboot. Repo `serverAppConfig` mandates Quadlet `.container` files. Historical: Caddy lost after reboot (Aug 26).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Frank