command-center/agent/memories/--home-blank-Projects-serverAppConfig--/raw_memories.md

Raw Memories

01a0d06e-c8b5-757b-a093-2081af1db7c6

updated_at: 1790503506 Context: 1970s Moscow cafeteria (stolovaya). Protagonist: Pyotr Arkadyevich Volkov (chef). Problem: The 'kotleta' (cutlet) lacked architecture, falling apart when eaten with a fork. Solution: Enclosing it in a pampushka bun to make it a 'hand-consumed' burger. Bureaucracy: The invention moved from a chef's observation to a formal memo, through a 9-link bureaucratic chain, eventually becoming a GOST standard (Meat cutlet in a soft bread bun). Key Philosophy: 'The standard gives the product a body. The kitchen gives it a life.' The 'half-millimeter' tolerance is the space where the individual's touch meets the state's law.

01a0e144-473f-71a5-96f8-0316232910aa

updated_at: 1790503503

Placement & Tensor Laws

• Override Logic: --override-tensor is first-match-wins. Specific patterns must precede broad sweeps (e.g., exps=CPU should be listed after specific blocks but before general ones).
• PCIe Strategy: -ot exps=CPU is the primary method to stop crossing the PCIe bus; it works only if --no-mmap is NOT used (as no-mmap forces tensors into anonymous memory, breaking the CPU-resident trick).
• Atomic Units: The unit of placement is the per-block expert tensor (512 experts = 1 tensor). There is no per-expert routing knob.
• Expert Distribution: Top 10% of experts carry 40.5% of samples. Hottest layers do not exist at -ot granularity because Top-8 routing fires on nearly every block.

Spec Decoding & Drafter Laws

• Composition: Legal combinations include [one context-backed drafter (MTP/DFlash/DSpark/Eagle3)] + [any number of ngrams]. Two context-backed drafters cannot share a process.
• KV Scaling: Drafter n_ctx automatically scales to the target's n_ctx.
• Drafter VRAM Traps: DSpark (no sliding window) can consume ~5.0 GiB at 262k context, while DFlash2 (sliding window) consumes only ~40 MiB.
• Device Pinning: Pin the drafter to the card holding the main model's output head (e.g., ROCm1 under -ts 1,1) to avoid SIGABRT on pre-allocated buffer mismatches.

Performance & Implementation

• Ubatch Optimum: For frank-medium, the peak is 512 (1193 tok/s). Below 512 is under-saturated; above 512 starts to see diminishing returns and higher VRAM costs.
• GGUF Extraction: Derive tensor extents from consecutive header offsets rather than quant type-size tables to avoid mis-bucketing tail elements into 'gap' buckets.
• Fleet State: Primary (192.168.0.235:7770), Large (192.168.0.143:7770), Small (192.168.0.143:7771).

01a0dfa8-7193-7235-980d-69e138218fcc

updated_at: 1790474002 Model: GLM-5.3-Flash-Uncensored-GGUF (orcarouter). Params: 321.3B MoE. Key Constraints: 1. Requires NVIDIA_TF32_OVERRIDE=0. 2. Must use -fa off for correct MLA output. 3. Use -ub 128 to mitigate microbatch-dependent long-context collapse. 4. Use -ot exps=CPU for offloading experts as Q6_K (~264GB) is slightly larger than available RAM (246GB). Features: Supports draft MTP (--spec-type draft-mtp). Context: Previous frank-medium unit was in a crash loop on ParoQuant MXFP6, highlighting the need for stable deployment of this new large MoE model.

01a0df97-1e50-77d7-9084-92708bfd1152

updated_at: 1790458743 Connectivity: Repeated 'Unable to connect' errors (ID 135168) occurred on dankserver using frank-medium. Framework: Introduced /guided-goal, a structured workflow requiring a 5-point interview (Success criteria, Verification, Attempt cap, Boundaries, Stop conditions) before executing a persistent autonomous objective loop. Current Project: Deploy https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF on dankserver using the https://github.com/Luce-Org/lucebox runtime.

01a0d66d-f9e2-7681-a5e3-e77b896dcc3f

updated_at: 1790407605 Homepage CSS Update: Flipped the stagger pattern in shared/homepage/style.css. Role and Inputs now use --dim (#2f8f52, bold), while Spec and Use use --fg (#33ff66). Removed the --input variable. Deployment: Committed cd07a66, pushed, and deployed to /srv/homepage. OpenWebUI/Authentik: OpenWebUI is running as a quadlet in dankclaw. Authentik integration (OAuth2/OIDC) is confirmed as the primary method for passing login state, matching the current containerized architecture.

01a0c81c-56dc-7359-ac24-6da36332ad84

updated_at: 1790068527

ComfyUI 0.37.0 Native Qwen-Image 2.1

• Released 2026-09-21, tag v0.37.0, CORE-423, PR #16400
• Path: comfy/ldm/qwen_image21/model.py (model detection + supported-models registration)
• Dankclaw image (comfyui-rocm:gfx1151, Containerfile.comfyui) has CUSTOM GGUF patches (apply_qwen21_patch.py: VAE 4D→5D expansion, in_channels shape[0] read, QwenImage21 arch registration)
• ACTION: On next image rebuild, check if native 0.37.0 path makes those patches redundant BEFORE re-applying
• Repo moved to Comfy-Org org (comfyanonymous/ComfyUI redirects)

OpenWebUI API Correction

• Real imagegen endpoint: POST /api/v1/images/generations (NOT /api/images/generate)
• Version in vault: 0.11.4

Calibre / Calibre-Web CVE Conflation

• CVE-2025-7404 belongs to Calibre-Web, NOT Calibre itself
• Calibre version corrected: 9.9.0 → 9.15.0

Vault Structural Facts

• Repo: /home/blank/Projects/serverAppConfig
• Docs: serverAppConfigs-docs/ (Obsidian vault, OKF v0.1 format)
• OKF frontmatter: type/description/tags/timestamp (YAML)
• 136 pages total, 127 non-reserved docs with FM
• Deployment: Quadlet/deploy-plan.py (NOT the old deployment.md description)
• GTT: 114 GiB (was incorrectly documented as 96)
• Hosts: Dankserver (192.168.0.235, media), Dankclaw (192.168.0.143, AI/dev)
• Tests: ./tests/run.sh → 336 seams
• Link checker: check-vault-links.py
• *arr docs moved to FrankBook-Pro/ section

Research Briefs Created

• Local TTS and Voice Cloning (OKF doc)
• AI Job Search and Resume Tools (12 verified projects)
• Most Popular Self-Hosted AI Apps Jun-Sep 2026 (top 15 by GitHub stars: OpenClaw 390k, Hermes 248k, Ollama, Dify, Open WebUI, ComfyUI)
• HTML report: popular-self-hosted-ai-apps.html (dark theme, star bar chart, per-app cards, homelab-fit table, 11 adoption best practices)

01a0c51e-53af-724a-9f4e-364858819889

updated_at: 1790063680

frank-medium chat template wiring

• Entry: dankserver/frank-medium/radiance-entrypoint.sh line 253: --chat-template "$CHAT_TEMPLATE"
• The CHAT_TEMPLATE env var is already plumbed; to switch templates, set it to the desired jinja file path.
• Qwen-Sharp template location: scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp (archived from ik_llama.cpp era for frank-large).
• The template is also referenced in tests/deployment_scoping.bats as a deployment-scoping guard.
• To activate: either copy the template to a stable path and set CHAT_TEMPLATE, or reference the .AGENTTRASH path directly.

frank-large swap (completed)

• Replaced Kairic Edge (4-bit) with an 8-bit model + DFlash2 spec decode drafter.
• Commits: 7689092 (swap), 600812d (docs: AGENTS.md, runbook, LLM index, top-level index, homepage).
• 8-bit is ~23% larger weight size than 4-bit Kairic Edge, explaining ~22 t/s vs higher throughput.
• DFlash2 draft acceptance is text-variety-dependent: 0.85 on trivial one-word answers, 0.519 on varied explanatory prompts. Do not read a single request as steady-state.
• Kairic Edge payload + runtime remain on disk as rollback target.
• Pending: purge 87 GiB assembled file + 31.7 GB corrupted model on dankclaw (awaiting approval).

01a0c5c1-0f4c-72bc-81b8-8b05f92d0ef1

updated_at: 1790063676

Homepage Architecture

• Location: shared/homepage/
• Files: services.json (main service list, ~35 entries), dankserver.services.json (host-specific), app.js (frontend renderer), style.css
• app.js determines CATEGORY from URL: 'admin' if path contains /admin, else 'user'
• Services filtered by svc.category !== CATEGORY
• WIP state: svc.wip boolean flag → state 'wip'
• Status from status.json → stateByName[s.name] gives 'up'/'down'
• Panel rendering: badge (state), name, description, links (open/docs)
• Summary line: UP/DOWN/WIP counts
• Bare mode toggle via ?bare=1 query param
• Deploy timestamp via window.DEPLOYED_AT baked at deploy time

Open WebUI Image Generation API

• Container: open-webui (podman) on host dankclaw
• Route file: /app/backend/open_webui/routers/images.py
• Endpoint: /api/images/generate
• Handler function: image_generations (line 592), called at line 575
• Config keys: image_generation.enabled, image_generation.engine, image_generation.model, image_generation.openai.params, etc.
• Access via: ssh dankclaw 'podman exec open-webui <cmd>'

01a0c6e4-d2b6-7078-a0c9-f492a78605a2

updated_at: 1790063108

frank-medium MXFP6 deployment (2026-09)

Model swap

• Model: just1moremodel/Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, Swift finetune)
• Template: Qwen-Sharp v22.5.0 (178 prompt tokens for a simple math question; 73,600 is the fallback sentinel)
• Drafter: DFlash2-FP8 (Qwen3.8-27B-based, not Swift-specific) — works fine, acceptance 85.7% easy / 58.3% varied
• max_model_len: 262144
• KV pool: 425,047 tokens (1.62x max_model_len), 14.42 GiB/card available

GPU_UTIL ceiling on 32GB R9700

• 0.99 is INFEASIBLE: vLLM startup check runs BEFORE weight load. 0.99 x 31.86 GiB = 31.54 GiB required, but only 31.23 GiB free at device-init. The 0.63 GiB gap is consumed by ROCm/HIP context + NCCL + radiance custom all-reduce buffers.
• 0.98 is the ceiling: 0.98 x 31.86 = 31.22 < 31.23. Prior production value.
• Symptom of 0.99: ValueError at engine init → crash loop. The 'registered' lines in journal are NOT the loop — they're ~20 sequential patch_*.py invocations each printing NOOP sitecustomize lines.

Weight sizes (affects KV pool)

• MXFP4: 19.37 GiB
• MXFP6: 25.16 GiB (5.79 GiB more → less KV headroom)

File paths

• Quadlet: dankserver/frank-medium/quadlet/frank-medium.container
• Entrypoint: dankserver/frank-medium/radiance-entrypoint.sh
• Repo: serverAppConfigs at http://192.168.0.143:3000/dank/serverAppConfigs.git
• Install path: ~/.config/containers/systemd/frank-medium.container

Deployment workflow

1. Edit quadlet + entrypoint + runbook
2. git commit + push origin main
3. Deploy (pulls latest, restarts container)
4. e2e gate (32/32)
5. Verify: health 200, model name, spec decode config, completion (prompt token count confirms template), tools, drafter acceptance

Commits

• e9a01cd: MXFP6 swap
• d5c8e40: GPU_UTIL 0.98 revert
• 24f7a77: measured boot shape in runbook

Open item

• Drafter is Qwen3.8-27B-based, not Swift-matched. Acceptance healthy (58-86%) so it's a net win, but a Swift-specific drafter would be the next lever if needed.

01a0c5e9-dee3-734a-946c-3cf809f9d7fd

updated_at: 1790027980

OMP Session Persistence

File Layout

• Session files: ~/.omp/agent/sessions/<project-slug>/<timestamp>_<id>.jsonl
• Project slug derived from cwd path.

Data Size Controls and Blob Externalization

• Strings >500,000 chars truncated with "[Session persistence truncated large content]", EXCEPT: signed/encrypted provider blocks, signature fields, complete Anthropic native web-search history blocks (must remain byte-exact for replay).
• Transient jsonlEvents field removed before persist.
• lineCount recomputed after truncation when both content (string) and lineCount (numeric) present.
• Image data URLs in image_url fields: ALWAYS content-addressed in blob store, replaced with blob:sha256:<hash>, regardless of length.
• Other base64 image payloads externalized at 1,024 chars threshold (image content/data payloads, image-generation results).
• Redundant OpenAI Responses thinkingSignature copies omitted when authoritative reasoning item already in providerPayload.
• On load, blob references resolved back to inline payload shapes.

Storage Abstractions

• SessionStorage interface: sync dir/existence/write/stat/list; async read, sliced read, write, guarded atomic write, rename, unlink, artifact-aware deletion, title update, writer creation, backend drain.
• FileSessionStorage: real local files.
• MemorySessionStorage: map/chunk-backed for non-persistent sessions and tests.
• IndexedSessionStorage: shared local index + ordered remote publication (Redis/SQL-backed).
• SessionStorageWriter: append, appendSync?, flush, flushSync?, isOpen, close, getError.

Error Behavior

• Persistence errors latched and rethrown by later flush/close/write; first logged once with session-file context.
• Failed atomic publication attempts authoritative repair.
• If storage may have published and repair cannot be proven durable: SessionPersistenceIndeterminateError fails closed with original + recovery errors.
• Writer close propagates first meaningful error.

Session Discovery

• getRecentSessions(sessionDir, limit?) → lightweight welcome metadata, default limit 4.
• findMostRecentSession(sessionDir) → newest by mtime.
• listSessions(sessionDir, storage) → project scope with lifecycle status.
• listAllSessions(storage) → all project scopes.
• resolveResumableSession(...) → local lookup then optional global fallback.
• Recent/most-recent scans: 4 KiB prefix only.
• Full lists: 4 KiB prefix + bounded 32 KiB tail for lifecycle status.
• Scans are stat-keyed and cached; large sets use bounded parallel workers.
• Per-directory scans recover newest orphaned EPERM backup when primary JSONL missing.
• Resume matching: case-insensitive, accepts session id prefixes, full filename prefixes, or id suffix after timestamp.

HistoryStorage (separate subsystem)

• SQLite at ~/.omp/agent/history.db
• Table: history(id, prompt, created_at, cwd, session_id)
• FTS5 index: history_fts with trigger-maintained sync.
• Deduplicates consecutive identical prompts via in-memory last-prompt cache.
• Inserts batched through async drain queue (~100ms delay) to avoid blocking turn execution.
• Use session files for conversation graph/state replay; use HistoryStorage for prompt history UX.

01a0aeb5-e0ba-742d-86da-df30e865107d

updated_at: 1790026831

Project: Private llama.cpp Fork (serverAppConfig repo)

Directory: llamacpp/ at repo root (planned, not yet created).

Phases: 0. Recon: git status, disk space, identify forks on GitHub, read runbooks, verify SSH to dankclaw/dankserver.

1. Clone mainline llama.cpp into llamacpp/.
2. Trim AGENTS.md — remove rules that block private-fork freedom (e.g. 'don't add dependencies', 'keep PRs small', 'no breaking API changes').
3. Merge charlie12345/llama.cpp — source of ROCmFP4 quantization for RDNA4. Test on frank-large (dankclaw, gfx1151 iGPU, port 7770). Must stop halogen service first per benchmark policy.
4. Merge ik_llama.cpp — improvements for unique setup. Runbook: frank-large-ik-uncensored.md.
5. Create managed skill: merge-llamacpp-forks (merge latest from all parent forks + TDD, noting quirks).
6. Optimize: terse, machine-optimal code. Hardware-aware: gfx1201 2x R9700 layer split, gfx1151 iGPU, CUDA 3090 x4 link.
7. Knowledge base: new section in serverAppConfigs-docs/ (OKF format).
8. HTML instruction manual: how to quantize models for R9700 pair.

Hardware topology:

• dankserver (192.168.0.112): 2x R9700 gfx1201, layer-split inference.
• dankclaw (192.168.0.143): RTX 3090 CUDA + gfx1151 iGPU. frank-large service on port 7770.
• the workstation / blankpad (dev boxes): Ryzen 5 PRO 5650U, RTX 4070 Max-Q. Has passwordless SSH to fleet.

Fork sources:

• Mainline: ggml-org/llama.cpp
• charlie12345/llama.cpp: ROCmFP4 quantization (RDNA4-focused). Also referenced as 'ROCmFPX fork' in runbooks.
• ik_llama.cpp: used on frank-large 2026-09-15 to 09-16.

Pitfalls encountered:

• frank-medium on dankserver was unreachable (10 retries exhausted, ~15.6s each). Service down.
• frank-large context overflow: prompt 144085 tokens + max_tokens 57344 > context 200192. Need to reduce max_tokens when prompt is large, or increase context.
• TDD skill mandates seam confirmation with user before writing tests — one consolidated question needed.
• Deploy planner: new top-level directory llamacpp/ should be a no-op (unknown paths don't map to services).

Status: Phase 0 (recon) was in progress when session was interrupted. No code written yet.

01a0c58d-9751-7094-bb98-3d7f54669e56

updated_at: 1790024021

Homepage Architecture

• Location: shared/homepage/
• Files: services.json, dankserver.services.json, app.js, style.css, index.html, admin.html, docs.html
• app.js determines category from URL path: /admin → 'admin', else → 'user'
• Services in JSON have fields: name, category, description, wip (bool), url, docs, showLink
• State resolution: if svc.wip is true → 'wip'; else look up name in status.json → 'up'/'down'
• Rendering: flat grid, one panel per service matching the current category
• No auto-refresh (Q41), bare mode via ?bare=1 (Q27), deploy timestamp baked in (Q43)

Deployment Pipeline

• deploy.sh (dev box): pre-flight (clean tree + on main), runs deploy-plan.py, git push, SSH to target for ff-only pull + revision check, execs deploy-remote.sh, then Playwright e2e gate
• deploy-plan.py: maps changed file paths → service names. shared/homepage/* + scripts/homepage-status.py → homepage. Docs-only → no-op. --service NAME bypasses planning.
• deploy-remote.sh (target): takes explicit service list (never infers). State store at ~/.local/state/serverAppConfigs/deploy-state/<service> with revision= and status=. Services marked pending before handler, deployed after success. Crash recovery: pending services retried next run. --pending (default) catches up services whose revision lags. Retired-service lockout protection (trash, not rm).
• Key invariants: master = deployed state (Q33), ff-only pull + exact revision match, service-scoped (unrelated services never restarted), deploy not complete until e2e gate passes
• E2e gate can take ~20 minutes (admin link-walk alone budgets 240s)

User Request Context (not yet implemented)

• Wanted sections on homepage: AI, Web Apps, Media Library Apps
• Wanted sections on admin: Media Library Apps, Observability, AI, Infrastructure
• Wanted separate labels: 'Work in Progress: Open' vs 'Work in Progress: Closed'
• Noted outdated AI descriptions and missing tautulli link on admin page

01a0c510-1716-724c-b99b-b1ee4df08178

updated_at: 1790020691

Calibre Content Server Deployment (Dankserver)

Image: lscr.io/linuxserver/calibre:9.15.0 — this is a DESKTOP build (Selkies GUI), NOT headless. The dedicated headless images are dormant (calibre 6.10/2017, below CVE fix threshold). Must run calibre-server as the s6 main service via Exec= in the Quadlet unit.

Port traps:

• Internal port 8080 is the image's own nginx → Address already in use if you try to bind calibre-server there. Use internal 8090 → host 8084.
• Default auth mode is Digest, not Basic. Every Basic-auth login returns 400 Unsupported authentication method. Fix: --auth-mode=basic.

Bootstrap password trap:

• echo "password" | ... stores password\n (trailing newline) → every login 401. Use printf '%s' on stdin.

pasta port-forward mode: On this host, pasta runs in port-forward mode. Only explicitly published ports are reachable from outside the container netns. The desktop GUI (if started) would be unreachable without an additional PublishPort line.

Final shape: Boot-persistent Quadlet (Restart=always), image pinned, userdb bootstrap idempotent in deploy handler, 3 new seam tests added.

Calibre-Web 0.6.27 SSO Analysis

• NO generic OIDC support. Login types: local, LDAP, GitHub, Google only.
• No authlib installed, no OIDC config keys.
• Upstream feature request open; maintainer declining more login options.
• Grafana's RP-mode OIDC path is NOT available.
• Correct path: guard chain (Anubis + Authentik + Caddy) with zero app changes. Calibre-Web's own login remains as a second layer (no disable mode), same posture as RomM.

Guard Chain Pattern (Fleet-Standard)

1. Outpost proxy provider: external_host = https://<domain>/, internal_host = http://caddy:<internal_port>
2. Caddy vhost: reverse_proxy anubis:8923 + post-auth @<name> handler on :<internal_port>
3. Anubis guard set + Caddy edge + provider list deployed together

Project Context

• Repo: serverAppConfigs — single source of truth for Dankserver (192.168.0.235, Ubuntu 24.04) and Dankclaw (192.168.0.143, Ubuntu 26.04)
• Deployment: Podman/Quadlet, service-scoped git deploy via scripts/deploy.sh
• Test suite: 328 tests via ./tests/run.sh, e2e gate 32/32 Playwright
• Docs vault: serverAppConfigs-docs/ with link integrity checked by test suite

01a0c09e-b740-7547-8216-e31373fa0316

updated_at: 1789968641

Why vllm-radiance TP=2 failed on Dankserver

• Hardware: 2x AMD Radeon AI PRO R9700 (gfx1201), chipset Gen4 x1 with AtomicOps Routing- (no PCIe atomics)
• RCCL collectives (allreduce, all_gather, broadcast, barrier) require PCIe atomics on upstream bridges — unavailable on this chipset
• PP (pipeline parallelism) also needs p2p send/recv using the same atomic primitive
• vLLM's layer-split mode isn't expressible in the needed form
• The radiance fork's RADIANCE_USE_R4D_AR (one-shot P2P push allreduce over gloo IPC handles) covers prefill+decode allreduce but does NOT intercept PyNcclCommunicator.init warmup allreduce or pynccl_comm.all_gather (direct RCCL calls)
• NCCL symmetric memory: unavailable (shipped RCCL advertises NCCL 2.30.4 but exposes no ncclDevComm* API)
• Workaround that worked: two single-GPU replicas (frank-medium-a/-b) behind a round-robin proxy on port 7770
• Corroborated by Codeberg StillDeadcode/vllm-radiance issue #22 (different host, same image, kernel 7.0.0-30, same enqueue.cc failure)
• Retired 2026-09-16; replaced by mainline llama.cpp with DFlash2 drafter

models.yml update (MTP → DFlash2 drafter swap)

• File: ~/.omp/agent/models.yml (NOT in git repo)
• Actual API settings unchanged: contextWindow=262144, maxTokens=32768, reasoning=true, input=[text,image]
• Stale comments fixed:
  • Model: Qwen3.8-27B-Uncensored-FP8 (30.9 GB / 7 shards), accept 4.24/7, throughput 100.0 vs 85.9 t/s (A/B), 100.0 vs 45.7 t/s
  • KV pool: 384,238 tokens / 1.47 concurrent (was 943,581 / 3.59 with MTP); DFlash2 drafter costs 2.1 GiB vs MTP's 461,540 tokens
• Pitfall: a schema parse error in models.yml drops the ENTIRE file silently (learned from the audio incident) — always verify 0 schema errors after edits
• Running sessions keep their process-start snapshot; changes take effect on next new session

01a0b5e1-46b7-7387-a64d-f6f69f79be10

updated_at: 1789847739

Fleet Model Swap Workflow (llm-fleet-model-swap skill)

• Read HF model card first: runtime pin, hard constraints, recommended args
• Check host for existing model files before downloading: ls /home/dank/models/, df -h /home, du -sh
• Artifact integrity: LFS oid in HF API = sha256; compare size AND sha256sum of on-host file
• Deploy via git flow: commit → push → deploy.sh dankclaw --service frank-small → health check → e2e
• Pre-deployment live verification + port-holder health-check trap check

vLLM MTP Speculative Decoding for Gemma-4-12B

• Main model: google/gemma-4-12B-it-qat-w4a16-ct (HuggingFace)
• MTP assistant: kunhunjon/gemma-4-12B-it-qat-assistant-w4a16-ct
• Target GPU: 3090 (24 GiB, sm_89 Ada)
• Fit parameters to maximize: max_model_len (full context), max-num-seqs (session count), gpu-memory-utilization
• Precedent: vLLM was previously on frank-medium (vllm-radiance replica, retired 2026-09-16)

3090 RGB Control (EVGA iCX3 EEPROM)

• Zone mode register read: 09 m0 m1 m2 m3 [field] led0 led1 led2 led3 sync
• m0-m3 = 0x00 means all zones MODE_OFF
• Write format: 0xC0 0x09 m0 m1 m2 m3 FF FF FF FF 0x00
• EEPROM save persists across reboots; no Linux-side daemon required
• LED counts in pkt[6-9] are informational (e.g., 04 05 3c 00 = per-zone LED counts)
• Tools: evga_probe.c (read), evga_raw.c (write) in /tmp on dankclaw

Post-Cutover Fleet Topology (2026-09-19)

• Host: Dankclaw (192.168.0.143)
• frank-small (:7771): Gemma4 Q4_K_M, llama.cpp, root systemd unit
• frank-large (:7770): CIRU Orca, rootless Quadlet (halogen)
• frank-medium: orcarouter Q8_0 + DFlash2
• All units enabled, self-restore on boot
• 331 bats + 32 e2e tests green

Operational Notes

• llamacpp (dankserver) provider can have transient 'Unable to connect' errors; auto-retry recovers (seen 2 consecutive failures then recovery)
• frank-small was previously on FreeToken (2026-09-02 → 09-15), then llama.cpp master 37b53fd with HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M, CTX_SIZE=1048576, PARALLEL=4, batch 8192/1024, q8_0 KV

01a0b693-7972-7104-bb4b-2bbec7f42acc

updated_at: 1789773472

frank-medium CACHE_RAM sizing law

Constraint: CACHE_RAM must exceed one full slot's prompt state size, which is n_ctx_slot × 68 KiB (64 KiB from 16 full-attention layers at 4 KV heads × 256 head_dim × f16 + 4 KiB MTP draft KV). At n_ctx_slot=196608: ~12.8 GiB. Must also stay under half host RAM (62.2 GiB → cap < 31 GiB). 16384 MiB satisfies both.

Why it's load-bearing for hybrid DeltaNet models: With --ctx-checkpoints 0, the recurrent cell (DeltaNet state) sits at the END of the prompt. If the RAM cache refuses the state snapshot, llama.cpp has no checkpoint to roll back to — it takes the 'forcing full prompt re-processing' path. The RAM cache is the ONLY thing that can carry a state across slots. A refused state is simply gone.

Historical evidence: 137 lines of prompt state size 11xxx MiB exceeds cache size limit 8192.000 MiB, skipping in 2 days (2026-09-16→18). Max observed: 11,388.99 MiB.

Verification: argv (--cache-ram 16384), code path, arithmetic, and historical refusal logs. The >8 GiB threshold was NOT exercised live (needs 123k+ token conversation ≈ 2-3 min prefill; AGENTS.md says not to run beside live service). Tell for future: refusal lines should only name states >16384 MiB; an eviction line naming an entry >8192 MiB would be direct proof.

Test seam pattern (tests/frank_medium_unit.bats)

New assertion: CACHE_RAM must clear one full slot's state (~12.8 GiB) AND stay under half host RAM. Fails on 8192, passes on 16384. This is the canonical way to guard against regressions when n_ctx_slot or model architecture changes.

Deploy housekeeping

Peer's uncommitted frank-small/vLLM WIP blocked deploy's dirty-tree gate. Pattern: git stash → deploy → git stash pop. Note: with peer WIP present, ./tests/run.sh is red (7 frank-small/frank-large seams fail) — full suite isn't a clean signal until that work lands.

frank-medium key facts (for future reference)

• llama.cpp master fb27a525 (2026-09-16), gfx1201, ROCm 7.14.0a20260624
• Layer-split across 2x R9700 (NOT tensor parallel: RCCL needs PCIe atomics, second card behind B650 chipset lacks them)
• Model: Qwen3.8-27B UD-Q8_K_XL (29.29 GiB), arch qwen35, 64 layers (48 DeltaNet + 16 full-attn)
• Drafter: DFlash2 Q8_0 (1.92 GiB), 5 layers, block_size 8, 2048-token sliding window
• n_slots=2, n_ctx_slot=196608, ubatch_size=1536
• VRAM: GPU0 ~33.1 GB used, GPU1 ~31.0 GB used; host RAM 13.4/62.2 GiB (cache is on-demand, unpinned)
• MTP engaged: draft_n 33 / accepted 25, 37.85 t/s
• Commit: 1acca43

01a0b5f3-22ec-7692-a98c-8290e5cc3476

updated_at: 1789762341

Incident: frank-medium GPU stranding by orphaned llama-server (2026-09-18)

Symptom

• curl http://192.168.0.235:7770/health → connection refused (rc=7)
• systemctl is-active frank-medium → inactive
• systemctl is-enabled frank-medium → enabled
• Unmasked (symlink to /home/dank/serverAppConfigs/scripts/frank-medium.service)
• NRestarts=0 → clean stop, not a crash loop

Root Cause

• An orphaned llama-server (PID 2519965, PPID 1) from ~/serverAppConfigs/third_party/llamacpp/build-merge was holding both R9700 GPUs
• Bound to 127.0.0.1:17770 with -c 16384 --split-mode layer --tensor-split 1,1 -ngl 99 -t 16
• Model: Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
• Timeline from sudo journal + /tmp/layersplit.log:
  • 08:14:02 frank-medium started
  • 09:19:44 frank-medium stopped (bench protocol)
  • 09:19:47 merge-build llama-server spawned
  • 09:20:00 log shows "model loaded" / "listening" — nothing after
  • ~12:15 user reports down (3h05m of silence)
• Prod couldn't self-recover: 35:30 split needs ~30.8 GiB ROCm0 + ~28.9 GiB ROCm1; stray owned ~15.5/17.1 leaving insufficient headroom

Diagnostic Pattern (reusable)

When a prod LLM unit is stopped but GPUs are busy:

1. rocm-smi --showmeminfo vram → check if used memory is non-trivial (>100 MiB baseline)
2. ps -eo pid,ppid,etime,args | grep llama → find the holder
3. Prove abandonment before killing:
   • /proc/<pid>/stat utime+stime identical across 10s sample (frozen CPU = idle)
   • ss -tnp | grep <port> → zero established connections
   • Check bind scope (loopback-only = not serving external traffic)
   • Last log mtime (hours stale = abandoned)
4. Check PPID: PPID=1 means orphaned (parent shell/harness exited)

Fix Procedure

1. Preserve stray's log: cp /tmp/layersplit.log /tmp/layersplit.log.preserved-YYYYMMDD
2. kill <pid> (exits <1s, cards drop to ~60 MiB baseline)
3. systemctl start frank-medium
4. Wait for health 503→200 (~20s when GGUF warm in page cache)
5. Verify geometry: cat /proc/$(systemctl show -p MainPID --value frank-medium)/cmdline | tr '\0' ' ' → confirm --tensor-split 35,30, ctx 393216, --parallel 2, --flash-attn on, --spec-type draft-mtp
6. Run real completion to confirm spec-decode: expect draft_n/accepted ratio ~90%, tg ~40 t/s
7. Fingerprint check: b11009-fb27a525d = pinned runtime

Fleet Topology (dankserver)

• Host: 192.168.0.235, user dank, SSH from blankpad
• GPUs: 2x AMD R9700 (32 GiB each) + 1x small (512 MiB)
• frank-medium: port 7770, tensor-split 35:30, ctx 393216, parallel 2, flash-attn on, spec-type draft-mtp
• Unit file: symlink /etc/systemd/system/frank-medium.service → /home/dank/serverAppConfigs/scripts/frank-medium.service
• Login shell on dankserver is ZSH — avoid words starting with = in SSH commands

Key Insight

Loopback scratch servers are invisible to endpoint monitors (they don't bind the prod port), so they strand the fleet silently instead of failing loudly. Two bench-protocol strandings occurred on the same day (2026-09-18): morning was a simple stop-without-restore; this one was worse because the orphan held GPUs.

Suggested Guards (not yet implemented)

• restore-frank-medium.sh trap in bench harness (EXIT handler to self-restore)
• Fleet check that alerts when GPUs are busy while no prod unit is active

Commit

• cfabe07 in serverAppConfigs-docs/log.md

01a0b30f-1e89-763e-b65d-48d87e6ee683

updated_at: 1789756946

Incident: frank-medium left down after bench protocol (2026-09-18)

• Symptom: connection refused on 192.168.0.235:7770. Unit was inactive (enabled, unmasked, NRestarts=0) — not a crash.
• Root cause: AGENTS.md bench stop-and-mask protocol executed from PWD=/home/dank via agent SSH. Timeline: stop+mask 02:47:00, unmask+start 04:21:27, final stop 04:25:59. Step 3 of the protocol (unmask/start + health verify) never ran for the last round.
• Recovery sequence (safe order):
  1. Confirm GPUs idle: both R9700s ~60 MiB, no llama-bench/llama-server procs.
  2. systemctl start frank-medium → active, health 200 first probe.
  3. Verify /proc/MainPID/cmdline matches deployed geometry: --tensor-split 35,30, --ctx-size 393216, --parallel 2, batch 6144x512, --spec-type draft-mtp.
  4. Behavioral proof: chat completion returns expected verbatim text.
• Pinned runtime fingerprint: b11009-fb27a525d
• Durable lesson: The restore half of the bench protocol is not optional. A session that benches owes the fleet a health 200 before it ends. Recommended fix: make the bench harness self-restoring (trap EXIT → start + health-verify) so a dead session can't strand the endpoint.

Guard chain topology (reference)

Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home/@data handlers) → app/static.

Key gotchas:

• BusyBox wget in caddy container: NO -H flag, use --header "Host: ...".
• 8016/8017 live in caddy container's netns; curl from host returns 000 (expected).
• Anubis container is distroless; podman exec sh fails.
• x-real-ip is masked to 10.89.1.22 (podman SNAT) for ALL clients; clearance is cookie-based.
• Homepage app.js does Promise.all([fetch services.json, fetch status.json]); on ANY fetch failure the grid stays empty and summary shows 'status unavailable: '.

01a0b23a-7a27-7792-b8da-97d7421b99c3

updated_at: 1789756944

Plex /mnt/entropy group permission fix on dankserver

Problem

Plex container on dankserver cannot read files in /mnt/entropy. Host-side permissions on /mnt/entropy must NOT be changed (no chmod/chgrp/chown/setfacl on dir or contents).

Diagnostic path (read-only)

1. ssh dankserver 'podman inspect plex' — check container groups, mounts, uid/gid mapping
2. ssh dankserver 'stat -c "%A %U:%G %n" /mnt/entropy; getfacl -p /mnt/entropy; ls -ln /mnt/entropy | head' — identify owning group of files
3. ssh dankserver 'findmnt /mnt/entropy' — determine mount type (NFS, mergerfs, local disk)
4. ssh dankserver 'id dank' — check if host user is already in the required group
5. Confirm container sees the tree: podman exec plex ls /mnt/entropy | wc -l
6. Probe a specific file: podman exec plex head -c 16 <file>

Fix approach (container-side only)

• Rootless quadlet: Use GroupAdd=keep-groups in the quadlet unit. This passes the host user's supplementary groups into the container. Prerequisite: host user dank must be a member of the group that owns /mnt/entropy files. Adding dank to that group is a host change (allowed), not a /mnt/entropy change.
• Rootful container: Use GroupAdd=<gid> with the numeric GID of the owning group directly.
• Alternative: --group-add keep-groups in compose if still compose-based.

Key fleet patterns

• Dankserver media stack (Plex, Jellyfin, RomM, Copyparty, Calibre-Web) uses compose.yaml in repo dankserver/ dir, but boot persistence requires quadlet units (compose-to-quadlet cutover in progress).
• Rootless podman user-unit spawns carry NO supplementary groups by default — must explicitly use GroupAdd=keep-groups or enumerate GIDs.
• Deploy workflow: edit repo at /home/blank/Projects/serverAppConfig → ./tests/run.sh → commit → push → scripts/deploy.sh dankserver --service plex. Deploy gate rejects dirty git tree.
• SSH alias: ssh dankserver (User dank, 192.168.0.235) from blankpad.
• Deletion policy: never rm, use .AGENTTRASH.

Plex local API probing (on dankserver)

• Identity: curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:32400/identity
• Token location: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml, extract with grep -oP 'X-Plex-Token="\K[^"]+'
• Token in curl: use curl -K with a mode-600 temp file (NOT -H on cmdline) to avoid leaking token in /proc/<pid>/cmdline.
• Library sections: curl -s -K <tokenfile> http://127.0.0.1:32400/library/sections
• PMS may return HTTP 000 on /identity while port 32400 is listening (pre-bind before HTTP layer ready); especially after long downtime. Wait or retry.

Session status

Interrupted before final fix applied. Next step: confirm the GID owning /mnt/entropy files, ensure dank is in that group (if not, sudo usermod -aG <group> dank), set GroupAdd=keep-groups in the plex quadlet unit, deploy, verify with podman exec plex head -c 1 <file>.

01a0b09a-aeae-75f5-9522-c5a3f1d97ac7

updated_at: 1789710503

Fleet Topology

• dankclaw (192.168.0.143): frank-large :7770, frank-small :7771 (llamacpp root units)
• dankserver (192.168.0.235): frank-medium :7770 (llamacpp root unit), media stack (podman user quadlets)
• frankbook-pro (192.168.0.112): *arr stack as quadlets inside podman machine
• All services configured from /home/dank/serverAppConfigs on each host
• Repo: /home/blank/Projects/serverAppConfig
• Quadlets run as user units (systemctl --user)

frank-medium 35:30 Tensor Split (verified post-deploy)

• Flags: --split-mode layer --tensor-split 35,30 --spec-type draft-mtp
• GPU0 (x16): 30.84 GiB used / 1.02 GiB free
• GPU1 (x1): 28.86 GiB used / 3.00 GiB free
• Bottleneck card headroom tripled (1.03 → 3.00 GiB); GPU0 is now the heavier card
• Utilization: x16 card 40%→60%; x1 card still 98% (structural: MTP drafter + output layers pinned to last-layers card)
• 36:29 fails floor (0.45 free) AND slack gate (2.22 gap) — layer split lever is exhausted; next lever is ctx cut
• MTP acceptance rate: 0.51–0.91 on live traffic
• Fit predicted 30.59/30.01; measured 30.84/28.86 (actual shift ~1.9 GiB, larger than fit's 0.58 GiB gap)
• Empty-body probe responses = queue backlog from live agent traffic, not split defect
• Runbook: serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md (commit 2fc0689)

frank-large max_tokens 400 Dead Zone

• Model: halogen, ctx 200192
• Dead zone condition: prompt + maxTokens > ctx AND prompt < 70% compaction trigger (140,134)
• Example: prompt 137,879 + 64,000 = 201,879 > 200,192
• Fix: maxTokens 57344 in ~/.omp/agent/models.yml
• Closes zone: worst-case 140,134 + 57,344 = 197,478 ≤ 200,192
• CRITICAL: OMP snapshots model config at process start; existing sessions are permanently pinned to the old value until restarted (use omp -r to resume)
• Wire-verified via tcpdump on dankclaw: fresh one-shot omp sends max_completion_tokens: 57344

Testing

• 326 seams passing (up from 321; a peer added a seerr test)

01a0ac72-2620-750f-82dd-79c84e00e98d

updated_at: 1789631572 DANKCLAW HARDWARE: AMD Strix Halo APU (gfx1151), Radeon 8060S iGPU. Has an M.2 dock slot used to carry an RTX 3090 at PCIe x4 (link downgraded from x16). This is distinct from dankserver (ASUS TUF GAMING B650-E WIFI) which has a free CPU-attached M.2_2 slot for an R9700 riser (~$30 M.2→PCIe x4 riser).

DANKCLAW LLM DEPLOYMENTS:

• frank-small: Qwen3.8-27B GSQ-RCO IQ3_S + MTP on RTX 3090 (M.2 dock, PCIe x4). Decode 61-82 t/s, prefill 430-1013 t/s. VRAM at 92% (22,614/24,576 MiB). PCIe utilization only ~3% during decode (compute-bound, not transport-bound). --parallel 1, 8 CPU threads.
• frank-large (halogen): runs on the iGPU (gfx1151), decode 37-48 t/s, MTP commit 1.60-1.88 tokens/round. Prefix caching confirmed working.

UNANSWERED QUESTION: User asked specifically about the M.2 slot in context of a Qwen Flash deployment running with pipeline parallelism on dankclaw. The assistant's response was aborted (OMP process exited) before answering. This likely refers to a llama.cpp pipeline-parallel setup where the Qwen Flash model's layers are split across the iGPU and the 3090 in the M.2 dock, using the M.2 slot as the interconnect between the two compute devices.

01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3

updated_at: 1789599423

Rootless Podman restart policy pitfall

• restart: unless-stopped only survives crashes, not explicit stops (e.g., kernel power-off sends a polite stop signal).
• Result: bare containers without a Quadlet unit are gone after reboot with no alert.

Repo serverAppConfig deployment pattern

• Every container must have a Quadlet .container file (git-tracked) to be boot-persistent and self-healing.
• The unit file is the container's birth certificate + pension plan + ghost in one.

Historical incidents (from story)

• Aug 26: Caddy never returned after reboot (no unit).
• Sep 11: Five observability services found unitless after reboot.

Persona note

• Model: frank-medium via llamacpp (dankserver). Persona: Soviet-style computer systems engineer with Munroe-esque humor. Uses "Comrade" address.

01a0a7ad-42e2-739d-bc1c-dc405ca02cbd

updated_at: 1789574671

Dankserver Dual R9700 RCCL Chipset Issue

Hardware: 2x AMD R9700 on ASUS board. Card #1 on CPU-attached PCIe x16. Card #2 on chipset-attached PCIe x16 slot (PCIEX16_2, which is x1 mode per ASUS spec).

Problem: RCCL (ROCm Collective Communications Library) fails on the chipset-attached card. Bare 2-rank torch.distributed probe results:

• broadcast: OK (CPU card) / FAIL (chipset card)
• all_reduce: OK / FAIL
• barrier: OK / FAIL
• send/recv (P2P): hang / FAIL

Implication: send/recv failure kills pipeline parallelism too (vLLM's PP handoff IS that primitive). No software-only escape.

Root cause: Chipset-attached PCIe lanes don't route atomic operations the way RCCL's kernels require. CPU-attached lanes do.

Fix (recommended): M.2→PCIe x4 riser (~$30) to move card #2 to M.2_2 slot. ASUS spec confirms M.2_1 and M.2_2 are CPU-attached; M.2_2 is free/empty. Restores atomic-op routing → RCCL works → TP=2 functional.

Alternative: Board with CPU x8/x8 bifurcation (~$250-400).

Fallback: llama.cpp layer split (no collectives needed, but needs GGUF and loses vLLM batching).

Bandwidth analysis: 27B TP=2 allreduce at batch 4 ≈ 10 MB per token-step ≈ 1.4 ms at Gen4 x4. Bandwidth is NOT the bottleneck; atomics routing is.

Software Stack

• vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14
• Shipped RCCL exposes no ncclDevComm* symbols → torch's symmetric-memory path can't run
• Open hypothesis (unverified): recompiling RCCL with -mprintf-kind=buffered might work, but no live printf/assert/__assertfail found in device source

Operational Notes

• dankserver went down during probing; needs physical power cycle
• Boot persistence should auto-restore: frank-medium trio, media stack, node-exporter
• To distinguish 'off' from 'wedged': ip neigh from peer (FAILED = off/NIC dead; wedged kernel still answers ARP)
• AGENTS.md benchmarking policy updated to cover any GPU experiment, not just benchmarks

Model Target

• kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
• Fork: charlie12345/rocmfpx (llama.cpp ROCm fork)

Skill Note

• 'grilling' skill does NOT exist in available skills list (returned 'Unknown skill: grilling')

01a0a87f-6b17-7492-995c-8c3e7b37e241

updated_at: 1789552363

Hardware Topology (dankclaw)

• RTX 3090 (24 GiB VRAM) in PCIe 4.0 x4 M.2 dock (downgraded from x16, measured ~29 GB/s sequential)
• AMD Strix Halo gfx1151 (64 GiB SVM-backed unified memory, RADV Vulkan driver; ROCm wheels do NOT support gfx1151)
• Combined GPU budget: ~88 GiB
• OS: Ubuntu 24.04
• Kernel cmdline: amd_iommu=off (for prefill gains; retired frank-npu)

ik_llama.cpp Build & Deployment

• Built with CUDA (for 3090) + Vulkan (for Strix Halo) backends
• Serves mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF at i1-Q4_K_M
• Deployed as root systemd unit (Restart=always) on port 7770, NOT a quadlet
• Context window: 262144 tokens
• Device split: --split-mode layer with -dev CUDA0,Vulkan0
• Shipped config: T16 (16 threads), no -ncmoe, no -thp
• Commits: 795b0b3, ad7a9c4
• Runbook: serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

Performance Envelope (verified single-instance)

Config	Decode (t/s)	Prefill (t/s)	Notes
Shipped T16	3.8-4.0	~192 (1.8k-32k ctx)	Production config
T8/T4/T2/T1	4.18/4.20/4.26/3.97	small-prompt only	Thread count barely matters
-ncmoe 42	4.54	—	22.0/24.6 GiB VRAM, fragile at 262k ctx
-ncmoe 48 -thp	4.60	—	110 GiB anon host buffer, co-tenant killer
--prefetch-experts	4.03	—	No effect (cache already resident)
Mainline llama.cpp 37b53fd	1.01	—	Ruled out entirely

Root-Cause: Why 30 t/s Decode Is Unreachable

1. Decode issues ~100 graph splits per token
2. Each split = GPU launch + sync across PCIe x4 (3090 in dock)
3. GPU SM utilization during decode: ~5%
4. ~90% of cycles = pause (thread barrier spin); T1 ≈ T16
5. CUDA graphs would collapse the overhead BUT capture ABORTS: qwen4exp hc ops use per-layer ggml_repeat_4d which is capture-illegal
6. Speculative decoding unavailable: GGUF has no MTP head (mradermacher stripped NextN from the checkpoint)
7. All experts on GPU: 84 GiB > 24 GiB 3090 VRAM — impossible
8. Path forward: mainline llama.cpp PR #28901 (hc-ops + capture-safe graphs) — re-evaluate when it lands

Prefill Ceiling

• ~192 t/s is the ceiling: compute-bound on CPU AVX512-VNNI
• Expert reads amortize across the batch; math saturates cores
• Halogen hit 30 t/s decode by holding entire 115.55 GiB w4b checkpoint resident in LPDDR inside one proprietary engine; 24 GB GPU + CPU split cannot replicate that residency with any public runtime

Operational Traps

• Unit has Restart=always: pkill'd servers respawn mid-experiment → phantom instances invalidate benchmarks. Always verify single-instance before measuring.
• Concurrent -j16 builds can starve probe processes
• PCIe x4 dock: do NOT assume x16 bandwidth; measured ~29 GB/s (not 130 GB/s)
• -ncmoe 42: +15-20% decode but only 2.5 GiB VRAM margin at 262k ctx — OOM risk
• -ncmoe 48 -thp: 110 GiB anon host buffer — will kill co-tenants

Trash (staged for user approval)

• dankclaw ~/.AGENTTRASH/20260916-ik-partfiles/ — 111 GiB (3 download parts, superseded)
• dankclaw ~/.AGENTTRASH/20260915-ik-swap/ — retired halogen quadlet
• repo tests/.AGENTTRASH/frank_large_quadlet.bats, scripts/.AGENTTRASH/frank-small*

01a0a902-7f58-77f4-9392-764f6cfbd59e

updated_at: 1789544414

AGENTS.md Rewrite Pattern (serverAppConfigs)

• Collapse chronological swap history into: (1) active-topology table (current state only), (2) runbook pointer table mapping each retired runtime → its docs/LLM/ runbook path.
• Add a 'Key Lessons Learned' section containing ONLY durable technical constraints/pitfalls (not narrative).
• In config sections (e.g. halogen), keep only invariants that survive any config change (pool/reserve matched pair, client contract, memory-model rule); push sizing history to runbook.
• Fix structural bugs (e.g. broken merged table rows) as part of the cleanup.

Durable LLM/ROCm/Quadlet Lessons (from AGENTS.md distillation)

• Layer-split-not-tensor: for multi-GPU ROCm, split by layer not tensor-parallel.
• mmap/PLE constraints apply to model loading on ROCm.
• HIP env collision: TEMP and HOST env vars can conflict with HIP runtime.
• GGML_CUDA_FA_ALL_QUANTS env var needed for flash-attention with quantized KV.
• MTP (Multi-Token Prediction) draft head has loader incompatibility on ROCm.
• For crash diagnosis: use gdb over logs (logs are often incomplete/misleading for segfaults).
• Swap-resize safe ordering: resize before use, not after.
• free command lies about available memory (doesn't account for reserved/pinned).
• systemctl disable can delete unit files (not just disable them).
• pkill -x for exact-name process kill (avoids substring matches).
• Rootless quadlet log retrieval has specific gotchas (see skill: rootless-quadlet-log-retrieval).

omp Skill Discovery

• omp ONLY discovers user skills at ~/.omp/agent/skills/.
• pi-era locations ~/.pi/agent/skills/ and ~/.agents/skills/ (vercel-labs skills CLI layout) are NOT scanned by omp.
• npx skills CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
• Newly installed skills surface on the NEXT session, not the current one.
• Skill sources: github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff), github.com/vercel-labs/skills (find-skills).
• 4 skills unrecoverable without the workstation online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base.

01a09e67-630f-74ed-a910-e997e01fbe54

updated_at: 1789368883 Source: serverAppConfigs-docs/LLM/halogen-flash-server.md (halogen-flash-server runbook), sections Memory model, Two failure modes, Co-tenancy, and troubleshooting table.

Core stall advice: With HALOGEN_FLASH_PIN_TRUNK=1 (default), the halogen container holds most host memory. If a co-tenant takes the remainder and host memory runs out, both the engine and the co-tenant can stall for minutes at full CPU with no disk activity and no output. This is documented behavior, not a crash, and it needs no restart. Do not treat full-CPU/no-output multi-minute stalls as OOM/crash/restart cases.

Last resort: HALOGEN_FLASH_PIN_TRUNK=0 gives the pinned memory back to the host but costs several times the decode speed; use only as last resort, not routine tuning.

Diagnostics/monitoring traps: free, MemAvailable, and Grafana can read wrong by about 68 GiB because locked weights are counted as reclaimable. A request that never returns then returns 503 engine_busy is a different admission issue: the pool could not reserve prompt + max_tokens.

Restart first-touch behavior: after restart, the first request can make the engine silent for ~45-60 s (has not answered PING, /health engine_unresponsive) while it first-touches the fresh KV pool; it self-recovers. HealthStartPeriod=300s covers this; do not restart the unit; retrying after the window succeeds.

KV pool verification: use podman logs frank-large | grep 'kv pool:' and match reserved count to requested count; /health does not prove pool size. If under budget, nothing may print; absence of WARNING is not evidence of headroom.

Memory freeing/trimming: cut the KV pool or arena (examples: 262144 hands back ~3.6 GiB from 401408; MAX_TOK 16384 = ~8.4 GiB). Measured host-left gain can be less than modelled because the engine keeps freed space as working slack, but contiguous block count may improve substantially; stall warning keys on that. Example verified trim: kv_pool_positions=401408 exact/no clamp, held 100.0 GiB (68.0 weights + 11.0 KV + 21.0 working), host left 9.1 GiB / 1396 contiguous blocks vs previous 7.1 GiB / 310, stall NOTE no longer printed, health ok, real completion succeeded.

Client config constraint (blankpad OMP ~/.omp/agent/): frank-large maxTokens must be <=65536 because halogen returns 400 above cap; prompt + max_tokens must fit the KV pool or the request queues ~3600 s then 503s. Retired dead flm/dankclaw-npu provider on :7771; roles: vision -> frank-large, tiny -> frank-small, task -> frank-large:medium.

Deployment/environment notes: generated Quadlet units cannot be systemctl --user enable on this host; boot persistence uses [Install] WantedBy=default.target plus linger. Before replacing the halogen container/unit, remove the old Quadlet unit from ~/.config/containers/systemd/ to avoid duplicate port 7770 claims. Known environmental e2e breakage: blankpad -> 192.168.0.112 Sonarr path issue; deploy used --skip-e2e. Tests: 285+ green except known pre-existing test-73 ComfyUI selector bug.

01a09835-91cf-72e7-8ece-e5316d4de48b

updated_at: 1789286336 Expanded frank-large KV pool from 262144 to 524288 positions (~14.4 GiB). Reduced HALOGEN_KV_SLOTS from 4 to 3 (~111 MiB freed per slot). Held memory stable at 103.4 GiB (68 weights + 14.4 KV pool + 21 working) within 121 GiB host budget. Admission is pool-bound, not slot-bound; caps at 3 simultaneous full-length requests.

frank-small defaults to sampling (checkpoint values: do_sample=true, temperature=1.0, top_k=20, top_p=0.95), creating asymmetry with frank-large which defaults to greedy. To force determinism on frank-small: send temperature: 0 per request or relaunch with --sampling-defaults none.

preserve_thinking cannot be set server-side. Enumerated 156 HALOGEN_* env vars; none control thinking/preserve. Must be sent per-request in chat_template_kwargs by the client.

API/Host quirks: frank-small returns 200 for unknown fields (200 proves nothing). max_tokens: 200 with empty content indicates thinking consumed budget. host_reserve_gib=24 protects n-gram table; file-cache shortage causes uninterruptible sleep, not OOM.

Tests: 285/286 pass (test 73 comfyui pre-existing selector bug).

01a09648-8eb8-7093-876a-116b11d4c5fb

updated_at: 1789246675 Deployment Stack: Podman compose on dankclaw. Marreta shares gluetun egress with searxng. Caddy TLS edge routes to Anubis (:8923) -> Authentik embedded outpost -> internal :8017 listener -> marreta upstream.

Auth Chain Workflow: Provider-first creation -> Route flip (vhost becomes reverse_proxy anubis:8923, internal @svc handler) -> Deploy. Verified via unauthenticated challenge probe, post-auth config grep, outpost DNS reachability.

Resolved Failure: Marreta login-wall due to ProxyProvider.application = None on marreta-proxy. Null invalidation_flow crashed Rust outpost serde. Null auth/authz flows compounded issue.

Fix Applied: Populated invalidation_flow, authentication_flow, authorization_flow + linked Application(name=Marreta, slug=marreta, provider=marreta-proxy). Restarted authentik-server + authentik-outpost.

Pitfalls & Constraints: Probe http://localhost:8017/ from host returns HTTP 000 (listener in Caddy netns); probe inside netns or grep config. Non-browser clients cannot solve JS challenge; audit service-to-service traffic for backend datasource URLs hitting public guarded hostnames. Exempt-by-construction: IdP (auth.), git smart-HTTP (Gitea), self-auth services. Authentik providers/applications are live-only (ak shell); no repo blueprint (captured in authentik-proxy-provider-app-link skill).

Verification: E2E regression test (tests/e2e/specs/16-marreta.spec.ts) asserts homepage tile passes through guard chain to Marreta app title, avoiding login-wall parking.

01a09686-789e-7042-8774-55507faa78b2

updated_at: 1789232480 SERVICE: frank-large (rootless Quadlet, boot-persistent) on dankclaw. Unit: frank-large.service | Container: frank-large. LOG_RETRIEVAL: podman logs frank-large (full); podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' (filtered); journalctl --user -u frank-large (journald). PORTS: Engine :8730 (container loopback, unauth), Front-end :8731, Published :7770. CONSTRAINTS: Weights pinned (~68 GiB). Env vars to avoid naming: TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads bare TEMP for JIT cache path). TEST_ENV: pyyaml missing for test suite (uv available, pip missing on system python3). Most suite failures are pre-existing environment issues, not commit regressions.