command-center/agent/memories/--home-blank-Projects-serverAppConfig--/raw_memories.md

# Raw Memories

## 01a11a28-0cdb-70df-9edd-7d10adfcd213
updated_at: 1791447271
## Halogen 400 max_tokens-does-not-fit Fix (frank-large, dankclaw)

### Root Cause
- HALOGEN_CTX (per-slot ceiling) was 200192. Harness budgeted max_tokens=57344 as a fixed allocation. When prompt_tokens (server-counted) + max_tokens > HALOGEN_CTX, engine returns 400 invalid_request_error.
- Harness estimates tokens at ~chars/4 (under-counts by ~3% vs server tokenizer), so harness "65% of 200k" ≈ 157k server tokens, exceeding the effective usable window of 200192 - 57344 = 142848.

### Fix Applied
- HALOGEN_CTX raised from 200192 → 262144 (native Qwen3.8-Flash-Next context window).
- HALOGEN_KV_POOL_POSITIONS raised to 524288 (2 slots × 262144).
- Harness models.yml contextWindow set to 217088 = 262144 - 45056 (maxTokens margin).
- Harness maxTokens set to 45056.
- Compaction threshold set to 85% of contextWindow → triggers at ~184524 estimated tokens.

### The Fit Law (formula)
```
engine_ceiling = 262144
harness_contextWindow = engine_ceiling - maxTokens = 217088
effective_max_tokens = min(maxTokens, contextWindow - est_tokens - reserve)
reserve = 4096 (safety margin)
```
With ~3% under-count robustness: a 10% under-count on est_tokens (2300–4450 tokens) still fits with ~43k margin.

### VRAM / Memory Budget (121.2 GiB unified, RTX 3090)
- Weights: ~62 GiB (Qwen3.8-Flash-Next)
- KV pool: 524288 positions × 28 KiB ≈ 13.3 GiB (measured 14.4 GiB)
- Arena: ~5.5 GiB
- Vision: ~2 GiB
- Total: ~83 GiB used, ~39 GiB remaining, 20 GiB reserve → 9.4 GiB measured margin
- Boot ledger NOTE fires (advisory, n-gram table served from disk page cache)
- Pin-gate floor on restart: MemAvailable ≥ ~78 GiB required

### Deploy Command
`dankclaw --service frank-large` (e2e gate 33/33, frank-large 20/20 including new fit-law seam test)

### Live Probe Verification
226,663-token prompt (26k past old ceiling) → HTTP 200, correct answer, ~137s at ~1600 t/s prefill. Decode 19.7–21.9 t/s, prompt-cache 99%+.

### Pitfalls
- HALOGEN_CTX is per-slot ceiling, not total KV budget. KV pool must be sized ≥ sum of all slot contexts.
- If KV pool is undersized, halogen clamps context silently at boot (check boot ledger for clamp message).
- The harness contextWindow must always be < HALOGEN_CTX by at least maxTokens to prevent 400s.
- frank-primary MAX_SEQS test (#180) is a pre-existing failure, unrelated.

### Files Modified
- `/home/blank/.omp/agent/models.yml` — contextWindow: 217088, maxTokens: 45056
- Dankclaw frank-large env (HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS=524288)
- Commit `76aaa25`: env + bats + AGENTS.md + runbook

## 01a10fc3-5009-7470-980a-fb58b594886f
updated_at: 1791422424
## Authentik Human-User Template (ridley shape)
Every human user in the fleet gets: username `<name>`, email `<name>@franksdank.cloud`, display name `Name`, active, path `users`, groups: Non-Admin Users + jellyfin_users + openwebui_users. This is the canonical template recorded in `serverAppConfigs-docs/Infrastructure/Authentik.md`.

## Audit-Before-Destroy Rule
Before deleting an Authentik user, check across: MFA device tables, `authentik_core_token`, `providers_proxy_proxysession`. Fleet column names differ from ORM names — `events.user` is denormalized JSON (not FK), `certificatekeypair` has no `user_id`. Events survive user deletion because of the denormalized user field, so audit trail integrity is preserved after delete.

## Jellyfin LDAP Outpost DN Stability
The DN format is `cn=<username>,ou=users,…`. Recreating a user with the same username preserves the Jellyfin user record (DN unchanged), so Jellyfin picks it up on the outpost's refresh cycle without user re-provisioning.

## Password Handling
Never touch argv, logs, or repo with passwords. Login verification uses `authenticate()` (PasswordStage code path): returns user for correct password, `None` for wrong one.

## Fleet Authentik Roster (as of 2026-10)
Superuser: akadmin. Humans: admin, alex (pk 15, recreated from pk 7 on 2026-10-08), ridley, kassie (pk 14). Plus QA and service accounts. The `admin` account is flagged as an open question (deactivate?).

## Commit Pattern for Docs-Only Changes
`bats tests/docs_completeness.bats` gate → `git add` → commit with detailed message → `git push -q origin main`. No deploy.sh needed for docs-only commits.

## 01a11414-45b0-744e-894d-0751abafe264
updated_at: 1791390151
## frank-large 400 max_tokens overflow — diagnosis and fix

### Error pattern
```
400 max_tokens 50013 does not fit: prompt is 150306 tokens and the context is 200192, leaving room for 49886.
```
Server rejects because promptTokens + maxTokens > contextWindow.

### Root cause
The harness computes `max_tokens = min(maxTokens, contextWindow − estimatedPromptTokens − 4096)` using a chars/4 token heuristic. On code-heavy prompts this estimate **under-runs** the server's actual BPE token count by ~2.8%. When the prompt is already near the compaction threshold, the 2.8% error (~127 tokens at 150k prompt size) is enough to push max_tokens over the remaining room by a small margin.

### Design constraints (from AGENTS.md / models.yml)
- contextWindow: 200192 (per-slot hard ceiling, HALOGEN_CTX=200192)
- Compaction triggers at 70% = 140,134 tokens
- Compaction budget: 57,344 tokens
- The clamp: `max_tokens = min(maxTokens, contextWindow − estimate − 4096)`
- reserveTokens in compaction logic: 16,384 (so compaction threshold = 200,192 − 16,384 = 183,808 for the internal compaction model call)
- maxTokens in models.yml: 45056 (44 × 1024)
- HALOGEN_MAX_TOKENS_CAP in frank-large.env must stay >= maxTokens value
- Worst-case estimate error at compaction threshold (183,808): ~5,147 tokens, leaving effective maxTokens <= 16,384

### Fix
- Lower maxTokens in models.yml OR add a larger safety buffer (the −4096 in the clamp is insufficient when estimate error is ~2.8%)
- Alternatively, lower the compaction threshold so the prompt never reaches sizes where estimate error matters
- The raw failed request is logged at `/home/blank/.omp/logs/http-400-requests/` for post-mortem

### frank-large config summary (as of this rollout)
- Engine: halogen-flash-server 0.16.2
- Model: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated (66.7 GB .hgn, r2 compacted)
- MTP: ALWAYS ON (HALOGEN_SPEC_ADAPT=0 — adaptive policy disabled because it halves decode)
- Vision: ON (image_url content parts, >=256px)
- Reasoning field: `reasoning_content`
- reasoning_effortMap: minimal/low→low, medium→medium, high/max→xhigh (any other value rejected, not clamped)
- Performance: 53-57 t/s greedy, ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than censored 27B
- TWO slots at HALOGEN_CTX=200192 each

## 01a10f30-003b-74c6-a60f-ae34c9eac5fa
updated_at: 1791316694
## Radiance Runtime (StillDeadcode) — frank-primary deployment

**Runtime**: radiance (StillDeadcode v1.0.7), image `stilldeadcode/radiance` digest-pinned (~287 MiB). This is NOT vLLM — completely different rules, flags, and architecture. Key distinction: native-precision checkpoints need no container or quantization step.

**Model**: `Qwen/Qwen3.8-Flash-Next-FP8` (official Qwen checkpoint, 172.8 GB, 131 shards). Served at native modelopt block-FP8 — zero requantization, zero calibration. Converted no-recipe to `qwen4exp-fp8-native.rad` (177,109 MiB = 173.0 GiB). Experts byte-stacked, PLE re-encoded at checkpoint's own bf16 scale.

**Chat template**: `huggingface.co/froggeric/Qwen-Fixed-Chat-Templates`

**Key radiance-specific concepts/flags**:
- `.rad` payload format (radiance's native model format)
- `libr4d` kernels (gfx1201 for RDNA4/R9700)
- `--tp-wire wht6` for tensor parallelism
- `--placement expert_tiered` for expert placement strategy
- `--expert-vs-cache-ratio 0.82` (ratio of pinned host pool to VRAM cache for experts)
- `--ngram-placement ram` (PLE pinned to host RAM)
- MTP (Multi-Token Prediction) spec decode, depth 3
- `--max-num-seqs`, `--max-model-len`, `--mnbt` (minimum batch tokens)

**Topology (dankserver 192.168.0.235:7770)**:
- Both R9700 GPUs, TP=2
- fp8 KV cache, 7.32 KiB/token
- max-model-len 200000, MNBT 2048, max-num-seqs 2
- 40 GiB/rank pinned host pool for experts
- PLE 47.7 GiB pinned to RAM

**Critical constraint**: A checkpoint served at its own precision needs no container/quantization. Quote from docs: "A checkpoint served at its own precision needs no container: radiance --model ~/models/Qwen/Qwen3.8-Flash-Next-FP8"

**Fleet history note**: Prior deployments used "radiance-vLLM" (tcclaviger fork) and "vllm-radiance" — these are different from the new StillDeadcode radiance standalone runtime. The new runtime is a completely different codebase with different CLI, flags, and model format.

**Cutover date**: 2026-10-06 00:55. Stable config commit: c80c250 (spec=3, seqs=2).

## 01a11006-6c8c-7288-9049-6d53c24be957
updated_at: 1791283043
## frank-large halogen-flash-server v2 (Swift1.5 Abliterated) — Cutover 2025-01

### Deployment State (post-cutover)
- Host: Dankclaw (192.168.0.143), Strix Halo iGPU gfx1151 (Radeon 8060S, ~114 GiB GTT)
- Container image: halogen-flash-server v2 (NOT the old halogen 0.1.4 from ghcr.io/peonist-ai/halogen)
- Model: Swift1.5 Abliterated, 62.11 GiB weights (incl. 0.84 GiB vision tower)
- KV pool: 400,384 positions ≈ 11.0 GiB, 2 slots over ONE pool, each request up to 200,192 ctx
- Prompt cache: ON, 12 entries, inplace
- Sampler: temp 1.0 / top_p 0.95 / top_k 20 (server-side, embedded in engine)
- Speculative decoding: draft_n=12, draft_n_accepted=9 (observed on test prompt)
- Watchdog: on, 180s
- Host RAM left after load: 15.6 GiB (tight but stable; engine pin gates all passed)
- Vision: works (512×512 red PNG probe returned "The image is red.")
- N-gram table: NOT pinned (mmap read-through confirmed in boot ledger)

### Performance (post-cutover measured)
- Prefill: 210 t/s
- Decode: 30.6 t/s
- Previous (halogen 0.1.4 qwen3.8-27b-p1w4d-d2): ~31.7 t/s mean across eight workloads (publisher claim)

### Rollback
Revert commit `31b1ec3` and redeploy. The 27B payload at `/home/dank/models/halogen-qwen3.8-27b/`, its fetch script, and halogen 0.1.4 image digest all remain on disk.

### Repo & Deploy Infrastructure
- Repo: `/home/blank/Projects/serverAppConfig` (singular "Config" in path)
- Gitea remote: `http://192.168.0.143:3000/dank/serverAppConfigs.git` (plural "Configs" in repo name)
- Cutover commits: `31b1ec3` (config), `2d89bf9` (docs)
- Deploy gate: 362 unit tests + 33 Playwright e2e tests must pass
- Docs updated: models.yml, runbook, index, AGENTS.md (317 insertions, 114 deletions across 3 files)

### Fleet Co-tenancy (Dankclaw)
- frank-large: halogen-flash-server v2, 62 GiB weights + 11 GiB KV (iGPU)
- frank-small: llama.cpp on RTX 3090
- Full AI/search/observability stack

### Operational Pitfalls
- `systemctl mask` FAILS on relinked units (unit file in /etc/systemd/system/ is a symlink into checkout). Stop is sufficient for bench windows; re-deploy restores the unit instead of unmasking.
- Engine stdout logs: `/home/dank/logs/frank_large.log` — boot ledger (KV pool, drafter state, clamps) lives there.
- Env tunables: `scripts/frank-large.env`; serve script: `scripts/serve-frank-large.sh`; systemd unit: `scripts/frank-large.service`.

### Benchmarking Policy (Dank fleet)
- STOP production service BEFORE benchmarking: `sudo systemctl stop frank-large`
- Use runtime's own harness for benchmarks; rolling your own bench requires asking first.
- After benching: unmask/restart (or re-deploy for symlinked units), verify health.
- `live-llm-benchmark-hygiene`, `live-endpoint-server-timing-ab`, `spec-decode-ab-single-gpu-host` skills govern methodology.

## 01a10fad-1e4d-71ba-95a6-376974fd706a
updated_at: 1791280376
## frank-small IQ3_S swap attempt (cancelled/rolled back)

### What was attempted
Replace frank-small (llama.cpp CUDA, RTX 3090) serving MiMo-V2.6-Distill-Qwen-9B Q6_K with `ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF` in IQ3_S quant, MTP, cap context 128k/slot, KV FP8 (q8_0), vision on, max parallel slots.

### Why it was rolled back
User issued explicit rollback order mid-swap. The Bonsai2-CRACK swap was also superseded. Both tasks were cancelled, not completed.

### Final live state (verified)
- **frank-small**: Swift RCO-IQ3_XXS + draft-mtp, 2×131072 context slots, health OK
- **Repo**: clean at commit `bb3532d`, pushed
- **Bonsai**: parked on disk (`models/Bonsai2-CRACK/` + `GitPackages/llama-prism/`), nothing committed

### Technical knowledge extracted from the sizing analysis

**IQ3_S 27B on RTX 3090 VRAM budget:**
- IQ3_S ≈ 3.44 bpw → ~11.6 GB weights for 27B
- MTP head (nextn layer) if included: small addition
- mmproj (vision tower): ~1.5 GiB runtime if separate file
- KV q8_0 for qwen35 hybrid (12 full-attn layers × 8 kv_heads × 128 dim × 2B × 2): ~24 KiB/token → ~3.0 GiB per 131072-token slot
- At 24 GB card: 11.6 (weights) + 1.5 (mmproj) + N×3.0 (KV) + buffers → fits ~2-3 slots at 128k with q8_0 KV
- Previous runbook: `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` (ISTA-DASLab GSQ-RCO IQ3_S ran 2026-09-16→09-17)

**MTP on qwen35 27B architecture:**
- MTP = property of the FILE (needs `qwen35.nextn_predict_layers` in GGUF metadata)
- MTP head tensors: `blk.<n>.nextn.*` (e.g. blk.48 for 48-layer 27B)
- Activate with `--spec-type draft-mtp` (sets `load_mtp`)
- Flash-Next MTP heads do NOT load on current loaders (need full per-layer `blk.<il>.hc_attn_norm.weight`, Flash-Next ships only blk.48). This is Flash-Next specific, not qwen35.
- qwen35 MTP worked historically: eemin W4A4 ROCm, ROCmFPX Q6_AGENT+MTP eras

**KV FP8 in llama.cpp:**
- "FP8" in llama.cpp = `q8_0` (8-bit quantized storage)
- Flags: `--cache-type-k q8_0 --cache-type-v q8_0`
- Requires `GGML_CUDA_FA_ALL_QUANTS=ON` in CUDA build (build-cuda rebuilt with this ON 2026-09-17)
- q8_0 KV halves f16 KV storage

**Vision on frank-small:**
- Requires `--mmproj` pointing at mmproj GGUF file
- Qwen3.8-27B checkpoint DOES carry a vision encoder (halogen engine ignores it, llama.cpp can use it)
- Must verify HF repo ships `mmproj-*.gguf`

**qwen35 27B architecture (for KV math):**
- 16 × (3 Gated DeltaNet + 1 Gated Attention) = 48 layers total, 12 full-attention layers
- Full-attn KV: 12 layers × 8 kv_heads × 128 head_dim

### Operational patterns
- **Mid-swap rollback**: User can cancel a swap order mid-flight. Handle by reverting to prior endpoint, verifying, and dropping stale todo items (use `drop` op, not `done`).
- **Bonsai2-CRACK**: parked on disk at `models/Bonsai2-CRACK/` + `GitPackages/llama-prism/` if revisited.
- **llama-prism**: the package name associated with Bonsai (likely a vision adapter for llama.cpp).

## 01a10f44-6c75-7548-86b1-a7a3603ca8af
updated_at: 1791270185
## Hardware / Stack Constraints
- Strix Halo iGPU with ~114 GiB GTT aperture is the compute target for ComfyUI.
- The discrete 3090 is permanently occupied by frank-small; ComfyUI MUST run on iGPU only.
- BF16 Qwen-Image checkpoint (~20 GB) is too heavy to occupy the iGPU while the box stays responsive. Use FP8 or Q6 quantized checkpoints instead.

## ComfyUI Goal Template (Qwen-Image Workflow)
Success criteria confirmed by user:
1. One Qwen-Image workflow JSON saved into repo's ComfyUI area (comfyui/workflows/).
2. One PNG actually generated through live ComfyUI API on Dankclaw's iGPU.
3. Short recommendations page (≤5 models/LoRAs with names, sizes, download sources).
4. Must use FP8 or Q6 quantization (user explicit constraint).

## Image Display in Terminal (Ghostty)
- Ghostty natively supports kitty graphics protocol; default `--image-storage-limit` is 320 MB per screen (0 disables images).
- Hand-rolled `\e_G` escape sequences emitting base64 PNG data are in-spec and work without any helper binaries.
- No image preview helpers installed by default (chafa, icat, viu, catimg all absent). `pacman -S chafa` would add `chafa file.png` with auto protocol detection.
- Two ~1-2 MB PNGs (~3 MB total b64) are well within ghostty's 320 MB limit.

## Harness Read Tool Image Behavior
- Plain `read <png>` (no selector): image-capable models receive text note + inline image block; max input 20 MiB (MAX_IMAGE_INPUT_BYTES).
- Text-only model role: receives metadata + `?q=` hint (no inline block) — this is documented behavior, not a failure.
- `read <png>?q=<question>`: routes to vision model, returns text answer.
- If current model role is text-only, PNGs will NOT display inline in chat; use `?q=` for vision analysis or kitty protocol for terminal display.

## 01a10e21-a0a6-7161-92a2-18c1cf17d7ed
updated_at: 1791260816
## Context Bloat Diagnosis
- nonMessageTokens (system prompt overhead) consistently ~37904-38171 tokens regardless of conversation length — this is the fixed floor before any user/assistant messages.
- Components measured: AGENTS.md (repo file), memory_summary.md (~413 chars, small), managed-skills descriptions (MAJOR bloat), user skills, tool definitions.
- Method to measure: `wc -c AGENTS.md`; count skills with `ls ~/.omp/agent/managed-skills | wc -l`; parse SKILL.md frontmatter descriptions with python for total desc bytes.
- Skills with descriptions are injected into context even when only names/descriptions are needed (not bodies). 175 skills × descriptions was the dominant bloat source.

## Skill Library Distillation Workflow
- Managed skills live at `~/.omp/agent/managed-skills/<name>/SKILL.md` (175 dirs, 1.7MB total).
- User skills at `~/.omp/agent/skills/` (9 items). Repo skills at `.omp/skills/`.
- Distillation: convert SKILL.md directories → vault reference pages (markdown in git repo, commit to Gitea). Removes them from context-injection path entirely.
- After distillation, move original dirs to `~/.AGENTTRASH/<date>-<label>/` (staging, reversible).
- User says `purge!` → `rm -rf` the staged dir permanently.
- Re-minting a skill: use `manage_skill` tool with content sourced from vault reference pages.

## Trash / Purge Policy
- `~/.AGENTTRASH/` is the canonical staging area for agent-initiated deletions.
- NEVER purge without explicit user approval (one-word "purge!" is sufficient).
- Multiple trash dirs can coexist; only purge the one approved.
- Verify deletion with `ls -d <path>` expecting "No such file or directory".

## Key Paths
- `~/.omp/agent/managed-skills/` — managed skill dirs (context-injected descriptions)
- `~/.omp/agent/skills/` — user skills
- `.omp/skills/` — repo-local skills
- `~/.AGENTTRASH/` — staged deletions awaiting purge approval
- Vault reference pages = git-committed markdown (e.g., commit 33eae71 on Gitea)

## Context Accounting Insight
- The fixed ~38k nonMessageTokens overhead is largely from skill descriptions + AGENTS.md + tool schemas.
- Removing skill directories from managed-skills eliminates their descriptions from future context windows entirely.
- After purge, managed-skills dir is empty (0 entries) → zero skill-description overhead.

## 01a10dd2-1b9e-7100-bdfe-2dd120a34962
updated_at: 1791256148
## Homelab Network Topology
- dankserver: 192.168.0.235 — AMD GPU (ROCm), runs frank-primary, SSH user `dank`
- dankclaw: 192.168.0.143 — RTX 3090, runs frank-small via llamacpp (used as inference provider for agent sessions)
- FrankBook-Pro (dev workstation): 192.168.0.112 — ~30GB RAM, no dedicated GPU (Mac mini M-series likely)

## Ornith-1.5-9B Architecture Facts (measured from GGUF header)
- Model: ornith-ai/Ornith-1.5-9B-GGUF, Q8_0 quant + BF16 mmproj
- qwen35 hybrid: 33 blocks, 9 full-attention layers, 4 KV heads, 256 k/v dim
- Native context: 262,144
- Dense (not MoE)
- nextn_predict_layers=1
- MTP head present in file — capability only, NOT enabled by default

## VRAM/Math Constraints on dankserver
- KV rate: 36 KiB/token at f16 (vs MiMo's measured 24.5 KiB/token) — this is architecture-determined, not quant-dependent
- Measured free VRAM before swap: 2,279 MiB
- Weights overhead for Q8_0: +1,900 MiB vs previous model
- Two 200,704 slots would need ~14.1 GiB KV + 1,900 MiB weights → impossible
- One native 262,144 slot fits with ~3.8 GiB projected free
- Harness compaction invariant: slot must be ≥ 191,148
- Ubatch lowered 2048 → 1024 (measured 616 MiB rung savings)
- No KV quantization: f16 K + f16 V (explicit user order)
- models.yml 200,704 context setting stays conservative-safe even with 262,144-capable model

## Consequences of Swap
- Subagent concurrency drops 2 → 1 (second request queues)
- Per-conversation window RISES: 200,704 → 262,144

## Deploy Workflow Gotchas
- deploy.sh has a whole-tree clean gate (includes untracked files, no path filtering) — blocks deployment if ANY untracked/modified files exist
- Never stash another writer's WIP under an active process (stash-pop races with live writers)
- Download sentinel: ~/.jobs/ornith-fetch/.done on dankclaw, sha256 verified before deploy
- Download speed on dankclaw: ~38 MB/s for GGUF files

## Coordination Lesson
- When a parallel writer (e.g., Obsidian/agent docs session) has strays in the tree, report the blocker to the owner rather than committing or stashing their work
- deploy.sh gate is by-design strict; bypass only with explicit owner approval

## 01a10af2-ce97-7257-9335-d246213db0e5
updated_at: 1791256146
## frank-primary 40k-token repetition/corruption diagnosis

**Endpoint**: frank-primary = tcclaviger RDNA4 vLLM fork, Qwen3.8-Flash-Next FP8, TP=2, expert offload, fp8 KV, no spec decode, MNBT 2048.
**Shape**: G: 3 slots × 200704 context, KV pool 3,166,500,000 bytes/rank ≈ 400,700 KV tokens, 2.00× concurrency. GPU KV cache size: 401,408 tokens. Price: 7,891.8 B/token.

**Symptom**: Repetition/corrupted outputs after ~40k tokens in a single conversation context.

**Key number**: 40,960 = 20 × 2048 (MNBT chunk boundary ×20). Also 40k/16 = 2500 blocks. Possible correlation with chunked-prefill or block-table boundaries.

**Top hypotheses (ranked by plausibility for a hard ~40k boundary)**:
1. **Hybrid model recurrent state + prefix cache eviction + recompute divergence**: Qwen3.8-Flash-Next is likely hybrid (DeltaNet/Gated Attention, same family as qwen3_5/qwen4exp). vLLM prefix caching evicts blocks under pool pressure; re-prefill recomputes recurrent state. If chunk boundaries differ between original prefill and re-prefill, the recurrent state diverges → repetition/corruption. This is a fork-specific correctness bug.
2. **fp8 KV precision at long context**: E4M3 has 3 mantissa bits; accumulated attention degradation at 40k+ may cross a threshold where repetition becomes visible. Gradual, not a hard boundary—less likely as sole cause.
3. **CUDA graph / MNBT chunk interaction at chunk 20**: 20th chunk of 2048 = 40,960. Possible off-by-one in attention mask or RoPE cos/sin table padding at a specific chunk index in the fork's attention kernel.
4. **Sliding-window attention boundary**: If some layers have a 32k or similar sliding window and the fork's implementation has an off-by-boundary bug past the window, degradation starts just after window+something.
5. **Block table overflow or RoPE table truncation**: less likely given 40k << 200,704 max.

**Proposed solutions (not yet confirmed)**:
- Test with `--enable-prefix-caching=false` to isolate hypothesis #1.
- Test with fp16 KV (disable `--kv-cache-dtype fp8`) to isolate hypothesis #2.
- Test MNBT 1024 or 4096 to see if the boundary shifts (isolates hypothesis #3).
- Check fork's hybrid allocator: does it checkpoint recurrent state at every preemption point? Compare `--ctx-checkpoints` equivalent in vLLM fork.
- Add a probe: force a preemption (saturate the pool with a second long conversation) at exactly 30k tokens and see if corruption starts earlier.

---

## frank-large rogue-session recovery (2025-06-03)

**What happened**: A rogue parallel session committed on top of frank-large's branch, clobbering uncommitted work:
- Serve script reverted to `--ulimit nofile=1048576` → container-create fails with rlimit type 7 (memlock)
- AGENTS.md rlimit lesson deleted
- Ornith-1.5-9B swap for frank-small left half-finished (deploy handler + seam twins still gated on MiMo-Q6K paths)

**Recovery steps**:
1. `git log --graph --all` to map two lineages (mine vs rogue's)
2. Rebase my 2 commits onto origin/main (which included rogue's commits)
3. Re-apply memlock-only fix (NOT nofile — nofile is the regression)
4. Complete the rogue's half-finished Ornith swap: flip deploy handler + seam from MiMo-Q6K → Ornith-Q8 (commits 8f53df7, 2930431)
5. Repair 9 broken vault wikilinks + index 7 untracked doc pages
6. Suite green, e2e 33/33, endpoint verified live

**Lesson recorded**: Always `git stash` or commit before parallel work; verify rlimit type (type 7 = memlock, type 1 = nofile — only memlock matters for pinned-memory GPU containers).

---

## frank-large live endpoint contract (verified 2025-06-03)

- Model: Ornith-1.5-9B (halogen container, ROCmFPX backend)
- `/health` → `{"status":"ok","version":{"api":"0.1.4","engine":"0.1.4","match":true},"slots":1,"slot_ctx":200700,"drafter_default":"dflash2","drafter_weights_loaded":true}`
- Boot ledger: `kv pool: 1 slot(s) × 200700 tokens = 13.15 GB`, checkpoint 35.9 GB @2.3 GB/s
- dtype census: `q4c 9.93 GB / fp8r 10.63 GB / i4l 12.35 GB`
- Drafter: 39–42 t/s greedy (verified via `timings.draft_n`/`draft_n_accepted`)
- `reasoning_effort: "high"` → **accepted** (200), not just xhigh|medium|low
- 400 contract: over-cap `max_tokens` → 400 naming 65536; `response_format` → 400 "no constrained decoding"
- Tool calls: real round-trip works (get_weather → OpenAI tool_calls format)
- e2e suite: 33 tests, FrankBook-Pro leg is trigger-gated project `9c50821` (sleeping mini can't fail unrelated deploy)
- Repo tree clean at `a6ad9af`, pushed.

## 01a10dd0-8791-71ff-993e-1979264c9275
updated_at: 1791233132
frank-primary ~40k-token output corruption — RESOLVED 2026-10-05: marginal DDR5 on dankserver.

SYMPTOMS: frank-primary (tcclaviger vLLM fork, Qwen3.8-Flash-Next FP8, 2x R9700, TP=2, expert offload, fp8 KV, no spec decode, MNBT 2048, 3 slots × 200704 context, KV pool ~400,700 tokens) produced repetition and garbage output after ~40k tokens in long agent conversations. Two failure eras: (1) 2026-10-04 gdn_hip NaN-state burst (guard-visible), (2) fault-free ~40k degradation (guard-invisible flips in KV/expert data).

ROOT CAUSE: 4 DIMMs running at 4800 MT/s (JEDEC floor on 5600-rated DIMMs). 4800 MT/s was in effect since the 2026-10-04 17:17 boot. Consumer non-ECC DDR5 raises NO MCE for silent bit flips. Dropping to 4600 MT/s (2026-10-05 12:21 boot) eliminated all symptoms: zero gdn_hip ERROR lines, zero MCE, live agent session past 140k context clean.

WHY SOFTWARE DIAGNOSIS FAILED:
- The prefix-caching OFF/ON experiment was CONFOUNDED — both arms ran on the same marginal RAM, so neither could prove/disprove the fork's QFN align-mode flaw. Prefix caching ships ON; FRANK_PRIMARY_NO_PREFIX_CACHING knob commented out for a future fork fix.
- A short in-band pattern check (60 s, 8 GiB of 246 GiB) covers ~0.5% of RAM — not a stability proof. It said "ruled out" while RAM was the cause.
- The gdn_hip guard catches NaN states in recurrent DeltaNet state but NOT silent flips in KV cache or expert weight data (guard-invisible).

METHODOLOGICAL LESSON (durable):
1. When a fault is DATA-SHAPED (NaN states, garbage tokens, repetition loops) and software reverts don't move it, suspect RAM EARLY — before deep-diving into runtime internals.
2. Consumer non-ECC memory raises no MCE for silent flips; absence of MCE is NOT evidence of RAM health.
3. A short in-band memory pattern check is NOT a stability proof for intermittent marginal-RAM corruption.
4. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a margin guarantee — especially under 4-DIMM load.
5. A software revert tested under the same marginal hardware is itself CONFUNDED and proves nothing about the software hypothesis.
6. For LLM serving specifically: KV cache and expert weight data live in system RAM (host swap regions, expert arenas) and are vulnerable to silent corruption that manifests as model output degradation indistinguishable from algorithmic bugs.

## 01a10a47-6406-7086-a869-d8f92b6dc5a8
updated_at: 1791185010
## Authentik User Creation Workflow

### Method: Django ORM via `ak shell`
- SSH to `dankclaw` (192.168.0.143)
- Run `ak shell` (gives Django ORM access with authentik's context loaded)
- Create user: `User.objects.create(username=..., email=..., is_active=True, is_superuser=False)`
- Set password: `user.set_password("..."); user.save()`
- Assign groups: `group = Group.objects.get(name="..."); group.users.add(user)`
- Password passed over stdin only (SSH pipe), never stored in repo files

### Group-Based Access Control (verified 2026-09)
Group membership is the **sole mechanism** that gates app access. No other ACLs exist per-app in this deployment.

**User-level groups:**
- `Non-Admin Users` — plain-user marker (what existing users like `alex`, `qa-e2e` carry)
- `jellyfin_users` → Jellyfin access (LDAP outpost filter: `(memberOf=cn=jellyfin_users,...)`)
- `openwebui_users` → OpenWebUI "user" role (via "Open WebUI roles" property mapping/entitlement)

**Admin-level groups (deny for user-level accounts):**
- `homepage-admins` → homepage /admin panel
- `jellyfin_admins` → Jellyfin admin
- `openwebui_admins` → OpenWebUI admin role
- `authentik Admins` → authentik admin UI
- `authentik Read-only` → authentik read-only UI

**Apps requiring NO specific group (any authenticated user works):**
SearXNG, Gitea, Calibre-Web, RomM, Copyparty, Plex, Seerr, Marreta, frank-* LLM endpoints

### Verification: Use ORM, Not Browser
- Authentik's flow UI is a React SPA (headless browser unreliable: inputs hydrate via URL state but submit button doesn't render)
- Instead verify via: `from authentik.core.models import authenticate; authenticate(user, password)`
- Also: `user.check_password("...")` returns True/False
- Default authentication flow: Identification → Password → (MFA only if authenticator enrolled) → Login
- MFA stage is pass-through when no authenticator enrolled

### Precedent Users
- `alex` has: `Non-Admin Users`, `jellyfin_users` (no openwebui_users)
- `qa-e2e` has similar pattern
- Judgment call: adding `openwebui_users` gives OpenWebUI login as `user` role; omitting means OpenWebUI won't grant a role via property mapping

## 01a100ed-f985-75f9-b64b-22c927319ec5
updated_at: 1791176847
## Frank-Primary GPU Memory Planning (dankserver, tcclaviger RDNA4 vLLM fork)

### Current Production Shape (E, 2026-10-02)
- Model: Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards)
- Image: docker.io/tcclaviger/vllm@sha256:da402966...
- MAXLEN=200704, SEQS=3, MNBT=2048
- kv_cache_memory_bytes=3.15e9 (pool)
- --expert-precise-memory ON
- R4D_LRU_GATHER_CHUNKS=32, R4D_LRU_GATHER_LANES=64 (+18.7% decode vs default)
- Prefetch ABSENT (36% precision, half gather traffic was waste)
- GPU KV cache: 398,658 tokens, max concurrency 1.99x for 200k tokens
- Expert arena: 17.22 GiB / 7,522 slots

### Critical Constraints (MEASURED)
1. **SEQS 2 CANNOT boot**: torch.fx ConstraintViolationError (query_start_loc.size()[0] marked dynamic but specialized to constant 3). MNBT cannot dodge it — buffer width is always SEQS+1. Only SEQS≥3 works.
2. **Arena lever = max_model_len**, NOT kv-cache-memory-bytes. Planner sizes expert cache as: budget - non-expert - KV_reserve(9190 B/tok × max_model_len) - graphs - overhead. Lower MAXLEN → bigger arena. Pool cut buys runtime headroom (surviving deep requests), NOT arena.
3. **KV price is shape-dependent**: 9190 B/token at 262144 ctx, 7892 B/token at 200704 ctx. Never extrapolate concurrency across geometries.
4. **MNBT 1024 is the lock** for optimal TTFT+arena tradeoff. 512 doubles TTFT at 4k ctx; 2048+ costs arena; 1024-4096 interchangeable on decode.
5. **Margin at this shape is fossilized estimator padding** — 4k probe moved high-water only ~220 KB. Activation buffers at MNBT 2048 are pre-allocated at boot (scale with batch not context). Real allocation pressure goes into fixed KV pool + arena, both pre-sized.

### Proposed Registry Patch (pending validation)
- KV price override: 9190 → 7892 B/token (+0.26 GiB to arena)
- Overhead override: 5.25 → ~4.5 GiB (+0.75 GiB to arena)
- Result: arena ~18.8 GiB / ~8,700 slots, driver ~31.7 of 31.86 GiB used
- Strategy: mount overlay first (reversible), bake after surviving week of real traffic
- Guard: fork's plan check compares planned vs measured overhead at boot

### Historical Failures (avoid)
- 2×262144 bf16 boot: 0.13 GiB free → died under real traffic
- 5×262144 boot: 0 bytes → died on 995-token request
- Both were different geometries (bigger windows, no precise-memory) but same mechanism: burst beyond observed high-water
- 2026-10-01 bf16 2×262144 shape: same ConstraintViolationError class as SEQS 2

### Rollback
- Git tag: frank-primary-fp8-mtp-20261001
- Legacy containers to clean before start: frank-primary-r9v, qwen-fp8

## 01a100c6-1871-7112-b7c8-08ecfdfa896f
updated_at: 1791176840
## frank-large model config (frank-large)
- Runtime: ROCmFPX fork of llama.cpp (pin in scripts/build-rocmfpx.sh). Required for ftype 114/115 (ROCmFPX tensor type); stock llama.cpp, SGLang, and gufo all refuse these files.
- Previous model: kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF, Q8_0_ROCMFPX_AGENT (ftype 115, 26.28 GiB, 8.39 bpw)
- New model: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF, Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf (ftype 114, 23.15 GiB)
- Draft head: Q4_0 MTP head (NOT Q8_0 — 4-bit head is faster end-to-end at equal-or-better acceptance because the cost is the projection over 248K vocab)
- AGENT variant keeps 187 output-head tensors at plain Q8_0 (vs 2 in non-AGENT), required for MTP serving
- mmproj: aligned repo has BF16 (931,145,888 B); uncensored repo has Q8_0 (0.59 GiB). BF16 currently deployed and works.
- Serve script: scripts/serve-frank-large.sh, env: scripts/frank-large.env
- Env-file override pattern: caller env vars captured before sourcing, restored after (prevents file from clobbering environment)
- Critical flags: --jinja (without it, chat_template_kwargs silently ignored), --reasoning-format deepseek, --alias frank-large, --ctx-checkpoints, --cache-ram 8192
- reasoning_effort values accepted: xhigh|medium|low only (harness levels folded onto this set)
- Prompt caching does NOT work on hybrid/recurrent memory in stock llama.cpp; budget full prefill every turn. Fork at commit 36e9acd may have MTP prompt-cache fix.

## frank-small context extension
- Target: 200700 tokens per individual stream (slot) without further quantization
- Strategy: reduce batch size to free VRAM for KV cache

## MiMo-V2.6-Distill-Qwen-9B GGUF
- Q8_0: size 9545979424 B, sha256 2fad0aa11bb9e7aa491ff12f768954f9dd0a6e7d4ce4a897ca73ec420f3b90ae
- Download target on dankclaw: /home/dank/models/MiMo-Q8_0/MiMo-V2.6-Distill-Qwen-9B-Q8_0.gguf
- Existing Q6K path used for mmproj reference (same bytes)

## Host & job patterns
- Host: dank@192.168.0.143 (dankclaw), Strix Halo iGPU gfx1151
- Model storage: /home/dank/models/<ModelName>/
- Detached jobs: ~/.jobs/ directory, pattern: setsid nohup bash -c '<cmds>' ~/.jobs/<name>.out 2>&1 < /dev/null > /dev/null & with .done/.rc markers
- HF API for file metadata: curl -s https://huggingface.co/api/models/<repo>/tree/main?recursive=true

## 01a10985-1eed-7260-ad2d-49a2af9c96c4
updated_at: 1791176837
## Expert Arena Definition — frank-primary (tcclaviger RDNA4 vLLM fork, dankserver:7770)

### Formula
arena_per_gpu = budget - non_expert - kv_reserve - graph_term - overhead_term
- budget = gpu_memory_utilization × total_vram (per rank, TP2)
- kv_reserve = 9190 B/token × max_model_len (registry constant price)
- graph_term = VLLM_CUDAGRAPH_MEM_BASE_MB + VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB × num_graphs (measured: 0 base + 12 MiB/graph = 0.07 GiB in shape G)
- overhead_term = 7.60 GiB default; --expert-precise-memory reduces to 5.25 GiB (fitted)

### Arena Levers (ranked by impact)
1. **max_model_len (MAXLEN)**: Lower MAXLEN → lower KV reserve → more arena. MAXLEN 262144→200704 freed 0.57 GiB = +248 slots.
2. **--expert-precise-memory**: Overhead 7.60→5.25 GiB, frees 2.35 GiB into arena (+1026 slots in shape E).
3. **VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB / _ESTIMATE_MB**: Control graph term (minor, 0.07 GiB in current shape).
4. **gpu_memory_utilization**: Controls budget (UTIL=0.982 is the law; 1.00 is UNBOOTABLE — worker init refuses budget >= device total; 0.13 GiB margin boots die under load).

### NOT arena levers (measured)
- **KV_BYTES / pool size**: Does NOT affect arena. Pool buys runtime headroom (prevents OOM on real requests), not arena space. 6.1e9→3.15e9 pool cut = zero arena change.
- **max_num_batched_tokens (MNBT)**: MNBT 2048 is ARENA-FREE by construction (registry overhead model is base + 0.22 GiB × ...). MNBT only affects compile range, not arena.

### Host offload
- `--expert-offload-mem` controls host-side expert cache: `host: 105.48 GiB across 2 ranks of 120.0 GiB usable (--expert-offload-mem)`
- If exceeds MemAvailable, warning is emitted; pinned_extra mechanism.

### Shape G (current, 2026-10-05)
- MAXLEN=200704, SEQS=3, MNBT=2048, UTIL=0.982, --expert-precise-memory ON
- Arena: 17.77 GiB / 7,762 slots of 2.344 MiB
- KV: 401,408 tokens / 2.00x concurrency for 200,704-token requests
- Driver used: 30.97 GiB of 31.86 (0.89 GiB free per card)
- GATHER: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, do not drop)

### Constraint: SEQS=2 cannot boot
- torch.fx ConstraintViolationError: query_start_loc.size()[0] specialized to constant (3)
- Root cause: model_runner sizes buffer to max_num_reqs+1, _dummy_run sets num_reqs=min(max_num_batched_tokens, max_num_reqs), width = SEQS+1 for any MNBT >= SEQS
- MNBT cannot dodge this; --enforce-eager bypasses but loses CUDA graphs
- SEQS=3 works with both MNBT 1024 and 2048

### Container/image
- Image: docker.io/tcclaviger/vllm@sha256:da4029666668b90a...
- Container: frank-primary-qwen on dankserver (192.168.0.235:7770)
- Config: scripts/frank-primary.env, scripts/serve-frank-primary.sh
- Code: expert_plan.py (line 818 prints the plan), expert_activation_registry

### Bug observed
- frank-primary (llamacpp on dankserver) hit repetition-loop errors at ~97K input tokens context. Three consecutive responses were repetition-stopped.

## 01a10981-24c4-75c2-b330-2af61176e8fd
updated_at: 1791171874
## Serving Architecture (dankserver / Frank stack)

### frank-primary (main)
- Model: uncensored Flash-Next MoE tower, shape G
- Provider: llamacpp on dankserver
- Hardware: R9700 GPUs, DDR5 memory (bandwidth-bound for expert loading)
- Arena: 17.77 GiB, 7,762 slots
- MNBT: 2048
- KV cost: ~7,900 bytes/token
- KV pool: ~400,000 tokens total capacity
- Throughput: ~38 t/s prose (measured: 1285 tokens in ~33.7s including 1.8s TTFT)
- Expert payload per decode step: ~1.5 GB (constrained by DDR5 bandwidth in low-tens GB/s)
- Metrics: `curl -s 192.168.0.235:7770/metrics | grep iteration` or `timings` object on completion

### frank-large
- 27B MoE on iGPU
- Quantization: Q6_0_ROCMFPX_AGENT + MTP4 (MTP = multi-token prediction, 4 draft tokens)
- Throughput: ~28 t/s prose
- Q6_0_ROCMFPX_AGENT format exists in only one loader (single GitHub fork), pinned by SHA due to prior segfault in set_rows

### frank-small
- MiMo Q6_K, 9B hybrid model on 3090
- KV: 2×200,704 f16 KV slots
- Architecture: no KV cache in 24/32 layers, only fixed-size recurrent state per slot → slot count (not context length) drives memory curve
- Config: 2×200,704 f16 KV

### Key Technical Insights
- Speculative decoding: draft head proposes 4 tokens, target verifies in single pass; SSE chunk count misreports throughput; usage counter is authoritative
- Prefix caching: highest-leverage optimization; warm cache turns ~7s TTFT into ~0.1s
- Attention is quadratic; long-context prefill cost exceeds early generation
- Tokenizer segmentation: model cannot see compound identifiers as single units; learns from neighbor relationships
- Sampling at T=0 is deterministic; T=0.7 introduces compounding fork divergence between runs

### Context Window
- nonMessageTokens: 37,667 (system prompt overhead)
- System prompt is large (~37.7k tokens) which is non-trivial relative to KV pool

## 01a10981-39c9-7454-b724-44156879c4d6
updated_at: 1791160938
## frank-primary shape G (2026-10-03) — live verified config

### Runtime & Identity
- Image: docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea (digest-pinned; :latest tag moved once)
- Model: Qwen3.8-Flash-Next-Uncensored-FP8, 172.8 GiB, 131 shards, at /home/dank/models/
- Container: frank-primary-qwen, serves as frank-primary on port 7770
- TP=2 on 2× R9700 (RDNA4), dankserver at 192.168.0.235
- Legacy containers cleaned on start: frank-primary-r9v, qwen-fp8

### Engine Shape (shape G)
- --max-model-len 200704 (not native 262144; window shrink from shape D)
- --max-num-seqs 3 (SEQS 2 UNBOOTABLE: torch.fx ConstraintViolationError, query_start_loc.size()[0] marked dynamic but specialized to constant 3)
- --max-num-batched-tokens 2048 (arena-free: registry overhead model anchors at nbt=4096, so ≤4096 all price same 5.25 GiB overhead)
- --gpu-memory-utilization 0.982 (THE LAW: 1.00 unbootable, worker-init ValueError on both ranks; 0.13-GiB-margin boots die under load)
- --kv-cache-memory-bytes 3166500000 (3.15e9 B/rank)
- --kv-cache-dtype fp8
- --expert-precise-memory (ON: reduces overhead from default 7.60 to fitted 5.25 GiB, puts 2.35 GiB into arena)
- VLLM_CUDAGRAPH_MEM_BASE_MB=0 / VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB=12 → 0.07 GiB graph term (measured, not 0.55 GiB analytic fiction)
- R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, keep)
- Prefetch OFF (36% precision measured)
- Spec decode: NONE (FRANK_PRIMARY_SPEC empty; MTP3+probabilistic-draft is open candidate, never A/B'd)

### Measured Resources at Shape G
- Expert arena: 17.77 GiB / 7,762 slots
- KV cache: 401,408 tokens, 7,892 B/token (shape-dependent, never extrapolate across geometries)
- Concurrency: 2.00x for 200,704-token requests
- VRAM: 30.97 GiB used of 31.86 per card, 0.89 GiB real free (held through live 4k-token generation)
- Budget: 31.29 GiB

### Arena Lever Mechanics (critical insight)
- ARENA = budget - non-expert - KV_reserve - graphs - overhead
- KV_reserve = registry constant 9190 B/token × max_model_len
- THE ARENA LEVER IS max_model_len, NOT kv_cache_memory_bytes (measured)
- --expert-precise-memory reduces overhead term → more arena
- KV pool size is runtime HEADROOM, not an arena lever (measured: 6.1e9→3.15e9 buys ZERO arena)
- MNBT ≤4096 is arena-free (overhead anchored at nbt=4096)

### Operational Pattern
- Check config: read scripts/frank-primary.env, git log on that file, SSH to dankserver for live podman inspect
- Host reboot detected by container uptime ≈ host uptime; container auto-restarts
- Health endpoint: http://127.0.0.1:7770/health → 200
- Cache mount /root/.cache skips 181s recompilation
- Git tag rollback: frank-primary-fp8-mtp-20261001

### Constraints / Pitfalls
- SEQS=2 fails at ANY MNBT (ConstraintViolationError is a SEQS property)
- Per-token KV price varies ~2.5% across geometries (7,701 vs 7,892 B/token)
- GPU_MAX_HW_QUEUES must be passed bare on run line (-e), all env keys prefixed FRANK_PRIMARY_ to prevent HIP leaks
- Three concurrent 67-token requests answered in parallel (~4.1s each) despite 2.00x long-req concurrency figure

## 01a1097b-bb20-77d1-b95f-64c86e9e8c0e
updated_at: 1791160787
## Skill routing pitfall: html-change-report is NOT for creative writing
- The html-change-report skill is specifically for git repo change reports (Keep a Changelog format, commit timelines, code excerpts). It should NOT be triggered for creative writing requests like "write a story about X."
- The assistant wasted a tool call reading the skill and then spent significant thinking time trying to adapt the change-report structure for a story before the user interrupted.
- Lesson: Only invoke html-change-report when the task involves summarizing/reporting actual git commits or repo changes. For creative content, just write the story directly in chat.

## User interruption pattern
- The user interrupted during the long thinking/planning phase (stopReason: aborted, errorMessage: Interrupted by user). The response took ~174 seconds total with ~111s TTFT before the tool call. When a creative writing request is made, avoid unnecessary tool calls that add latency before producing the actual content.

## 01a10978-0a99-75e0-877e-8d062824fbb7
updated_at: 1791160751
## Fleet Topology (as of 2026-10-05)
- Host 192.168.0.235:7770 → frank-primary (the serving model for this conversation)
- Host 192.168.0.143:7770 → frank-large
- Host 192.168.0.143:7771 → frank-small (fastest, Q6_K rung, 2x200704 f16-KV shape)
- Host 192.168.0.112 → up, role unclear (possibly another node)
- All health endpoints return 200 in <10ms
- Provider label: "llamacpp (dankserver)"

## Pitfalls
- `bc` is NOT installed on the host. Use python3 for float math in bash scripts instead of `bc`.
- `python3` IS available and working.

## Project: serverAppConfig
- Path: /home/blank/Projects/serverAppConfig
- Naming convention: frank-primary, frank-large, frank-small (quantization rungs: Q5_K_M → Q6_K)
- Key shape params tracked: arena size (GiB + slots), KV tokens, UTIL ratio, MNBT, CUDA-graph plan
- Shape G is current for frank-primary: arena 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x, UTIL 0.982
- frank-small: Q6_K at 2x200704 f16-KV shape
- frank-large: Q6 uncensored
- UTIL law: 0.982 (forcing 1.00 is unbootable)

## User preferences
- Casual tone, enjoys creative tangents (asked for a Soviet-burger story mid-benchmark)
- Home dir: /home/blank
- Calls the assistant "Chief"

## Token rates observed
- All three endpoints generated 256 completion tokens sequentially in ~19.5s total wall time (including prefill).
- Individual decode rates not cleanly measured due to bc missing; python3 rerun was interrupted.

## 01a108e0-8f6c-703e-9ba5-a77404744afc
updated_at: 1791160220
## ASUS ProArt B850-CREATOR WIFI NEO — BIOS Reset & RAM OC Recovery

### BIOS Reset Methods
1. **CLRTC header short**: Short pins 1–2 for 5–10 seconds (power off, unplug). No physical CMOS button on this board.
2. **Load Optimized Defaults**: Press F5 inside BIOS, then F10 to save.
3. **CR2032 battery removal**: Works with DIMMs seated.
- All methods work with DIMMs installed — no need to remove memory.
- CMOS clear does NOT affect SPD EEPROM on modules.

### RAM OC Failure Recovery (AM5 ASUS)
- First try: reseat DIMMs (common cause of instability mistaken for bad OC).
- Then: clear CMOS via CLRTC header or F5 Load Optimized Defaults.
- Board supports XMP and AEMP (AMD Enhanced Memory Profiles).
- After recovery, rebuild profile from kit SPD with margin below spec.

### DDR5 Speed Limits
- **Spec sheet (authoritative)**: 8400+ MT/s (OC) for 2-DIMM; 8200+ MT/s for 4-DIMM.
- **Marketing claims**: "up to 9000 MT/s" — not the validated ceiling.
- **4-DIMM derating**: Real-world ceiling with 4 sticks is well below the 2-stick number. CPU memory controller is the binding constraint at 4-DIMM.
- User achieved 4600 MHz clock = 9200 MT/s before failure — ~9.5% above spec, beyond realistic 4-DIMM headroom.

### Unit Conversion
- MT/s = MHz × 2 (DDR5 is double data rate)
- Task Manager shows MHz (clock); CPU-Z DRAM tab shows MT/s (data rate) for the same number.
- 8400 MT/s = 4200 MHz clock.

### Key Pitfalls
- Do NOT confuse marketing max (9000) with spec-sheet validated max (8400+).
- 4-DIMM configs derate significantly below 2-DIMM ceiling.
- At frequencies well above JEDEC (e.g., 9200 MT/s on spec=8400), sub-timings (tRFC, tRP, command rate) and thermals become binding constraints, not base clock.
- BIOS may retain a failed profile across soft reboots; full CMOS clear is needed to wipe it.

### Sources
- ASUS techspec page: https://www.asus.com/us/motherboards-components/motherboards/proart/proart-b850-creator-wifi-neo/techspec/
- ASUS Clear CMOS FAQ: https://www.asus.com/support/faq/1040820/

## 01a0feba-078d-70b7-bdf0-aacc26bf5078
updated_at: 1791150294
## tcclaviger vLLM fork logging — verified levers and traps (frank-primary, dankserver image da402966)

### Verified logging env vars (present in container image, tested via scout sweep of source + strings):
- `VLLM_LOGGING_LEVEL=DEBUG` — DEBUG across all `vllm.*` loggers; includes full request/response bodies in `api_server.py:1053,521`. Heavy, not for steady-state.
- `VLLM_LOG_STATS_INTERVAL=N` (default 10.0s) — controls throughput/KV/prefix-cache telemetry tick rate; only ticks while requests in flight. Setting to 1 gives 10x resolution, nearly free.
- `VLLM_LOGGING_CONFIG_PATH=/x.json` — stock dictConfig, per-module levels (e.g. root INFO + `vllm.model_executor.offloader` DEBUG). Useful for surgical silence/boost.
- `VLLM_DEBUG_WORKSPACE=1` — asserts + usage trace of workspace manager (the module behind `query_start_loc`/workspace constraint failures).
- `VLLM_DEBUG_DUMP_PATH` — torch.compile graph dump.
- `VLLM_LOG_MODEL_INSPECTION` — model inspection detail.
- `VLLM_PATTERN_MATCH_DEBUG` — per-graph pattern-match level.
- `VLLM_TRACE_FUNCTION=1` — full Python call trace to a dir. Last-resort, enormous volume.
- `VLLM_DEBUG_LOG_API_SERVER_RESPONSE` — logs full API response payloads.
- `--disable-log-stats` — CLI flag to silence stats entirely.
- `AMD_LOG_LEVEL=1..3` — HIP runtime trace (crash forensics). **Critical caveat**: in this image the reader library is `/usr/lib/x86_64-linux-gnu/libamdhip64.so.4`, NOT `/opt/rocm` — a grep against `/opt/rocm` is a false negative. Enormous volume; probe windows only.

### What does NOT exist in this image:
- Any `R4D_*` or `CLAV_*` *logging/debug* env switch in `r4d.so`/`parohip`/`fp8hip` (strings sweep empty; `R4D_LRU_GATHER_*` vars are tuning only, not logging).
- `HSA_ENABLE_DEBUG` in the HSA runtime (verified absent via strings on `libhsa-runtime64.so.1`).
- `perf` or `rocprof` tooling in the container.
- The offloader (`model_executor/offloader/*.py`) has only 8 `logger.debug` calls total — per-module DEBUG bump there buys little; useful offloader data stays at INFO (plan lines) + external `frank-primary-expert-activity.py`.

### The log file trap:
`serve-frank-primary.sh` line ~156 does `podman run -d ... > "$LOG"`. This writes only the container ID (one line), NOT engine stdout/stderr. Engine output goes to journald (via `LogConfig` type in podman) and `podman logs`. The `/home/dank/logs/frank_primary.log` file is misleading. Fix: either drop it, or add a companion `podman logs -f` tee.

### Recommended permanent shape:
- Pass through `VLLM_LOG_STATS_INTERVAL` (default 1) for permanent higher-resolution telemetry.
- Env-gated (default empty, using `${FRANK_PRIMARY_X:+-e X=...}` pattern already used for `GPU_MAX_HW_QUEUES`) pass-throughs for `VLLM_LOGGING_LEVEL` and `AMD_LOG_LEVEL` for incident windows.
- Restart cost per apply: ~8–9 min (compile cache mount keeps compilation at ~4s; ~7 min is payload load).

### Scout methodology trap:
Grepping `/opt/rocm` for `AMD_LOG_LEVEL` in this image returns nothing (false negative). The actual library path is `/usr/lib/x86_64-linux-gnu/libamdhip64.so.4`. Always verify library paths with `ldconfig -p` or `find` before concluding a string is absent.

## 01a0fd77-162f-7451-9285-566736703023
updated_at: 1790972101
## frank-primary live config (as of probe)
- FRANK_PRIMARY_MAXLEN=200704 (NOT 262144 as AGENTS.md claims)
- FRANK_PRIMARY_SEQS=3
- FRANK_PRIMARY_KV_BYTES=2200000000 (~2.2 GB, NOT 6.1e9)
- FRANK_PRIMARY_MNBT=2048
- ulimit -l: unlimited
- AGENTS.md row (bf16 KV / 2 seqs × 262144 / expert-precise-memory) is STALE — live argv is ground truth.

## Host topology
- frank-primary: llamacpp on 'dankserver', health at 192.168.0.235:7770, container name 'frank-primary-glm' (podman)
- frank-small: llamacpp on 'dankclaw-3090'
- Dev workstation: /home/blank/Projects/serverAppConfig, /home/blank/.omp/agent/models.yml

## Shell escaping pitfall (bash tool)
- The bash tool's command parameter mangles `$(...)` command substitution and `\$` escapes.
- `\$(cmd)` → syntax error at varying col depending on context.
- Workaround: write a script to disk (e.g. /home/blank/.check-primary.sh) then execute it.
- Do NOT rely on inline command substitution in the bash tool parameter.

## Crash pattern post-probe
- After the config check script ran (upload+exec via python), frank-primary entered a tight retry loop:
  'The socket connection was closed unexpectedly' × 10 retries, each ~15.5s timeout.
- Likely cause: OOM-kill or segfault when KV allocation was attempted or re-probed.
- The `-T` flag error in `date` output is a red herring from the script, not the crash cause.

## Models repo location
- /home/blank/.omp/agent/models.yml contains frank model definitions.

## 01a0fe18-c297-766a-b631-5889c2444dc2
updated_at: 1790971702
## frank-primary log access (dankserver 192.168.0.235)

### Architecture
- frank-primary.service is a ROOT systemd unit on dankserver.
- ExecStart runs scripts/serve-frank-primary.sh which does `podman run -d --name frank-primary-qwen ...` (detached), then `exec podman wait frank-primary-qwen`.
- Because the container is detached (-d), stdout does NOT flow into the systemd unit's stdout. The unit journal only has script-level output (container ID printout, gate results, podman wait status).

### Log locations
1. **Engine logs (primary):** `podman logs frank-primary-qwen` — only available while the container is alive (not after removal).
2. **File log:** `/home/dank/logs/frank_primary.log` — the serve script redirects podman run output to this file (`> "$LOG" 2>&1`). Contains engine startup and runtime output.
3. **systemd journal:** `journalctl -u frank-primary` — only script-level lines (container ID, gates, exec podman wait). Does NOT contain engine stdout/stderr.
4. **Large ERROR lines:** journald splits long lines; read via `journalctl -o json`, NOT `-o cat` (per runbook note at line 381 of dankserver-qwen38-fp8-expert-offload-tuning.md).

### Pitfalls
- Do NOT expect `journalctl -u frank-primary` to show vLLM engine output — it's a detached container pattern.
- `podman logs` dies when the container is removed/recreated (deploy-remote.sh removes and recreates the container on deploy).
- The rootless-quadlet-log-retrieval skill does NOT apply — this is a root unit with direct podman run, not a rootless quadlet.
- ExecStop uses `podman stop -t 120 frank-primary-qwen` (120s drain timeout).
- Legacy containers that may hold GPU/name registrations: frank-primary-r9v, qwen-fp8, glm-probe.

## 01a0fc98-aecb-778a-bad8-f3d0b1e54464
updated_at: 1790958824
## frank-primary Autoresearch Harness Architecture

### Service Identity
- Host: Dankserver (192.168.0.235), deploy user `dank`
- Runtime: vLLM fork (tcclaviger), model orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8
- Hardware: 2× R9700, TP=2, expert offload, port 7770
- Container name: frank-primary-qwen, systemd unit: frank-primary
- Boot cost: ~7 min warm; health returns 503 during load

### Benchmarking Discipline (AGENTS.md enforced)
1. STOP production unit (mask fails on relinked units—stop alone sufficient for bench window)
2. Candidate boots AS THE SERVICE: same serve script, same container name, same port 7770
3. NO scratch ports on LLM hosts (single GPU, can't co-tenant)
4. Measure via in-container client (measure.py) hitting localhost:7770
5. EXIT trap = unconditional restore: stop candidate, start prod unit, verify health
6. Failed restore forces exit 1; bench failure still restores

### Harness Layout
- Orchestrator: /home/blank/Projects/serverAppConfig/scripts/autoresearch/frank-primary-window.sh (invoked via SSH from dev box)
- Staging dir on Dankserver: /home/dank/.autoresearch-frank-primary/
  - serve.sh (candidate serve script)
  - frank-primary.env (candidate knobs)
  - measure.py (in-container measure client)
  - hash (fingerprint)
  - run.json (result: metrics + boot + kv + fingerprint)
  - serve.out / measure.out / fail.log
  - last/ (memo of last fully successful window: run.json + hash)

### Key Env Variables
- FRANK_PRIMARY_AR_STAGE (staging dir path)
- FRANK_PRIMARY_AR_UNIT (systemd unit name)
- FRANK_PRIMARY_AR_CTR (container name)
- FRANK_PRIMARY_AR_BOOT_TIMEOUT (default 1200s)
- FRANK_PRIMARY_AR_RESTORE_TIMEOUT (default 1200s)
- FRANK_PRIMARY_AR_MEASURE_TIMEOUT (default 1800s)
- FRANK_PRIMARY_SERVED_NAME (model served name)

### Speed Optimization Constraints
- Allowed: alter context size, MNBT, expert arena, prefetch, seqs/slots
- NOT allowed: quantization changes (includes KV cache dtype fp8 vs bf16)
- Current AGENTS.md has contradiction: topology table says bf16 KV / no MTP / 2×262144; detail section says MTP4 / 200k×3 / fp8 KV. Trust actual scripts/frank-primary.env + serve-frank-primary.sh.

### SSH Access Pattern
- Dev box → dankserver via SSH alias (deploy scripts use this)
- Window script runs AS deploy user on Dankserver
- Health endpoints: http://127.0.0.1:7770/health and /v1/models

## 01a0fb3e-f1d3-7330-bd66-2b997c1a0264
updated_at: 1790938297
## frank-small reconfiguration (2026-10-02)

**Model**: bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via llama.cpp on RTX 3090 (24,576 MiB VRAM)

**Topology**: 3 parallel slots, n_ctx_slot = CTX_SIZE/PARALLEL = 540672/3 = 180,224 tokens per conversation slot. KV pool total = 540,672 tokens.

**VRAM budget formula** (empirically calibrated):
- Non-KV overhead text-only: ~7,390 MiB
- Non-KV overhead WITH vision tower: ~8,859 MiB (projector encoder buffers add ~1.4 GiB for a 918 MB mmproj file)
- Safety headroom: 2,048 MiB (Frank's 2-3 GiB band requirement)
- KV budget = 24,576 - 8,859 - 2,048 = 13,669 MiB available for KV
- Solved: pool 540,672 tokens fits. First attempt at 3×196,608 (589,824 total) booted healthy but left only 1,153 MiB free → recalibrated down.

**Final VRAM**: 21,684 used / 2,441 free. Within band.

**Performance**: Decode ~106.8 t/s, prefill ~3,814 t/s (temp 0). ~5% faster decode than 2×262,144 shape.

**Critical pitfalls**:
1. `n_ctx_slot = CTX_SIZE / PARALLEL` — this is llama.cpp's formula. 3 slots × native 262,144 = 786,432 tokens KV alone (18.4 GiB) — won't fit. Must solve for pool size.
2. Vision adds ~1,469 MiB overhead (measured 8,859 vs 7,390 MiB). Bats budget gate must price non-KV at measured 9 GiB, not text-only.
3. Legacy top-level `images` field in /v1/chat/completions is SILENTLY IGNORED by llama.cpp. Only `image_url` content parts within `messages[].content[]` array reach the vision tower. Same red image → "red" via image_url, "White" (hallucinated) via legacy images field.
4. frank-primary's compaction request is ~183,501 tokens — EXCEEDS frank-small's 180,224 token slots. Returns hard 400 exceed_context_size_error. Rollback if needed: PARALLEL=2 CTX_SIZE=524288 + contextWindow: 262144.
5. models.yml `contextWindow` field requires harness restart to take effect — running omp sessions keep old value until restart.

**Deployment artifacts**:
- commits: a028502 (3 slots + vision), 7fe881d (pool 540672 + recalibrated gate), 934285c (runbook/docs/AGENTS.md)
- scripts: scripts/serve-frank-small.sh, scripts/frank-small.service, scripts/frank-small.env
- tests/run.sh: 364 passed, frank-small seam 13 passed, Playwright e2e 33 passed
- Host: dank@192.168.0.143, port 7771, alias frank-small

## frank-large reconfiguration

**Model**: kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (highest quality quant) on gfx1151 iGPU (Dankclaw)
**Topology**: 1 full context session, whatever runtime it requires
**Port**: 7770

## Pending approvals
- Purge /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ (~59 GiB) — user approval needed

## 01a0f190-490b-765e-b62d-21e28c3c0ec0
updated_at: 1790760416
Hardware Context: Dankserver features 2x AMD R9700 (gfx1201) and 246 GiB DDR5-3600 RAM running at 42 GB/s peak bandwidth. 

DeepSeek 4.1 Flash Research:
- Model: 552B parameters.
- Quantization Analysis:
    - `vcruz305 Q2_K`: Backbone ~191 GiB. Fits entirely in page cache. Recommended for efficiency.
    - `pfeifferj GSQ-RCO 3.0-bit`: Backbone ~192.6 GiB. High quality (61.0% MMLU-Pro). 
    - `Lucebox ROCMFP23`: 189 GiB fp8 backbone (previously identified as the 'thrash' case).

Performance & Constraints:
- Bandwidth Limit: 42 GB/s bandwidth results in a realistic decode speed of 6–11 tok/s.
- Porting Status: No native HIP build currently exists for DS4.1; requires conversion from CUDA (PR #28696 is conversion-only). 
- Action Item: Build from fork using `-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201`.
- Vault Update: Update `Dankserver System Specs.md` with 246 GiB RAM, 42 GB/s bandwidth, and `ram-bandwidth-bound-moe-ceiling` skill.

## 01a0eb77-a0b2-7748-8c3c-c2eee29d34d4
updated_at: 1790724034
Current state of frank-large:
- Model: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (31.46 GiB dense)
- Context: 262,144 tokens (full window)
- Drafter: DFlash2 at depth 4
- KV Cache: q8_0
- Template: Qwen-Sharp v22.4.0
- Current Runtime: llama.cpp (unsloth b11030 prebuilt)

Target Change:
- New Runtime: gufo (https://github.com/gufo-org/gufo)
- Goal: Maintain the high-precision (q8_0) and large context (262k) characteristics of the current setup while adopting the gufo execution engine.

## 01a0ea08-9e09-7608-8003-81ab447e507e
updated_at: 1790641878
Dankware is a retired workstation/client. 

**Purge Status:**
- `~/.omp` (command-center): 5 commits ahead of `main`. 'Dankware' replaced by nameless 'workstation' rows.
- `serverAppConfigs`: Cleaned; currently on branch `docs/purge-retired-workstation` (unpushed).

**Functional Artifacts (Preserved):**
- `packages/dankware.pkgs`: Filename is the lookup key (load-bearing).
- `scripts/install.sh`: Contains `--host` case list (load-bearing).

**Contextual Notes:**
- Blankpad (192.168.0.83) is a distinct machine and was left untouched.
- Audit confirmed zero subagent residue in the repo for this operation.

## 01a0e983-6c65-77e6-813a-c655e46ce06f
updated_at: 1790628098
Context: frank-primary model on R9V architecture (RDNA4/gfx1201) with 2 GPUs (TP=2). 

Key Configs:
- R9V_PLE_READY_TIMEOUT=1800 (f187165)
- R9V_ENABLE_PREFIX_CACHING=1 (41e9189)
- VLLM_PLE_PINNED_RESERVE_BYTES=54400261120
- R9V_FULL_MUTABLE_CACHE=0
- Q8_0 GGUF payload (6 shards)
- MTP4 speculative-config (draft TP=2, fp8)
- Context: 131072, max_num_seqs: 1, prefill_chunk: 4096.

Git Status:
- main: ba2100e (prefix caching deployed)
- autoresearch/...-20260927: PLE timeout, bench matrix runbook, doc-link fixes.
- autoresearch/...-20260928: prefix caching (E8-qualified, 9x/63x reuse).

Deployment Workflow:
Repo env files (Frank's git overrides) layer on host R9V state (/home/dank/r9v-state/q8xl/setup.json).

## 01a0e091-9152-75fc-b81b-8e311474eea8
updated_at: 1790624016
Model: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF (Q_8). Drafter: Q_8 (DFlash2). Speculative Decoding: --spec-type draft-mtp,ngram-mod (layered). Vision: Enabled. Architecture Note: Flash-Next MTP heads require full per-layer hc_attn_norm.weight (do not ship only blk.48). Memory Context: Q_8 quant is ~180 GiB; fits in Dankclaw's 114 GiB GTT + host RAM via mmap. Label: frank-primary.

## 01a0d06e-c8b5-757b-a093-2081af1db7c6
updated_at: 1790503506
Context: 1970s Moscow cafeteria (stolovaya). Protagonist: Pyotr Arkadyevich Volkov (chef). Problem: The 'kotleta' (cutlet) lacked architecture, falling apart when eaten with a fork. Solution: Enclosing it in a pampushka bun to make it a 'hand-consumed' burger. Bureaucracy: The invention moved from a chef's observation to a formal memo, through a 9-link bureaucratic chain, eventually becoming a GOST standard (Meat cutlet in a soft bread bun). Key Philosophy: 'The standard gives the product a body. The kitchen gives it a life.' The 'half-millimeter' tolerance is the space where the individual's touch meets the state's law.

## 01a0e144-473f-71a5-96f8-0316232910aa
updated_at: 1790503503
### Placement & Tensor Laws
- **Override Logic:** `--override-tensor` is first-match-wins. Specific patterns must precede broad sweeps (e.g., `exps=CPU` should be listed after specific blocks but before general ones).
- **PCIe Strategy:** `-ot exps=CPU` is the primary method to stop crossing the PCIe bus; it works only if `--no-mmap` is NOT used (as no-mmap forces tensors into anonymous memory, breaking the CPU-resident trick).
- **Atomic Units:** The unit of placement is the per-block expert tensor (512 experts = 1 tensor). There is no per-expert routing knob.
- **Expert Distribution:** Top 10% of experts carry 40.5% of samples. Hottest layers do not exist at -ot granularity because Top-8 routing fires on nearly every block.

### Spec Decoding & Drafter Laws
- **Composition:** Legal combinations include [one context-backed drafter (MTP/DFlash/DSpark/Eagle3)] + [any number of ngrams]. Two context-backed drafters cannot share a process.
- **KV Scaling:** Drafter `n_ctx` automatically scales to the target's `n_ctx`. 
- **Drafter VRAM Traps:** DSpark (no sliding window) can consume ~5.0 GiB at 262k context, while DFlash2 (sliding window) consumes only ~40 MiB.
- **Device Pinning:** Pin the drafter to the card holding the main model's output head (e.g., ROCm1 under `-ts 1,1`) to avoid `SIGABRT` on pre-allocated buffer mismatches.

### Performance & Implementation
- **Ubatch Optimum:** For frank-medium, the peak is 512 (1193 tok/s). Below 512 is under-saturated; above 512 starts to see diminishing returns and higher VRAM costs.
- **GGUF Extraction:** Derive tensor extents from **consecutive header offsets** rather than quant type-size tables to avoid mis-bucketing tail elements into 'gap' buckets.
- **Fleet State:** Primary (192.168.0.235:7770), Large (192.168.0.143:7770), Small (192.168.0.143:7771).

## 01a0dfa8-7193-7235-980d-69e138218fcc
updated_at: 1790474002
Model: GLM-5.3-Flash-Uncensored-GGUF (orcarouter). Params: 321.3B MoE. Key Constraints: 1. Requires `NVIDIA_TF32_OVERRIDE=0`. 2. Must use `-fa off` for correct MLA output. 3. Use `-ub 128` to mitigate microbatch-dependent long-context collapse. 4. Use `-ot exps=CPU` for offloading experts as Q6_K (~264GB) is slightly larger than available RAM (246GB). Features: Supports draft MTP (`--spec-type draft-mtp`). Context: Previous `frank-medium` unit was in a crash loop on ParoQuant MXFP6, highlighting the need for stable deployment of this new large MoE model.

## 01a0df97-1e50-77d7-9084-92708bfd1152
updated_at: 1790458743
Connectivity: Repeated 'Unable to connect' errors (ID 135168) occurred on dankserver using frank-medium. Framework: Introduced /guided-goal, a structured workflow requiring a 5-point interview (Success criteria, Verification, Attempt cap, Boundaries, Stop conditions) before executing a persistent autonomous objective loop. Current Project: Deploy https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF on dankserver using the https://github.com/Luce-Org/lucebox runtime.

## 01a0d66d-f9e2-7681-a5e3-e77b896dcc3f
updated_at: 1790407605
Homepage CSS Update: Flipped the stagger pattern in shared/homepage/style.css. Role and Inputs now use --dim (#2f8f52, bold), while Spec and Use use --fg (#33ff66). Removed the --input variable. Deployment: Committed cd07a66, pushed, and deployed to /srv/homepage. OpenWebUI/Authentik: OpenWebUI is running as a quadlet in dankclaw. Authentik integration (OAuth2/OIDC) is confirmed as the primary method for passing login state, matching the current containerized architecture.

## 01a0c81c-56dc-7359-ac24-6da36332ad84
updated_at: 1790068527
## ComfyUI 0.37.0 Native Qwen-Image 2.1
- Released 2026-09-21, tag v0.37.0, CORE-423, PR #16400
- Path: comfy/ldm/qwen_image21/model.py (model detection + supported-models registration)
- Dankclaw image (comfyui-rocm:gfx1151, Containerfile.comfyui) has CUSTOM GGUF patches (apply_qwen21_patch.py: VAE 4D→5D expansion, in_channels shape[0] read, QwenImage21 arch registration)
- ACTION: On next image rebuild, check if native 0.37.0 path makes those patches redundant BEFORE re-applying
- Repo moved to Comfy-Org org (comfyanonymous/ComfyUI redirects)

## OpenWebUI API Correction
- Real imagegen endpoint: POST /api/v1/images/generations (NOT /api/images/generate)
- Version in vault: 0.11.4

## Calibre / Calibre-Web CVE Conflation
- CVE-2025-7404 belongs to Calibre-Web, NOT Calibre itself
- Calibre version corrected: 9.9.0 → 9.15.0

## Vault Structural Facts
- Repo: /home/blank/Projects/serverAppConfig
- Docs: serverAppConfigs-docs/ (Obsidian vault, OKF v0.1 format)
- OKF frontmatter: type/description/tags/timestamp (YAML)
- 136 pages total, 127 non-reserved docs with FM
- Deployment: Quadlet/deploy-plan.py (NOT the old deployment.md description)
- GTT: 114 GiB (was incorrectly documented as 96)
- Hosts: Dankserver (192.168.0.235, media), Dankclaw (192.168.0.143, AI/dev)
- Tests: ./tests/run.sh → 336 seams
- Link checker: check-vault-links.py
- *arr docs moved to FrankBook-Pro/ section

## Research Briefs Created
- Local TTS and Voice Cloning (OKF doc)
- AI Job Search and Resume Tools (12 verified projects)
- Most Popular Self-Hosted AI Apps Jun-Sep 2026 (top 15 by GitHub stars: OpenClaw 390k, Hermes 248k, Ollama, Dify, Open WebUI, ComfyUI)
- HTML report: popular-self-hosted-ai-apps.html (dark theme, star bar chart, per-app cards, homelab-fit table, 11 adoption best practices)

## 01a0c51e-53af-724a-9f4e-364858819889
updated_at: 1790063680
## frank-medium chat template wiring
- Entry: `dankserver/frank-medium/radiance-entrypoint.sh` line 253: `--chat-template "$CHAT_TEMPLATE"`
- The `CHAT_TEMPLATE` env var is already plumbed; to switch templates, set it to the desired jinja file path.
- Qwen-Sharp template location: `scripts/.AGENTTRASH/chat-template.jinja.ik-llama-qwen-sharp` (archived from ik_llama.cpp era for frank-large).
- The template is also referenced in `tests/deployment_scoping.bats` as a deployment-scoping guard.
- To activate: either copy the template to a stable path and set CHAT_TEMPLATE, or reference the .AGENTTRASH path directly.

## frank-large swap (completed)
- Replaced Kairic Edge (4-bit) with an 8-bit model + DFlash2 spec decode drafter.
- Commits: 7689092 (swap), 600812d (docs: AGENTS.md, runbook, LLM index, top-level index, homepage).
- 8-bit is ~23% larger weight size than 4-bit Kairic Edge, explaining ~22 t/s vs higher throughput.
- DFlash2 draft acceptance is text-variety-dependent: 0.85 on trivial one-word answers, 0.519 on varied explanatory prompts. Do not read a single request as steady-state.
- Kairic Edge payload + runtime remain on disk as rollback target.
- Pending: purge 87 GiB assembled file + 31.7 GB corrupted model on dankclaw (awaiting approval).

## 01a0c5c1-0f4c-72bc-81b8-8b05f92d0ef1
updated_at: 1790063676
## Homepage Architecture
- Location: `shared/homepage/`
- Files: `services.json` (main service list, ~35 entries), `dankserver.services.json` (host-specific), `app.js` (frontend renderer), `style.css`
- `app.js` determines `CATEGORY` from URL: `'admin'` if path contains `/admin`, else `'user'`
- Services filtered by `svc.category !== CATEGORY`
- WIP state: `svc.wip` boolean flag → state 'wip'
- Status from `status.json` → `stateByName[s.name]` gives 'up'/'down'
- Panel rendering: badge (state), name, description, links (open/docs)
- Summary line: UP/DOWN/WIP counts
- Bare mode toggle via `?bare=1` query param
- Deploy timestamp via `window.DEPLOYED_AT` baked at deploy time

## Open WebUI Image Generation API
- Container: `open-webui` (podman) on host `dankclaw`
- Route file: `/app/backend/open_webui/routers/images.py`
- Endpoint: `/api/images/generate`
- Handler function: `image_generations` (line 592), called at line 575
- Config keys: `image_generation.enabled`, `image_generation.engine`, `image_generation.model`, `image_generation.openai.params`, etc.
- Access via: `ssh dankclaw 'podman exec open-webui <cmd>'`

## 01a0c6e4-d2b6-7078-a0c9-f492a78605a2
updated_at: 1790063108
## frank-medium MXFP6 deployment (2026-09)

### Model swap
- Model: just1moremodel/Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, Swift finetune)
- Template: Qwen-Sharp v22.5.0 (178 prompt tokens for a simple math question; 73,600 is the fallback sentinel)
- Drafter: DFlash2-FP8 (Qwen3.8-27B-based, not Swift-specific) — works fine, acceptance 85.7% easy / 58.3% varied
- max_model_len: 262144
- KV pool: 425,047 tokens (1.62x max_model_len), 14.42 GiB/card available

### GPU_UTIL ceiling on 32GB R9700
- 0.99 is INFEASIBLE: vLLM startup check runs BEFORE weight load. 0.99 x 31.86 GiB = 31.54 GiB required, but only 31.23 GiB free at device-init. The 0.63 GiB gap is consumed by ROCm/HIP context + NCCL + radiance custom all-reduce buffers.
- 0.98 is the ceiling: 0.98 x 31.86 = 31.22 < 31.23. Prior production value.
- Symptom of 0.99: ValueError at engine init → crash loop. The 'registered' lines in journal are NOT the loop — they're ~20 sequential patch_*.py invocations each printing NOOP sitecustomize lines.

### Weight sizes (affects KV pool)
- MXFP4: 19.37 GiB
- MXFP6: 25.16 GiB (5.79 GiB more → less KV headroom)

### File paths
- Quadlet: dankserver/frank-medium/quadlet/frank-medium.container
- Entrypoint: dankserver/frank-medium/radiance-entrypoint.sh
- Repo: serverAppConfigs at http://192.168.0.143:3000/dank/serverAppConfigs.git
- Install path: ~/.config/containers/systemd/frank-medium.container

### Deployment workflow
1. Edit quadlet + entrypoint + runbook
2. git commit + push origin main
3. Deploy (pulls latest, restarts container)
4. e2e gate (32/32)
5. Verify: health 200, model name, spec decode config, completion (prompt token count confirms template), tools, drafter acceptance

### Commits
- e9a01cd: MXFP6 swap
- d5c8e40: GPU_UTIL 0.98 revert
- 24f7a77: measured boot shape in runbook

### Open item
- Drafter is Qwen3.8-27B-based, not Swift-matched. Acceptance healthy (58-86%) so it's a net win, but a Swift-specific drafter would be the next lever if needed.

## 01a0c5e9-dee3-734a-946c-3cf809f9d7fd
updated_at: 1790027980
## OMP Session Persistence

### File Layout
- Session files: `~/.omp/agent/sessions/<project-slug>/<timestamp>_<id>.jsonl`
- Project slug derived from cwd path.

### Data Size Controls and Blob Externalization
- Strings >500,000 chars truncated with `"[Session persistence truncated large content]"`, EXCEPT: signed/encrypted provider blocks, signature fields, complete Anthropic native web-search history blocks (must remain byte-exact for replay).
- Transient `jsonlEvents` field removed before persist.
- `lineCount` recomputed after truncation when both `content` (string) and `lineCount` (numeric) present.
- Image data URLs in `image_url` fields: ALWAYS content-addressed in blob store, replaced with `blob:sha256:<hash>`, regardless of length.
- Other base64 image payloads externalized at 1,024 chars threshold (image content/data payloads, image-generation results).
- Redundant OpenAI Responses `thinkingSignature` copies omitted when authoritative reasoning item already in `providerPayload`.
- On load, blob references resolved back to inline payload shapes.

### Storage Abstractions
- `SessionStorage` interface: sync dir/existence/write/stat/list; async read, sliced read, write, guarded atomic write, rename, unlink, artifact-aware deletion, title update, writer creation, backend drain.
- `FileSessionStorage`: real local files.
- `MemorySessionStorage`: map/chunk-backed for non-persistent sessions and tests.
- `IndexedSessionStorage`: shared local index + ordered remote publication (Redis/SQL-backed).
- `SessionStorageWriter`: append, appendSync?, flush, flushSync?, isOpen, close, getError.

### Error Behavior
- Persistence errors latched and rethrown by later flush/close/write; first logged once with session-file context.
- Failed atomic publication attempts authoritative repair.
- If storage may have published and repair cannot be proven durable: `SessionPersistenceIndeterminateError` fails closed with original + recovery errors.
- Writer close propagates first meaningful error.

### Session Discovery
- `getRecentSessions(sessionDir, limit?)` → lightweight welcome metadata, default limit 4.
- `findMostRecentSession(sessionDir)` → newest by mtime.
- `listSessions(sessionDir, storage)` → project scope with lifecycle status.
- `listAllSessions(storage)` → all project scopes.
- `resolveResumableSession(...)` → local lookup then optional global fallback.
- Recent/most-recent scans: 4 KiB prefix only.
- Full lists: 4 KiB prefix + bounded 32 KiB tail for lifecycle status.
- Scans are stat-keyed and cached; large sets use bounded parallel workers.
- Per-directory scans recover newest orphaned EPERM backup when primary JSONL missing.
- Resume matching: case-insensitive, accepts session id prefixes, full filename prefixes, or id suffix after timestamp.

### HistoryStorage (separate subsystem)
- SQLite at `~/.omp/agent/history.db`
- Table: `history(id, prompt, created_at, cwd, session_id)`
- FTS5 index: `history_fts` with trigger-maintained sync.
- Deduplicates consecutive identical prompts via in-memory last-prompt cache.
- Inserts batched through async drain queue (~100ms delay) to avoid blocking turn execution.
- Use session files for conversation graph/state replay; use HistoryStorage for prompt history UX.

## 01a0aeb5-e0ba-742d-86da-df30e865107d
updated_at: 1790026831
## Project: Private llama.cpp Fork (serverAppConfig repo)

**Directory**: `llamacpp/` at repo root (planned, not yet created).

**Phases**:
0. Recon: git status, disk space, identify forks on GitHub, read runbooks, verify SSH to dankclaw/dankserver.
1. Clone mainline llama.cpp into `llamacpp/`.
2. Trim AGENTS.md — remove rules that block private-fork freedom (e.g. 'don't add dependencies', 'keep PRs small', 'no breaking API changes').
3. Merge charlie12345/llama.cpp — source of ROCmFP4 quantization for RDNA4. Test on frank-large (dankclaw, gfx1151 iGPU, port 7770). Must stop halogen service first per benchmark policy.
4. Merge ik_llama.cpp — improvements for unique setup. Runbook: `frank-large-ik-uncensored.md`.
5. Create managed skill: merge-llamacpp-forks (merge latest from all parent forks + TDD, noting quirks).
6. Optimize: terse, machine-optimal code. Hardware-aware: gfx1201 2x R9700 layer split, gfx1151 iGPU, CUDA 3090 x4 link.
7. Knowledge base: new section in `serverAppConfigs-docs/` (OKF format).
8. HTML instruction manual: how to quantize models for R9700 pair.

**Hardware topology**:
- dankserver (192.168.0.112): 2x R9700 gfx1201, layer-split inference.
- dankclaw (192.168.0.143): RTX 3090 CUDA + gfx1151 iGPU. frank-large service on port 7770.
- dankware/blankpad (dev box): Ryzen 5 PRO 5650U, RTX 4070 Max-Q. Has passwordless SSH to fleet.

**Fork sources**:
- Mainline: ggml-org/llama.cpp
- charlie12345/llama.cpp: ROCmFP4 quantization (RDNA4-focused). Also referenced as 'ROCmFPX fork' in runbooks.
- ik_llama.cpp: used on frank-large 2026-09-15 to 09-16.

**Pitfalls encountered**:
- frank-medium on dankserver was unreachable (10 retries exhausted, ~15.6s each). Service down.
- frank-large context overflow: prompt 144085 tokens + max_tokens 57344 > context 200192. Need to reduce max_tokens when prompt is large, or increase context.
- TDD skill mandates seam confirmation with user before writing tests — one consolidated question needed.
- Deploy planner: new top-level directory `llamacpp/` should be a no-op (unknown paths don't map to services).

**Status**: Phase 0 (recon) was in progress when session was interrupted. No code written yet.

## 01a0c58d-9751-7094-bb98-3d7f54669e56
updated_at: 1790024021
## Homepage Architecture
- Location: `shared/homepage/`
- Files: services.json, dankserver.services.json, app.js, style.css, index.html, admin.html, docs.html
- `app.js` determines category from URL path: `/admin` → 'admin', else → 'user'
- Services in JSON have fields: name, category, description, wip (bool), url, docs, showLink
- State resolution: if `svc.wip` is true → 'wip'; else look up name in status.json → 'up'/'down'
- Rendering: flat grid, one panel per service matching the current category
- No auto-refresh (Q41), bare mode via `?bare=1` (Q27), deploy timestamp baked in (Q43)

## Deployment Pipeline
- **deploy.sh** (dev box): pre-flight (clean tree + on main), runs deploy-plan.py, git push, SSH to target for ff-only pull + revision check, execs deploy-remote.sh, then Playwright e2e gate
- **deploy-plan.py**: maps changed file paths → service names. `shared/homepage/*` + `scripts/homepage-status.py` → homepage. Docs-only → no-op. `--service NAME` bypasses planning.
- **deploy-remote.sh** (target): takes explicit service list (never infers). State store at `~/.local/state/serverAppConfigs/deploy-state/<service>` with `revision=` and `status=`. Services marked `pending` before handler, `deployed` after success. Crash recovery: pending services retried next run. `--pending` (default) catches up services whose revision lags. Retired-service lockout protection (trash, not rm).
- Key invariants: master = deployed state (Q33), ff-only pull + exact revision match, service-scoped (unrelated services never restarted), deploy not complete until e2e gate passes
- E2e gate can take ~20 minutes (admin link-walk alone budgets 240s)

## User Request Context (not yet implemented)
- Wanted sections on homepage: AI, Web Apps, Media Library Apps
- Wanted sections on admin: Media Library Apps, Observability, AI, Infrastructure
- Wanted separate labels: 'Work in Progress: Open' vs 'Work in Progress: Closed'
- Noted outdated AI descriptions and missing tautulli link on admin page

## 01a0c510-1716-724c-b99b-b1ee4df08178
updated_at: 1790020691
## Calibre Content Server Deployment (Dankserver)

**Image**: `lscr.io/linuxserver/calibre:9.15.0` — this is a DESKTOP build (Selkies GUI), NOT headless. The dedicated headless images are dormant (calibre 6.10/2017, below CVE fix threshold). Must run `calibre-server` as the s6 main service via `Exec=` in the Quadlet unit.

**Port traps**:
- Internal port 8080 is the image's own nginx → `Address already in use` if you try to bind calibre-server there. Use internal 8090 → host 8084.
- Default auth mode is **Digest**, not Basic. Every Basic-auth login returns `400 Unsupported authentication method`. Fix: `--auth-mode=basic`.

**Bootstrap password trap**:
- `echo "password" | ...` stores `password\n` (trailing newline) → every login 401. Use `printf '%s'` on stdin.

**pasta port-forward mode**: On this host, pasta runs in port-forward mode. Only explicitly published ports are reachable from outside the container netns. The desktop GUI (if started) would be unreachable without an additional `PublishPort` line.

**Final shape**: Boot-persistent Quadlet (`Restart=always`), image pinned, userdb bootstrap idempotent in deploy handler, 3 new seam tests added.

## Calibre-Web 0.6.27 SSO Analysis

- NO generic OIDC support. Login types: local, LDAP, GitHub, Google only.
- No `authlib` installed, no OIDC config keys.
- Upstream feature request open; maintainer declining more login options.
- Grafana's RP-mode OIDC path is NOT available.
- Correct path: guard chain (Anubis + Authentik + Caddy) with zero app changes. Calibre-Web's own login remains as a second layer (no disable mode), same posture as RomM.

## Guard Chain Pattern (Fleet-Standard)

1. Outpost proxy provider: `external_host = https://<domain>/`, `internal_host = http://caddy:<internal_port>`
2. Caddy vhost: `reverse_proxy anubis:8923` + post-auth `@<name>` handler on `:<internal_port>`
3. Anubis guard set + Caddy edge + provider list deployed together

## Project Context

- Repo: serverAppConfigs — single source of truth for Dankserver (192.168.0.235, Ubuntu 24.04) and Dankclaw (192.168.0.143, Ubuntu 26.04)
- Deployment: Podman/Quadlet, service-scoped git deploy via `scripts/deploy.sh`
- Test suite: 328 tests via `./tests/run.sh`, e2e gate 32/32 Playwright
- Docs vault: `serverAppConfigs-docs/` with link integrity checked by test suite

## 01a0c09e-b740-7547-8216-e31373fa0316
updated_at: 1789968641
## Why vllm-radiance TP=2 failed on Dankserver
- Hardware: 2x AMD Radeon AI PRO R9700 (gfx1201), chipset Gen4 x1 with AtomicOps Routing- (no PCIe atomics)
- RCCL collectives (allreduce, all_gather, broadcast, barrier) require PCIe atomics on upstream bridges — unavailable on this chipset
- PP (pipeline parallelism) also needs p2p send/recv using the same atomic primitive
- vLLM's layer-split mode isn't expressible in the needed form
- The radiance fork's RADIANCE_USE_R4D_AR (one-shot P2P push allreduce over gloo IPC handles) covers prefill+decode allreduce but does NOT intercept PyNcclCommunicator.__init__ warmup allreduce or pynccl_comm.all_gather (direct RCCL calls)
- NCCL symmetric memory: unavailable (shipped RCCL advertises NCCL 2.30.4 but exposes no ncclDevComm* API)
- Workaround that worked: two single-GPU replicas (frank-medium-a/-b) behind a round-robin proxy on port 7770
- Corroborated by Codeberg StillDeadcode/vllm-radiance issue #22 (different host, same image, kernel 7.0.0-30, same enqueue.cc failure)
- Retired 2026-09-16; replaced by mainline llama.cpp with DFlash2 drafter

## models.yml update (MTP → DFlash2 drafter swap)
- File: ~/.omp/agent/models.yml (NOT in git repo)
- Actual API settings unchanged: contextWindow=262144, maxTokens=32768, reasoning=true, input=[text,image]
- Stale comments fixed:
  - Model: Qwen3.8-27B-Uncensored-FP8 (30.9 GB / 7 shards), accept 4.24/7, throughput 100.0 vs 85.9 t/s (A/B), 100.0 vs 45.7 t/s
  - KV pool: 384,238 tokens / 1.47 concurrent (was 943,581 / 3.59 with MTP); DFlash2 drafter costs 2.1 GiB vs MTP's 461,540 tokens
- Pitfall: a schema parse error in models.yml drops the ENTIRE file silently (learned from the `audio` incident) — always verify 0 schema errors after edits
- Running sessions keep their process-start snapshot; changes take effect on next new session

## 01a0b5e1-46b7-7387-a64d-f6f69f79be10
updated_at: 1789847739
## Fleet Model Swap Workflow (llm-fleet-model-swap skill)
- Read HF model card first: runtime pin, hard constraints, recommended args
- Check host for existing model files before downloading: `ls /home/dank/models/`, `df -h /home`, `du -sh`
- Artifact integrity: LFS oid in HF API = sha256; compare size AND sha256sum of on-host file
- Deploy via git flow: commit → push → deploy.sh dankclaw --service frank-small → health check → e2e
- Pre-deployment live verification + port-holder health-check trap check

## vLLM MTP Speculative Decoding for Gemma-4-12B
- Main model: google/gemma-4-12B-it-qat-w4a16-ct (HuggingFace)
- MTP assistant: kunhunjon/gemma-4-12B-it-qat-assistant-w4a16-ct
- Target GPU: 3090 (24 GiB, sm_89 Ada)
- Fit parameters to maximize: max_model_len (full context), max-num-seqs (session count), gpu-memory-utilization
- Precedent: vLLM was previously on frank-medium (vllm-radiance replica, retired 2026-09-16)

## 3090 RGB Control (EVGA iCX3 EEPROM)
- Zone mode register read: `09 m0 m1 m2 m3 [field] led0 led1 led2 led3 sync`
- m0-m3 = 0x00 means all zones MODE_OFF
- Write format: `0xC0 0x09 m0 m1 m2 m3 FF FF FF FF 0x00`
- EEPROM save persists across reboots; no Linux-side daemon required
- LED counts in pkt[6-9] are informational (e.g., 04 05 3c 00 = per-zone LED counts)
- Tools: evga_probe.c (read), evga_raw.c (write) in /tmp on dankclaw

## Post-Cutover Fleet Topology (2026-09-19)
- Host: Dankclaw (192.168.0.143)
- frank-small (:7771): Gemma4 Q4_K_M, llama.cpp, root systemd unit
- frank-large (:7770): CIRU Orca, rootless Quadlet (halogen)
- frank-medium: orcarouter Q8_0 + DFlash2
- All units enabled, self-restore on boot
- 331 bats + 32 e2e tests green

## Operational Notes
- llamacpp (dankserver) provider can have transient 'Unable to connect' errors; auto-retry recovers (seen 2 consecutive failures then recovery)
- frank-small was previously on FreeToken (2026-09-02 → 09-15), then llama.cpp master 37b53fd with HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M, CTX_SIZE=1048576, PARALLEL=4, batch 8192/1024, q8_0 KV

## 01a0b693-7972-7104-bb4b-2bbec7f42acc
updated_at: 1789773472
## frank-medium CACHE_RAM sizing law

**Constraint**: CACHE_RAM must exceed one full slot's prompt state size, which is `n_ctx_slot × 68 KiB` (64 KiB from 16 full-attention layers at 4 KV heads × 256 head_dim × f16 + 4 KiB MTP draft KV). At n_ctx_slot=196608: ~12.8 GiB. Must also stay under half host RAM (62.2 GiB → cap < 31 GiB). 16384 MiB satisfies both.

**Why it's load-bearing for hybrid DeltaNet models**: With `--ctx-checkpoints 0`, the recurrent cell (DeltaNet state) sits at the END of the prompt. If the RAM cache refuses the state snapshot, llama.cpp has no checkpoint to roll back to — it takes the 'forcing full prompt re-processing' path. The RAM cache is the ONLY thing that can carry a state across slots. A refused state is simply gone.

**Historical evidence**: 137 lines of `prompt state size 11xxx MiB exceeds cache size limit 8192.000 MiB, skipping` in 2 days (2026-09-16→18). Max observed: 11,388.99 MiB.

**Verification**: argv (`--cache-ram 16384`), code path, arithmetic, and historical refusal logs. The >8 GiB threshold was NOT exercised live (needs 123k+ token conversation ≈ 2-3 min prefill; AGENTS.md says not to run beside live service). Tell for future: refusal lines should only name states >16384 MiB; an eviction line naming an entry >8192 MiB would be direct proof.

## Test seam pattern (tests/frank_medium_unit.bats)

New assertion: CACHE_RAM must clear one full slot's state (~12.8 GiB) AND stay under half host RAM. Fails on 8192, passes on 16384. This is the canonical way to guard against regressions when n_ctx_slot or model architecture changes.

## Deploy housekeeping

Peer's uncommitted frank-small/vLLM WIP blocked deploy's dirty-tree gate. Pattern: `git stash` → deploy → `git stash pop`. Note: with peer WIP present, `./tests/run.sh` is red (7 frank-small/frank-large seams fail) — full suite isn't a clean signal until that work lands.

## frank-medium key facts (for future reference)

- llama.cpp master fb27a525 (2026-09-16), gfx1201, ROCm 7.14.0a20260624
- Layer-split across 2x R9700 (NOT tensor parallel: RCCL needs PCIe atomics, second card behind B650 chipset lacks them)
- Model: Qwen3.8-27B UD-Q8_K_XL (29.29 GiB), arch qwen35, 64 layers (48 DeltaNet + 16 full-attn)
- Drafter: DFlash2 Q8_0 (1.92 GiB), 5 layers, block_size 8, 2048-token sliding window
- n_slots=2, n_ctx_slot=196608, ubatch_size=1536
- VRAM: GPU0 ~33.1 GB used, GPU1 ~31.0 GB used; host RAM 13.4/62.2 GiB (cache is on-demand, unpinned)
- MTP engaged: draft_n 33 / accepted 25, 37.85 t/s
- Commit: 1acca43

## 01a0b5f3-22ec-7692-a98c-8290e5cc3476
updated_at: 1789762341
## Incident: frank-medium GPU stranding by orphaned llama-server (2026-09-18)

### Symptom
- `curl http://192.168.0.235:7770/health` → connection refused (rc=7)
- `systemctl is-active frank-medium` → inactive
- `systemctl is-enabled frank-medium` → enabled
- Unmasked (symlink to /home/dank/serverAppConfigs/scripts/frank-medium.service)
- NRestarts=0 → clean stop, not a crash loop

### Root Cause
- An orphaned `llama-server` (PID 2519965, PPID 1) from `~/serverAppConfigs/third_party/llamacpp/build-merge` was holding both R9700 GPUs
- Bound to 127.0.0.1:17770 with `-c 16384 --split-mode layer --tensor-split 1,1 -ngl 99 -t 16`
- Model: Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
- Timeline from sudo journal + /tmp/layersplit.log:
  - 08:14:02 frank-medium started
  - 09:19:44 frank-medium stopped (bench protocol)
  - 09:19:47 merge-build llama-server spawned
  - 09:20:00 log shows "model loaded" / "listening" — nothing after
  - ~12:15 user reports down (3h05m of silence)
- Prod couldn't self-recover: 35:30 split needs ~30.8 GiB ROCm0 + ~28.9 GiB ROCm1; stray owned ~15.5/17.1 leaving insufficient headroom

### Diagnostic Pattern (reusable)
When a prod LLM unit is stopped but GPUs are busy:
1. `rocm-smi --showmeminfo vram` → check if used memory is non-trivial (>100 MiB baseline)
2. `ps -eo pid,ppid,etime,args | grep llama` → find the holder
3. Prove abandonment before killing:
   - `/proc/<pid>/stat` utime+stime identical across 10s sample (frozen CPU = idle)
   - `ss -tnp | grep <port>` → zero established connections
   - Check bind scope (loopback-only = not serving external traffic)
   - Last log mtime (hours stale = abandoned)
4. Check PPID: PPID=1 means orphaned (parent shell/harness exited)

### Fix Procedure
1. Preserve stray's log: `cp /tmp/layersplit.log /tmp/layersplit.log.preserved-YYYYMMDD`
2. `kill <pid>` (exits <1s, cards drop to ~60 MiB baseline)
3. `systemctl start frank-medium`
4. Wait for health 503→200 (~20s when GGUF warm in page cache)
5. Verify geometry: `cat /proc/$(systemctl show -p MainPID --value frank-medium)/cmdline | tr '\0' ' '` → confirm --tensor-split 35,30, ctx 393216, --parallel 2, --flash-attn on, --spec-type draft-mtp
6. Run real completion to confirm spec-decode: expect draft_n/accepted ratio ~90%, tg ~40 t/s
7. Fingerprint check: b11009-fb27a525d = pinned runtime

### Fleet Topology (dankserver)
- Host: 192.168.0.235, user dank, SSH from blankpad
- GPUs: 2x AMD R9700 (32 GiB each) + 1x small (512 MiB)
- frank-medium: port 7770, tensor-split 35:30, ctx 393216, parallel 2, flash-attn on, spec-type draft-mtp
- Unit file: symlink /etc/systemd/system/frank-medium.service → /home/dank/serverAppConfigs/scripts/frank-medium.service
- Login shell on dankserver is ZSH — avoid words starting with `=` in SSH commands

### Key Insight
Loopback scratch servers are invisible to endpoint monitors (they don't bind the prod port), so they strand the fleet silently instead of failing loudly. Two bench-protocol strandings occurred on the same day (2026-09-18): morning was a simple stop-without-restore; this one was worse because the orphan held GPUs.

### Suggested Guards (not yet implemented)
- `restore-frank-medium.sh` trap in bench harness (EXIT handler to self-restore)
- Fleet check that alerts when GPUs are busy while no prod unit is active

### Commit
- cfabe07 in serverAppConfigs-docs/log.md

## 01a0b30f-1e89-763e-b65d-48d87e6ee683
updated_at: 1789756946
## Incident: frank-medium left down after bench protocol (2026-09-18)

- **Symptom:** connection refused on 192.168.0.235:7770. Unit was `inactive` (enabled, unmasked, NRestarts=0) — not a crash.
- **Root cause:** AGENTS.md bench stop-and-mask protocol executed from PWD=/home/dank via agent SSH. Timeline: stop+mask 02:47:00, unmask+start 04:21:27, final stop 04:25:59. Step 3 of the protocol (unmask/start + health verify) never ran for the last round.
- **Recovery sequence (safe order):**
  1. Confirm GPUs idle: both R9700s ~60 MiB, no llama-bench/llama-server procs.
  2. `systemctl start frank-medium` → active, health 200 first probe.
  3. Verify /proc/MainPID/cmdline matches deployed geometry: `--tensor-split 35,30`, `--ctx-size 393216`, `--parallel 2`, batch 6144x512, `--spec-type draft-mtp`.
  4. Behavioral proof: chat completion returns expected verbatim text.
- **Pinned runtime fingerprint:** b11009-fb27a525d
- **Durable lesson:** The restore half of the bench protocol is not optional. A session that benches owes the fleet a health 200 before it ends. Recommended fix: make the bench harness self-restoring (trap EXIT → start + health-verify) so a dead session can't strand the endpoint.

## Guard chain topology (reference)

Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home/@data handlers) → app/static.

Key gotchas:
- BusyBox wget in caddy container: NO -H flag, use `--header "Host: ..."`.
- 8016/8017 live in caddy container's netns; curl from host returns 000 (expected).
- Anubis container is distroless; podman exec sh fails.
- x-real-ip is masked to 10.89.1.22 (podman SNAT) for ALL clients; clearance is cookie-based.
- Homepage app.js does Promise.all([fetch services.json, fetch status.json]); on ANY fetch failure the grid stays empty and summary shows 'status unavailable: <err>'.

## 01a0b23a-7a27-7792-b8da-97d7421b99c3
updated_at: 1789756944
## Plex /mnt/entropy group permission fix on dankserver

### Problem
Plex container on dankserver cannot read files in /mnt/entropy. Host-side permissions on /mnt/entropy must NOT be changed (no chmod/chgrp/chown/setfacl on dir or contents).

### Diagnostic path (read-only)
1. `ssh dankserver 'podman inspect plex'` — check container groups, mounts, uid/gid mapping
2. `ssh dankserver 'stat -c "%A %U:%G %n" /mnt/entropy; getfacl -p /mnt/entropy; ls -ln /mnt/entropy | head'` — identify owning group of files
3. `ssh dankserver 'findmnt /mnt/entropy'` — determine mount type (NFS, mergerfs, local disk)
4. `ssh dankserver 'id dank'` — check if host user is already in the required group
5. Confirm container sees the tree: `podman exec plex ls /mnt/entropy | wc -l`
6. Probe a specific file: `podman exec plex head -c 16 <file>`

### Fix approach (container-side only)
- **Rootless quadlet**: Use `GroupAdd=keep-groups` in the quadlet unit. This passes the host user's supplementary groups into the container. Prerequisite: host user `dank` must be a member of the group that owns /mnt/entropy files. Adding dank to that group is a host change (allowed), not a /mnt/entropy change.
- **Rootful container**: Use `GroupAdd=<gid>` with the numeric GID of the owning group directly.
- **Alternative**: `--group-add keep-groups` in compose if still compose-based.

### Key fleet patterns
- Dankserver media stack (Plex, Jellyfin, RomM, Copyparty, Calibre-Web) uses compose.yaml in repo `dankserver/` dir, but boot persistence requires quadlet units (compose-to-quadlet cutover in progress).
- Rootless podman user-unit spawns carry NO supplementary groups by default — must explicitly use `GroupAdd=keep-groups` or enumerate GIDs.
- Deploy workflow: edit repo at `/home/blank/Projects/serverAppConfig` → `./tests/run.sh` → commit → push → `scripts/deploy.sh dankserver --service plex`. Deploy gate rejects dirty git tree.
- SSH alias: `ssh dankserver` (User dank, 192.168.0.235) from blankpad.
- Deletion policy: never `rm`, use `.AGENTTRASH`.

### Plex local API probing (on dankserver)
- Identity: `curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:32400/identity`
- Token location: `/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml`, extract with `grep -oP 'X-Plex-Token="\K[^"]+'`
- Token in curl: use `curl -K` with a mode-600 temp file (NOT `-H` on cmdline) to avoid leaking token in `/proc/<pid>/cmdline`.
- Library sections: `curl -s -K <tokenfile> http://127.0.0.1:32400/library/sections`
- PMS may return HTTP 000 on /identity while port 32400 is listening (pre-bind before HTTP layer ready); especially after long downtime. Wait or retry.

### Session status
Interrupted before final fix applied. Next step: confirm the GID owning /mnt/entropy files, ensure `dank` is in that group (if not, `sudo usermod -aG <group> dank`), set `GroupAdd=keep-groups` in the plex quadlet unit, deploy, verify with `podman exec plex head -c 1 <file>`.

## 01a0b09a-aeae-75f5-9522-c5a3f1d97ac7
updated_at: 1789710503
## Fleet Topology
- dankclaw (192.168.0.143): frank-large :7770, frank-small :7771 (llamacpp root units)
- dankserver (192.168.0.235): frank-medium :7770 (llamacpp root unit), media stack (podman user quadlets)
- frankbook-pro (192.168.0.112): *arr stack as quadlets inside podman machine
- All services configured from /home/dank/serverAppConfigs on each host
- Repo: /home/blank/Projects/serverAppConfig
- Quadlets run as user units (systemctl --user)

## frank-medium 35:30 Tensor Split (verified post-deploy)
- Flags: --split-mode layer --tensor-split 35,30 --spec-type draft-mtp
- GPU0 (x16): 30.84 GiB used / 1.02 GiB free
- GPU1 (x1): 28.86 GiB used / 3.00 GiB free
- Bottleneck card headroom tripled (1.03 → 3.00 GiB); GPU0 is now the heavier card
- Utilization: x16 card 40%→60%; x1 card still 98% (structural: MTP drafter + output layers pinned to last-layers card)
- 36:29 fails floor (0.45 free) AND slack gate (2.22 gap) — layer split lever is exhausted; next lever is ctx cut
- MTP acceptance rate: 0.51–0.91 on live traffic
- Fit predicted 30.59/30.01; measured 30.84/28.86 (actual shift ~1.9 GiB, larger than fit's 0.58 GiB gap)
- Empty-body probe responses = queue backlog from live agent traffic, not split defect
- Runbook: serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md (commit 2fc0689)

## frank-large max_tokens 400 Dead Zone
- Model: halogen, ctx 200192
- Dead zone condition: prompt + maxTokens > ctx AND prompt < 70% compaction trigger (140,134)
- Example: prompt 137,879 + 64,000 = 201,879 > 200,192
- Fix: maxTokens 57344 in ~/.omp/agent/models.yml
- Closes zone: worst-case 140,134 + 57,344 = 197,478 ≤ 200,192
- CRITICAL: OMP snapshots model config at process start; existing sessions are permanently pinned to the old value until restarted (use `omp -r` to resume)
- Wire-verified via tcpdump on dankclaw: fresh one-shot omp sends max_completion_tokens: 57344

## Testing
- 326 seams passing (up from 321; a peer added a seerr test)

## 01a0ac72-2620-750f-82dd-79c84e00e98d
updated_at: 1789631572
DANKCLAW HARDWARE: AMD Strix Halo APU (gfx1151), Radeon 8060S iGPU. Has an M.2 dock slot used to carry an RTX 3090 at PCIe x4 (link downgraded from x16). This is distinct from dankserver (ASUS TUF GAMING B650-E WIFI) which has a free CPU-attached M.2_2 slot for an R9700 riser (~$30 M.2→PCIe x4 riser).

DANKCLAW LLM DEPLOYMENTS:
- frank-small: Qwen3.8-27B GSQ-RCO IQ3_S + MTP on RTX 3090 (M.2 dock, PCIe x4). Decode 61-82 t/s, prefill 430-1013 t/s. VRAM at 92% (22,614/24,576 MiB). PCIe utilization only ~3% during decode (compute-bound, not transport-bound). --parallel 1, 8 CPU threads.
- frank-large (halogen): runs on the iGPU (gfx1151), decode 37-48 t/s, MTP commit 1.60-1.88 tokens/round. Prefix caching confirmed working.

UNANSWERED QUESTION: User asked specifically about the M.2 slot in context of a Qwen Flash deployment running with pipeline parallelism on dankclaw. The assistant's response was aborted (OMP process exited) before answering. This likely refers to a llama.cpp pipeline-parallel setup where the Qwen Flash model's layers are split across the iGPU and the 3090 in the M.2 dock, using the M.2 slot as the interconnect between the two compute devices.

## 01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3
updated_at: 1789599423
## Rootless Podman restart policy pitfall
- `restart: unless-stopped` only survives *crashes*, not *explicit stops* (e.g., kernel power-off sends a polite stop signal).
- Result: bare containers without a Quadlet unit are gone after reboot with no alert.

## Repo `serverAppConfig` deployment pattern
- Every container must have a Quadlet `.container` file (git-tracked) to be boot-persistent and self-healing.
- The unit file is the container's birth certificate + pension plan + ghost in one.

## Historical incidents (from story)
- Aug 26: Caddy never returned after reboot (no unit).
- Sep 11: Five observability services found unitless after reboot.

## Persona note
- Model: `frank-medium` via llamacpp (dankserver). Persona: Soviet-style computer systems engineer with Munroe-esque humor. Uses "Comrade" address.

## 01a0a7ad-42e2-739d-bc1c-dc405ca02cbd
updated_at: 1789574671
## Dankserver Dual R9700 RCCL Chipset Issue

**Hardware**: 2x AMD R9700 on ASUS board. Card #1 on CPU-attached PCIe x16. Card #2 on chipset-attached PCIe x16 slot (PCIEX16_2, which is x1 mode per ASUS spec).

**Problem**: RCCL (ROCm Collective Communications Library) fails on the chipset-attached card. Bare 2-rank torch.distributed probe results:
- broadcast: OK (CPU card) / FAIL (chipset card)
- all_reduce: OK / FAIL
- barrier: OK / FAIL
- send/recv (P2P): hang / FAIL

**Implication**: send/recv failure kills pipeline parallelism too (vLLM's PP handoff IS that primitive). No software-only escape.

**Root cause**: Chipset-attached PCIe lanes don't route atomic operations the way RCCL's kernels require. CPU-attached lanes do.

**Fix (recommended)**: M.2→PCIe x4 riser (~$30) to move card #2 to M.2_2 slot. ASUS spec confirms M.2_1 and M.2_2 are CPU-attached; M.2_2 is free/empty. Restores atomic-op routing → RCCL works → TP=2 functional.

**Alternative**: Board with CPU x8/x8 bifurcation (~$250-400).

**Fallback**: llama.cpp layer split (no collectives needed, but needs GGUF and loses vLLM batching).

**Bandwidth analysis**: 27B TP=2 allreduce at batch 4 ≈ 10 MB per token-step ≈ 1.4 ms at Gen4 x4. Bandwidth is NOT the bottleneck; atomics routing is.

## Software Stack
- vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14
- Shipped RCCL exposes no ncclDevComm* symbols → torch's symmetric-memory path can't run
- Open hypothesis (unverified): recompiling RCCL with -mprintf-kind=buffered might work, but no live printf/assert/__assertfail found in device source

## Operational Notes
- dankserver went down during probing; needs physical power cycle
- Boot persistence should auto-restore: frank-medium trio, media stack, node-exporter
- To distinguish 'off' from 'wedged': ip neigh from peer (FAILED = off/NIC dead; wedged kernel still answers ARP)
- AGENTS.md benchmarking policy updated to cover any GPU experiment, not just benchmarks

## Model Target
- kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
- Fork: charlie12345/rocmfpx (llama.cpp ROCm fork)

## Skill Note
- 'grilling' skill does NOT exist in available skills list (returned 'Unknown skill: grilling')

## 01a0a87f-6b17-7492-995c-8c3e7b37e241
updated_at: 1789552363
## Hardware Topology (dankclaw)
- RTX 3090 (24 GiB VRAM) in PCIe 4.0 x4 M.2 dock (downgraded from x16, measured ~29 GB/s sequential)
- AMD Strix Halo gfx1151 (64 GiB SVM-backed unified memory, RADV Vulkan driver; ROCm wheels do NOT support gfx1151)
- Combined GPU budget: ~88 GiB
- OS: Ubuntu 24.04
- Kernel cmdline: amd_iommu=off (for prefill gains; retired frank-npu)

## ik_llama.cpp Build & Deployment
- Built with CUDA (for 3090) + Vulkan (for Strix Halo) backends
- Serves mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF at i1-Q4_K_M
- Deployed as root systemd unit (Restart=always) on port 7770, NOT a quadlet
- Context window: 262144 tokens
- Device split: --split-mode layer with -dev CUDA0,Vulkan0
- Shipped config: T16 (16 threads), no -ncmoe, no -thp
- Commits: 795b0b3, ad7a9c4
- Runbook: serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

## Performance Envelope (verified single-instance)
| Config | Decode (t/s) | Prefill (t/s) | Notes |
|---|---|---|---|
| Shipped T16 | 3.8-4.0 | ~192 (1.8k-32k ctx) | Production config |
| T8/T4/T2/T1 | 4.18/4.20/4.26/3.97 | small-prompt only | Thread count barely matters |
| -ncmoe 42 | 4.54 | — | 22.0/24.6 GiB VRAM, fragile at 262k ctx |
| -ncmoe 48 -thp | 4.60 | — | 110 GiB anon host buffer, co-tenant killer |
| --prefetch-experts | 4.03 | — | No effect (cache already resident) |
| Mainline llama.cpp 37b53fd | 1.01 | — | Ruled out entirely |

## Root-Cause: Why 30 t/s Decode Is Unreachable
1. Decode issues ~100 graph splits per token
2. Each split = GPU launch + sync across PCIe x4 (3090 in dock)
3. GPU SM utilization during decode: ~5%
4. ~90% of cycles = pause (thread barrier spin); T1 ≈ T16
5. CUDA graphs would collapse the overhead BUT capture ABORTS: qwen4exp hc ops use per-layer ggml_repeat_4d which is capture-illegal
6. Speculative decoding unavailable: GGUF has no MTP head (mradermacher stripped NextN from the checkpoint)
7. All experts on GPU: 84 GiB > 24 GiB 3090 VRAM — impossible
8. Path forward: mainline llama.cpp PR #28901 (hc-ops + capture-safe graphs) — re-evaluate when it lands

## Prefill Ceiling
- ~192 t/s is the ceiling: compute-bound on CPU AVX512-VNNI
- Expert reads amortize across the batch; math saturates cores
- Halogen hit 30 t/s decode by holding entire 115.55 GiB w4b checkpoint resident in LPDDR inside one proprietary engine; 24 GB GPU + CPU split cannot replicate that residency with any public runtime

## Operational Traps
- Unit has Restart=always: pkill'd servers respawn mid-experiment → phantom instances invalidate benchmarks. Always verify single-instance before measuring.
- Concurrent -j16 builds can starve probe processes
- PCIe x4 dock: do NOT assume x16 bandwidth; measured ~29 GB/s (not 130 GB/s)
- -ncmoe 42: +15-20% decode but only 2.5 GiB VRAM margin at 262k ctx — OOM risk
- -ncmoe 48 -thp: 110 GiB anon host buffer — will kill co-tenants

## Trash (staged for user approval)
- dankclaw ~/.AGENTTRASH/20260916-ik-partfiles/ — 111 GiB (3 download parts, superseded)
- dankclaw ~/.AGENTTRASH/20260915-ik-swap/ — retired halogen quadlet
- repo tests/.AGENTTRASH/frank_large_quadlet.bats, scripts/.AGENTTRASH/frank-small*

## 01a0a902-7f58-77f4-9392-764f6cfbd59e
updated_at: 1789544414
## AGENTS.md Rewrite Pattern (serverAppConfigs)
- Collapse chronological swap history into: (1) active-topology table (current state only), (2) runbook pointer table mapping each retired runtime → its docs/LLM/ runbook path.
- Add a 'Key Lessons Learned' section containing ONLY durable technical constraints/pitfalls (not narrative).
- In config sections (e.g. halogen), keep only invariants that survive any config change (pool/reserve matched pair, client contract, memory-model rule); push sizing history to runbook.
- Fix structural bugs (e.g. broken merged table rows) as part of the cleanup.

## Durable LLM/ROCm/Quadlet Lessons (from AGENTS.md distillation)
- Layer-split-not-tensor: for multi-GPU ROCm, split by layer not tensor-parallel.
- mmap/PLE constraints apply to model loading on ROCm.
- HIP env collision: `TEMP` and `HOST` env vars can conflict with HIP runtime.
- `GGML_CUDA_FA_ALL_QUANTS` env var needed for flash-attention with quantized KV.
- MTP (Multi-Token Prediction) draft head has loader incompatibility on ROCm.
- For crash diagnosis: use gdb over logs (logs are often incomplete/misleading for segfaults).
- Swap-resize safe ordering: resize before use, not after.
- `free` command lies about available memory (doesn't account for reserved/pinned).
- `systemctl disable` can delete unit files (not just disable them).
- `pkill -x` for exact-name process kill (avoids substring matches).
- Rootless quadlet log retrieval has specific gotchas (see skill: rootless-quadlet-log-retrieval).

## omp Skill Discovery
- omp ONLY discovers user skills at `~/.omp/agent/skills/`.
- pi-era locations `~/.pi/agent/skills/` and `~/.agents/skills/` (vercel-labs `skills` CLI layout) are NOT scanned by omp.
- `npx skills` CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
- Newly installed skills surface on the NEXT session, not the current one.
- Skill sources: github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff), github.com/vercel-labs/skills (find-skills).
- 4 skills unrecoverable without dankware (192.168.0.226) online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base.

## 01a09e67-630f-74ed-a910-e997e01fbe54
updated_at: 1789368883
Source: serverAppConfigs-docs/LLM/halogen-flash-server.md (halogen-flash-server runbook), sections Memory model, Two failure modes, Co-tenancy, and troubleshooting table.

Core stall advice: With HALOGEN_FLASH_PIN_TRUNK=1 (default), the halogen container holds most host memory. If a co-tenant takes the remainder and host memory runs out, both the engine and the co-tenant can stall for minutes at full CPU with no disk activity and no output. This is documented behavior, not a crash, and it needs no restart. Do not treat full-CPU/no-output multi-minute stalls as OOM/crash/restart cases.

Last resort: HALOGEN_FLASH_PIN_TRUNK=0 gives the pinned memory back to the host but costs several times the decode speed; use only as last resort, not routine tuning.

Diagnostics/monitoring traps: free, MemAvailable, and Grafana can read wrong by about 68 GiB because locked weights are counted as reclaimable. A request that never returns then returns 503 engine_busy is a different admission issue: the pool could not reserve prompt + max_tokens.

Restart first-touch behavior: after restart, the first request can make the engine silent for ~45-60 s (has not answered PING, /health engine_unresponsive) while it first-touches the fresh KV pool; it self-recovers. HealthStartPeriod=300s covers this; do not restart the unit; retrying after the window succeeds.

KV pool verification: use podman logs frank-large | grep 'kv pool:' and match reserved count to requested count; /health does not prove pool size. If under budget, nothing may print; absence of WARNING is not evidence of headroom.

Memory freeing/trimming: cut the KV pool or arena (examples: 262144 hands back ~3.6 GiB from 401408; MAX_TOK 16384 = ~8.4 GiB). Measured host-left gain can be less than modelled because the engine keeps freed space as working slack, but contiguous block count may improve substantially; stall warning keys on that. Example verified trim: kv_pool_positions=401408 exact/no clamp, held 100.0 GiB (68.0 weights + 11.0 KV + 21.0 working), host left 9.1 GiB / 1396 contiguous blocks vs previous 7.1 GiB / 310, stall NOTE no longer printed, health ok, real completion succeeded.

Client config constraint (blankpad OMP ~/.omp/agent/): frank-large maxTokens must be <=65536 because halogen returns 400 above cap; prompt + max_tokens must fit the KV pool or the request queues ~3600 s then 503s. Retired dead flm/dankclaw-npu provider on :7771; roles: vision -> frank-large, tiny -> frank-small, task -> frank-large:medium.

Deployment/environment notes: generated Quadlet units cannot be systemctl --user enable on this host; boot persistence uses [Install] WantedBy=default.target plus linger. Before replacing the halogen container/unit, remove the old Quadlet unit from ~/.config/containers/systemd/ to avoid duplicate port 7770 claims. Known environmental e2e breakage: blankpad -> 192.168.0.112 Sonarr path issue; deploy used --skip-e2e. Tests: 285+ green except known pre-existing test-73 ComfyUI selector bug.

## 01a09835-91cf-72e7-8ece-e5316d4de48b
updated_at: 1789286336
Expanded frank-large KV pool from 262144 to 524288 positions (~14.4 GiB). Reduced HALOGEN_KV_SLOTS from 4 to 3 (~111 MiB freed per slot). Held memory stable at 103.4 GiB (68 weights + 14.4 KV pool + 21 working) within 121 GiB host budget. Admission is pool-bound, not slot-bound; caps at 3 simultaneous full-length requests.

frank-small defaults to sampling (checkpoint values: do_sample=true, temperature=1.0, top_k=20, top_p=0.95), creating asymmetry with frank-large which defaults to greedy. To force determinism on frank-small: send temperature: 0 per request or relaunch with --sampling-defaults none.

preserve_thinking cannot be set server-side. Enumerated 156 HALOGEN_* env vars; none control thinking/preserve. Must be sent per-request in chat_template_kwargs by the client.

API/Host quirks: frank-small returns 200 for unknown fields (200 proves nothing). max_tokens: 200 with empty content indicates thinking consumed budget. host_reserve_gib=24 protects n-gram table; file-cache shortage causes uninterruptible sleep, not OOM.

Tests: 285/286 pass (test 73 comfyui pre-existing selector bug).

## 01a09648-8eb8-7093-876a-116b11d4c5fb
updated_at: 1789246675
Deployment Stack: Podman compose on dankclaw. Marreta shares gluetun egress with searxng. Caddy TLS edge routes to Anubis (:8923) -> Authentik embedded outpost -> internal :8017 listener -> marreta upstream.

Auth Chain Workflow: Provider-first creation -> Route flip (vhost becomes reverse_proxy anubis:8923, internal @svc handler) -> Deploy. Verified via unauthenticated challenge probe, post-auth config grep, outpost DNS reachability.

Resolved Failure: Marreta login-wall due to ProxyProvider.application = None on marreta-proxy. Null invalidation_flow crashed Rust outpost serde. Null auth/authz flows compounded issue.

Fix Applied: Populated invalidation_flow, authentication_flow, authorization_flow + linked Application(name=Marreta, slug=marreta, provider=marreta-proxy). Restarted authentik-server + authentik-outpost.

Pitfalls & Constraints: Probe http://localhost:8017/ from host returns HTTP 000 (listener in Caddy netns); probe inside netns or grep config. Non-browser clients cannot solve JS challenge; audit service-to-service traffic for backend datasource URLs hitting public guarded hostnames. Exempt-by-construction: IdP (auth.), git smart-HTTP (Gitea), self-auth services. Authentik providers/applications are live-only (ak shell); no repo blueprint (captured in authentik-proxy-provider-app-link skill).

Verification: E2E regression test (tests/e2e/specs/16-marreta.spec.ts) asserts homepage tile passes through guard chain to Marreta app title, avoiding login-wall parking.

## 01a09686-789e-7042-8774-55507faa78b2
updated_at: 1789232480
SERVICE: frank-large (rootless Quadlet, boot-persistent) on dankclaw. Unit: frank-large.service | Container: frank-large.
LOG_RETRIEVAL: podman logs frank-large (full); podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' (filtered); journalctl --user -u frank-large (journald).
PORTS: Engine :8730 (container loopback, unauth), Front-end :8731, Published :7770.
CONSTRAINTS: Weights pinned (~68 GiB). Env vars to avoid naming: TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads bare TEMP for JIT cache path).
TEST_ENV: pyyaml missing for test suite (uv available, pip missing on system python3). Most suite failures are pre-existing environment issues, not commit regressions.