# Raw Memories

## 01a0ac72-2620-750f-82dd-79c84e00e98d
updated_at: 1789631572
DANKCLAW HARDWARE: AMD Strix Halo APU (gfx1151), Radeon 8060S iGPU. Has an M.2 dock slot used to carry an RTX 3090 at PCIe x4 (link downgraded from x16). This is distinct from dankserver (ASUS TUF GAMING B650-E WIFI) which has a free CPU-attached M.2_2 slot for an R9700 riser (~$30 M.2→PCIe x4 riser).

DANKCLAW LLM DEPLOYMENTS:
- frank-small: Qwen3.8-27B GSQ-RCO IQ3_S + MTP on RTX 3090 (M.2 dock, PCIe x4). Decode 61-82 t/s, prefill 430-1013 t/s. VRAM at 92% (22,614/24,576 MiB). PCIe utilization only ~3% during decode (compute-bound, not transport-bound). --parallel 1, 8 CPU threads.
- frank-large (halogen): runs on the iGPU (gfx1151), decode 37-48 t/s, MTP commit 1.60-1.88 tokens/round. Prefix caching confirmed working.

UNANSWERED QUESTION: User asked specifically about the M.2 slot in context of a Qwen Flash deployment running with pipeline parallelism on dankclaw. The assistant's response was aborted (OMP process exited) before answering. This likely refers to a llama.cpp pipeline-parallel setup where the Qwen Flash model's layers are split across the iGPU and the 3090 in the M.2 dock, using the M.2 slot as the interconnect between the two compute devices.

## 01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3
updated_at: 1789599423
## Rootless Podman restart policy pitfall
- `restart: unless-stopped` only survives *crashes*, not *explicit stops* (e.g., kernel power-off sends a polite stop signal).
- Result: bare containers without a Quadlet unit are gone after reboot with no alert.

## Repo `serverAppConfig` deployment pattern
- Every container must have a Quadlet `.container` file (git-tracked) to be boot-persistent and self-healing.
- The unit file is the container's birth certificate + pension plan + ghost in one.

## Historical incidents (from story)
- Aug 26: Caddy never returned after reboot (no unit).
- Sep 11: Five observability services found unitless after reboot.

## Persona note
- Model: `frank-medium` via llamacpp (dankserver). Persona: Soviet-style computer systems engineer with Munroe-esque humor. Uses "Comrade" address.

## 01a0a7ad-42e2-739d-bc1c-dc405ca02cbd
updated_at: 1789574671
## Dankserver Dual R9700 RCCL Chipset Issue

**Hardware**: 2x AMD R9700 on ASUS board. Card #1 on CPU-attached PCIe x16. Card #2 on chipset-attached PCIe x16 slot (PCIEX16_2, which is x1 mode per ASUS spec).

**Problem**: RCCL (ROCm Collective Communications Library) fails on the chipset-attached card. Bare 2-rank torch.distributed probe results:
- broadcast: OK (CPU card) / FAIL (chipset card)
- all_reduce: OK / FAIL
- barrier: OK / FAIL
- send/recv (P2P): hang / FAIL

**Implication**: send/recv failure kills pipeline parallelism too (vLLM's PP handoff IS that primitive). No software-only escape.

**Root cause**: Chipset-attached PCIe lanes don't route atomic operations the way RCCL's kernels require. CPU-attached lanes do.

**Fix (recommended)**: M.2→PCIe x4 riser (~$30) to move card #2 to M.2_2 slot. ASUS spec confirms M.2_1 and M.2_2 are CPU-attached; M.2_2 is free/empty. Restores atomic-op routing → RCCL works → TP=2 functional.

**Alternative**: Board with CPU x8/x8 bifurcation (~$250-400).

**Fallback**: llama.cpp layer split (no collectives needed, but needs GGUF and loses vLLM batching).

**Bandwidth analysis**: 27B TP=2 allreduce at batch 4 ≈ 10 MB per token-step ≈ 1.4 ms at Gen4 x4. Bandwidth is NOT the bottleneck; atomics routing is.

## Software Stack
- vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14
- Shipped RCCL exposes no ncclDevComm* symbols → torch's symmetric-memory path can't run
- Open hypothesis (unverified): recompiling RCCL with -mprintf-kind=buffered might work, but no live printf/assert/__assertfail found in device source

## Operational Notes
- dankserver went down during probing; needs physical power cycle
- Boot persistence should auto-restore: frank-medium trio, media stack, node-exporter
- To distinguish 'off' from 'wedged': ip neigh from peer (FAILED = off/NIC dead; wedged kernel still answers ARP)
- AGENTS.md benchmarking policy updated to cover any GPU experiment, not just benchmarks

## Model Target
- kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
- Fork: charlie12345/rocmfpx (llama.cpp ROCm fork)

## Skill Note
- 'grilling' skill does NOT exist in available skills list (returned 'Unknown skill: grilling')

## 01a0a87f-6b17-7492-995c-8c3e7b37e241
updated_at: 1789552363
## Hardware Topology (dankclaw)
- RTX 3090 (24 GiB VRAM) in PCIe 4.0 x4 M.2 dock (downgraded from x16, measured ~29 GB/s sequential)
- AMD Strix Halo gfx1151 (64 GiB SVM-backed unified memory, RADV Vulkan driver; ROCm wheels do NOT support gfx1151)
- Combined GPU budget: ~88 GiB
- OS: Ubuntu 24.04
- Kernel cmdline: amd_iommu=off (for prefill gains; retired frank-npu)

## ik_llama.cpp Build & Deployment
- Built with CUDA (for 3090) + Vulkan (for Strix Halo) backends
- Serves mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF at i1-Q4_K_M
- Deployed as root systemd unit (Restart=always) on port 7770, NOT a quadlet
- Context window: 262144 tokens
- Device split: --split-mode layer with -dev CUDA0,Vulkan0
- Shipped config: T16 (16 threads), no -ncmoe, no -thp
- Commits: 795b0b3, ad7a9c4
- Runbook: serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

## Performance Envelope (verified single-instance)
| Config | Decode (t/s) | Prefill (t/s) | Notes |
|---|---|---|---|
| Shipped T16 | 3.8-4.0 | ~192 (1.8k-32k ctx) | Production config |
| T8/T4/T2/T1 | 4.18/4.20/4.26/3.97 | small-prompt only | Thread count barely matters |
| -ncmoe 42 | 4.54 | — | 22.0/24.6 GiB VRAM, fragile at 262k ctx |
| -ncmoe 48 -thp | 4.60 | — | 110 GiB anon host buffer, co-tenant killer |
| --prefetch-experts | 4.03 | — | No effect (cache already resident) |
| Mainline llama.cpp 37b53fd | 1.01 | — | Ruled out entirely |

## Root-Cause: Why 30 t/s Decode Is Unreachable
1. Decode issues ~100 graph splits per token
2. Each split = GPU launch + sync across PCIe x4 (3090 in dock)
3. GPU SM utilization during decode: ~5%
4. ~90% of cycles = pause (thread barrier spin); T1 ≈ T16
5. CUDA graphs would collapse the overhead BUT capture ABORTS: qwen4exp hc ops use per-layer ggml_repeat_4d which is capture-illegal
6. Speculative decoding unavailable: GGUF has no MTP head (mradermacher stripped NextN from the checkpoint)
7. All experts on GPU: 84 GiB > 24 GiB 3090 VRAM — impossible
8. Path forward: mainline llama.cpp PR #28901 (hc-ops + capture-safe graphs) — re-evaluate when it lands

## Prefill Ceiling
- ~192 t/s is the ceiling: compute-bound on CPU AVX512-VNNI
- Expert reads amortize across the batch; math saturates cores
- Halogen hit 30 t/s decode by holding entire 115.55 GiB w4b checkpoint resident in LPDDR inside one proprietary engine; 24 GB GPU + CPU split cannot replicate that residency with any public runtime

## Operational Traps
- Unit has Restart=always: pkill'd servers respawn mid-experiment → phantom instances invalidate benchmarks. Always verify single-instance before measuring.
- Concurrent -j16 builds can starve probe processes
- PCIe x4 dock: do NOT assume x16 bandwidth; measured ~29 GB/s (not 130 GB/s)
- -ncmoe 42: +15-20% decode but only 2.5 GiB VRAM margin at 262k ctx — OOM risk
- -ncmoe 48 -thp: 110 GiB anon host buffer — will kill co-tenants

## Trash (staged for user approval)
- dankclaw ~/.AGENTTRASH/20260916-ik-partfiles/ — 111 GiB (3 download parts, superseded)
- dankclaw ~/.AGENTTRASH/20260915-ik-swap/ — retired halogen quadlet
- repo tests/.AGENTTRASH/frank_large_quadlet.bats, scripts/.AGENTTRASH/frank-small*

## 01a0a902-7f58-77f4-9392-764f6cfbd59e
updated_at: 1789544414
## AGENTS.md Rewrite Pattern (serverAppConfigs)
- Collapse chronological swap history into: (1) active-topology table (current state only), (2) runbook pointer table mapping each retired runtime → its docs/LLM/ runbook path.
- Add a 'Key Lessons Learned' section containing ONLY durable technical constraints/pitfalls (not narrative).
- In config sections (e.g. halogen), keep only invariants that survive any config change (pool/reserve matched pair, client contract, memory-model rule); push sizing history to runbook.
- Fix structural bugs (e.g. broken merged table rows) as part of the cleanup.

## Durable LLM/ROCm/Quadlet Lessons (from AGENTS.md distillation)
- Layer-split-not-tensor: for multi-GPU ROCm, split by layer not tensor-parallel.
- mmap/PLE constraints apply to model loading on ROCm.
- HIP env collision: `TEMP` and `HOST` env vars can conflict with HIP runtime.
- `GGML_CUDA_FA_ALL_QUANTS` env var needed for flash-attention with quantized KV.
- MTP (Multi-Token Prediction) draft head has loader incompatibility on ROCm.
- For crash diagnosis: use gdb over logs (logs are often incomplete/misleading for segfaults).
- Swap-resize safe ordering: resize before use, not after.
- `free` command lies about available memory (doesn't account for reserved/pinned).
- `systemctl disable` can delete unit files (not just disable them).
- `pkill -x` for exact-name process kill (avoids substring matches).
- Rootless quadlet log retrieval has specific gotchas (see skill: rootless-quadlet-log-retrieval).

## omp Skill Discovery
- omp ONLY discovers user skills at `~/.omp/agent/skills/`.
- pi-era locations `~/.pi/agent/skills/` and `~/.agents/skills/` (vercel-labs `skills` CLI layout) are NOT scanned by omp.
- `npx skills` CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
- Newly installed skills surface on the NEXT session, not the current one.
- Skill sources: github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff), github.com/vercel-labs/skills (find-skills).
- 4 skills unrecoverable without dankware (192.168.0.226) online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base.

## 01a09e67-630f-74ed-a910-e997e01fbe54
updated_at: 1789368883
Source: serverAppConfigs-docs/LLM/halogen-flash-server.md (halogen-flash-server runbook), sections Memory model, Two failure modes, Co-tenancy, and troubleshooting table.

Core stall advice: With HALOGEN_FLASH_PIN_TRUNK=1 (default), the halogen container holds most host memory. If a co-tenant takes the remainder and host memory runs out, both the engine and the co-tenant can stall for minutes at full CPU with no disk activity and no output. This is documented behavior, not a crash, and it needs no restart. Do not treat full-CPU/no-output multi-minute stalls as OOM/crash/restart cases.

Last resort: HALOGEN_FLASH_PIN_TRUNK=0 gives the pinned memory back to the host but costs several times the decode speed; use only as last resort, not routine tuning.

Diagnostics/monitoring traps: free, MemAvailable, and Grafana can read wrong by about 68 GiB because locked weights are counted as reclaimable. A request that never returns then returns 503 engine_busy is a different admission issue: the pool could not reserve prompt + max_tokens.

Restart first-touch behavior: after restart, the first request can make the engine silent for ~45-60 s (has not answered PING, /health engine_unresponsive) while it first-touches the fresh KV pool; it self-recovers. HealthStartPeriod=300s covers this; do not restart the unit; retrying after the window succeeds.

KV pool verification: use podman logs frank-large | grep 'kv pool:' and match reserved count to requested count; /health does not prove pool size. If under budget, nothing may print; absence of WARNING is not evidence of headroom.

Memory freeing/trimming: cut the KV pool or arena (examples: 262144 hands back ~3.6 GiB from 401408; MAX_TOK 16384 = ~8.4 GiB). Measured host-left gain can be less than modelled because the engine keeps freed space as working slack, but contiguous block count may improve substantially; stall warning keys on that. Example verified trim: kv_pool_positions=401408 exact/no clamp, held 100.0 GiB (68.0 weights + 11.0 KV + 21.0 working), host left 9.1 GiB / 1396 contiguous blocks vs previous 7.1 GiB / 310, stall NOTE no longer printed, health ok, real completion succeeded.

Client config constraint (blankpad OMP ~/.omp/agent/): frank-large maxTokens must be <=65536 because halogen returns 400 above cap; prompt + max_tokens must fit the KV pool or the request queues ~3600 s then 503s. Retired dead flm/dankclaw-npu provider on :7771; roles: vision -> frank-large, tiny -> frank-small, task -> frank-large:medium.

Deployment/environment notes: generated Quadlet units cannot be systemctl --user enable on this host; boot persistence uses [Install] WantedBy=default.target plus linger. Before replacing the halogen container/unit, remove the old Quadlet unit from ~/.config/containers/systemd/ to avoid duplicate port 7770 claims. Known environmental e2e breakage: blankpad -> 192.168.0.112 Sonarr path issue; deploy used --skip-e2e. Tests: 285+ green except known pre-existing test-73 ComfyUI selector bug.

## 01a09835-91cf-72e7-8ece-e5316d4de48b
updated_at: 1789286336
Expanded frank-large KV pool from 262144 to 524288 positions (~14.4 GiB). Reduced HALOGEN_KV_SLOTS from 4 to 3 (~111 MiB freed per slot). Held memory stable at 103.4 GiB (68 weights + 14.4 KV pool + 21 working) within 121 GiB host budget. Admission is pool-bound, not slot-bound; caps at 3 simultaneous full-length requests.

frank-small defaults to sampling (checkpoint values: do_sample=true, temperature=1.0, top_k=20, top_p=0.95), creating asymmetry with frank-large which defaults to greedy. To force determinism on frank-small: send temperature: 0 per request or relaunch with --sampling-defaults none.

preserve_thinking cannot be set server-side. Enumerated 156 HALOGEN_* env vars; none control thinking/preserve. Must be sent per-request in chat_template_kwargs by the client.

API/Host quirks: frank-small returns 200 for unknown fields (200 proves nothing). max_tokens: 200 with empty content indicates thinking consumed budget. host_reserve_gib=24 protects n-gram table; file-cache shortage causes uninterruptible sleep, not OOM.

Tests: 285/286 pass (test 73 comfyui pre-existing selector bug).

## 01a09648-8eb8-7093-876a-116b11d4c5fb
updated_at: 1789246675
Deployment Stack: Podman compose on dankclaw. Marreta shares gluetun egress with searxng. Caddy TLS edge routes to Anubis (:8923) -> Authentik embedded outpost -> internal :8017 listener -> marreta upstream.

Auth Chain Workflow: Provider-first creation -> Route flip (vhost becomes reverse_proxy anubis:8923, internal @svc handler) -> Deploy. Verified via unauthenticated challenge probe, post-auth config grep, outpost DNS reachability.

Resolved Failure: Marreta login-wall due to ProxyProvider.application = None on marreta-proxy. Null invalidation_flow crashed Rust outpost serde. Null auth/authz flows compounded issue.

Fix Applied: Populated invalidation_flow, authentication_flow, authorization_flow + linked Application(name=Marreta, slug=marreta, provider=marreta-proxy). Restarted authentik-server + authentik-outpost.

Pitfalls & Constraints: Probe http://localhost:8017/ from host returns HTTP 000 (listener in Caddy netns); probe inside netns or grep config. Non-browser clients cannot solve JS challenge; audit service-to-service traffic for backend datasource URLs hitting public guarded hostnames. Exempt-by-construction: IdP (auth.), git smart-HTTP (Gitea), self-auth services. Authentik providers/applications are live-only (ak shell); no repo blueprint (captured in authentik-proxy-provider-app-link skill).

Verification: E2E regression test (tests/e2e/specs/16-marreta.spec.ts) asserts homepage tile passes through guard chain to Marreta app title, avoiding login-wall parking.

## 01a09686-789e-7042-8774-55507faa78b2
updated_at: 1789232480
SERVICE: frank-large (rootless Quadlet, boot-persistent) on dankclaw. Unit: frank-large.service | Container: frank-large.
LOG_RETRIEVAL: podman logs frank-large (full); podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' (filtered); journalctl --user -u frank-large (journald).
PORTS: Engine :8730 (container loopback, unauth), Front-end :8731, Published :7770.
CONSTRAINTS: Weights pinned (~68 GiB). Env vars to avoid naming: TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads bare TEMP for JIT cache path).
TEST_ENV: pyyaml missing for test suite (uv available, pip missing on system python3). Most suite failures are pre-existing environment issues, not commit regressions.