# Frank Fleet Memory Consolidation

## 1. Fleet Topology & Identity

### Hosts
- **dankclaw** (192.168.0.143): Strix Halo APU (gfx1151, ~114 GiB GTT) + M.2 docked RTX 3090 (24 GiB, PCIe x4). Hosts `frank-large` (Halogen) and `frank-small` (Llama.cpp).
- **dankserver** (192.168.0.235): ASUS B650-E, 2x AMD R9700 (gfx1201, 32 GiB each), 4x DIMMs DDR5. Hosts `frank-primary` (VLLM/Radiance) and `frank-medium` (Llama.cpp).
- **frankbook-pro** (192.168.0.112): M-series Mac mini (~30 GiB RAM). Role unclear/secondary.

### Services (Live as of 2026-10-05)
- **frank-large** (:7770): Halogen-flash-server v2, Qwen3.8-27B Uncensored (Q6_0/Q8_0/ROCmFP4), MTP/DFlash2 drafter, HALOGEN_CTX=262144, KV pool 524288 pos.
- **frank-small** (:7771): Llama.cpp, MiMo-V2.6-9B (Q6_K/Q5_K_M) or Qwen3.8-9B (IQ3_S), 2x200704 f16-KV.
- **frank-primary** (:7770): tcclaviger VLLM fork, Qwen3.8-Flash-Next FP8, TP=2, SEQS=3, MNBT=2048, MAXLEN=200704, Arena ~17.77 GiB.
- **frank-medium** (:7770): Llama.cpp, Qwen3.8-27B UD-Q8_K_XL, Tensor Split 35:30, CACHE_RAM 16384 MiB.
- **Authentik**: `ridley` shape template, Django ORM creation, Group-based ACLs.
- **ComfyUI**: Strix Halo iGPU, FP8/Q6 quantization, Ghostty image protocol.

## 2. Hardware & Memory Constraints

### RAM Stability (dankserver)
- **Critical**: 4x DIMMs at 4800 MT/s caused silent corruption (NaN/garbage) at ~40k tokens in `frank-primary`. **Fix**: Drop to 4600 MT/s.
- **Lesson**: Consumer non-ECC RAM raises no MCE for silent flips. Short in-band checks miss intermittent corruption.
- **Shape E**: 3.15e9 B KV pool, 17.77 GiB Arena, UTIL 0.982.

### GPU & PCIe
- **dankclaw 3090**: PCIe x4 dock (~29 GB/s). Decode bottlenecked by graph splits + PCIe sync. 30 t/s unreachable without mainline PR #28901.
- **dankserver R9700**: Chipset-attached PCIe (Gen4 x1) fails RCCL atomics. **Fix**: M.2→PCIe x4 riser for 2nd card.
- **Strix Halo**: 64 GiB Unified, RADV Vulkan. HALOGEN_CTX must be 262144 (native Qwen3.8-Flash-Next).

### KV & Context
- **frank-large**: `maxTokens` 57344 (harness) + HALOGEN_CTX 262144. 400 error if `prompt + max > ctx`.
- **frank-medium**: `CACHE_RAM` 16384 MiB required for 196608 ctx DeltaNet state. 8192 MiB caused refusals.
- **frank-small**: 2x200704 f16-KV slots.

## 3. Runtime Environments

### Halogen (dankclaw)
- **Container**: `halogen-flash-server v2` (Swift1.5 Abliterated).
- **Env**: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`.
- **Pitfalls**: `HALOGEN_FLASH_PIN_TRUNK=1` causes multi-minute CPU stalls under co-tenancy. `HALOGEN_FLASH_PIN_TRUNK=0` as last resort.
- **Boot**: First request may be silent ~45-60s (first-touch KV pool).

### Llama.cpp (dankclaw/dankserver)
- **Forks**: `ik_llama.cpp` (CUDA+Vulkan), `rocmfpx` (ROCmFP4).
- **Flags**: `--cache-ram 16384` (medium), `--expert-precise-memory` (primary), `--max-model-len 200704`.
- **MTP**: Qwen3.8-Flash-Next MTP head needs full per-layer loading (Flash-Next specific).

### VLLM (dankserver)
- **Fork**: tcclaviger RDNA4 vLLM.
- **Seqs**: `SEQS=3` mandatory (SEQS=2 fails `torch.fx` ConstraintViolationError).
- **MTP**: Speculative decoding disabled/optional.

## 4. Operational Hygiene

### Bench Protocol
1. `sudo systemctl stop <service>`
2. Run benchmark
3. `sudo systemctl start <service>`
4. `curl -s <port>/health` → 200
5. **Trap**: `trap EXIT 'systemctl start <service>'; curl -s <port>/health` for self-restoring harnesses.

### Quadlet Persistence
- **Rootless Podman**: `restart: unless-stopped` only survives crashes. Bare containers vanish on reboot.
- **Pattern**: Git-tracked `.container` files in `~/.config/containers/systemd/`.
- **Historical**: Caddy lost after reboot (Aug 26); 5 observability services lost (Sep 11).

### Git & Deploy
- **Repo**: `/home/blank/Projects/serverAppConfig`.
- **Gate**: `deploy.sh` rejects dirty trees (untracked/modified files).
- **Pattern**: `git stash` → deploy → `git stash pop`.

## 5. Troubleshooting Knowledge Base

### max_tokens 400 Overflow
- **Symptom**: `prompt is X tokens and the context is Y, leaving room for Z`. `X+max > Y`.
- **Fix**: Adjust `maxTokens` in `models.yml` (e.g., 57344) or `HALOGEN_CTX` (262144).
- **Note**: OMP snapshots at process start; existing sessions need restart (`omp -r`).

### Orphaned `llama-server` (frank-medium)
- **Symptom**: Unit stopped, GPUs busy.
- **Diag**: `ps -eo pid,ppid,args | grep llama`, `rocm-smi --showmeminfo`.
- **Fix**: `kill <pid>`, `systemctl start`, verify geometry.

### Context Bloat
- **Source**: 175 managed-skills descriptions (~38k nonMessageTokens).
- **Fix**: Distill to vault reference pages, move dirs to `~/.AGENTTRASH/`, purge on approval.

### Authentik User Creation
- **Method**: `ak shell` on `dankclaw`, Django ORM (`User.objects.create`...), `ak shell` sets password, assign groups (`Non-Admin Users`, `jellyfin_users`).
- **ACLs**: Group membership is sole gate.

### ComfyUI (iGPU)
- **Target**: Strix Halo iGPU (3090 occupied).
- **Quant**: FP8 or Q6.
- **Display**: Ghostty `\e_G` base64 PNG, or `chafa` helper.

## 6. Skills & Playbooks

- **llm-fleet-model-swap**: Check HF model card, download sha256, commit → push → deploy.sh → health.
- **halogen-flash-server-memory**: Memory model, two failure modes (stall, 400), co-tenancy.
- **frank-primary-ram-stability**: DDR5 4800 vs 4600 MT/s, chunk boundaries, prefix cache eviction.
- **frank-medium-bench-protocol**: Restore half optional? No. Self-restoring harness recommended.
- **authentik-user-creation**: Django ORM workflow, group-based access.
- **orchestration-quadlet**: Rootless Podman boot persistence.
- **context-bloat-distillation**: Skill directory to vault migration.
- **comfyui-qwen-workflow**: iGPU image generation, ghostty display.
- **expert-arena-definition**: `arena = budget - non-expert - kv_reserve - graphs - overhead`.
- **skill-routing-pitfall**: `html-change-report` for git commits, not creative writing.