diff --git a/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md b/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md index bc15cdc..67543ea 100644 --- a/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md +++ b/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md @@ -1,3 +1,4 @@ +- Dankclaw podman 5.7 quadlet gotchas (verified live during the Anubis rollout, 2026-08-28): (1) podman's reference parser rejects mixed-case ghcr.io refs at parse time — always use lowercase ghcr.io/techarohq/anubis:vX.Y.Z; ghcr resolves case-insensitively at pull. (2) The v1 quadlet generator cannot enable units on this host, and registry-prefixed image refs fail container create with "image not known" even when the image is in storage — run containers from a localhost/name:tag alias tag instead (deploy handler pulls canonical ref then podman tags the alias). (3) The generator joins bare HealthCmd into ONE argv element with \x20-escaped spaces → shell-form exec of a missing shell; distroless ko-build images (entrypoint /ko-app/anubis, no /bin/sh) need JSON exec-form health probes: HealthCmd=["/ko-app/anubis","--healthcheck"] — the generator passes a JSON array through verbatim. (4) Guard image pulls with podman image exists canonical-ref first; cold anonymous ghcr token requests can 403 on first fetch — retry once, second attempt resolves locally. (5) Anubis chain shape: Caddy TLS :443 → Anubis verify :8923 → accepted traffic back to Caddy internal listener :8016 → per-vhost routing; guarded set = homepage/search/hermes/prometheus/romm/files; exempt by construction: auth (IdP itself), plex (plain HTTP), gitea git smart-HTTP (git clients can't solve JS challenges), grafana (own login). _(context: Repo /home/dank/Projects/serverAppConfigs; docs already cover the chain in serverAppConfigs-docs/Infrastructure/Anubis.md and Architecture/Network Architecture.md; memory covers the podman/quadlet runtime quirks that bite during deploys on this host specifically.)_ - MTP draft heads for Qwen3.8-Flash-Next are INCOMPATIBLE with the STRIX_LEAN Uncensored model on the charlie12345/ROCmFPX qwen4exp binary. Both attempted heads — EasiiX Q8_0 (4.1 GB) and agentionai ROCmFP4-FAST (2.28 GiB) — fail with missing tensor 'blk.0.hc_attn_norm.weight'. The qwen4exp binary expects this hyper-connection attn norm tensor in the draft GGUF, but neither draft head carries it. MTP only ever worked on the 27B model (kingjones-Qwen3.8-27B-STRIX-MTP with mtp-Qwen3.8-27B-Q4_0.gguf), never on any Flash-Next variant. The Flash-Next STRIX (non-LEAN) that ran before the 2026-08-28 session never loaded a draft model at all. Current production config: SPEC_TYPE=ngram-mod only. To get MTP working on STRIX_LEAN, need either a draft head that includes hc_attn_norm tensors or a binary that doesn't require them for draft models. _(context: frank-default-large MTP integration attempt on Dankclaw, 2026-08-28. Downloaded two MTP heads from HuggingFace, both failed to load against the STRIX_LEAN Uncensored model. Service crash-looped on each attempt. Fell back to ngram-mod only. Files: scripts/frank-default-large.env, scripts/serve-frank-large.sh.)_ - The MTP draft head GGUF mtp-Qwen3.8-Flash-Next-Q8_0.gguf (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, 4.1 GB) is INCOMPATIBLE with the STRIX_LEAN Uncensored Flash-Next model on the charlie12345/ROCmFPX binary (build-qwen38). The draft GGUF is missing tensor blk.0.hc_attn_norm.weight (hyper-connection attention norm) that the qwen4exp architecture expects. The service crash-loops: main model loads fine (~59s), then draft model load fails and exits. MTP only ever worked on the older 27B model (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT + its own mtp-Qwen3.8-27B-Q4_0.gguf head). The Flash-Next STRIX (non-LEAN) that ran before this session never loaded a draft model either — it ran ngram-only. Current production config: SPEC_TYPE=ngram-mod (no draft sidecar). TODO: find a compatible MTP head for STRIX_LEAN or build one from the Flash-Next checkpoint. _(context: frank-default-large MTP draft head integration on Dankclaw, 2026-08-28)_ - When running Python scripts on a remote host (Dankclaw) that need to manage local services (systemctl, sed), use subprocess.run(cmd, shell=True) for local execution — NOT nested SSH back to the same host. Nested SSH to localhost fails (no key auth for self-loopback). Also: before git pull --ff-only on a remote, verify the branch with git branch --show-current — Dankclaw was on frank-large-strix-model not main, so pulls were no-ops and new files didn't appear. _(context: bench-spec-decode.py ran on Dankclaw but used ssh() to SSH back to itself for systemctl/sed. All SSH calls failed with Permission denied. Also: Dankclaw was on frank-large-strix-model branch, not main, so git pull --ff-only was a no-op and the new file didn't appear.)_