diff --git a/agent/memories/sharpshooter/tmp-1tpw73408x3e/state.json b/agent/memories/sharpshooter/tmp-1tpw73408x3e/state.json index 2a0b1a5..cec83ee 100644 --- a/agent/memories/sharpshooter/tmp-1tpw73408x3e/state.json +++ b/agent/memories/sharpshooter/tmp-1tpw73408x3e/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1792379255984 + "lastConsolidatedAt": 1792385569648 } diff --git a/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/architecture.md b/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/architecture.md index 368396e..babdbc7 100644 --- a/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/architecture.md +++ b/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/architecture.md @@ -1,118 +1,53 @@ -## Runtime Stack +## Documentation law (2026-10-18) -**llama.cpp** is the inference runtime for all `frank-zealot`-class endpoints. The `frank-zealot` service runs llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build, under a root systemd unit. +**One fact, one owner.** The `serverAppConfigs` repo owns every fleet fact; +`AGENTS.md` restates only what a deploy gate or the harness needs and otherwise +links. `serverAppConfigs-docs/` (Obsidian vault) carries the lore: one page per +endpoint (`[[frank-archon]]`, `[[frank-dragoon]]`, `[[frank-zealot]]`) holding +Current shape, **Eras** (one row per payload swap: dates, payload, runtime, +shape, why replaced, rollback), Laws, wire contract, files. `log.md` records +incidents and laws only — no per-swap ledgers, no bench tables. A retired +payload is an Eras row, never a page of its own. `scripts/.env` beats +any page: where they disagree, the page is the bug. -## Model Inventory +## Model swaps (2026-10-18) -### frank-zealot (Dankclaw: 192.168.0.143:7771) +Swap the model, don't instrument the swap: edit the trio +(`scripts/.env` + serve script + unit) → `./tests/run.sh` → PR → +`deploy.sh --service ` → `/health` + one chat / tool-call / vision +probe → done. **No benchmarks, A/Bs, ladders, or acceptance tables unless +Frank explicitly asks for data.** If a payload misbehaves, revert the commit and +redeploy. When a bench IS ordered, the GPU-safety procedure (stop + mask, no +scratch ports, never `systemctl disable`) is `AGENTS.md` → *Model swaps* and +*Deployment completeness*. -**Ternary-Bonsai-2-27B-PQ2_0.gguf** (prism-ml/Ternary-Bonsai-2-27B-gguf) +## Where facts live -- Runtime: llama.cpp -- Format: PQ2_0 quantized GGUF -- 27B parameter class -- Vision: native BF16 mmproj projector for image input -- MTP: OFF (no `nextn` tensors in checkpoint) -- Context: 256k per-slot window (`--ctx-size 401408 --parallel 2` = 200,704 tokens effective) -- Batch: 4096 / ubatch 2048 (CUDA optimum) -- Log verbosity: 4 -- Ports: 7771 on Dankclaw +|Need|Owner| +|---|---| +|Topology, ports, payload, shape|repo `AGENTS.md` → Fleet topology; `scripts/.env` is the authority| +|Endpoint lore, eras, laws, rollback|vault `serverAppConfigs-docs/LLM/.md`| +|Harness model entries (`contextWindow`, `maxTokens`, `reasoningContentField`, `modelRoles`)|`~/.omp/agent/models.yml`| +|Incidents and operational laws|vault `serverAppConfigs-docs/log.md`| +|Service/infra pages|vault `Dankclaw/`, `Dankserver/`, `Infrastructure/`, `Architecture/`| -### frank-dragoon (Dankclaw: 192.168.0.143:7770) +Models.yml resolution rules that are harness-local: a default model resolves as +a `defaultProvider` + `defaultModel` pair; a subagent override is a bare model +`id` and resolves only when unique across providers; two ports on one host need +two provider blocks — the file cannot express "same host, different port". -**Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated** (`.hgn` format) +There is **no audio input anywhere in the fleet** (gone with the Gemma4 +retirement, 2026-10-01). All three chat endpoints take text + images. -- Runtime: halogen-flash-server 0.16.2 (digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`) -- Format: v2 `.hgn` (flash-server exclusive) -- MTP head: INSIDE checkpoint (`HALOGEN_MTP_HEAD` unset) -- Vision: ON (v2 tower loads beside checkpoint) -- Base: abliterated (27B-era was censored finetune) -- Slots: 2 × 262,144 ctx (KV pool 524,288 positions) -- Server sampler: 1.0/0.95/20 -- Prompt cache: ON by default +## Retired laws -### frank-archon (Dankserver: 192.168.0.235:7770) - -**StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe** (`.rad` format) - -- Runtime: radiance 1.0.13 (digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`) -- Format: fp8-KV `.rad` (121,969,901,568 B = 116,322 MiB) -- MTP head: MTP depth 2, MNBT 2048, Froggeric v22.5 template -- Vision: 27-layer Qwen-VL tower native -- KV: 3 × 200,000 fp8-KV (`--max-num-seqs 3`) -- 30,720 MiB of 30,969 MiB claimable -- 44 GiB/rank pinned host pool, PLE pinned to RAM -- Base-alignment: censored (abliterated family lives on frank-dragoon) - -## Resolution Rules - -Official model ids are StarCraft unit names, on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: `frank-archon` → `192.168.0.235:7770`, `frank-dragoon` → `192.168.0.143:7770`, `frank-zealot` → `192.168.0.143:7771`. - -The default model resolves as a `defaultProvider` + `defaultModel` pair (provider must serve the model); a subagent override uses a bare model `id`, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port". - -When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 **all three endpoints accept images** (`input: [text, image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap, while `frank-archon` takes them natively and `frank-zealot` through its BF16 mmproj. - -Reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — `models.yml` sets `reasoningContentField` per endpoint to match. - -There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01. - -## Boot Persistence & Self-Healing (MANDATORY) - -Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from `compose.yaml` or a manual `podman run` dies silently at reboot — rootless Podman does not auto-start bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. Never report such a service as "up". - -Unit requirements: - -- **Boot persistence** — `[Install] WantedBy=default.target`, installed to `~/.config/containers/systemd/.container`, enabled in the user manager, with linger on for the deploy user (`loginctl show-user dank -p Linger` → `Linger=yes`). -- **Self-healing** — `Restart=always` (or `on-failure`) with a short `RestartSec`; quadlet health check where the service has one. -- **Repo provenance** — the `.container` file lives in this repo (`//quadlet/`), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth. - -Before declaring a service deployed: `systemctl --user is-enabled ` → `enabled`, `is-active` → `active`, and the container answers its health check. - -A running container with no unit is a gap: write the unit, test it, deploy it. - -This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under `dankclaw/observability/quadlet/` and their compose definitions were removed — one owner per container name. The stack is **configured and delivering as of 2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, loopback :9093) → host-side Signal relay (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as `host.containers.internal`; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311. - -Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See `serverAppConfigs-docs/Infrastructure/Observability.md`. - -## Benchmarking Policy (MANDATORY) - -**STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench requires asking first. - -Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. - -A gufo-era law: **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** - -Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real* text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently returns nothing. Generate **at least 128 tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise and the run-to-run spread swamps every real difference. - -This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: `ip neigh show ` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP. - -1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU): - `sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon` - - Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090): - `sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot` - - Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2): - `sudo systemctl stop frank-archon && sudo systemctl mask frank-archon` -2. Bench. - NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in `/etc/systemd/system/` is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking. -3. Unmask and start; verify `curl http://192.168.0.:7770/health`. - -**CRITICAL:** never `systemctl disable` a system unit on these hosts — it deletes the unit file from `/etc/systemd/system/`, not just the symlink. `mask` symlinks to `/dev/null` and is safe/reversible. - -## Subagent Lifecycle (MANDATORY) - -A subagent that appears stuck must have an explicit recovery path defined before the subagent is considered deployed. "Stuck" includes: silent completion halt, repeated identical responses, or extended latency without token emission. - -Recovery requirements: - -- **Explicit timeout** — a hard timeout (≤30s) on the subagent's response channel, with a defined action (retry, fallback, or escalate). -- **State checkpoint** — the subagent maintains checkpoints at decision boundaries, allowing resumption from a known good state. -- **Health signal** — the subagent publishes a heartbeat (≤5s interval) that triggers the recovery path when missed. - -A subagent without these three properties is a gap: write the timeout, add the checkpoint, wire the heartbeat, then deploy. - -This rule exists because subagent hangs were observed in early deployments (2026-09-xx: silent completion halt; 2026-09-xx: repeated identical responses). The deployment script must verify the heartbeat channel is live before reporting "deployed". - -Before declaring a subagent deployed: verify the heartbeat endpoint returns within the timeout window, and confirm the recovery path is wired (retry/fallback/escalation logic exists in the codepath). - -A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.`, \ No newline at end of file +- The old "Benchmarking Policy (MANDATORY)" wording demanded a bench harness per + payload; superseded 2026-10-18 by the swap law above. +- A "Subagent Lifecycle (MANDATORY)" rule (heartbeat ≤5 s, checkpoints, hard + timeout) was recorded here 2026-09 and **never implemented anywhere**; it + described nothing that exists and is deleted 2026-10-18. Do not re-add it + without a real implementation. +- The fleet model inventory that used to live in this memory file drifted out of + date within hours (it still claimed Bonsai-2-**27B** at `--ctx-size 401408` + and a dragoon 524,288-position pool). Memory files do not get deploys — the + repo does. That is why it is pointers now. diff --git a/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/state.json b/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/state.json index e383ec3..5792cd3 100644 --- a/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/state.json +++ b/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792381956673, + "lastConsolidatedAt": 1792385631740, "lastResult": { "at": 1792379504653, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261018-204442-7c8a99d-2u1dm4kttza8t/state.json b/agent/memories/sharpshooter/wt-20261018-204442-7c8a99d-2u1dm4kttza8t/state.json index 87c6b68..e4f6f7e 100644 --- a/agent/memories/sharpshooter/wt-20261018-204442-7c8a99d-2u1dm4kttza8t/state.json +++ b/agent/memories/sharpshooter/wt-20261018-204442-7c8a99d-2u1dm4kttza8t/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1792381842302 + "lastConsolidatedAt": 1792385502315 } diff --git a/agent/memories/sharpshooter/wt-20261018-204501-6d1fc1f-3fct0wvty8jvu/state.json b/agent/memories/sharpshooter/wt-20261018-204501-6d1fc1f-3fct0wvty8jvu/state.json index a363c14..2801894 100644 --- a/agent/memories/sharpshooter/wt-20261018-204501-6d1fc1f-3fct0wvty8jvu/state.json +++ b/agent/memories/sharpshooter/wt-20261018-204501-6d1fc1f-3fct0wvty8jvu/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792381870670, + "lastConsolidatedAt": 1792385401831, "lastResult": { "at": 1792381870670, "sessions": 1, diff --git a/dotfiles/noctalia/.local/state/noctalia/settings.toml b/dotfiles/noctalia/.local/state/noctalia/settings.toml index 8dadaa1..a04d0c3 100644 --- a/dotfiles/noctalia/.local/state/noctalia/settings.toml +++ b/dotfiles/noctalia/.local/state/noctalia/settings.toml @@ -275,13 +275,13 @@ transition = [ "fade" ] interval_seconds = 1200 [wallpaper.default] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxmpwl.jpg" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_28r759.jpg" [wallpaper.last] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxmpwl.jpg" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_28r759.jpg" [wallpaper.monitors.DP-1] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxmpwl.jpg" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_28r759.jpg" [wallpaper.monitors.DP-2] path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxex69.jpg" @@ -290,7 +290,7 @@ transition = [ "fade" ] path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9e2w3.jpg" [wallpaper.monitors.eDP-1] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxmpwl.jpg" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_28r759.jpg" [weather] unit = "imperial"