live: heartbeat sync 2026-10-18T20:54:41-07:00 (Refs #11)

agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/architecture.md

@@ -22,7 +22,7 @@
 
 **Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated** (`.hgn` format)
 
-- Runtime: halogen-flash-server 0.16.2 (digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`) 
+- Runtime: halogen-flash-server 0.16.2 (digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`)
 - Format: v2 `.hgn` (flash-server exclusive)
 - MTP head: INSIDE checkpoint (`HALOGEN_MTP_HEAD` unset)
 - Vision: ON (v2 tower loads beside checkpoint)
@@ -62,7 +62,7 @@ Every deployed, working Podman container MUST have a self-healing, boot-persiste
 
 Unit requirements:
 
-- **Boot persistence** — `[Install] WantedBy=default.target`, installed to `~/.config/containers/systemd/<name>.container`, enabled in the user manager, with linger on for the deploy user (`loginctl show-user dank -p Linger` → `Linger=yes`). 
+- **Boot persistence** — `[Install] WantedBy=default.target`, installed to `~/.config/containers/systemd/<name>.container`, enabled in the user manager, with linger on for the deploy user (`loginctl show-user dank -p Linger` → `Linger=yes`).
 - **Self-healing** — `Restart=always` (or `on-failure`) with a short `RestartSec`; quadlet health check where the service has one.
 - **Repo provenance** — the `.container` file lives in this repo (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.
 
@@ -82,7 +82,7 @@ Scratch ports are not used on LLM hosts — a test process cannot share the sing
 
 A gufo-era law: **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
 
-Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real** text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently returns nothing. Generate **at least 128 tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise and the run-to-run spread swamps every real difference.
+Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real* text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently returns nothing. Generate **at least 128 tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise and the run-to-run spread swamps every real difference.
 
 This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.
 
@@ -115,4 +115,4 @@ This rule exists because subagent hangs were observed in early deployments (2026
 
 Before declaring a subagent deployed: verify the heartbeat endpoint returns within the timeout window, and confirm the recovery path is wired (retry/fallback/escalation logic exists in the codepath).
 
-A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.
+A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.`,