live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/llm-endpoint-behavior-battery-a-b/SKILL.md

@@ -0,0 +1,30 @@
+---
+name: llm-endpoint-behavior-battery-a-b
+description: "A/B two boots of a live OpenAI-compatible LLM endpoint (payload swap trial) with one identical behavior+throughput battery: refusals, thinking-token census, tools, vision, needle, decode/prefill rates."
+---
+
+Run the SAME battery against both boots of an endpoint swap (candidate vs shipped payload). Never compare across different prompts, never benchmark beside the live service (stop + mask first), never measure the first request.
+
+## Harness
+`/tmp/halogen-eval/battery.py <tag> [all|tail]` (stdlib only, `urllib`): appends one JSON line per probe to `/tmp/halogen-eval/battery-<tag>.jsonl`, summary to stderr. `tail` = long-context + throughput only. Stage assets next to it: a corpus text file, a base64 512 px solid-color PNG.
+
+Probe order (each is a gate, all comparable across boots):
+1. **warmup** — discarded. First request after boot is a graph-capture artifact (measured 3x spread: 2.6 vs 1258 t/s prefill).
+2. **sanity** — exact-answer arithmetic, `temperature=0`.
+3. **refusals** — 5 hostile prompts, count refusal-marker regex hits. The abliteration/base-alignment census.
+4. **thinking census** — 6 fixed questions, `temperature=1.0`, `max_tokens=4096`; report mean `usage.completion_tokens` + per-item. A removed instruction fine-tune shows here first (measured: mean 784 vs the card's 3271 for the base model).
+5. **tools** — 1-turn (named-arg `tool_calls`) then 2-turn (tool result → answer).
+6. **vision** — OpenAI `image_url` content part only (legacy top-level `images` is silently ignored); solid color, expect the exact word; budget >=512 tokens or thinking eats it and empty `content` + `finish_reason=length` reads as blindness.
+7. **needle** at planted depth 0.85 and 0.10 of a token-budgeted haystack.
+8. **decode** — greedy, `max_tokens=512`, long-form prompt: `completion_tokens/secs`.
+9. **prefill** — one ~58k-token prompt twice: pass 1 cold, pass 2 cached; the ratio is the prompt-cache proof (measured 136x warm, 1.4x when already cached).
+
+## Laws that bite
+- **Token-budget every prompt by measurement, never chars/4.** Measure chars/token on the actual corpus (`prompt_tokens` of one probe ÷ its char count; wikitext-2 ≈ 4.33 for a HF-tokenizer Qwen-class model — chars/4 under-bills ~8 %). Over-window prompts come back HTTP 400 whose message names the exact token count and the ceiling — treat that as ground truth, it is the cheapest tokenizer oracle available.
+- **Corpus splitting:** check the separator before trusting a paragraph budget. Wikitext-2 raw blocks separate with `"\n \n"`, and a naive parquet→text dump joins rows with a SINGLE newline: a `"\n\n"`/blank-line split then returns ONE paragraph = the whole file, so every budget silently becomes "everything". Verify with `text.count("\n\n")` and the split's part count before running.
+- **Catch `HTTPError` in the POST helper and record the body.** A 400 is a probe result, not a crash — otherwise the battery dies mid-run and you re-run the whole thing against a live GPU.
+- **Report decode from a quiet host.** The same shape measured 25.6 / 37.9 / 46.8 t/s while a 62 GB download hammered page cache; only the idle-host number is real.
+- **Server-side samplers exist.** If the endpoint pins temperature/top_p/top_k, omit `temperature` for the thinking census (match production) and pin `temperature=0` only for exact-answer probes.
+
+## Verdict framing
+Behavior gates (sanity/refusals/tools/vision/needle) are the contract; PPL/KL from the engine's own checkpoint tools is advisory; throughput deltas under ~5 % are noise unless a mechanical control (a prompt shorter than the knob under test) proves the knob was the only variable.