command-center/agent/managed-skills/llm-endpoint-behavior-battery-a-b/SKILL.md

name, description

name	description
llm-endpoint-behavior-battery-a-b	A/B two boots of a live OpenAI-compatible LLM endpoint (payload swap trial) with one identical behavior+throughput battery: refusals, thinking-token census, tools, vision, needle, decode/prefill rates.

Run the SAME battery against both boots of an endpoint swap (candidate vs shipped payload). Never compare across different prompts, never benchmark beside the live service (stop + mask first), never measure the first request.

Harness

/tmp/halogen-eval/battery.py <tag> [all|tail] (stdlib only, urllib): appends one JSON line per probe to /tmp/halogen-eval/battery-<tag>.jsonl, summary to stderr. tail = long-context + throughput only. Stage assets next to it: a corpus text file, a base64 512 px solid-color PNG.

Probe order (each is a gate, all comparable across boots):

1. warmup — discarded. First request after boot is a graph-capture artifact (measured 3x spread: 2.6 vs 1258 t/s prefill).
2. sanity — exact-answer arithmetic, temperature=0.
3. refusals — 5 hostile prompts, count refusal-marker regex hits. The abliteration/base-alignment census.
4. thinking census — 6 fixed questions, temperature=1.0, max_tokens=4096; report mean usage.completion_tokens + per-item. A removed instruction fine-tune shows here first (measured: mean 784 vs the card's 3271 for the base model).
5. tools — 1-turn (named-arg tool_calls) then 2-turn (tool result → answer).
6. vision — OpenAI image_url content part only (legacy top-level images is silently ignored); solid color, expect the exact word; budget >=512 tokens or thinking eats it and empty content + finish_reason=length reads as blindness.
7. needle at planted depth 0.85 and 0.10 of a token-budgeted haystack.
8. decode — greedy, max_tokens=512, long-form prompt: completion_tokens/secs.
9. prefill — one ~58k-token prompt twice: pass 1 cold, pass 2 cached; the ratio is the prompt-cache proof (measured 136x warm, 1.4x when already cached).

Laws that bite

• Token-budget every prompt by measurement, never chars/4. Measure chars/token on the actual corpus (prompt_tokens of one probe ÷ its char count; wikitext-2 ≈ 4.33 for a HF-tokenizer Qwen-class model — chars/4 under-bills ~8 %). Over-window prompts come back HTTP 400 whose message names the exact token count and the ceiling — treat that as ground truth, it is the cheapest tokenizer oracle available.
• Corpus splitting: check the separator before trusting a paragraph budget. Wikitext-2 raw blocks separate with "\n \n", and a naive parquet→text dump joins rows with a SINGLE newline: a "\n\n"/blank-line split then returns ONE paragraph = the whole file, so every budget silently becomes "everything". Verify with text.count("\n\n") and the split's part count before running.
• Catch HTTPError in the POST helper and record the body. A 400 is a probe result, not a crash — otherwise the battery dies mid-run and you re-run the whole thing against a live GPU.
• Report decode from a quiet host. The same shape measured 25.6 / 37.9 / 46.8 t/s while a 62 GB download hammered page cache; only the idle-host number is real.
• Server-side samplers exist. If the endpoint pins temperature/top_p/top_k, omit temperature for the thinking census (match production) and pin temperature=0 only for exact-answer probes.

Verdict framing

Behavior gates (sanity/refusals/tools/vision/needle) are the contract; PPL/KL from the engine's own checkpoint tools is advisory; throughput deltas under ~5 % are noise unless a mechanical control (a prompt shorter than the knob under test) proves the knob was the only variable.