command-center/agent/managed-skills/fleet-second-llm-endpoint/SKILL.md

name, description

name	description
fleet-second-llm-endpoint	Add a second LLM endpoint to a host that already serves one on the Dank fleet (new port, unit shape, co-tenancy budget, harness provider/roles, monitoring, cutover, verification) — with the port-bind race, the reasoning_effort template trap, and the tools/--jinja trap that silently break subagents.

Adding a second LLM endpoint to a Dank fleet host

Use when a host must serve two models at once (e.g. an iGPU runtime plus a second card), or when one endpoint is moving and a new one takes the freed hardware. Rebuilt from the 2026-09-16 Dankclaw split (halogen frank-large on :7770 + llama.cpp frank-small on :7771).

0. Decide the shape before touching hardware

• One provider per HOST:PORT in ~/.omp/agent/models.yml (on the box where the harness runs, not on the servers). A second endpoint is a new provider block, not a second model in the existing one.
• Read the budget as an inequality, not a wish. For a unified-memory runtime the engine's own fit check is the arbiter (budget = host − pinned weights − reserve; need = KV + arena + vision); for a discrete card, weight the KV by counting only the full-attention layers — and remember --parallel N DIVIDES --ctx-size, it does not multiply.
• A co-tenant is a budget line, not a footnote. Free host RAM by changing the POOL or the ARENA (the reserve is a fit-check term, never an allocation), and drop optional charges (a vision tower is ~2 GiB) rather than hoping.

1. Repo first, and retire in the same commit

1. New service files: scripts/<svc>.env (documented tunables), .service (root unit for a native binary; a Quadlet only for a container), serve-<svc>.sh (defaults + hard guards for missing artifacts).
2. Planner: add the service to SERVICE_ORDER AND the target's service set in scripts/deploy-plan.py, map its paths (including "the unit/env/serve trio selects only me"), add it to the inventories in deploy.sh and deploy-remote.sh, and the dispatch case.
3. Retire what the cutover obsoletes into .AGENTTRASH/ with a suffix — and check what pointed at it: bench harnesses, a chat template only the old runtime read, and scripts/test-changed.sh mappings are the usual victims (a mapping to a test file retired earlier fails the pre-commit wrapper silently for weeks).
4. Seams: one bats file per service asserting the invariants (pool == 2 × ctx, KV bytes ≤ card budget, port separation), not just constant values. A negative grep must strip comments (grep -vE '^\s*#' file | grep -q ...) or it will match the prose that explains the very thing you removed.

2. The port-bind race (rootless podman)

pasta binds the published port at container CREATE. A predecessor that has just stopped still holds it, so a naive handler starts a doomed container:

Deactivated successfully          <- predecessor gone
pasta: Listen failed for HOST TCP port */PORT: Address already in use
exit 126                          <- only Restart=on-failure saves you

Order the handler: retire the old manager's unit → stop THIS service's own unit (no-op on a first cutover) → wait for every socket on the port to clear (listener, accepted, TIME_WAIT) → install → start. Without the second step a redeploy deadlocks against its own gate.

3. Wire the surfaces, and refuse impossible signals

• Harness roles: point the small/fast + image roles at the new endpoint, and move the vision role with the vision tower (only one endpoint should own each capability).
• shared/homepage/services.json: a card per endpoint, probed from the monitor's own host (loopback is legal there; the LAN address otherwise).
• Prometheus: only add a job if the runtime actually exposes Prometheus text. Halogen publishes /health, /cache and /v1/* and nothing else — scraping it would be red by construction, so the job is REMOVED with the reason written down rather than left as a permanent red.

4. Cutover, then verify by driving it

./scripts/deploy.sh <target> --service <large> --service <small>   # order matters

Then, per endpoint: /health; /v1/models reports the alias; a chat completion returns content AND reasoning_content; a request carrying tools returns tool_calls; an image request is answered (or refused with a named reason); the runtime's own speculation counters move; and the unit is enabled / generated with linger on. Read the engine's boot ledger for a silent clamp.

Finally spend one real request through the harness per changed role — routing labels lie, the servers' own journals do not: journalctl -u <svc> shows which endpoint actually served it.

Traps that look like model faults

symptom	cause
500 tools param requires --jinja flag	the endpoint's runtime lacks the Jinja chat-template path, so every tool-using agent on it is broken. mainline llama.cpp defaults it on — pin it
500 Jinja Exception: Unexpected reasoning effort <level>	the model's own template has a narrower accepted set (Qwen3.8: xhigh/medium/low only) than the harness's levels; fold them in compat.reasoningEffortMap
the same request works by hand but a subagent dies	subagents run at a different thinking level — check both
the container dies instantly although the predecessor "stopped"	the port race above