39 lines
3.2 KiB
Markdown
39 lines
3.2 KiB
Markdown
## LLM Endpoint Inventory
|
||
|
||
### frank-secondary (Dankclaw, port 7771)
|
||
|
||
**Current**: Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K (llama.cpp, RTX 3090, 200,704-token slots, vision via BF16 mmproj).
|
||
|
||
**Target**: Underdog-Saluki-27B-1.0 (llama.cpp, RTX 3090, 400,000-token slots, vision via BF16 mmproj).
|
||
|
||
Both run on the same hardware with the same runtime stack. The swap is model-only; container image and systemd unit remain unchanged. Vision support persists via the publisher's BF16 mmproj pass-through (`--mmproj`).
|
||
|
||
**Context window law**: 400,000 tokens is the per-slot hard ceiling (`--ctx-size 401408 --parallel 2`). The harness clamps to `min(45056, contextWindow − est − 4096)` to keep room for the MTP head. With 400K context, a 45K token request leaves 355K for MTP head + estimation buffer.
|
||
|
||
### frank-primary (Dankserver, port 7770)
|
||
|
||
**Radiance 1.0.13**, ukisai/Swift1.5-Qwen3.8-Flash-Next BF16, 49,152 experts GPTQ w4 + 24 bf16-protected experts, 200K max-model-len, 47.7 GiB n-gram pinned to host RAM, 2 R9700s TP=2 (`--tp-wire wht6`), MTP spec decode depth 2, `--max-num-seqs 3`, `--max-model-len 200000`.
|
||
|
||
### frank-large (Dankclaw, port 7770)
|
||
|
||
**Halogen-flash-server 0.16.2**, Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, 262,144 context per-slot, 2 slots × 262,144 ctx, MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0`), vision ON (v2 tower loads beside checkpoint), 524,288 positions KV pool.
|
||
|
||
## Runtime Stack
|
||
|
||
- **frank-primary**: Radiance 1.0.13 (StillDeadcode, digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`), plain `podman run`, root systemd unit, 2 R9700s TP=2.
|
||
- **frank-large**: Halogen-flash-server 0.16.2 (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`), digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit, Strix Halo iGPU.
|
||
- **frank-secondary**: llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build, root systemd unit, RTX 3090.
|
||
|
||
All LLM services are root systemd units with `Restart=always`. Container exit codes drive service restarts (`ExecStop=podman stop -t 60 <name>`). Boot persistence requires quadlet units in `~/.config/containers/systemd/<name>.container`, enabled in user manager, with linger on for the deploy user.
|
||
|
||
## Model Config Resolution
|
||
|
||
Official model ids are StarCraft unit names: `frank-archon` → `192.168.0.235:7770` (`FRANK_PRIMARY_SERVED_NAME`), `frank-dragoon` → `192.168.0.143:7770` (`HALOGEN_MODEL_ID`), `frank-zealot` → `192.168.0.143:7771` (`ALIAS`).
|
||
|
||
**Per-slot vs pool**: llama.cpp reports per-slot window (`--ctx-size 401408 --parallel 2`), not the pool. A 2×200K pool means 200K per slot, not 400K total.
|
||
|
||
**Reasoning field**: llama.cpp and SGLang emit `reasoning_content`; vLLM emits `reasoning`. `models.yml` sets `reasoningContentField` per endpoint to match.
|
||
|
||
**Vision**: All three endpoints accept `input: [text, image]`. Radiance runs the checkpoint's 27-layer Qwen-VL tower (patching, pad-token expansion, 3-part rotary, `--mm-max-patches` available). Halogen v2 loads the vision tower beside the checkpoint. Llama.cpp passes the model's BF16 mmproj via `--mmproj`.
|
||
|
||
**Audio**: No audio input anywhere in the fleet (left with FLM NPU endpoint 2026-09-11, returned 2026-09-17, left again 2026-10-01). |