2.8 KiB
Model Capabilities
Context Window
400,000 tokens is the fleet-wide per-slot ceiling. The harness clamps max_tokens to min(45056, contextWindow − est − 4096) to preserve MTP head and estimation buffer. With 400K context, a 45K token request leaves 355K for MTP head + estimation.
Multimodal
All three LLM endpoints accept input: [text, image]. Vision is native: Radiance and Halogen v2 load their vision towers beside the checkpoint; llama.cpp passes the BF16 mmproj via --mmproj.
Audio
No audio input in the fleet. The FLM NPU endpoint had audio 2026-09-17 → 2026-10-01, but neither MiMo nor Qwen3.5 tower carries it.
MTP (Multi-Token Prediction)
- frank-primary (Radiance): MTP spec decode depth 2,
--max-num-seqs 3. - frank-large (Halogen): MTP spec decode ALWAYS-ON (
HALOGEN_SPEC_ADAPT=0), 2 slots × 262,144 ctx, harness window 217,088 = ceiling − 45k estimate headroom. - frank-secondary (llama.cpp): MTP OFF (no
nextntensors exist in Qwen3.5-9B Q6_K GGUF).
Benchmarking Policy
STOP the production service BEFORE benchmarking. A live service starves the GPU and corrupts results.
Procedure:
- Stop and mask (self-healing would restart it):
- Dankclaw
frank-large:sudo systemctl stop frank-large && sudo systemctl mask frank-large - Dankclaw
frank-secondary:sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary - Dankserver
frank-primary:sudo systemctl stop frank-primary && sudo systemctl mask frank-primary
- Dankclaw
- Bench.
- Unmask and start; verify
curl http://192.168.0.<host>:7770/health.
CRITICAL: systemctl disable deletes the unit file from /etc/systemd/system/; mask symlinks to /dev/null and is safe/reversible. Re-deploy after benching instead of unmasking.
Gufo bench: Cannot rank speculative-decode settings; will invert the ranking. Use for shape, not for A/B. Generate at least 128 tokens — at 8 tokens, inter_token is first-token noise and run-to-run spread swamps real differences. For A/B, replay real text through the live server and read the Server-Timing header (ttft;dur=402.485 in the name half, split on = keys the dict ttft;dur).
GPU experiment law: A bare 2-rank RCCL/torch.distributed probe against a live service can wedge the GPU driver (host went hard down ~20 minutes, power cycle required). If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Distinguish: ip neigh show <ip> from a peer — FAILED means off/NIC dead, merely wedged kernel usually still answers ARP.
Live measurement: llama.cpp servers report honest per-request numbers in the /completion timings object (prefill/decode rates, draft_n, draft_n_accepted).
Scratch ports: Not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.