Commit Graph
3 Commits
Author SHA1 Message Date
Frank 43ec77fee6 inference: frank-large returns to halogen, frank-small takes the 3090
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.

frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.

The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.

Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
2026-09-16 13:58:00 -07:00
Frank 6b647c1558 frank-medium: swap ROCmFPX llama.cpp runtime for vllm-radiance MXFP4
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).

- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
  serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
  the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
  mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
  with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
2026-09-15 20:59:55 -07:00
Frank 69f87c8483 feat(llm): rename inference services dropping 'default' from names
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).

Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
2026-09-03 16:15:43 -07:00