frank-small: revert vLLM cutover — back to llama.cpp Gemma4-12B Q4_K_M
The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
This commit is contained in:
Frank committed
1 parent
bab8f7f839
commit
374041cd7a
15 files changed
+495
-637
No files matched your search
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
|
||||
the gfx1151 iGPU) and `frank-small` (vLLM on the RTX 3090). Caddy on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | vLLM `gemma-x86_64-cu129` dev build + audio extra (rootless Quadlet) | google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin) + Google's bf16 MTP assistant on the RTX 3090, 262144 window / 289,647-token bf16 KV pool / 2 full-context sessions, native text+image+audio — the fleet's multimodal endpoint | 7771 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -68,8 +68,8 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — frank-large and frank-small by their `dankclaw/`
|
||||
quadlet/Dockerfile directories, frank-medium by its unit/env/serve trio in `scripts/` — so unrelated
|
||||
their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
|
||||
and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
|
||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||
runs on a host without passwordless sudo; the value never enters the repo.
|
||||
|
||||
@@ -232,10 +232,8 @@ merely wedged kernel usually still answers ARP.
|
||||
`~/.config/containers/systemd/frank-large.container` aside and
|
||||
`systemctl --user daemon-reload` — the quadlet generator re-creates that
|
||||
unit from the file on every reload, so the source is the lever, not `mask`.
|
||||
- Dankclaw `frank-small` (rootless Quadlet since 2026-09-19, vLLM on the 3090):
|
||||
`systemctl --user stop frank-small`, then move its
|
||||
`~/.config/containers/systemd/frank-small.container` aside and
|
||||
`systemctl --user daemon-reload` (same quadlet-source lever as frank-large).
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-medium` (system unit):
|
||||
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
|
||||
2. Bench.
|
||||
@@ -265,10 +263,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-19 `frank-small` is the fleet's multimodal endpoint
|
||||
(`input: [text, image, audio]`) — vLLM's native Gemma4 vision AND audio
|
||||
towers, both verified live. `frank-large` is text-only because its halogen
|
||||
deployment loads no vision tower.
|
||||
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
|
||||
`frank-large` is text-only because its halogen deployment loads no vision
|
||||
tower. There is **no audio** input anywhere in the fleet: that capability left
|
||||
with the FLM NPU endpoint on 2026-09-11.
|
||||
|
||||
### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -303,37 +301,31 @@ that as "tight and fragmented", not "out of memory".
|
||||
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
Rollback: [[frank-large-ik-uncensored]].
|
||||
|
||||
### frank-small — vLLM on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (digest-pinned, audio
|
||||
extra pip-installed into `localhost/vllm/vllm-openai:frank-small` — see
|
||||
`dankclaw/frank-small/Dockerfile`) as a **rootless Quadlet**
|
||||
(`dankclaw/frank-small/quadlet/frank-small.container`), serving
|
||||
`google/gemma-4-12B-it-qat-w4a16-ct` (compressed-tensors w4a16, int4 Marlin
|
||||
weights, 10.26 GiB) with Google's **bf16** MTP assistant
|
||||
(`google/gemma-4-12B-it-assistant`, 845 MB). Port 7771, 262,144 window,
|
||||
bf16 KV pool of 289,647 tokens, **2 concurrent full-context sessions**
|
||||
(the 3rd queues), CUDA graphs on, `ENFORCE_EAGER=0`.
|
||||
Mainline llama.cpp master `37b53fd` (CUDA build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
|
||||
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
|
||||
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
|
||||
(four FULL 262,144-token slots — the measured card ceiling), batch
|
||||
**8192/1024**, greedy default. Deployed flags are in
|
||||
`scripts/frank-small.env`; the serve script refuses to start if the
|
||||
weights, the projector, the sidecar or the binary are missing.
|
||||
|
||||
The w4a16-ct assistant CANNOT draft in vLLM (the drafter path skips the
|
||||
Marlin kernel — packed int4 misread, acceptance 1.00); the bf16 assistant
|
||||
measures acceptance 2.80–3.56. fp8 KV is hardware-dead on sm_86 (Ampere
|
||||
has no fp8 tensor cores), so bf16 is the only option.
|
||||
|
||||
Dankclaw has **no nvidia-container-toolkit**: the unit wires CUDA by hand
|
||||
(`Device=` for the four nvidia nodes + host `libcuda`/`libnvidia-ml`
|
||||
mounted read-only into two library paths). Host driver is
|
||||
`nvidia-driver-610` 610.57.04 on Ubuntu 26.04; `sm_86` is the correct
|
||||
compute capability — no arch override.
|
||||
|
||||
Verified at deploy (2026-09-19): `/v1/models` reports `frank-small`; chat
|
||||
returns content plus `reasoning` (vLLM's gemma4 parser field — the harness
|
||||
maps it via `compat.reasoningContentField`); a tool call returns
|
||||
`tool_calls`; an image is described; a 440 Hz WAV is described (native
|
||||
audio encoder). Measured: pp@67k 1,293 t/s, pp@213k 608 t/s,
|
||||
tg@213k ~13.6 t/s (bandwidth-bound), tg@85k ~17.6 t/s.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md`.
|
||||
Rollback: [[frank-small-gemma4-12b]] (llama.cpp, weights still on host).
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; an image request is answered from the projector;
|
||||
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
|
||||
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
|
||||
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
|
||||
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
100 is WORSE at long ctx) is tabled in the runbook.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
|
||||
|
||||
@@ -715,7 +707,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-small (vLLM Gemma4-12B w4a16, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -727,7 +719,7 @@ Retired (rollback targets and history):
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 09-19 | `frank-small-gemma4-12b.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
Reference in new issue
Block a user