frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"uid": "llm-fleet",
|
||||
"title": "LLM fleet \u2014 capacity and reuse",
|
||||
"description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU). frank-primary (Dankserver, the Lucebox DS4.1 engine since the 2026-09-26 cutover) exports no metrics and has no panels here. The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.",
|
||||
"description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* across all three endpoints — frank-small (Dankclaw, RTX 3090), frank-large (Dankclaw, gfx1151 iGPU) and frank-primary (Dankserver, 2x R9700, llama.cpp again since the 2026-09-26 swap back from Lucebox DS4.1). The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.",
|
||||
"tags": [
|
||||
"llm",
|
||||
"capacity"
|
||||
@@ -96,7 +96,7 @@
|
||||
},
|
||||
{
|
||||
"expr": "job:llm_spec_accept_length:rate5m{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} mean accepted length (vLLM)",
|
||||
"legendFormat": "{{job}} mean accepted length",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
@@ -110,7 +110,7 @@
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "Requests waiting (queue depth)",
|
||||
"description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). vLLM: the scheduler's own waiting/running gauges against MAXSEQS=8.",
|
||||
"description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). All three endpoints are llama.cpp since the 2026-09-26 frank-primary swap.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -138,7 +138,7 @@
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "Decode and prefill throughput",
|
||||
"description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
|
||||
"description": "llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. The series carry the job label of their scrape, so the endpoint picker selects between the endpoints.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -166,7 +166,7 @@
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Decode steps and batch size",
|
||||
"description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).",
|
||||
"description": "Per-step counters: steps per second and the busy-slot count per step (the effective batch). With frank-primary's --parallel 1 the slot count stays at 0 or 1 and says nothing about the card; with frank-small's 4 slots it tracks the agent-fleet burst shape.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
|
||||
@@ -36,8 +36,8 @@
|
||||
Pick a model in the chat's model selector — the three models the fleet runs are
|
||||
described <a href="#models">below</a>. Open WebUI keeps your chat history; the raw
|
||||
model endpoints do not.</p>
|
||||
<p>There is no audio input anywhere on the fleet. Text in, and — on
|
||||
<code>frank-small</code> only — images too.</p>
|
||||
<p>Images go to <code>frank-primary</code> or <code>frank-small</code>
|
||||
(both see them). Audio (WAV) goes to <code>frank-small</code> only.</p>
|
||||
</section>
|
||||
|
||||
<section id="models">
|
||||
@@ -54,12 +54,12 @@
|
||||
<li><strong>Not for:</strong> images. It cannot see them.</li>
|
||||
</ul>
|
||||
|
||||
<h2>frank-medium — the primary model</h2>
|
||||
<h2>frank-primary — the primary model</h2>
|
||||
<ul>
|
||||
<li><strong>Model:</strong> Qwen3.8-27B-Uncensored (MXFP4/W4A8, 19.4 GiB).</li>
|
||||
<li><strong>Capacity:</strong> 262k-token context shared by up to eight conversations at once; sees images.</li>
|
||||
<li><strong>Model:</strong> Qwen3.8-Flash-Next (Q8_0, 175.3 GiB) with MTP + n-gram speculative drafting on both R9700s.</li>
|
||||
<li><strong>Capacity:</strong> 262k-token context, exactly one conversation at a time; sees images.</li>
|
||||
<li><strong>Inputs:</strong> text, image.</li>
|
||||
<li><strong>Best for:</strong> software engineering, system administration, and instruction tasks. Beats Claude Sonnet 5 per Artificial Analysis.</li>
|
||||
<li><strong>Best for:</strong> software engineering, system administration, and instruction tasks.</li>
|
||||
</ul>
|
||||
|
||||
<h2>frank-small — the fast utility model</h2>
|
||||
@@ -72,7 +72,7 @@
|
||||
|
||||
<p class="foot-note">Which one to pick? Small utility work, RAG, search, and
|
||||
screenshots: <code>frank-small</code>. Software engineering, sysadmin, and
|
||||
instruction tasks: <code>frank-medium</code>. Cybersecurity and debugging:
|
||||
instruction tasks: <code>frank-primary</code>. Cybersecurity and debugging:
|
||||
<code>frank-large</code> (with Frank's permission).</p>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -281,12 +281,12 @@
|
||||
"showLink": false
|
||||
},
|
||||
{
|
||||
"name": "frank-medium",
|
||||
"name": "frank-primary",
|
||||
"category": "user",
|
||||
"section": "AI",
|
||||
"details": {
|
||||
"role": "Primary model",
|
||||
"spec": "Qwen3.8-27B-Uncensored (MXFP4/W4A8, 19.4 GiB) — 262k-token context across up to eight conversations; sees images. Beats Claude Sonnet 5 per Artificial Analysis.",
|
||||
"spec": "Qwen3.8-Flash-Next (Q8_0, 175.3 GiB) — full 262k-token context, one conversation; sees images. Spec-drafted (MTP + n-gram) on both R9700s.",
|
||||
"inputs": ["text", "image"],
|
||||
"use": "Software engineering, system administration, and instruction tasks."
|
||||
},
|
||||
|
||||
@@ -79,35 +79,46 @@ scrape_configs:
|
||||
# this container, not the guarded vhosts: these are machine clients and
|
||||
# Prometheus cannot solve an Anubis challenge.
|
||||
#
|
||||
# One metric namespace: llamacpp:* — frank-large and frank-small, both
|
||||
# llama.cpp with --metrics and both verified 2026-09-20 to export
|
||||
# One metric namespace: llamacpp:* — frank-large, frank-small and
|
||||
# frank-primary, all llama.cpp with /metrics (the family verified from
|
||||
# /metrics on 2026-09-16 and 2026-09-20) and all exporting
|
||||
# requests_deferred (the queue depth, the only form llama.cpp shows the
|
||||
# capacity wall in), spec_decode_num_drafts_total, tokens_predicted_total,
|
||||
# and the prompt_tokens_cached_total / prompt_tokens_total reuse pair.
|
||||
#
|
||||
# frank-large is scraped over host.containers.internal, not its own LAN address:
|
||||
# this Prometheus is a bridge-networked container (Network=dankclaw_default),
|
||||
# and on this netavark host a container can only reach the host's LAN IP at a
|
||||
# PUBLISHED port — a host-process bind like llama-server's 0.0.0.0:7770 is
|
||||
# refused at 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
|
||||
# frank-large and frank-small are scraped over host.containers.internal,
|
||||
# not their own LAN addresses: this Prometheus is a bridge-networked
|
||||
# container (Network=dankclaw_default), and on this netavark host a
|
||||
# container can only reach the host's LAN IP at a PUBLISHED port — a
|
||||
# host-process bind like llama-server's 0.0.0.0:7770 is refused at
|
||||
# 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
|
||||
# why frank-small was red here for days (up{job="frank-small"} == 0,
|
||||
# "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
|
||||
# exporting: the target address was unreachable by construction, not the
|
||||
# service down. Same host, same rule, so both local endpoints use the gateway.
|
||||
# (2026-09-26) frank-medium is replaced by frank-primary (lucebox DS4.1),
|
||||
# which exposes no metrics endpoint (/metrics -> 404), so there is no job
|
||||
# for it here. The endpoint's liveness is covered by the homepage probe;
|
||||
# the documented remedy is a blackbox exporter, which is not deployed.
|
||||
# service down. Same host, same rule, so both local endpoints use the
|
||||
# gateway.
|
||||
#
|
||||
# (2026-09-26) frank-primary is llama.cpp again (the unsloth b11030-mix
|
||||
# prebuilt serving Qwen3.8-Flash-Next Q8_0) on Dankserver, a DIFFERENT
|
||||
# machine: the netavark published-port quirk above is same-host only, so
|
||||
# plain LAN routing reaches 192.168.0.235:7770 and the job returns.
|
||||
- job_name: "frank-small"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["host.containers.internal:7771"]
|
||||
labels:
|
||||
service: frank-small
|
||||
|
||||
|
||||
- job_name: "frank-large"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["host.containers.internal:7770"]
|
||||
labels:
|
||||
service: frank-large
|
||||
|
||||
- job_name: "frank-primary"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:7770"]
|
||||
labels:
|
||||
service: frank-primary
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
# Alerting and recording rules for the LLM fleet.
|
||||
#
|
||||
# One metric namespace: llamacpp:* — frank-small and frank-large, both
|
||||
# llama.cpp with --metrics (the family verified from /metrics on 2026-09-16
|
||||
# and 2026-09-20). frank-primary (Dankserver, the Lucebox DS4.1 engine since
|
||||
# the 2026-09-26 cutover) answers 404 on /metrics, so nothing here selects
|
||||
# it; its liveness is covered by the homepage probe, and the documented
|
||||
# remedy is a blackbox exporter (not deployed).
|
||||
# One metric namespace: llamacpp:* — frank-small, frank-large and
|
||||
# frank-primary, all llama.cpp with /metrics (the family verified from
|
||||
# /metrics on 2026-09-16 and 2026-09-20). frank-primary is scraped across
|
||||
# the LAN at 192.168.0.235:7770 (the netavark published-port quirk that
|
||||
# forces the gateway for the Dankclaw endpoints is same-host only).
|
||||
#
|
||||
# Every expression is keyed to a series that EXISTS. Nothing matches on
|
||||
# absence: when an endpoint's runtime leaves the fleet, its selectors are
|
||||
@@ -111,7 +110,6 @@ groups:
|
||||
description: >-
|
||||
Prometheus cannot reach the LLM endpoint. frank-small and
|
||||
frank-large are root units on Dankclaw: `systemctl is-active
|
||||
frank-small` / `frank-large`. Both scraped endpoints export the
|
||||
llamacpp:* family. frank-primary (Dankserver) is not scraped —
|
||||
its runtime exports no metrics; check the homepage probe or
|
||||
`systemctl --user is-active frank-primary` on Dankserver.
|
||||
frank-small` / `frank-large`. frank-primary is a root unit on
|
||||
Dankserver: `systemctl is-active frank-primary`. All three
|
||||
scraped endpoints export the llamacpp:* family.
|
||||
|
||||
Reference in New Issue
Block a user