diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 88dddee..6cceb03 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -35,7 +35,7 @@ graph LR Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install: -- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code. +- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code. - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus. - **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking. - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it. @@ -85,21 +85,17 @@ The rule is the same as for every other service: the only off-host path is a gua | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` | | `comfyui` | `comfyui:8188` | UI probe, no native exporter | | `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes | -| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series | +| `frank-medium` | `192.168.0.235:7770` | llama.cpp on Dankserver — `llamacpp:` counters, including the spec-decode pair | | `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair | The LLM endpoints were added on 2026-09-16, after a night spent diagnosing frank-medium by hand, and re-cut the same day when Dankclaw split into two -endpoints. vLLM exports exactly what that diagnosis needed: -`vllm:kv_cache_usage_perc` (how full the per-card pool is), -`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and -the prefix-cache counters `vllm:prefix_cache_hits_total` / -`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says -whether a pinned conversation is being served warm or re-prefilled cold. What -those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]]; -llama.cpp's counters (prompt and decode throughput, KV fill, and the -`spec_decode_*` pair that proves MTP is running) are read against -[[frank-small-qwen38-gsq-rco]]. +endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium +now runs mainline llama.cpp, and the endpoint exports llama.cpp's own +counters — prompt and decode throughput, KV fill, and the `spec_decode_*` +pair that proves the DFlash2 drafter is running. What those numbers mean on +this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter +family on frank-small is read against [[frank-small-qwen38-gsq-rco]]. **`frank-large` is deliberately NOT scraped.** The halogen engine publishes `/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md new file mode 100644 index 0000000..21a81f7 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -0,0 +1,231 @@ +--- +type: configuration +title: frank-medium llama.cpp + DFlash2 Runbook +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards. +tags: + - frank-medium + - dankserver + - llama-cpp + - dflash2 + - qwen35 + - gfx1201 + - r9700 + - layer-split +timestamp: 2026-09-16 +--- + +# frank-medium — llama.cpp + DFlash2 + +The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp** +serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL` +with the **z-lab DFlash2** block-diffusion drafter, as **one process across +both Radeon AI PRO R9700 cards**, layer-split. + +It replaced the vllm-radiance MXFP4 replica pair + routing proxy +([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in +the same day the pair went live. It is also the box's **only** LLM runtime: the +replicas, the proxy and their unit files are retired. + +## Why one process and layer split + +The endpoint's whole history is one constraint: **the two cards cannot take +part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer, +and ROCr refuses to supply it to a device without PCIe atomics; the second +R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so +every collective that touches it dies with `hipErrorIllegalState`. That rules +out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**, +whose stage handoff is a `send`/`recv` of the hidden state on the device +backend (`vllm/distributed/parallel_state.py`), the very primitive that fails. +The full matrix is in [[Dual-R9700-Cooperation]]. + +llama.cpp's `--split-mode layer` needs **no collective at all**: layers are +distributed across the cards in order and the boundary activation is a plain +per-backend copy, once per forward pass. Measured on this host with the +predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on): + +| Split | pp4096 | tg128 | +|---|---:|---:| +| one R9700 (`-sm none`) | 1160.9 | 19.42 | +| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** | +| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 | + +Tensor split buys ~40 % decode and loses 7× prefill. The workload here is +prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the +shape — prefill nearly doubles and one model serves from both cards. + +## Hardware + +| Property | Value | +|---|---| +| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) | +| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it | +| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** | +| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` | +| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap | + +## Runtime + +Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by +`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624` +toolchain (TheRock nightly) that is **bundled next to the binary** — +`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1 +packages. + +Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used +earlier the same day (its tip, `c49ebdb`, predates both): + +- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise + convolution plus the candidate selector, detected from the draft GGUF + (`selector_top_k > 0`) and served through the ordinary + `--spec-type draft-dflash` path. +- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's + single trained head. + +Build with: + +```bash +bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc) +``` + +> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code +> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`, +> `../lib/hipblaslt/library/`). `cp -r SRC DST/` with a missing `DST` +> creates `DST` as a copy of `SRC` and **flattens it one level**, and the +> failure is invisible at startup: the engine loads 29 GiB of weights, answers +> `/health`, and dies on the **first request** with +> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No +> such file or directory for GPU arch : gfx1201`. The build script now declares +> the destination and copies contents. + +## Model and drafter + +| File | Bytes | Role | +|---|---:|---| +| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB | +| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB | + +Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified — +a truncated 31 GB GGUF is otherwise a load-time crash). + +The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of +the same size, on Unsloth's importance matrix) of an abliterated Swift +fine-tune: orcarouter's single refusal direction projected out of 131 tensors, +vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image +endpoint is frank-small. + +**The file also carries the model's MTP head** (`blk.64.nextn.*`). With +`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the +log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` — +because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining) +`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the +head in; it is not enabled by default because the DFlash2 path is the measured +one. + +## Deployment + +| File | Role | +|---|---| +| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments | +| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments | +| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` | + +```bash +./scripts/deploy.sh dankserver --service frank-medium +``` + +The handler verifies the runtime and payload exist (and refuses otherwise), +retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`, +waits for **port 7770 to be released** (the retired proxy held it on the host +network), relinks and restarts the unit, and gates on +`http://127.0.0.1:7770/health`. + +Deployed flags: + +| Flag | Value | Why | +|---|---|---| +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective | +| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | +| `--ctx-size` / `--parallel` | `131072` / `1` | one slot; `--parallel` **divides** the context | +| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | +| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows | +| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers | +| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint | +| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited | +| `--no-mmproj` | — | text only | + +## Verified serving behaviour (2026-09-16) + +- Load: `model loaded` ~29 s after start with a warm page cache, then + `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s. +- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports + `modalities: {vision: false}`, one slot, and the sampler defaults above. +- DFlash2 engaged — the init line is the proof, not the flag: + + ``` + common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash' + common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00 + common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true + ``` + + `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is + what distinguishes it from DFlash 1. +- Both cards hold the model: at 131072 context, `rocm-smi` reports + **17.9 GiB on GPU 0 and 22.0 GiB on GPU 1** (the drafter and the attention + layers' KV sit on the second). +- A real completion (60-token prompt, 43 output tokens incl. reasoning): + **28.3 tok/s**, `draft_n = 49`, `draft_n_accepted = 36` → **73 % acceptance**. + The reasoning trace arrives in `reasoning_content`. + +## Observability + +The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace +(`--metrics`), which is what the fleet's scrape and dashboard read — the +`vllm:` series the predecessor exported no longer exist: + +| Need | Series | +|---|---| +| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` | +| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` | +| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` | +| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) | + +`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the +drafter actually running": the flags can all be right while the draft model +failed to load. + +## Client contract + +OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and +`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces +are projected into `reasoning_content`. A request that sends no sampler fields +gets the server defaults above; a client that sends its own wins. Text only. + +## Troubleshooting + +- **Loads, then dies on the first request with a rocBLAS Tensile error** — the + bundled device-library depth (see the bundling trap above); check + `build-rdna4-rocm714/lib/rocblas/library/` exists and is populated. +- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be + `0,1`; the iGPU enumerates as device 2 and must stay hidden. +- **Drafting not engaged** — read the journal for + `common_speculative_impl_draft_dflash`; a missing drafter file makes the + serve script exit before llama-server starts, and a non-DFlash2 drafter (or + an older llama.cpp) fails the load instead. +- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is + correct for `draft-dflash`. It is not a truncated download: + `bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes. +- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a + gfx1200 sibling binary links, loads the model and then segfaults on these + cards. + +## See Also + +- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment + (replicas + proxy) and the PCIe-atomics evidence +- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on + 2026-09-15, whose `--split-mode layer` measurements this service reuses +- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a + collective, and every software escape that is closed +- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX + 3090, MTP, the image path) +- [[Dankserver System Specs]] — hardware inventory diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index ed193e1..a79e3e4 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -16,6 +16,12 @@ tags: timestamp: 2026-09-16 --- +> **SUPERSEDED 2026-09-16.** This page is the record of the retired +> vllm-radiance MXFP4 deployment. The ACTIVE frank-medium is mainline +> llama.cpp with the DFlash2 drafter ([[frank-medium-llamacpp-dflash2]]); +> the PCIe-atomics evidence in this page remains the reason the runtime is +> layer-split llama.cpp rather than tensor-parallel vLLM. + # frank-medium vllm-radiance MXFP4 Runbook The ACTIVE frank-medium deployment (since 2026-09-15/16): the diff --git a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md index 36b1bda..afbf213 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md +++ b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md @@ -16,7 +16,7 @@ timestamp: 2026-09-15 > **PRIOR deployment (retired 2026-09-16).** This page documents the > llama.cpp era of `frank-medium`, which served the endpoint for part of > 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE -> deployment is [[frank-medium-radiance-mxfp4]]; this page is kept as the +> deployment is [[frank-medium-llamacpp-dflash2]]; this page is kept as the > record of the runtime it replaced. # frank-medium ROCmFPX Runbook diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 640c8d5..4f14556 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -114,12 +114,19 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the - vllm-radiance MXFP4 stack serving - just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon - AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, plus - the PCIe-atomics evidence that rules out TP=2 on this host (since - 2026-09-15/16) +- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since + 2026-09-16: mainline llama.cpp (gfx1201 HIP build) serving + ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with + the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0) on + Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the + :7770 endpoint — no tensor parallelism, because the second card cannot + dispatch an RCCL collective +- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance + MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on + Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a + round-robin proxy; kept as the record of the deployment + [[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence + that rules out TP=2 on this host (since 2026-09-15/16) - [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix (collectives *and* P2P `send`/`recv` both fail on card #2, which rules out @@ -157,6 +164,11 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) +- [[frank-small-qwen38-gsq-rco]] — the ACTIVE frank-small runbook since + 2026-09-16: mainline llama.cpp serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO- + GGUF at IQ3_S (own MTP head + BF16 vision projector) on Dankclaw's + RTX 3090, port 7771 — the 3090 freed when halogen took the iGPU back for + frank-large - [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) - [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index 325abcf..2b4508d 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -1,7 +1,7 @@ { "uid": "llm-fleet", "title": "LLM fleet — capacity and reuse", - "description": "The signals that explain frank-medium under load, from the 2026-09-16 capacity work: per-card KV fill, queue depth, prefix-cache reuse, and request latency. frank-small (llama.cpp on Dankclaw's RTX 3090) exports throughput counters and appears in the same endpoint picker; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against.", + "description": "The signals that explain frank-medium under load, from the 2026-09-16 capacity work. The panels read the endpoint's own counters: KV fill, queue depth, prefix-cache reuse, and request latency. frank-small (llama.cpp on Dankclaw's RTX 3090) exports throughput counters and appears in the same endpoint picker; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.", "tags": [ "llm", "frank-medium", diff --git a/shared/homepage/services.json b/shared/homepage/services.json index c6aa557..b297533 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -233,7 +233,7 @@ { "name": "frank-medium", "category": "admin", - "description": "The medium LLM (Qwen3.8-27B Uncensored MXFP4, vllm-radiance) on Dankserver — two R9700s behind one :7770 endpoint.", + "description": "The medium LLM (Qwen3.8-27B Uncensored UD-Q8_K_XL with the DFlash2 drafter, mainline llama.cpp) on Dankserver — both R9700s in one layer-split process behind the :7770 endpoint.", "url": "", "docs": "", "probe": "192.168.0.235:7770",