frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
This commit is contained in:
@@ -0,0 +1,135 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
|
||||
description: ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
|
||||
tags:
|
||||
- frank-primary
|
||||
- dankserver
|
||||
- llamacpp
|
||||
- unsloth
|
||||
- qwen38-flash-next
|
||||
- q8_0
|
||||
- mtp
|
||||
- ngram-mod
|
||||
- spec-decode
|
||||
- vision
|
||||
- rocm
|
||||
- gfx1201
|
||||
- r9700
|
||||
timestamp: 2026-09-26
|
||||
---
|
||||
|
||||
# frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
|
||||
|
||||
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**):
|
||||
unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
|
||||
(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
|
||||
with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
|
||||
**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the
|
||||
Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|
||||
([[frank-primary-lucebox-ds4]]), which itself replaced
|
||||
[[frank-medium-radiance-mxfp4]].
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **One runtime for the fleet.** All three endpoints are llama.cpp now —
|
||||
one metric namespace (`llamacpp:*`), one reasoning contract
|
||||
(`reasoning_content`), one tool contract (`--jinja`). The two-day
|
||||
experiment with a single-vendor engine (lucebox) ended because Qwen
|
||||
Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1
|
||||
fp2/fp3 stream, plus vision, plus the full 262k window, at comparable
|
||||
speed.
|
||||
- **The placement law, not a preference:** the routed experts and the
|
||||
51B-param n-gram (PLE) embedding table are **computed in host RAM**
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`) and never streamed over PCIe. The
|
||||
x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
|
||||
computing in RAM does not care (20+ t/s). The dense weights, attention,
|
||||
projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
|
||||
expert pages live in mmap page cache — `--no-mmap` is forbidden (it
|
||||
forces the table into anonymous memory; the cgroup kill is silent) and
|
||||
pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
|
||||
- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
|
||||
ports with full PCIe atomics (the ProArt B850-Creator swap,
|
||||
2026-09-20), but this deployment uses llama.cpp's layer split —
|
||||
`-ot` tensor overrides do not compose with tensor-parallel mode, and
|
||||
`exps=CPU` is the override the placement law needs. The 27B-dense
|
||||
equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in
|
||||
this shape; Flash-Next Q8 with CPU experts lands in the same class
|
||||
(measured numbers below).
|
||||
- **Spec chain: `draft-mtp,ngram-mod`.** MTP is a property of the FILE:
|
||||
the shared-Q8_0 head (`mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`,
|
||||
2,786,568,256 B) is loaded via `--model-draft` and borrows the target's
|
||||
embedding tensors — which is why `per_layer_token_embd.weight` must be
|
||||
CPU-resident in the override set (a GPU-resident borrow crashes the
|
||||
draft load with `invalid vector subscript`). `ngram-mod` is draftless,
|
||||
so it chains legally behind the drafter. Depth 2: each longer
|
||||
verification round re-batches the CPU expert reads, so depth buys less
|
||||
than bandwidth here.
|
||||
- **Vision, explicitly.** This build never autoloads projectors — without
|
||||
`--mmproj` the server comes up healthy and text-only and answers image
|
||||
requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector
|
||||
(907,542,944 B) is named in the serve script.
|
||||
|
||||
## Deployment
|
||||
|
||||
|Piece|Value|
|
||||
|---|---|
|
||||
|Unit|`scripts/frank-primary.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|
||||
|Serve script|`scripts/serve-frank-primary.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|
||||
|Env tunables|`scripts/frank-primary.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|
||||
|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`|
|
||||
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-primary.sh` (sha256 + byte-count verified)|
|
||||
|Port / alias|7770 / `frank-primary` — unchanged from the lucebox and radiance eras; no client moves|
|
||||
|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)|
|
||||
|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request|
|
||||
|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)|
|
||||
|Observability|`frank-primary` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
|
||||
|
||||
### Cutover mechanics
|
||||
|
||||
The deployer (`deploy_llm_primary` in `scripts/deploy-remote.sh`) gates on
|
||||
every artifact at its declared byte count (`stat -c %s`, never `wc -c` —
|
||||
which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
|
||||
retires the lucebox user quadlet to dated home trash
|
||||
(`~/.AGENTTRASH/20260926-llamacpp-cutover/`), waits for port 7770 to be
|
||||
truly free, relinks + enables + restarts the root unit, and waits up to
|
||||
1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
|
||||
|
||||
## Verification (2026-09-26 deploy)
|
||||
|
||||
- `/v1/models` reports `frank-primary`.
|
||||
- Chat completion returns `content` + `reasoning_content`.
|
||||
- Tool call returns well-formed `tool_calls` (`--jinja`).
|
||||
- Real image request answered from the projector (not a health check —
|
||||
health cannot see vision).
|
||||
- Spec chain live: `draft acceptance` lines in the unit journal; ngram-mod
|
||||
has no startup banner (identify behaviourally: `draft_n` > 0 where text
|
||||
repeats).
|
||||
- `reasoning_effort: high` → 200 (the template's `{xhigh, medium, low}`
|
||||
gate is folded by `reasoningEffortMap` in `models.yml`).
|
||||
- VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long
|
||||
completion.
|
||||
|
||||
_Measured numbers land here after the deploy battery._
|
||||
|
||||
## Rollback
|
||||
|
||||
The lucebox DS4.1 material is fully retained on the host: image
|
||||
`localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era
|
||||
repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
|
||||
[[frank-primary-lucebox-ds4]]). Restore = move the `.container` back,
|
||||
uninstall the root unit (`systemctl disable --now frank-primary` — the
|
||||
link, not the file; never `systemctl disable` a unit whose file must
|
||||
survive), `daemon-reload`, start the user quadlet. The Qwen payload
|
||||
(~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a
|
||||
refetch.
|
||||
|
||||
## Fleet context
|
||||
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
|
||||
b11030-mix binary family on gfx1151.
|
||||
- [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
|
||||
3090; the only audio-capable endpoint.
|
||||
- The harness contract lives in `/home/blank/.omp/agent/models.yml`
|
||||
(`llamacpp (dankserver)` provider, `contextWindow 262144`,
|
||||
`maxTokens 65536`, `input: [text, image]`).
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-primary Lucebox DeepSeek V4 Runbook
|
||||
description: ACTIVE frank-primary runbook — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaces frank-medium (vllm-radiance TP=2) on :7770, cutover 2026-09-26.
|
||||
description: RETIRED 2026-09-26 frank-primary runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-primary-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
|
||||
tags:
|
||||
- frank-primary
|
||||
- dankserver
|
||||
@@ -18,14 +18,22 @@ timestamp: 2026-09-26
|
||||
|
||||
# frank-primary Lucebox DeepSeek V4 Runbook
|
||||
|
||||
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): the
|
||||
> [!warning] RETIRED 2026-09-26 (same day it went live)
|
||||
> Replaced by [[frank-primary-llamacpp-qwen38-flash-next-q8]] — unsloth
|
||||
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
|
||||
> served for hours; the rollback material (image `localhost/lucebox-rocm:
|
||||
> gfx1201` and the 378 GiB payload) is retained on the host, and the
|
||||
> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`.
|
||||
|
||||
The dankserver LLM endpoint (:7770, alias **frank-primary**) as it stood on
|
||||
2026-09-26: the
|
||||
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
|
||||
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
|
||||
bytes, LFS oid
|
||||
`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
|
||||
single Radeon AI PRO R9700 (gfx1201). It replaces
|
||||
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired on
|
||||
2026-09-26.
|
||||
single Radeon AI PRO R9700 (gfx1201). It replaced
|
||||
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the
|
||||
same day.
|
||||
|
||||
## Why this shape
|
||||
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-21):
|
||||
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
|
||||
process across both Radeon AI PRO R9700 cards (just1moremodel
|
||||
Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
|
||||
R4D one-shot allreduce), and TWO endpoints on Dankclaw — stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151; frank-large, unsloth Qwen3.8-27B UD-Q8_K_XL +
|
||||
the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, port 7770) and mainline
|
||||
llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
|
||||
the RTX 3090, port 7771).
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-26):
|
||||
**llama.cpp on all three endpoints** — Dankserver runs unsloth llama.cpp
|
||||
(the b11030-mix prebuilt, ROCm gfx120X) serving frank-primary
|
||||
(unsloth Qwen3.8-Flash-Next Q8_0 + the shared Q8_0 MTP head + ngram-mod +
|
||||
vision) across both Radeon AI PRO R9700s on :7770, and Dankclaw runs two
|
||||
endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large,
|
||||
unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the
|
||||
gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS
|
||||
Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -22,11 +22,10 @@ graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[vllm-radiance 0.9.3, TP=2
|
||||
frank-medium, port 7770]
|
||||
C[Qwen3.8-27B MXFP4 + DFlash2 drafter
|
||||
ONE process on both cards, 262144 ctx, up to 8 seqs]
|
||||
A --> B2 --> C
|
||||
B2[unsloth llama.cpp b11030-mix, ROCm gfx120X
|
||||
frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next Q8_0 + shared-Q8_0 MTP head
|
||||
layer split on both cards, 262144 ctx / 1 slot, ngram-mod, vision]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -53,20 +52,34 @@ frank-small, port 7771]
|
||||
|
||||
| Server | GPU | VRAM / Memory | OS |
|
||||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 |
|
||||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
|
||||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
|
||||
vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the
|
||||
z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind
|
||||
:7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test
|
||||
whether the endpoint's historical fp8 KV had been degrading W4A8 quality).
|
||||
The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21
|
||||
shape) is the on-disk rollback. The three-arm battery that chose TP=2, the
|
||||
auto-sized KV pool, and verified capabilities (tools, reasoning)
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE
|
||||
frank-primary runbook (2026-09-26, root systemd unit, port 7770):
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
routed experts and the 51B-param n-gram table computed in host RAM
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`), the shared Q8_0 MTP head
|
||||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||||
conversation, `reasoning_content`, the placement law, the cutover and
|
||||
rollback
|
||||
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||||
rollback target; active for hours): the Lucebox DS4.1 engine
|
||||
(localhost/lucebox-rocm:gfx1201) serving
|
||||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||||
the fleet's only non-llama.cpp runtime detour
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||||
[[frank-primary-lucebox-ds4]], then
|
||||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||||
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
|
||||
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
|
||||
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
|
||||
2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
|
||||
battery that chose TP=2, the auto-sized KV pool, and the verified
|
||||
capability battery (tools, reasoning)
|
||||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||||
@@ -181,9 +194,9 @@ frank-small, port 7771]
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
|
||||
Kairic Edge runtime since 2026-09-21, frank-small on the RTX 3090)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||||
the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
|
||||
@@ -124,12 +124,18 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-primary-lucebox-ds4]] — the ACTIVE frank-primary runbook:
|
||||
the Lucebox runtime serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB
|
||||
GGUF, 43-layer MoE, 256 routed experts top-6) on Dankserver's R9700
|
||||
with the routed experts computed on CPU from the mmapped page cache —
|
||||
replaces frank-medium on :7770 (cutover 2026-09-26)
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
|
||||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — the ACTIVE
|
||||
frank-primary runbook: unsloth llama.cpp (b11030-mix prebuilt, ROCm
|
||||
gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
|
||||
GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
|
||||
behind ngram-mod, BF16 vision, the experts and the 51B n-gram table
|
||||
computed in host RAM, the placement law, the cutover, and the rollback
|
||||
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
|
||||
rollback target): the Lucebox runtime serving
|
||||
DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed
|
||||
experts top-6) on Dankserver's R9700 with the routed experts computed
|
||||
on CPU from the mmapped page cache — held :7770 for hours
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
|
||||
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
|
||||
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
|
||||
W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
|
||||
|
||||
Reference in New Issue
Block a user