Frank
8500600d09
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
149 lines
7.0 KiB
Markdown
149 lines
7.0 KiB
Markdown
---
|
|
type: configuration
|
|
title: frank-primary Lucebox DeepSeek V4 Runbook
|
|
description: RETIRED 2026-09-26 frank-primary runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-primary-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
|
|
tags:
|
|
- frank-primary
|
|
- dankserver
|
|
- lucebox
|
|
- deepseek-v4
|
|
- ds4
|
|
- rocmfp23
|
|
- gfx1201
|
|
- r9700
|
|
- quadlet
|
|
- moe
|
|
timestamp: 2026-09-26
|
|
---
|
|
|
|
# frank-primary Lucebox DeepSeek V4 Runbook
|
|
|
|
> [!warning] RETIRED 2026-09-26 (same day it went live)
|
|
> Replaced by [[frank-primary-llamacpp-qwen38-flash-next-q8]] — unsloth
|
|
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
|
|
> served for hours; the rollback material (image `localhost/lucebox-rocm:
|
|
> gfx1201` and the 378 GiB payload) is retained on the host, and the
|
|
> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`.
|
|
|
|
The dankserver LLM endpoint (:7770, alias **frank-primary**) as it stood on
|
|
2026-09-26: the
|
|
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
|
|
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
|
|
bytes, LFS oid
|
|
`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
|
|
single Radeon AI PRO R9700 (gfx1201). It replaced
|
|
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the
|
|
same day.
|
|
|
|
## Why this shape
|
|
|
|
DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
|
|
per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
|
|
down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
|
|
the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
|
|
so it is served the way llama.cpp serves big models by default: the GGUF is
|
|
mmapped, pages pull from disk/page cache on demand, dense work (attention,
|
|
shared expert, embeddings, output projection) and the MLA KV cache run on
|
|
the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
|
|
(upgraded for this deployment) holds the working set of active experts; the
|
|
rest streams from the 1.8 T NVMe.
|
|
|
|
Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
|
|
this size that allocation fails and the backend falls back to **hybrid
|
|
expert placement + layered decode** — that fallback is the intended serving
|
|
mode here, not an error.
|
|
|
|
## Image
|
|
|
|
`localhost/lucebox-rocm:gfx1201`, built on the host from
|
|
`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
|
|
|
|
```
|
|
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
|
|
cd lucebox
|
|
# Three local modifications (repo-tracked at
|
|
# dankserver/frank-primary/lucebox-gfx1201.patch):
|
|
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
|
|
# base already ships them in /opt/rocm; the apt repo does not serve
|
|
# them) and the three runtime math libs.
|
|
# 2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
|
|
# 3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
|
|
# requires, CMake >= 3.25).
|
|
# 3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
|
|
# `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
|
|
# unconditionally, but the macro is never defined by the build
|
|
# system, so the in-guard definition was dead and the use failed.
|
|
git apply ../lucebox-gfx1201.patch
|
|
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
|
|
--build-arg ROCM_VERSION=7.14.1-full \
|
|
-t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
|
|
```
|
|
|
|
- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
|
|
it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
|
|
code-object compatible.
|
|
- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
|
|
radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
|
|
on a 7.x driver can segfault at model load. Only `-full` tags exist for
|
|
7.14.x.
|
|
- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
|
|
|
|
## Model
|
|
|
|
`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
|
|
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
|
|
400 against HF's xethub CDN for this repo; see the fetch note in
|
|
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
|
|
`deploy_llm_primary`.
|
|
|
|
## Deployed flags (2026-09-26)
|
|
|
|
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (the unit
|
|
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
|
|
flag set and cannot express the DS4 flags):
|
|
|
|
| Setting | Value |
|
|
|---|---|
|
|
| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
|
|
| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
|
|
| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
|
|
| Chunk | `--chunk 8192` (the qualified profile's chunk) |
|
|
| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
|
|
| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
|
|
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
|
|
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
|
|
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
|
|
| Identity | `--model-name frank-primary`, `--host 0.0.0.0 --port 7770` |
|
|
|
|
## API surface
|
|
|
|
OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
|
|
`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
|
|
metadata — the fleet contract is unchanged from frank-medium.
|
|
|
|
## Cutover procedure
|
|
|
|
1. Image built and model fetched (gates in `deploy_llm_primary`).
|
|
2. `deploy.sh` → `deploy_llm_primary`: stops and **masks** the radiance
|
|
unit (`frank-medium`), waits for :7770 to free, installs
|
|
`dankserver/frank-primary/quadlet/frank-primary.container`, starts
|
|
`frank-primary.service`, waits for health (1800 s start period — first
|
|
start page-ins the dense weights and builds the graphs).
|
|
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
|
|
call, and an Open WebUI completion.
|
|
4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
|
|
checkpoint payload stays on disk.
|
|
|
|
## Rollback
|
|
|
|
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
|
|
systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
|
`localhost/radiance-vllm:0.13.0-e060895` image and the
|
|
`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
|
|
|
|
## Verification (2026-09-26)
|
|
|
|
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
|
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
|
state._
|