frank-primary Lucebox DeepSeek V4 Runbook

The ACTIVE dankserver LLM endpoint (:7770, alias frank-primary): the Lucebox runtime serving Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF (377.2 GiB, 405,349,859,328 bytes, LFS oid 2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2) on a single Radeon AI PRO R9700 (gfx1201). It replaces frank-medium-radiance-mxfp4 (vllm-radiance TP=2), retired on 2026-09-26.

Why this shape

DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3 down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM, so it is served the way llama.cpp serves big models by default: the GGUF is mmapped, pages pull from disk/page cache on demand, dense work (attention, shared expert, embeddings, output projection) and the MLA KV cache run on the R9700, and the routed experts are computed on CPU. The 246 GiB RAM (upgraded for this deployment) holds the working set of active experts; the rest streams from the 1.8 T NVMe.

Lucebox's --ds4-fused-decode requests a monolithic GPU load; on a model this size that allocation fails and the backend falls back to hybrid expert placement + layered decode — that fallback is the intended serving mode here, not an error.

Image

localhost/lucebox-rocm:gfx1201, built on the host from github.com/Luce-Org/lucebox Dockerfile.rocm:

git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
# Three local modifications (repo-tracked at
# dankserver/frank-primary/lucebox-gfx1201.patch):
#   1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
#      base already ships them in /opt/rocm; the apt repo does not serve
#      them) and the three runtime math libs.
#   2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
#      3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
#      requires, CMake >= 3.25).
#   3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
#      `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
#      unconditionally, but the macro is never defined by the build
#      system, so the in-guard definition was dead and the use failed.
git apply ../lucebox-gfx1201.patch
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
             --build-arg ROCM_VERSION=7.14.1-full \
             -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .

The prebuilt ghcr.io/luce-org/lucebox-hub:rocm is not usable as-is: it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not code-object compatible.
ROCM_VERSION=7.14.1-full matches the host driver's ROCm 7.x line (the radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace on a 7.x driver can segfault at model load. Only -full tags exist for 7.14.x.
The binary lands at /opt/lucebox-hub/server/build/luce_server.

Model

/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf, fetched with huggingface_hub (the xet-native path — plain curl full GETs 400 against HF's xethub CDN for this repo; see the fetch note in scripts/hf-fetch.sh). Byte count and sha256 are deploy-gated in deploy_llm_primary.

Deployed flags (2026-09-26)

From dankserver/frank-primary/frank-primary-entrypoint.sh (the unit bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash flag set and cannot express the DS4 flags):

Setting	Value
Model	`DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB)
Device	`--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out)
Window	`--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small)
Chunk	`--chunk 8192` (the qualified profile's chunk)
KV dtype	`--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch)
Decode	`--ds4-fused-decode` (falls back to hybrid expert placement on this host)
Prefill	`--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost)
Routing	`--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05)
Draft	none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode
Identity	`--model-name frank-primary`, `--host 0.0.0.0 --port 7770`

API surface

OpenAI Chat Completions (streaming + tool metadata), /v1/models, /health, OpenAI Responses, Anthropic Messages, and Open WebUI model metadata — the fleet contract is unchanged from frank-medium.

Cutover procedure

Image built and model fetched (gates in deploy_llm_primary).
deploy.sh → deploy_llm_primary: stops and masks the radiance unit (frank-medium), waits for :7770 to free, installs dankserver/frank-primary/quadlet/frank-primary.container, starts frank-primary.service, waits for health (1800 s start period — first start page-ins the dense weights and builds the graphs).
Verify: /health 200 is not enough — a real chat completion, a tool call, and an Open WebUI completion.
Retire the radiance unit to the home trash (never rm); the MXFP6 checkpoint payload stays on disk.

Rollback

Unmask and start the radiance unit (systemctl --user unmask frank-medium && systemctl --user start frank-medium) and stop/mask frank-primary. The localhost/radiance-vllm:0.13.0-e060895 image and the /models/Qwen3.8-27B-PARO-MXFP6 checkpoint remain on disk.

Verification (2026-09-26)

Filled in after the cutover: health, real completion, tool call, Open WebUI completion, measured prefill/decode, systemctl --user is-enabled/is-active state.

6.4 KiB Raw Blame History