serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	frank-primary Lucebox DeepSeek V4 Runbook	RETIRED 2026-09-26 frank-primary runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by frank-primary-llamacpp-qwen38-flash-next-q8 (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.	frank-primary dankserver lucebox deepseek-v4 ds4 rocmfp23 gfx1201 r9700 quadlet moe	2026-09-26

frank-primary Lucebox DeepSeek V4 Runbook

[!warning] RETIRED 2026-09-26 (same day it went live) Replaced by frank-primary-llamacpp-qwen38-flash-next-q8 — unsloth llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime served for hours; the rollback material (image localhost/lucebox-rocm: gfx1201 and the 378 GiB payload) is retained on the host, and the quadlet-era files live in dankserver/.AGENTTRASH/20260926-llamacpp-cutover/.

The dankserver LLM endpoint (:7770, alias frank-primary) as it stood on 2026-09-26: the Lucebox runtime serving Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF (377.2 GiB, 405,349,859,328 bytes, LFS oid 2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2) on a single Radeon AI PRO R9700 (gfx1201). It replaced frank-medium-radiance-mxfp4 (vllm-radiance TP=2), retired earlier the same day.

Why this shape

DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3 down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM, so it is served the way llama.cpp serves big models by default: the GGUF is mmapped, pages pull from disk/page cache on demand, dense work (attention, shared expert, embeddings, output projection) and the MLA KV cache run on the R9700, and the routed experts are computed on CPU. The 246 GiB RAM (upgraded for this deployment) holds the working set of active experts; the rest streams from the 1.8 T NVMe.

Lucebox's --ds4-fused-decode requests a monolithic GPU load; on a model this size that allocation fails and the backend falls back to hybrid expert placement + layered decode — that fallback is the intended serving mode here, not an error.

Image

localhost/lucebox-rocm:gfx1201, built on the host from github.com/Luce-Org/lucebox Dockerfile.rocm:

git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
# Three local modifications (repo-tracked at
# dankserver/frank-primary/lucebox-gfx1201.patch):
#   1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
#      base already ships them in /opt/rocm; the apt repo does not serve
#      them) and the three runtime math libs.
#   2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
#      3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
#      requires, CMake >= 3.25).
#   3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
#      `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
#      unconditionally, but the macro is never defined by the build
#      system, so the in-guard definition was dead and the use failed.
git apply ../lucebox-gfx1201.patch
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
             --build-arg ROCM_VERSION=7.14.1-full \
             -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .

• The prebuilt ghcr.io/luce-org/lucebox-hub:rocm is not usable as-is: it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not code-object compatible.
• ROCM_VERSION=7.14.1-full matches the host driver's ROCm 7.x line (the radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace on a 7.x driver can segfault at model load. Only -full tags exist for 7.14.x.
• The binary lands at /opt/lucebox-hub/server/build/luce_server.

Model

/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf, fetched with huggingface_hub (the xet-native path — plain curl full GETs 400 against HF's xethub CDN for this repo; see the fetch note in scripts/hf-fetch.sh). Byte count and sha256 are deploy-gated in deploy_llm_primary.

Deployed flags (2026-09-26)

From dankserver/frank-primary/frank-primary-entrypoint.sh (the unit bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash flag set and cannot express the DS4 flags):

Setting	Value
Model	DeepSeek-V4.1-Flash-ROCMFP23.gguf (377.2 GiB)
Device	--target-device hip:0 (HIP_VISIBLE_DEVICES=0; the Raphael iGPU at index 2 stays out)
Window	--max-ctx 131072 (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small)
Chunk	--chunk 8192 (the qualified profile's chunk)
KV dtype	--cache-type-k q4_0 --cache-type-v q4_0 (the qualified launch)
Decode	--ds4-fused-decode (falls back to hybrid expert placement on this host)
Prefill	--ds4-prefill sparse (the batched prefill; exact runs at token-wise decode cost)
Routing	--ds4-expert-top-k 6 (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05)
Draft	none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode
Identity	--model-name frank-primary, --host 0.0.0.0 --port 7770

API surface

OpenAI Chat Completions (streaming + tool metadata), /v1/models, /health, OpenAI Responses, Anthropic Messages, and Open WebUI model metadata — the fleet contract is unchanged from frank-medium.

Cutover procedure

1. Image built and model fetched (gates in deploy_llm_primary).
2. deploy.sh → deploy_llm_primary: stops and masks the radiance unit (frank-medium), waits for :7770 to free, installs dankserver/frank-primary/quadlet/frank-primary.container, starts frank-primary.service, waits for health (1800 s start period — first start page-ins the dense weights and builds the graphs).
3. Verify: /health 200 is not enough — a real chat completion, a tool call, and an Open WebUI completion.
4. Retire the radiance unit to the home trash (never rm); the MXFP6 checkpoint payload stays on disk.

Rollback

Unmask and start the radiance unit (systemctl --user unmask frank-medium && systemctl --user start frank-medium) and stop/mask frank-primary. The localhost/radiance-vllm:0.13.0-e060895 image and the /models/Qwen3.8-27B-PARO-MXFP6 checkpoint remain on disk.

Verification (2026-09-26)

Filled in after the cutover: health, real completion, tool call, Open WebUI completion, measured prefill/decode, systemctl --user is-enabled/is-active state.

Dual-R9700 follow-up (2026-09-28 probe era)

After this single-card runbook was retired, the DS41 engine got a dual-R9700 attempt on the feat/ds41-engine branch (main rejects the checkpoint with unsupported model architecture 'deepseek41'). Findings that outlive the probe:

Budget both cards explicitly, or you waste half the box. The engine sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a cache of things still streaming from NVMe. Pin all four:

env	card	meaning
LUCE_EXPERT_BUDGET_MB	hip:0	primary hot experts (~11.96 MiB each)
LUCE_EXPERT_SECONDARY_BUDGET_MB	hip:1	secondary experts (needs LUCE_DS4_MOE_TP*)
LUCE_EXPERT_STREAM_CACHE_MB	hip:1	cap the auto-sized stream cache
LUCE_EXPERT_STREAM_DEVICE=1	hip:1	which card holds the slots and computes them

Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB, card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.

Expert TP DOES work on gfx1201+gfx1201. A retired quadlet recorded LUCE_DS4_MOE_TP_INPROC as "qualified for gfx1151+gfx1201, not gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary tier comes up clean: [deepseek4-moe-tp] enabled mode=in-process local=hip:0 secondary=hip:1. It is the only lever that materially raises resident expert count, and resident count is the entire bottleneck. Confirm every run with expert ownership: and hybrid experts ready: lines.

The placement file must live under the mounted models dir. JSON {"owners": [...]}, 15,360 entries indexed layer*384+expert, values primary/secondary/stream. It is a ranking, not a promise: each device keeps what fits its budget and demotes the rest. Anywhere outside /models/... gives cannot open expert placement and a SILENT single-tier fallback — a silent-wrongness failure, not a crash.

V4.1 refuses layer split and layer-major prefill — only --ds4-prefill exact (token-by-token) is legal on a 32 GiB card and --ds4-fused-decode is refused there. With ~22% of experts resident each token pulls ~187 cold experts × ~12 MiB; if /proc/<pid>/io read_bytes climbs into terabytes the model is larger than the box's RAM and you are in page-cache thrash at the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound decode, not a broken run.

Harness traps, each of which produced a wrong number: never env -i the podman call (rootless podman runs as dank; env -i HOME=/root makes PODMAN try /root/.config and die stat /root/.config: permission denied before the container exists — pass container env with -e); never put a large prompt on curl's argv (Argument list too long at ~128 KB — use --data-binary @file); size prompts by growing the file until wc -c reaches the byte target AND assert depth against the engine's own usage.prompt_tokens (nominal 4 chars/token was really 5.73; a chunk loop that appends a fixed ~152-byte line per nominal 200-byte chunk undershot 30%); the server runs as the container's PID 1, so tear down with podman rm -f, never host pkill; drop page cache between arms (sync; echo 3 > /proc/sys/vm/drop_caches) or the first arm reads warm and every later one is a lie.

Read the placement log — it is the whole diagnosis (arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304 hash_layers=0 — hash_layers=0 is correct for V4.1; engram: layers=[1,14] … 2 table(s) embedded (not mapped) — the 189.13 GiB Engram tables are never mapped, so they do not evict the expert page cache).