141 lines
6.4 KiB
Markdown
141 lines
6.4 KiB
Markdown
---
|
|
type: configuration
|
|
title: frank-primary Lucebox DeepSeek V4 Runbook
|
|
description: ACTIVE frank-primary runbook — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaces frank-medium (vllm-radiance TP=2) on :7770, cutover 2026-09-26.
|
|
tags:
|
|
- frank-primary
|
|
- dankserver
|
|
- lucebox
|
|
- deepseek-v4
|
|
- ds4
|
|
- rocmfp23
|
|
- gfx1201
|
|
- r9700
|
|
- quadlet
|
|
- moe
|
|
timestamp: 2026-09-26
|
|
---
|
|
|
|
# frank-primary Lucebox DeepSeek V4 Runbook
|
|
|
|
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): the
|
|
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
|
|
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
|
|
bytes, LFS oid
|
|
`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
|
|
single Radeon AI PRO R9700 (gfx1201). It replaces
|
|
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired on
|
|
2026-09-26.
|
|
|
|
## Why this shape
|
|
|
|
DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
|
|
per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
|
|
down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
|
|
the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
|
|
so it is served the way llama.cpp serves big models by default: the GGUF is
|
|
mmapped, pages pull from disk/page cache on demand, dense work (attention,
|
|
shared expert, embeddings, output projection) and the MLA KV cache run on
|
|
the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
|
|
(upgraded for this deployment) holds the working set of active experts; the
|
|
rest streams from the 1.8 T NVMe.
|
|
|
|
Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
|
|
this size that allocation fails and the backend falls back to **hybrid
|
|
expert placement + layered decode** — that fallback is the intended serving
|
|
mode here, not an error.
|
|
|
|
## Image
|
|
|
|
`localhost/lucebox-rocm:gfx1201`, built on the host from
|
|
`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
|
|
|
|
```
|
|
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
|
|
cd lucebox
|
|
# Three local modifications (repo-tracked at
|
|
# dankserver/frank-primary/lucebox-gfx1201.patch):
|
|
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
|
|
# base already ships them in /opt/rocm; the apt repo does not serve
|
|
# them) and the three runtime math libs.
|
|
# 2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
|
|
# 3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
|
|
# requires, CMake >= 3.25).
|
|
# 3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
|
|
# `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
|
|
# unconditionally, but the macro is never defined by the build
|
|
# system, so the in-guard definition was dead and the use failed.
|
|
git apply ../lucebox-gfx1201.patch
|
|
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
|
|
--build-arg ROCM_VERSION=7.14.1-full \
|
|
-t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
|
|
```
|
|
|
|
- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
|
|
it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
|
|
code-object compatible.
|
|
- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
|
|
radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
|
|
on a 7.x driver can segfault at model load. Only `-full` tags exist for
|
|
7.14.x.
|
|
- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
|
|
|
|
## Model
|
|
|
|
`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
|
|
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
|
|
400 against HF's xethub CDN for this repo; see the fetch note in
|
|
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
|
|
`deploy_llm_primary`.
|
|
|
|
## Deployed flags (2026-09-26)
|
|
|
|
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (the unit
|
|
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
|
|
flag set and cannot express the DS4 flags):
|
|
|
|
| Setting | Value |
|
|
|---|---|
|
|
| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
|
|
| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
|
|
| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
|
|
| Chunk | `--chunk 8192` (the qualified profile's chunk) |
|
|
| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
|
|
| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
|
|
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
|
|
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
|
|
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
|
|
| Identity | `--model-name frank-primary`, `--host 0.0.0.0 --port 7770` |
|
|
|
|
## API surface
|
|
|
|
OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
|
|
`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
|
|
metadata — the fleet contract is unchanged from frank-medium.
|
|
|
|
## Cutover procedure
|
|
|
|
1. Image built and model fetched (gates in `deploy_llm_primary`).
|
|
2. `deploy.sh` → `deploy_llm_primary`: stops and **masks** the radiance
|
|
unit (`frank-medium`), waits for :7770 to free, installs
|
|
`dankserver/frank-primary/quadlet/frank-primary.container`, starts
|
|
`frank-primary.service`, waits for health (1800 s start period — first
|
|
start page-ins the dense weights and builds the graphs).
|
|
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
|
|
call, and an Open WebUI completion.
|
|
4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
|
|
checkpoint payload stays on disk.
|
|
|
|
## Rollback
|
|
|
|
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
|
|
systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
|
`localhost/radiance-vllm:0.13.0-e060895` image and the
|
|
`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
|
|
|
|
## Verification (2026-09-26)
|
|
|
|
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
|
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
|
state._
|