frank-primary: Lucebox DeepSeek-V4.1-Flash-ROCMFP23 quadlet + deploy handler
Replaces frank-medium (vllm-radiance TP=2) on :7770. The 377.2 GiB 43-layer MoE (256 routed experts, top-6) is mmapped: dense work and the MLA KV cache on the R9700, routed experts computed on CPU from the page cache (246 GiB RAM). The unit bypasses the image entrypoint (Qwen/DFlash-only flag set) and execs luce_server with the DS4 launch shape; --ds4-fused-decode's monolithic-load fallback IS the serving mode here. deploy_llm_primary gates on image + model byte count, stops/masks the radiance unit, waits for :7770, installs and starts. Runbook: frank-primary-lucebox-ds4.md.
This commit is contained in:
Frank
@@ -0,0 +1,127 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-primary Lucebox DeepSeek V4 Runbook
|
||||
description: ACTIVE frank-primary runbook — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaces frank-medium (vllm-radiance TP=2) on :7770, cutover 2026-09-26.
|
||||
tags:
|
||||
- frank-primary
|
||||
- dankserver
|
||||
- lucebox
|
||||
- deepseek-v4
|
||||
- ds4
|
||||
- rocmfp23
|
||||
- gfx1201
|
||||
- r9700
|
||||
- quadlet
|
||||
- moe
|
||||
timestamp: 2026-09-26
|
||||
---
|
||||
|
||||
# frank-primary Lucebox DeepSeek V4 Runbook
|
||||
|
||||
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): the
|
||||
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
|
||||
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
|
||||
bytes, LFS oid
|
||||
`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
|
||||
single Radeon AI PRO R9700 (gfx1201). It replaces
|
||||
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired on
|
||||
2026-09-26.
|
||||
|
||||
## Why this shape
|
||||
|
||||
DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
|
||||
per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
|
||||
down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
|
||||
the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
|
||||
so it is served the way llama.cpp serves big models by default: the GGUF is
|
||||
mmapped, pages pull from disk/page cache on demand, dense work (attention,
|
||||
shared expert, embeddings, output projection) and the MLA KV cache run on
|
||||
the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
|
||||
(upgraded for this deployment) holds the working set of active experts; the
|
||||
rest streams from the 1.8 T NVMe.
|
||||
|
||||
Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
|
||||
this size that allocation fails and the backend falls back to **hybrid
|
||||
expert placement + layered decode** — that fallback is the intended serving
|
||||
mode here, not an error.
|
||||
|
||||
## Image
|
||||
|
||||
`localhost/lucebox-rocm:gfx1201`, built on the host from
|
||||
`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
|
||||
|
||||
```
|
||||
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
|
||||
cd lucebox
|
||||
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
|
||||
--build-arg ROCM_VERSION=7.14.1-full \
|
||||
-t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
|
||||
```
|
||||
|
||||
- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
|
||||
it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
|
||||
code-object compatible.
|
||||
- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
|
||||
radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
|
||||
on a 7.x driver can segfault at model load. Only `-full` tags exist for
|
||||
7.14.x.
|
||||
- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
|
||||
|
||||
## Model
|
||||
|
||||
`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
|
||||
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
|
||||
400 against HF's xethub CDN for this repo; see the fetch note in
|
||||
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
|
||||
`deploy_llm_primary`.
|
||||
|
||||
## Deployed flags (2026-09-26)
|
||||
|
||||
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (the unit
|
||||
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
|
||||
flag set and cannot express the DS4 flags):
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
|
||||
| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
|
||||
| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
|
||||
| Chunk | `--chunk 8192` (the qualified profile's chunk) |
|
||||
| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
|
||||
| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
|
||||
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
|
||||
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
|
||||
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
|
||||
| Identity | `--model-name frank-primary`, `--host 0.0.0.0 --port 7770` |
|
||||
|
||||
## API surface
|
||||
|
||||
OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
|
||||
`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
|
||||
metadata — the fleet contract is unchanged from frank-medium.
|
||||
|
||||
## Cutover procedure
|
||||
|
||||
1. Image built and model fetched (gates in `deploy_llm_primary`).
|
||||
2. `deploy.sh` → `deploy_llm_primary`: stops and **masks** the radiance
|
||||
unit (`frank-medium`), waits for :7770 to free, installs
|
||||
`dankserver/frank-primary/quadlet/frank-primary.container`, starts
|
||||
`frank-primary.service`, waits for health (1800 s start period — first
|
||||
start page-ins the dense weights and builds the graphs).
|
||||
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
|
||||
call, and an Open WebUI completion.
|
||||
4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
|
||||
checkpoint payload stays on disk.
|
||||
|
||||
## Rollback
|
||||
|
||||
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
|
||||
systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
||||
`localhost/radiance-vllm:0.13.0-e060895` image and the
|
||||
`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
|
||||
|
||||
## Verification (2026-09-26)
|
||||
|
||||
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
||||
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
||||
state._
|
||||
@@ -124,6 +124,11 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-primary-lucebox-ds4]] — the ACTIVE frank-primary runbook:
|
||||
the Lucebox runtime serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB
|
||||
GGUF, 43-layer MoE, 256 routed experts top-6) on Dankserver's R9700
|
||||
with the routed experts computed on CPU from the mmapped page cache —
|
||||
replaces frank-medium on :7770 (cutover 2026-09-26)
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
|
||||
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
|
||||
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
|
||||
|
||||
Reference in New Issue
Block a user