Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
223 lines
12 KiB
Markdown
223 lines
12 KiB
Markdown
---
|
||
type: configuration
|
||
title: frank-archon Lucebox DeepSeek V4 Runbook
|
||
description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); the image is still in rootless storage but the payload is gone from the host, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
|
||
tags:
|
||
- frank-archon
|
||
- dankserver
|
||
- lucebox
|
||
- deepseek-v4
|
||
- ds4
|
||
- rocmfp23
|
||
- gfx1201
|
||
- r9700
|
||
- quadlet
|
||
- moe
|
||
timestamp: 2026-09-26
|
||
---
|
||
|
||
# frank-archon Lucebox DeepSeek V4 Runbook
|
||
|
||
> [!warning] RETIRED 2026-09-26 (same day it went live)
|
||
> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
|
||
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
|
||
> served for hours; the image `localhost/lucebox-rocm:gfx1201` is still in
|
||
> dank's rootless storage, but the 378 GiB payload was deleted from the host
|
||
> (`ls /home/dank/models/` on Dankserver, 2026-10-18: `StillDeadcode`, `rad`,
|
||
> `tpl.jinja` only), and the quadlet-era repo files were deleted by `8500600`
|
||
> (2026-09-26) — they survive only in git history, not in a `.AGENTTRASH`
|
||
> directory (`dankserver/.AGENTTRASH/` holds only `20260916-vllm-radiance`).
|
||
|
||
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
|
||
2026-09-26: the
|
||
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
|
||
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
|
||
bytes, LFS oid
|
||
`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
|
||
single Radeon AI PRO R9700 (gfx1201). It replaced
|
||
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the
|
||
same day.
|
||
|
||
## Why this shape
|
||
|
||
DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
|
||
per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
|
||
down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
|
||
the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
|
||
so it is served the way llama.cpp serves big models by default: the GGUF is
|
||
mmapped, pages pull from disk/page cache on demand, dense work (attention,
|
||
shared expert, embeddings, output projection) and the MLA KV cache run on
|
||
the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
|
||
(upgraded for this deployment) holds the working set of active experts; the
|
||
rest streams from the 1.8 T NVMe.
|
||
|
||
Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
|
||
this size that allocation fails and the backend falls back to **hybrid
|
||
expert placement + layered decode** — that fallback is the intended serving
|
||
mode here, not an error.
|
||
|
||
## Image
|
||
|
||
`localhost/lucebox-rocm:gfx1201`, built on the host from
|
||
`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
|
||
|
||
```
|
||
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
|
||
cd lucebox
|
||
# Three local modifications (tracked at the time as
|
||
# dankserver/frank-primary/lucebox-gfx1201.patch, deleted by 8500600):
|
||
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
|
||
# base already ships them in /opt/rocm; the apt repo does not serve
|
||
# them) and the three runtime math libs.
|
||
# 2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
|
||
# 3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
|
||
# requires, CMake >= 3.25).
|
||
# 3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
|
||
# `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
|
||
# unconditionally, but the macro is never defined by the build
|
||
# system, so the in-guard definition was dead and the use failed.
|
||
git apply ../lucebox-gfx1201.patch
|
||
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
|
||
--build-arg ROCM_VERSION=7.14.1-full \
|
||
-t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
|
||
```
|
||
|
||
- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
|
||
it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
|
||
code-object compatible.
|
||
- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
|
||
radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
|
||
on a 7.x driver can segfault at model load. Only `-full` tags exist for
|
||
7.14.x.
|
||
- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
|
||
|
||
## Model
|
||
|
||
`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
|
||
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
|
||
400 against HF's xethub CDN for this repo; see the fetch note in
|
||
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
|
||
`deploy_llm_archon`.
|
||
|
||
## Deployed flags (2026-09-26)
|
||
|
||
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (deleted by `8500600`;
|
||
recover it with `git show 8500600^:dankserver/frank-primary/frank-primary-entrypoint.sh`) — the unit
|
||
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
|
||
flag set and cannot express the DS4 flags):
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
|
||
| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
|
||
| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
|
||
| Chunk | `--chunk 8192` (the qualified profile's chunk) |
|
||
| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
|
||
| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
|
||
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
|
||
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
|
||
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
|
||
| Identity | `--model-name frank-archon`, `--host 0.0.0.0 --port 7770` |
|
||
|
||
## API surface
|
||
|
||
OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
|
||
`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
|
||
metadata — the fleet contract is unchanged from frank-medium.
|
||
|
||
## Cutover procedure
|
||
|
||
1. Image built and model fetched (gates in `deploy_llm_archon`).
|
||
2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
|
||
unit (`frank-medium`), waits for :7770 to free, installs
|
||
`dankserver/frank-primary/quadlet/frank-primary.container` (also deleted by
|
||
`8500600`; recover it from `git show 8500600^`), starts
|
||
`frank-archon.service`, waits for health (1800 s start period — first
|
||
start page-ins the dense weights and builds the graphs).
|
||
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
|
||
call, and an Open WebUI completion.
|
||
4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
|
||
checkpoint payload did not stay — it is gone from the host (see Rollback).
|
||
|
||
## Rollback
|
||
|
||
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
|
||
systemctl --user start frank-medium`) and stop/mask frank-archon. The
|
||
`localhost/radiance-vllm:0.13.0-e060895` image is still in dank's rootless
|
||
storage; the `/models/Qwen3.8-27B-PARO-MXFP6` checkpoint is not — Dankserver's
|
||
`/home/dank/models/` holds only `StillDeadcode`, `rad` and `tpl.jinja` as of
|
||
2026-10-18, so this rollback needs a re-fetch of the payload.
|
||
|
||
## Verification (2026-09-26)
|
||
|
||
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
||
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
||
state._
|
||
|
||
## Dual-R9700 follow-up (2026-09-28 probe era)
|
||
|
||
After this single-card runbook was retired, the DS41 engine got a dual-R9700
|
||
attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
|
||
`unsupported model architecture 'deepseek41'`). Findings that outlive the
|
||
probe:
|
||
|
||
**Budget both cards explicitly, or you waste half the box.** The engine
|
||
sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
|
||
every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
|
||
looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
|
||
cache of things still streaming from NVMe. Pin all four:
|
||
|
||
| env | card | meaning |
|
||
|---|---|---|
|
||
| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
|
||
| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
|
||
| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
|
||
| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
|
||
|
||
Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
|
||
card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
|
||
|
||
**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
|
||
`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
|
||
gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
|
||
tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
|
||
secondary=hip:1`. It is the only lever that materially raises resident
|
||
expert count, and resident count is the entire bottleneck. Confirm every run
|
||
with `expert ownership:` and `hybrid experts ready:` lines.
|
||
|
||
**The placement file must live under the mounted models dir.** JSON
|
||
`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
|
||
`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
|
||
keeps what fits its budget and demotes the rest. Anywhere outside
|
||
`/models/...` gives `cannot open expert placement` and a SILENT single-tier
|
||
fallback — a silent-wrongness failure, not a crash.
|
||
|
||
**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
|
||
exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
|
||
is refused there. With ~22% of experts resident each token pulls ~187 cold
|
||
experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
|
||
the model is larger than the box's RAM and you are in page-cache thrash at
|
||
the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
|
||
with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
|
||
decode, not a broken run.
|
||
|
||
**Harness traps, each of which produced a wrong number:** never `env -i` the
|
||
podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
|
||
PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
|
||
before the container exists — pass container env with `-e`); never put a
|
||
large prompt on curl's argv (`Argument list too long` at ~128 KB — use
|
||
`--data-binary @file`); size prompts by growing the file until `wc -c`
|
||
reaches the byte target AND assert depth against the engine's own
|
||
`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
|
||
that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
|
||
30%); the server runs as the container's PID 1, so tear down with
|
||
`podman rm -f`, never host `pkill`; drop page cache between arms
|
||
(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
|
||
every later one is a lie.
|
||
|
||
Read the placement log — it is the whole diagnosis
|
||
(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
|
||
hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
|
||
… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
|
||
mapped, so they do not evict the expert page cache).
|