Files

T

Frank 8500600d09 frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0

The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).

The deployed contract:

- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
  and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
  -ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
  over PCIe loses at 3.7 t/s, computing in RAM does not care).
  per_layer_token_embd.weight=CPU is required by the shared MTP head's
  borrow path (a GPU-resident borrow crashes the draft load).
  mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
  (mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
  draftless ngram-mod matcher chained behind it — the only legal way to
  layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
  autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
  Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
  by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
  with Qwen-recommended sampling riding compat.extraBody.

The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.

Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.

Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).

models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].

Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).

Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.

2026-09-26 19:45:28 -07:00

7.0 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-primary Lucebox DeepSeek V4 Runbook

[!warning] RETIRED 2026-09-26 (same day it went live) Replaced by frank-primary-llamacpp-qwen38-flash-next-q8 — unsloth llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime served for hours; the rollback material (image localhost/lucebox-rocm: gfx1201 and the 378 GiB payload) is retained on the host, and the quadlet-era files live in dankserver/.AGENTTRASH/20260926-llamacpp-cutover/.

The dankserver LLM endpoint (:7770, alias frank-primary) as it stood on 2026-09-26: the Lucebox runtime serving Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF (377.2 GiB, 405,349,859,328 bytes, LFS oid 2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2) on a single Radeon AI PRO R9700 (gfx1201). It replaced frank-medium-radiance-mxfp4 (vllm-radiance TP=2), retired earlier the same day.

Why this shape

DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3 down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM, so it is served the way llama.cpp serves big models by default: the GGUF is mmapped, pages pull from disk/page cache on demand, dense work (attention, shared expert, embeddings, output projection) and the MLA KV cache run on the R9700, and the routed experts are computed on CPU. The 246 GiB RAM (upgraded for this deployment) holds the working set of active experts; the rest streams from the 1.8 T NVMe.

Lucebox's --ds4-fused-decode requests a monolithic GPU load; on a model this size that allocation fails and the backend falls back to hybrid expert placement + layered decode — that fallback is the intended serving mode here, not an error.

Image

localhost/lucebox-rocm:gfx1201, built on the host from github.com/Luce-Org/lucebox Dockerfile.rocm:

git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
# Three local modifications (repo-tracked at
# dankserver/frank-primary/lucebox-gfx1201.patch):
#   1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
#      base already ships them in /opt/rocm; the apt repo does not serve
#      them) and the three runtime math libs.
#   2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
#      3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
#      requires, CMake >= 3.25).
#   3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
#      `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
#      unconditionally, but the macro is never defined by the build
#      system, so the in-guard definition was dead and the use failed.
git apply ../lucebox-gfx1201.patch
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
             --build-arg ROCM_VERSION=7.14.1-full \
             -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .

The prebuilt ghcr.io/luce-org/lucebox-hub:rocm is not usable as-is: it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not code-object compatible.
ROCM_VERSION=7.14.1-full matches the host driver's ROCm 7.x line (the radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace on a 7.x driver can segfault at model load. Only -full tags exist for 7.14.x.
The binary lands at /opt/lucebox-hub/server/build/luce_server.

Model

/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf, fetched with huggingface_hub (the xet-native path — plain curl full GETs 400 against HF's xethub CDN for this repo; see the fetch note in scripts/hf-fetch.sh). Byte count and sha256 are deploy-gated in deploy_llm_primary.

Deployed flags (2026-09-26)

From dankserver/frank-primary/frank-primary-entrypoint.sh (the unit bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash flag set and cannot express the DS4 flags):

Setting	Value
Model	`DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB)
Device	`--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out)
Window	`--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small)
Chunk	`--chunk 8192` (the qualified profile's chunk)
KV dtype	`--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch)
Decode	`--ds4-fused-decode` (falls back to hybrid expert placement on this host)
Prefill	`--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost)
Routing	`--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05)
Draft	none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode
Identity	`--model-name frank-primary`, `--host 0.0.0.0 --port 7770`

API surface

OpenAI Chat Completions (streaming + tool metadata), /v1/models, /health, OpenAI Responses, Anthropic Messages, and Open WebUI model metadata — the fleet contract is unchanged from frank-medium.

Cutover procedure

Image built and model fetched (gates in deploy_llm_primary).
deploy.sh → deploy_llm_primary: stops and masks the radiance unit (frank-medium), waits for :7770 to free, installs dankserver/frank-primary/quadlet/frank-primary.container, starts frank-primary.service, waits for health (1800 s start period — first start page-ins the dense weights and builds the graphs).
Verify: /health 200 is not enough — a real chat completion, a tool call, and an Open WebUI completion.
Retire the radiance unit to the home trash (never rm); the MXFP6 checkpoint payload stays on disk.

Rollback

Unmask and start the radiance unit (systemctl --user unmask frank-medium && systemctl --user start frank-medium) and stop/mask frank-primary. The localhost/radiance-vllm:0.13.0-e060895 image and the /models/Qwen3.8-27B-PARO-MXFP6 checkpoint remain on disk.

Verification (2026-09-26)

Filled in after the cutover: health, real completion, tool call, Open WebUI completion, measured prefill/decode, systemctl --user is-enabled/is-active state.

7.0 KiB Raw Blame History