From acd14ab7dfedc9630638b8300779ab3a798081de Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 26 Sep 2026 15:29:59 -0700 Subject: [PATCH] frank-primary: Lucebox DeepSeek-V4.1-Flash-ROCMFP23 quadlet + deploy handler Replaces frank-medium (vllm-radiance TP=2) on :7770. The 377.2 GiB 43-layer MoE (256 routed experts, top-6) is mmapped: dense work and the MLA KV cache on the R9700, routed experts computed on CPU from the page cache (246 GiB RAM). The unit bypasses the image entrypoint (Qwen/DFlash-only flag set) and execs luce_server with the DS4 launch shape; --ds4-fused-decode's monolithic-load fallback IS the serving mode here. deploy_llm_primary gates on image + model byte count, stops/masks the radiance unit, waits for :7770, installs and starts. Runbook: frank-primary-lucebox-ds4.md. --- .../frank-primary/frank-primary-entrypoint.sh | 55 ++++++++ .../frank-primary-healthcheck.py | 14 ++ .../quadlet/frank-primary.container | 85 ++++++++++++ scripts/deploy-plan.py | 9 ++ scripts/deploy-remote.sh | 49 ++++++- .../LLM/frank-primary-lucebox-ds4.md | 127 ++++++++++++++++++ serverAppConfigs-docs/index.md | 5 + tests/deployment_scoping.bats | 9 ++ 8 files changed, 352 insertions(+), 1 deletion(-) create mode 100755 dankserver/frank-primary/frank-primary-entrypoint.sh create mode 100755 dankserver/frank-primary/frank-primary-healthcheck.py create mode 100644 dankserver/frank-primary/quadlet/frank-primary.container create mode 100644 serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md diff --git a/dankserver/frank-primary/frank-primary-entrypoint.sh b/dankserver/frank-primary/frank-primary-entrypoint.sh new file mode 100755 index 0000000..c1a0d70 --- /dev/null +++ b/dankserver/frank-primary/frank-primary-entrypoint.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# frank-primary entrypoint: exec the native luce_server with the DeepSeek V4 +# launch shape for this host (2x R9700 gfx1201, 246 GiB RAM). +# +# The image's own entrypoint (server/scripts/entrypoint.sh) only forwards a +# fixed Qwen/DFlash flag set and cannot express the DS4 flags, so this unit +# bypasses it (PodmanArgs=--entrypoint) and execs the binary directly. The +# C++ server reads LUCE_* environment variables straight from the process +# environment, so no env plumbing is needed here. +# +# LAUNCH SHAPE (server/docs/DS4.md, server/docs/RECOMMENDED_SETUPS.md): +# --ds4-fused-decode requests a monolithic GPU load; on a 377 GiB model +# that allocation fails and the backend falls back +# to hybrid expert placement + layered decode (dense +# work on the R9700, routed experts computed on CPU +# from the mmapped page cache). That fallback IS the +# intended serving mode on this host. +# --ds4-prefill sparse the batched prefill; use for prompts beyond a few +# hundred tokens (exact is token-wise decode cost). +# --ds4-expert-top-k 6 the model default. Do NOT reduce on adaptive +# (qtype-105/106) artifacts: measured degenerate +# output at top-4 (DS4.md, 2026-08-05). +# --chunk 8192 the qualified Strix Halo profile's chunk. +# --max-ctx 131072 the qualified Strix Halo window; DS4's MLA KV is +# compressed, so the pool is small. +# --cache-type-k/v q4_0 the qualified launch's KV dtype. +# +# No DSpark draft is published for DeepSeek-V4.1-Flash, so this is plain +# autoregressive decode (no --draft, no LUCE_DS4_SPEC). +set -euo pipefail + +MODEL="${FRANK_PRIMARY_MODEL:-/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf}" +BIN="${FRANK_PRIMARY_BIN:-/opt/lucebox-hub/server/build/luce_server}" + +if [[ ! -f "$MODEL" ]]; then + echo "frank-primary: model missing at $MODEL" >&2 + exit 1 +fi +if [[ ! -x "$BIN" ]]; then + echo "frank-primary: binary missing at $BIN (image build failed?)" >&2 + exit 1 +fi + +exec "$BIN" "$MODEL" \ + --host 0.0.0.0 \ + --port 7770 \ + --model-name frank-primary \ + --target-device hip:0 \ + --max-ctx 131072 \ + --chunk 8192 \ + --cache-type-k q4_0 \ + --cache-type-v q4_0 \ + --ds4-fused-decode \ + --ds4-prefill sparse \ + --ds4-expert-top-k 6 diff --git a/dankserver/frank-primary/frank-primary-healthcheck.py b/dankserver/frank-primary/frank-primary-healthcheck.py new file mode 100755 index 0000000..1decc9f --- /dev/null +++ b/dankserver/frank-primary/frank-primary-healthcheck.py @@ -0,0 +1,14 @@ +#!/usr/bin/env python3 +"""frank-primary health probe: 200 on /health only once the engine serves. + +File-based (not an inline HealthCmd) because the podman 5.x v1 quadlet +generator splits HealthCmd on spaces into an argv array. +""" +import sys +import urllib.request + +try: + with urllib.request.urlopen("http://127.0.0.1:7770/health", timeout=10) as r: + sys.exit(0 if r.status == 200 else 1) +except Exception: + sys.exit(1) diff --git a/dankserver/frank-primary/quadlet/frank-primary.container b/dankserver/frank-primary/quadlet/frank-primary.container new file mode 100644 index 0000000..3040496 --- /dev/null +++ b/dankserver/frank-primary/quadlet/frank-primary.container @@ -0,0 +1,85 @@ +# Quadlet unit: frank-primary (rootless, boot-persistent). +# +# Install path on dankserver: ~/.config/containers/systemd/frank-primary.container +# Unit name: frank-primary.service (container name: frank-primary) +# +# The dankserver LLM endpoint (:7770, alias frank-primary): the Lucebox +# runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm: +# gfx1201 built from Dockerfile.rocm with LUCE_HIP_ARCHES=gfx1201 and +# ROCM_VERSION=7.14.1-full to match the host driver's ROCm 7.x line) +# serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, +# 256 routed experts + 1 shared per layer, top-6 routing) on a single +# Radeon AI PRO R9700 (gfx1201). The model does not fit VRAM: dense work +# and the MLA KV cache sit on the R9700, the routed experts are computed +# on CPU from the mmapped page cache (246 GiB RAM holds the working set; +# the rest streams from NVMe). Lazy mmap page-in is the engine default — +# no flag. +# +# The image's entrypoint only forwards a fixed Qwen/DFlash flag set and +# cannot express the DS4 flags, so the unit bypasses it (--entrypoint) and +# runs the repo's frank-primary-entrypoint.sh, which execs the native +# luce_server with the DS4 launch shape (see that script's header). +# +# SHAPE. Host network, :7770, served model name frank-primary, OpenAI /v1 +# (Chat Completions with tool metadata, Responses, Anthropic Messages, +# Open WebUI metadata) — the fleet contract is unchanged from +# frank-medium. +# +# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack. HIP_VISIBLE_ +# DEVICES=0 pins the first R9700 (the Raphael iGPU enumerates as index 2 +# and stays out). The second R9700 is not used: the in-process +# heterogeneous expert-parallel path (LUCE_DS4_MOE_TP_INPROC) is qualified +# for gfx1151+gfx1201 pairs, not gfx1201+gfx1201. +# +# MOUNTS. /models (ro — the GGUF is mmapped, nothing writes), and the +# entrypoint + healthcheck scripts straight out of this checkout +# (deploy.sh git-pulls the checkout before dispatching). +# +# IMAGE. localhost/-aliased tag: this host's podman quadlet path cannot +# resolve registry-prefixed refs from local storage at create time (same +# as frank-medium). Pull=never keeps boots off the registry. +# +# CUTOVER. This unit REPLACES frank-medium on :7770. The deploy handler +# (deploy_llm_primary in scripts/deploy-remote.sh) stops and masks the +# radiance unit before starting this one; the radiance unit is retired to +# the home trash after frank-primary is verified (trash policy, never rm). +# +# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the +# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident). +[Unit] +Description=frank-primary — Lucebox DeepSeek-V4.1-Flash-ROCMFP23 on R9700 (gfx1201), experts in RAM +After=network-online.target + +[Container] +ContainerName=frank-primary +Image=localhost/lucebox-rocm:gfx1201 +# Host network: luce_server binds 0.0.0.0:7770 directly; no PublishPort. +AddDevice=/dev/kfd +AddDevice=/dev/dri +GroupAdd=keep-groups +PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host --entrypoint /usr/local/bin/frank-primary-entrypoint.sh +# First R9700 only (index pin excludes the Raphael iGPU at index 2). +Environment=HIP_VISIBLE_DEVICES=0 +Environment=ROCR_VISIBLE_DEVICES=0 +# The 377 GiB GGUF, mmapped; pages pull from disk/page cache on demand. +Volume=/home/dank/models:/models:ro +# Entrypoint + healthcheck from this checkout, mounted read-only. +Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-entrypoint.sh:/usr/local/bin/frank-primary-entrypoint.sh:ro +Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-healthcheck.py:/usr/local/bin/frank-primary-healthcheck.py:ro +# 200 only once the engine serves; first start page-ins the dense weights +# and builds the graphs, which takes a while. HealthCmd is split on spaces +# into an argv array by the podman 5.x v1 generator, hence the file probe. +HealthCmd=python3 /usr/local/bin/frank-primary-healthcheck.py +HealthInterval=60s +HealthTimeout=30s +HealthRetries=3 +HealthStartPeriod=1800s +Pull=never + +[Service] +TimeoutStartSec=1800 +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=default.target diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index d3a3ff2..9475e36 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -64,6 +64,7 @@ SERVICE_ORDER = ( "frank-large", "frank-small", "frank-medium", + "frank-primary", ) DANKCLAW_SERVICES = { @@ -114,6 +115,7 @@ DANKSERVER_SERVICES = { "lidarr", "readarr", "frank-medium", + "frank-primary", "node-exporter", } @@ -381,6 +383,13 @@ def services_for_path(path: str, target: str) -> set[str]: # scripts/.AGENTTRASH/20260920-llama-layersplit/ and plans nothing.) _add(services, "frank-medium") return services + if len(parts) >= 2 and parts[1] == "frank-primary": + # frank-primary is the Lucebox DeepSeek V4 Quadlet (2026-09-26 + # cutover, replacing frank-medium on :7770): its entrypoint and + # healthcheck live under this directory and the container mounts + # them straight out of this checkout, so any file here changes + # deployed state. + _add(services, "frank-primary") return services if parts[0] == "frankbook-pro": diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index a6ba645..7b333fe 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -95,7 +95,7 @@ service_is_allowed() { dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\ dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\ dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\ - dankserver:frank-medium|dankserver:node-exporter|\ + dankserver:frank-medium|dankserver:frank-primary|dankserver:node-exporter|\ frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\ frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\ frankbook-pro:readarr) @@ -1331,6 +1331,52 @@ deploy_llm_medium() { } } +# deploy_llm_primary — frank-primary is the Lucebox DeepSeek V4 Quadlet +# (the 2026-09-26 cutover that replaces frank-medium on :7770). The model +# is a 377 GiB GGUF and the image is a local build, so the gates are: +# image present, model file present at its declared byte count (a partial +# download must never be served). +deploy_llm_primary() { + local missing=0 + local model="/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf" + local want_size=405349859328 + if ! podman image exists localhost/lucebox-rocm:gfx1201; then + echo "ERROR: lucebox image absent (localhost/lucebox-rocm:gfx1201)" >&2 + echo "frank-primary: podman build --build-arg LUCE_HIP_ARCHES=gfx1201 --build-arg ROCM_VERSION=7.14.1-full -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm . (see runbook)" >&2 + missing=1 + fi + local got_size + got_size="$(wc -c < "$model" 2>/dev/null || echo 0)" + if [[ "$got_size" != "$want_size" ]]; then + echo "ERROR: model absent or incomplete at $model (have $got_size, want $want_size)" >&2 + missing=1 + fi + [[ "$missing" -eq 0 ]] || return 1 + + # Cutover: frank-primary takes :7770 from frank-medium. Stop and mask + # the radiance unit (masking is mandatory: Restart=on-failure would + # restart it and re-take the port). Retiring the radiance unit (moving + # it to the home trash) happens after frank-primary is verified — this + # handler only releases the port. + systemctl --user stop frank-medium 2>/dev/null || true + systemctl --user mask frank-medium + # 7770 is a host-network bind: nothing new can be created until the old + # holder released it (the same gate as the 2026-09-16 cutover). + sleep 2 + wait_port_free 7770 || return 1 + + install_quadlet frank-primary dankserver/frank-primary/quadlet frank-primary + # No enable: generated units cannot be enabled on this host (v1 quadlet + # generator); boot persistence is the unit's [Install] WantedBy=default.target. + systemctl --user start frank-primary.service + # First start page-ins the dense weights and builds the graphs; + # HealthStartPeriod in the unit is 1800s to match. + wait_container_healthy frank-primary 1800 || { + podman logs --tail 40 frank-primary 2>&1 || true + return 1 + } +} + wait_for_health() { local service="$1" url="$2" limit="$3" local i @@ -1518,6 +1564,7 @@ deploy_service() { frank-large) deploy_llm_large ;; frank-small) deploy_llm_small ;; frank-medium) deploy_llm_medium ;; + frank-primary) deploy_llm_primary ;; hermes|hermes-dashboard|hermes-webui) deploy_hermes ;; romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;; jellyfin) deploy_jellyfin ;; diff --git a/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md b/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md new file mode 100644 index 0000000..ded11f0 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md @@ -0,0 +1,127 @@ +--- +type: configuration +title: frank-primary Lucebox DeepSeek V4 Runbook +description: ACTIVE frank-primary runbook — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaces frank-medium (vllm-radiance TP=2) on :7770, cutover 2026-09-26. +tags: + - frank-primary + - dankserver + - lucebox + - deepseek-v4 + - ds4 + - rocmfp23 + - gfx1201 + - r9700 + - quadlet + - moe +timestamp: 2026-09-26 +--- + +# frank-primary Lucebox DeepSeek V4 Runbook + +The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): the +[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving +`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328 +bytes, LFS oid +`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a +single Radeon AI PRO R9700 (gfx1201). It replaces +[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired on +2026-09-26. + +## Why this shape + +DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert +per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3 +down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB +the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM, +so it is served the way llama.cpp serves big models by default: the GGUF is +mmapped, pages pull from disk/page cache on demand, dense work (attention, +shared expert, embeddings, output projection) and the MLA KV cache run on +the R9700, and the routed experts are computed on CPU. The 246 GiB RAM +(upgraded for this deployment) holds the working set of active experts; the +rest streams from the 1.8 T NVMe. + +Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model +this size that allocation fails and the backend falls back to **hybrid +expert placement + layered decode** — that fallback is the intended serving +mode here, not an error. + +## Image + +`localhost/lucebox-rocm:gfx1201`, built on the host from +`github.com/Luce-Org/lucebox` `Dockerfile.rocm`: + +``` +git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git +cd lucebox +podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \ + --build-arg ROCM_VERSION=7.14.1-full \ + -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm . +``` + +- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**: + it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not + code-object compatible. +- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the + radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace + on a 7.x driver can segfault at model load. Only `-full` tags exist for + 7.14.x. +- The binary lands at `/opt/lucebox-hub/server/build/luce_server`. + +## Model + +`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`, +fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs +400 against HF's xethub CDN for this repo; see the fetch note in +`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in +`deploy_llm_primary`. + +## Deployed flags (2026-09-26) + +From `dankserver/frank-primary/frank-primary-entrypoint.sh` (the unit +bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash +flag set and cannot express the DS4 flags): + +| Setting | Value | +|---|---| +| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) | +| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) | +| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) | +| Chunk | `--chunk 8192` (the qualified profile's chunk) | +| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) | +| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) | +| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) | +| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) | +| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode | +| Identity | `--model-name frank-primary`, `--host 0.0.0.0 --port 7770` | + +## API surface + +OpenAI Chat Completions (streaming + tool metadata), `/v1/models`, +`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model +metadata — the fleet contract is unchanged from frank-medium. + +## Cutover procedure + +1. Image built and model fetched (gates in `deploy_llm_primary`). +2. `deploy.sh` → `deploy_llm_primary`: stops and **masks** the radiance + unit (`frank-medium`), waits for :7770 to free, installs + `dankserver/frank-primary/quadlet/frank-primary.container`, starts + `frank-primary.service`, waits for health (1800 s start period — first + start page-ins the dense weights and builds the graphs). +3. Verify: `/health` 200 is not enough — a real chat completion, a tool + call, and an Open WebUI completion. +4. Retire the radiance unit to the home trash (never `rm`); the MXFP6 + checkpoint payload stays on disk. + +## Rollback + +Unmask and start the radiance unit (`systemctl --user unmask frank-medium && +systemctl --user start frank-medium`) and stop/mask frank-primary. The +`localhost/radiance-vllm:0.13.0-e060895` image and the +`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk. + +## Verification (2026-09-26) + +_Filled in after the cutover: health, real completion, tool call, Open WebUI +completion, measured prefill/decode, `systemctl --user is-enabled/is-active` +state._ diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index b899542..7d4baa0 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -124,6 +124,11 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change +- [[frank-primary-lucebox-ds4]] — the ACTIVE frank-primary runbook: + the Lucebox runtime serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB + GGUF, 43-layer MoE, 256 routed experts top-6) on Dankserver's R9700 + with the routed experts computed on CPU from the mmapped page cache — + replaces frank-medium on :7770 (cutover 2026-09-26) - [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index 30d0ee3..6f950ce 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -69,6 +69,15 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" [ -z "$output" ] } +@test "deployment planner maps the frank-primary container dir to frank-primary only" { + run python3 "$PLANNER" --target dankserver --paths \ + dankserver/frank-primary/quadlet/frank-primary.container \ + dankserver/frank-primary/frank-primary-entrypoint.sh \ + dankserver/frank-primary/frank-primary-healthcheck.py + [ "$status" -eq 0 ] + [ "$output" = "frank-primary" ] +} + @test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" { run python3 "$PLANNER" --target dankclaw --paths \ scripts/frank-small.env scripts/frank-small.service scripts/serve-frank-small.sh