dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0

Browse Source 
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).

The deployed contract:

- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
  and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
  -ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
  over PCIe loses at 3.7 t/s, computing in RAM does not care).
  per_layer_token_embd.weight=CPU is required by the shared MTP head's
  borrow path (a GPU-resident borrow crashes the draft load).
  mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
  (mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
  draftless ngram-mod matcher chained behind it — the only legal way to
  layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
  autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
  Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
  by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
  with Qwen-recommended sampling riding compat.extraBody.

The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.

Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.

Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).

models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].

Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).

Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
This commit is contained in:
Frank committed 2026-09-26 19:45:28 -07:00
1 parent d89813363e
commit 8500600d09
25 files changed
+856 -544
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_primary_unit.bats
+143 -69
View File
@@ -1,94 +1,168 @@
#!/usr/bin/env bats
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
#
# This file asserts the LUCEBOX shape (the 2026-09-26 cutover that replaced
# frank-medium): a rootless Quadlet running the lucebox DS4.1 engine on the
# two R9700s, with the model mmapped out of /home/dank/models and the
# entrypoint/healthcheck mounted straight out of this checkout.
# This file asserts the llama.cpp ROOT-UNIT shape (the 2026-09-26 swap that
# replaced the Lucebox DS4.1 Quadlet): the unsloth b11030-mix prebuilt on
# the two R9700s (gfx1201) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0
# with the shared Q8_0 MTP head, the BF16 projector, and the
# draft-mtp + ngram-mod chain. The seam pins what the flags alone do not
# say: the CPU-override placement law (experts and the 51B n-gram table
# are COMPUTED in host RAM, never streamed), mmap staying on, the
# shared-head borrow requirement (per_layer_token_embd on CPU), the
# explicit projector (never autoloaded by this build), HIP env hygiene,
# and the deployer's byte-count gates over a six-part payload.
load helpers
UNIT="$REPO_ROOT/dankserver/frank-primary/quadlet/frank-primary.container"
EP="$REPO_ROOT/dankserver/frank-primary/frank-primary-entrypoint.sh"
HC="$REPO_ROOT/dankserver/frank-primary/frank-primary-healthcheck.py"
UNIT="$REPO_ROOT/scripts/frank-primary.service"
SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "the frank-primary quadlet file and its mounted scripts exist" {
@test "frank-primary unit, serve script and env exist in scripts/" {
[ -f "$UNIT" ]
[ -x "$EP" ]
[ -f "$HC" ]
# The healthcheck is python; a syntax error would leave the container
# permanently unhealthy with the failure hidden inside HealthCmd.
python3 -m py_compile "$HC"
[ -x "$SERVE" ]
[ -f "$ENV_FILE" ]
}
@test "frank-primary is boot-persistent and self-healing" {
# Rootless Podman does not auto-start bare containers and restart:
# unless-stopped does not survive shutdown's explicit stop. The unit must
# carry both the boot symlink and a restart policy (2026-08-26 incident:
# a compose-only Caddy never came back after a reboot).
grep -q '^WantedBy=default.target' "$UNIT"
grep -qE '^Restart=(always|on-failure)' "$UNIT"
@test "frank-primary is boot-persistent, self-healing and unprivileged" {
# Root unit parity with frank-large/frank-small: enable is the [Install]
# section, the restart policy survives a crash, and the mmap'd payload
# needs an unbounded memlock.
grep -q '^WantedBy=multi-user.target$' "$UNIT"
grep -q '^Restart=on-failure$' "$UNIT"
grep -q '^RestartSec=' "$UNIT"
grep -q '^User=dank$' "$UNIT"
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
}
@test "frank-primary binds the host network and needs no published port" {
# luce_server binds 0.0.0.0:7770 directly, so the container takes the
# host netns. A PublishPort would be dead config.
grep -q -- '--network=host' "$UNIT"
! grep -q '^PublishPort=' "$UNIT"
@test "frank-primary runs the repo's own serve script with the repo's env" {
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh$' "$UNIT"
grep -q 'FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env' "$UNIT"
}
@test "frank-primary reaches both R9700s and hides the Raphael iGPU" {
# /dev/kfd + /dev/dri via AddDevice; HIP_VISIBLE_DEVICES=0,1 keeps the
# host's Raphael iGPU (index 2) out of the enumeration so the engine's
# device picks cannot land on it.
grep -q '^AddDevice=/dev/kfd' "$UNIT"
grep -q '^AddDevice=/dev/dri' "$UNIT"
grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT"
@test "frank-primary drafts with the shared MTP head chained behind ngram-mod" {
# The chain is draft-mtp (context-backed) + ngram-mod (draftless): the
# only legal way to layer two spec implementations in one process. The
# shared Q8_0 head borrows the target's embedding tensors — which is
# why per_layer_token_embd.weight must be CPU-resident (a GPU-resident
# borrow crashes the draft load with "invalid vector subscript").
grep -q -- '--spec-type draft-mtp,ngram-mod' "$SERVE"
grep -q '^SPEC_DRAFT_N_MAX=2$' "$ENV_FILE"
grep -q -- '--spec-draft-n-max "$SPEC_DRAFT_N_MAX"' "$SERVE"
grep -q 'DRAFTER_FILE="$MODEL_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf"' "$SERVE"
grep -q -- '--spec-draft-device "$SPEC_DRAFT_DEVICE"' "$SERVE"
}
@test "frank-primary never names a bare HIP-reserved env var" {
# The HIP runtime reads bare TEMP for its JIT cache path (the 2026-08-13
# incident: TEMP=1.0 sent the JIT cache to path "1.0" and segfaulted every
# load). HOST and GPU_MAX_HEAP_SIZE are the same class. The unit's
# Environment= lines must not define them; the entrypoint exports
# GPU_MAX_HW_QUEUES (explicitly not GPU_MAX_HEAP_SIZE).
! grep -qE '^Environment=(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$UNIT"
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$EP"
@test "the bandwidth-huge tensors are computed in host RAM, not streamed" {
# The x4-pair law: streaming experts over PCIe loses (3.7 tok/s) while
# computing them in RAM does not care (20+ tok/s on the same model
# shape). exps=CPU plus the n-gram embedding table on CPU; the PLE
# table must NEVER carry a GPU target (the 23 -> 13.4 t/s regression).
grep -q '^OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU$' "$ENV_FILE"
grep -q -- '--override-tensor "$ot"' "$SERVE"
# ...and mmap must stay on: --no-mmap forces the 51B-param table into
# anonymous memory and the cgroup kill is silent.
! grep -q -- '--no-mmap' "$SERVE"
! grep -q '^MMPROJ_ON_GPU' "$ENV_FILE"
}
@test "the unit mounts the entrypoint and healthcheck from the checkout" {
# deploy.sh git-pulls this checkout before dispatching, so a Volume
# reference is how an edit here reaches the container without an image
# rebuild. Absolute host paths, read-only.
grep -q 'Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-entrypoint.sh:/usr/local/bin/frank-primary-entrypoint.sh:ro' "$UNIT"
grep -q 'Volume=/home/dank/serverAppConfigs/dankserver/frank-primary/frank-primary-healthcheck.py:/usr/local/bin/frank-primary-healthcheck.py:ro' "$UNIT"
@test "the projector is explicit, never autoloaded" {
# This build never autoloads mmproj from the model directory: without
# the flag the server comes up healthy and text-only, answering image
# requests with 500 (the frank-medium 2026-09-17 trap).
grep -q -- '--mmproj "$MMPROJ_FILE"' "$SERVE"
grep -q 'MMPROJ_FILE="$MODEL_ROOT/mmproj-BF16.gguf"' "$SERVE"
}
@test "first-start patience is configured on both sides of the health check" {
# Cold start page-ins the dense weights and builds the graphs; the
# container must not be killed as unhealthy while that happens, and the
# deployer must wait the same order of magnitude.
grep -q '^HealthStartPeriod=1800s' "$UNIT"
grep -q '^TimeoutStartSec=1800$' "$UNIT"
grep -q 'wait_container_healthy frank-primary 1800' \
"$REPO_ROOT/scripts/deploy-remote.sh"
@test "frank-primary is vision-serving but holds the single fleet port" {
grep -q '^PORT=7770$' "$ENV_FILE"
grep -q '^BIND_HOST=0.0.0.0$' "$ENV_FILE"
grep -q -- '--alias frank-primary' "$SERVE"
grep -q -- '--host "$BIND_HOST"' "$SERVE"
grep -q -- '--port "$PORT"' "$SERVE"
}
@test "the deployer refuses to start without image and complete payload" {
# A 377 GiB partial download must never be served: the handler gates on
# the image being present and the model file existing at its exact
# declared byte count (stat -c %s, never wc -c — wc read the whole stream
# for minutes on 2026-09-26).
grep -q 'deploy_llm_primary() {' "$REPO_ROOT/scripts/deploy-remote.sh"
grep -q 'podman image exists localhost/lucebox-rocm:gfx1201' \
"$REPO_ROOT/scripts/deploy-remote.sh"
grep -q 'want_size=405349859328' "$REPO_ROOT/scripts/deploy-remote.sh"
grep -q 'stat -c %s' "$REPO_ROOT/scripts/deploy-remote.sh"
@test "one stream, the model's full native window" {
local ctx parallel
ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
parallel="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
[ -n "$ctx" ] && [ -n "$parallel" ]
[ "$ctx" -eq 262144 ]
[ "$parallel" -eq 1 ]
grep -q -- '--ctx-size "$CONTEXT_SIZE"' "$SERVE"
grep -q -- '--parallel "$PARALLEL_SLOTS"' "$SERVE"
}
@test "the deployer gates on the port actually being free" {
# 7770 is a host-network bind: the previous holder must release it before
# the new container can be created, or pasta's bind fails at create time
# and the failure hides behind Restart=.
grep -q 'wait_port_free 7770' "$REPO_ROOT/scripts/deploy-remote.sh"
@test "reasoning is extracted to reasoning_content and tools are legal" {
# The harness contract: llama.cpp endpoints emit reasoning_content
# (the deepseek format is the one that fills it), and tools require
# --jinja (the ik_llama 500 lesson). The Qwen3.8 template gates
# reasoning_effort to {xhigh, medium, low} — folded in models.yml.
grep -q -- '--jinja' "$SERVE"
grep -q -- '--reasoning auto' "$SERVE"
grep -q -- '--reasoning-format deepseek' "$SERVE"
grep -q -- '--reasoning-budget -1' "$SERVE"
}
@test "the greedy server default is explicit" {
# Deterministic e2e probes; the harness overrides sampling per request.
grep -q -- '--temp 0' "$SERVE"
grep -q -- '--top-p 1.0' "$SERVE"
grep -q -- '--top-k 0' "$SERVE"
grep -q -- '--min-p 0.0' "$SERVE"
}
@test "both R9700s are visible and the Raphael iGPU is not" {
# The host has a third (Raphael) GPU the server must never place
# tensors on; the bundled ROCm runtime is the only one linked.
grep -q 'export HIP_VISIBLE_DEVICES=0,1$' "$SERVE"
grep -q 'export LD_LIBRARY_PATH="$RUNTIME_DIR' "$SERVE"
}
@test "no bare HIP-reserved env var is defined" {
# HIP reads bare TEMP for its JIT cache path (2026-08-13: TEMP=1.0
# segfaulted every load), and HOST/GPU_MAX_HEAP_SIZE are the same
# class. The bind address is BIND_HOST, not HOST.
! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
grep -q '^BIND_HOST=' "$ENV_FILE"
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
}
@test "the serve script refuses to start on a half-finished download" {
# Six split parts plus the head and the projector: a missing tail part
# fails at LOAD time with a confusing partial-mmap error, so every
# artifact is checked before the server execs.
grep -q 'missing asset' "$SERVE"
grep -q 'download-frank-primary.sh' "$SERVE"
}
@test "the prompt-cache cap is set below the RAM the host can spare" {
grep -q '^CACHE_RAM=16384$' "$ENV_FILE"
grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
}
@test "the deployer retires the lucebox quadlet, gates every part and waits out the port" {
# The cutover order: the user-scope quadlet is stopped and its
# generated .container moved to dated home trash BEFORE the root unit
# binds 7770 (nothing starts until the port is released — the 2026-09-26
# and 2026-09-16 cutover failures were both here).
grep -q 'retire_lucebox_primary_quadlet' "$REMOTE_DEPLOY"
grep -q 'systemctl --user stop frank-primary.service' "$REMOTE_DEPLOY"
grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"
grep -q 'systemctl restart frank-primary.service' "$REMOTE_DEPLOY"
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
}
@test "the deployer gates the six-part payload at declared byte counts" {
# stat -c %s, never wc -c — wc read the whole 377 GiB lucebox file for
# minutes on 2026-09-26 before it could reach the cutover. A partial
# download must never be served.
grep -q 'LLM_PRIMARY_PART_SIZES' "$REMOTE_DEPLOY"
grep -q 'stat -c %s' "$REMOTE_DEPLOY"
grep -q '2786568256' "$REMOTE_DEPLOY"
grep -q '907542944' "$REMOTE_DEPLOY"
grep -q 'wait_for_health frank-primary http://127.0.0.1:7770/health 1800' "$REMOTE_DEPLOY"
}