The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling;
decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the
root-cause evidence chain, tuning levers, and co-tenancy traps
- AGENTS.md: fleet topology row + Active Services section for the ik
deployment; halogen demoted to rollback; bench policy unit type fixed
- LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE,
halogen-flash-server retired)
New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records
the hardware swap, the self-contained gfx1201 build, the measured layer-vs-
tensor decision (llama-bench multi-device syntax gotcha included), the
tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm
posture. AGENTS.md, README, docs indexes, Dankserver System Specs,
deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename;
freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED.
Work log entry added.
Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger
printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous
2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also
documents the ~45-60 s first-request PING silence after a restart (the
healthcheck's HealthStartPeriod covers it; do not restart the unit).
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).
Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.
Answering "does the FreeToken runtime use sampling arguments?" -- yes, and
nothing in our config revealed it. serve-frank-small.sh passes no sampling
flags; the behaviour is the runtime default:
freetoken/server/args.py:31 sampling_defaults: str = "model"
which fills temperature/top_k/top_p from the checkpoint's
generation_config.json -- this one ships do_sample:true, temp 1.0, top_k 20,
top_p 0.95. Proved live: three identical requests with no sampling params
produced three different reasoning traces (487/540/577 chars).
The asymmetry with frank-large is the part that will bite someone: a silent
client samples on frank-small and decodes greedy on frank-large, so the same
request is reproducible on one endpoint and not the other. Documented with the
determinism options (explicit temperature:0, or --sampling-defaults none).
Also recorded two request-level traps found during the probe: thinking eats the
whole max_tokens budget and returns empty content with finish_reason length,
and the API accepts unknown fields with a 200, so a 200 proves nothing about
whether a key was understood.
Ledger confirms the change landed: "3 slot(s) over one 524288-position pool",
"slots: 1 -> 3 POOLED, 111.0 MiB each", held 103.5 -> 103.4 GiB.
Recorded the oversubscription nuance that the old row missed: the slots were
never the binding constraint on context. 4 x 262144 exceeds the 524288 pool,
so admission was pool-bound all along -- two full-length conversations, or
four at 131k. Lowering the cap trims simultaneous requests, not capacity.
Seven restarts measured. New section "The reserve cannot buy air": reserve 16
was deployed and held exactly what 15 holds — 103.5 GiB, 7.1 GiB left to the
host. The flag is a term in the fit-check inequality, not a reservation, so it
has only two regimes: below the pool's requirement it changes nothing
observable, above it the pool silently disappears.
Sizing rule now defers to the engine's printed requirement (budget >= 36.5 for
524288 at max_tok 32768, from "need ~35.0 GiB plus 1.5 of margin") over my
modelled 37.0, and the ledger table carries the reserve-16 row.
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.
Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.
Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:
kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all
so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.
Two corrections to what I wrote earlier today:
1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
spare it" line, print ONLY when the fit check had to clamp something. The
09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
of a warning in old logs was never evidence of headroom.
2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
stated budget -- and has been the whole time. It boots and serves with
12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
floor, not a wall.
The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
The first draft of these notes estimated the doubled pool at 103.5 GiB
held / ~5 GiB host remainder. Both numbers were wrong — they costed the
KV alone and ignored that the arena shrinks when MAX_TOK drops. Measured:
pool 262144 / max_tok 32768 -> 96.3 GiB held, 12.1 GiB host left
pool 524288 / max_tok 16384 -> 94.6 GiB held, 13.8 GiB host left
The doubled pool is 1.7 GiB cheaper overall. Also records the failed
first attempt verbatim, since the silent-revert failure mode is the thing
a future operator most needs to recognise, and adds the HALOGEN_MAX_TOK
row to the deployed-flags table (the "baked 32768" line it overrode was
stale).
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
Prior docs claimed unknown tags silently serve llama3.2:1b. Live probes on the
live endpoint disprove it: /v1/chat/completions answers HTTP 200 with a flat
{"error":"Failed to load <tag> model!"} body and no choices; /v1/completions
answers a nested OpenAI-shaped server_error (code 500) under HTTP 200. Valid
tags route per-request to the named model (response echoes the requested tag);
no restart needed to switch between registry tags. Corrected in the runbook,
the AGENTS.md runtime section, the serve-script header comment, and the
historical FastFlowLM incident note.
frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, port 52626) is live on
Dankclaw. Documents the first-deploy crash-loop root cause: iommu=off on the
kernel cmdline breaks the XDNA2 SVA bind (amdxdna_drm_open ret -19), so every
/dev/accel/accel0 open fails with ENODEV; fixed with iommu=pt + reboot.
Whole-registry single-port serving with per-request routing by exact FLM tag;
unknown tags fall back to llama3.2:1b (no 404). Health = GET /v1/version.
Vision + audio towers proven live (image_url color probe; audio control-pair
token-count probe). Updates AGENTS.md fleet table + runtime section, both
index pages, and the change log.
- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook:
runtime build recipe (WEBUI=OFF gotcha), deployed serving profile,
fa-on-works-unpatched evidence, MTP heads documented unloadable,
swap procedure + 2026-08-30 validation record
- agentionai-fast-imatrix marked RETIRED (rollback record kept)
- freetoken-frank-default-small: model/lineage/template sections now
Huihui abliterated FP8; new operational quirk: chat API 422 without
explicit model field
- LLM index + AGENTS fleet table/runtime sections current; both
services' history chains fixed (small was still Tiel in AGENTS.md)
- worklog entry added; models.json contextWindow for large corrected
to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Frank