Engine v1.2.0 (Codeberg e95d397, Docker Hub tag 1.2.0, index digest
665775ed...) removes --expert-vs-cache-ratio. The explicit
--vram-weights-mib 26071 / --vram-kv-mib 3543 pair transcribes the live
1.0.13 boot ledger at ratio 0.82 (weights 25.46 GiB, KV 3.46 GiB per
card); bracket arms scale KV linearly in (1-r): 0.90 -> 1968, 0.95 -> 984.
Serve script, autoresearch harness, and bats gates follow; the bats
shape asserts are realigned to the deployed seqs=3/spec=2 sweep result
(95923d3, df6b5bb) they failed to track.
- download-frank-primary-swift15.sh: ukisai/Swift1.5-Qwen3.8-Flash-Next
(BF16, 131 shards + index + configs, 360037355038 B = 335.3 GiB, ungated)
via the vendored hfget.sh; verify = listed sizes + 131-shard weight-map
count + every shard's sha256 against the Hub LFS oid. Pattern cloned
from scripts/download-frank-primary.sh.
- qwen4exp-swift15-w4dn8-i8-hc8m.recipe: routed experts with DOWN at
block-fp8 (rtn codes=fp8_e4m3 block=128x128 scale=bf16, the served
FP8-native tile form; moe_gemm_fp8a8 reads it; minicpm5-2b is the
shipped bf16->fp8a8 precedent) and gate_up at w4nl64 GPTQ cd=3;
hc/MTP-fc fp8, trunk i8, lm_head i8, PLE fixed scale, draft_head u2
(inert). Protected-expert cast lines carry the BASE checkpoint's names
as a warm start; the derivation step rewrites them from Swift's own
calibration recording before the recipe is ever run (law documented
in the file).
Frank's order the same night. The 1024->2048 rung measured +4.6 %
prefill for 616 MiB on the 3090 (2026-10-05 boot pair); the 112k cut
left 1,751 MiB free, so it rides with ~1.1 GiB predicted margin.
Decode is ubatch-invariant; batch 4096 >= 2048 keeps the seam happy.
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.
serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).
frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
autoresearch.sh (dev box) scp's the candidate scripts/frank-primary.env
+ the bench script to dankserver and runs it over ssh.
scripts/autoresearch-frank-primary.sh (dankserver) follows the bench
policy: stop frank-primary (relink unit: stop, not mask), boot a
scratch radiance container (arbench) on the same two cards with the
candidate argv, poll /health, run ONE self-contained greedy completion
(fixed ~140k-char real-text prompt -> ~32k tokens, 256-token decode),
parse the engine timings into METRIC lines, tear the scratch down, and
restore production via a trap. Fresh prefix-cache tier per run for cold
prefill. Baseline validated: decode_tps=31.70, prefill_tps=465.4,
ttft_ms=93745, draft_accept=0.831 at prompt_n=43627.
The engine's kernel selector reads $RADIANCE_HOME/tune/<machine>.tune at
boot; $RADIANCE_HOME defaults to /opt/radiance, which lives in the image,
so a cache written by a --rm scratch container dies with it. Pre-create
$FRANK_PRIMARY_STATE_DIR/tune and bind-mount it at
/opt/radiance/share/radiance/tune so the rad-tune cache (generated in a
stopped-service window) survives and is read at every production boot.
The engine's create is not recursive: booting with only the state dir
mounted left the SSD prefix tier off ('cannot create
/data/kvcache/frank-primary: No such file or directory'). The serve
script now mkdir -p's the full path before podman run.
The first deploy refused: floor 178000 > actual 177109 MiB - the plan-only
estimate (172.76 GiB) prices the data planes but not the container's
metadata tail. Floor 177000 sits just under the real payload.
Plan-only measured the .rad at 172.76 GiB (176,906 MiB) - the checkpoint's
own fp8 mass, byte-stacked experts plus the PLE re-encoded at the
checkpoint's scale - not the ~161 GiB first estimate. Floor 158000 -> 178000.
Auto cache-sizing is OFF on this box: the boot ledger refused it at
36.9 GB (2 x full-entry) floor vs 35.7 GB free, so every deep agent
request re-prefills its whole 79-81k context (84.9-179.8 s measured on
serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. Pin
HALOGEN_CACHE_MB=16384 (>= one full-slot entry at ~70 KB/token) — the
documented override; the per-store reserve re-check keeps the host safe.
Also: the serve script's 'podman run -d > LOG' captured only the
container ID — the runbook's boot-ledger grep read a one-line hex file
and the ledger existed nowhere on disk. Foreground 'podman run' makes
the ledger land in the log file as documented; exit-code and ExecStop
semantics are unchanged. Test seam updated for both.
The reader opens every index-named file in sorted order and emplaces
EVERY tensor each file physically holds, first-wins (checkpoint.cpp
add_shard + shard_list). The raw model-000NN shards carry tensors named
...ple.ple_embedding.ngram_embedding.shard_0.weight ([2500012,160] fp8)
- the identical name the plugin asks the merged table under - so
ple.merged.safetensors lost the emplace race and the plan read one raw
shard as the codes (declared [320001536,160] vs codes [2500012,160]).
Rename to 00-ple.merged.safetensors: '00-' wins the sort, the merged
bf16 tensor wins the name, table_bf16 declares and the rtn rule makes
the fp8 container form.
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
Run 3 rewrote the staged index THROUGH the relative symlink into the raw
checkpoint: the raw index.json was restamped with the stacked layout
(605 KB smaller), the raw verify then failed 143/144 and the overnight
run died before stacking. Two fixes:
- stack-qwen38-official-fp8.py: index/config writes are now temp+os.replace
(an atomic swap replaces the symlink itself; the raw tree stays pristine).
- build-frank-primary-rad.sh: the stage's index.json is COPIED like
config.json; only shards stay relative symlinks.
- release the numpy frombuffer view before closing the source mmap
(Python 3.14 BufferError: cannot close exported pointers exist).
- stacked scale planes rename: checkpoint.cpp scale_name() appends
_scale_inv to the STACKED tensor name verbatim (no .weight segment) -
...experts.gate_up_proj_scale_inv, not ...gate_up_proj.weight_scale_inv.
Boot ledger lines, 19,404 used / 4,721 MiB free (the q8_0 proof), decode arms (66.1 t/s @0.8, 66.9 @0, 58.9 @recipe 1.0 with acceptance 52.1%/52.2%/44.8%), vision Red probe, reasoning_effort xhigh/high contract, e2e 12-crawl transient + full-gate green rerun. Unit Description updated in-repo; the relinked unit picks it up at the next deploy/restart (no bounce for cosmetics). 360/360 suite green.
The parallel session's uncommitted swap, committed at Frank's direction
(commit whatever's in the working tree):
- llama.cpp master on the RTX 3090 now serves ukisai's
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE
131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through
the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to
rollback target.
- env/serve/download + seam twins updated together (deployer payload gates
rode the previous commit's deploy-remote.sh).
- docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology
and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135
seam green).
- comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/
workflows/ (12 KiB).
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:
- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
--quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
(experts byte-stacked into the plugin's stacked layout; PLE's 128
scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
scale; the rtn rule re-encodes at that same scale - codes round-trip
exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
TTM preflight fixed for kernels with pages_limit only (no pages_max -
an absent knob is skipped, not read as zero); pages_limit raised to
52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
+ health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
Dockerfile); payload + image stay on disk until the window closes.
deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
FreeToken's /health returns 200 with {"status":"loading"} while the
engine loads — the FastAPI front end binds before the scheduler is up.
Measured 2026-10-06: the deploy gate went green after 6s while 131 shards
were still paging in (vLLM 503'd during load, which made the bare curl
honest there). The frank-primary gate now loops until status:ok; the
MainPID stray-port guard is unchanged.
Measured first boot: the auto attention backend is qsa_sparse, which
overrides page size to 64 and refuses --num-tokens 300000 ('nearest valid
values: 299968 or 300032'). 299,968 x 24 KiB = 7.0 GiB still clears the
177,356-token harness worst case and one native 262,144 conversation.
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201
built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201
(scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated
--no-deps install per docs/install_amd.md).
Shape (first-boot conservative): ONE R9700 (second card idle;
--tensor-parallel-size exists for future 2-card tuning), KV pool
--num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token
harness worst case and one native 262,144 conversation), MoE LRU
--moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens
36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml
flips reasoningContentField on the harness side). Experts + PLE pin
~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity.
Container identity renamed to the unit name (Frank's call): frank-primary;
vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image
gate becomes podman image exists (a local build carries no RepoDigests;
the old digest grep refused exactly this shape). Rollback tag:
frank-primary-vllm-shapeG-20261003.
- FROM pinned to sha256:a223aee1... (the pulled rocm/pytorch rocm7.14
pytorch 2.11.0 manifest); a re-pushed tag cannot change the stack.
- The pip-ROCm torch ships libhiprtc.so.7 only; the PR's _cpu_moe links
-lhiprtc and ld needs the unversioned symlink in the already-linked
_rocm_sdk_core/lib (first build died: cannot find -lhiprtc).
The rogue session's swap left this env reprice uncommitted in the tree
(2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn
layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16
lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because
the compaction invariant needs slot >= 191,148. The only lever that
keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K
stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch
script manifest already flipped in 3ee88d7. MEASURE after boot; under
1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
The rogue session's Ornith commits (3ee88d7/77decb5) swapped the env and
fetch script but left the deploy handler gates and the seam asserting
the MiMo-Q6K paths — the seam failures 195-203 and a deploy that gates
a file the unit no longer reads. Flip both to Ornith-Q8 (Q8_0 GGUF +
BF16 mmproj), keeping the anti-drift twin structure intact.
The 11-admin spec walked links that terminate on FrankBook-Pro
(192.168.0.112), which sleeps when idle — every deploy gate timed out
there (measured 2026-10-05, two deploy windows). Split: the frankbook
links move to tests/e2e/specs/19-frankbook.spec.ts under a frankbook
project that run.sh excludes by default and deploy.sh triggers ONLY
when a frankbook blast-radius path changed (the spec, frankbook-pro/,
the homepage registry, the Caddyfile arr vhosts). deploy.sh computes
the trigger from the same BASE..HEAD revision range the plan uses.
The first constants were wrong in three ways, found while resuming the
checkpoint (the file was COMPLETE; the manifest was not):
- checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not
35,880,329,102 / 40874e7a...), 1352 tensors per the card;
- the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole
at /tokenizer (model-card docker-run parity; the engine's
HALOGEN_TOKENIZER default is the directory) — there is no
frank-large.tokenizer.gguf;
- plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the
fetch script size-checks them and sha-verifies only the two LFS
objects (checkpoint, tokenizer.json).
Gates flipped together (serve + deployer twins): tokenizer dir must be a
real directory and every named file a regular file, no symlinks (HF
snapshot links dangle in the mount). Fetch made idempotent: a file at
the published size skips the transfer (resuming it is a CDN 416 under
set -e). Seam rewritten to the same contract, 17/17.
curl drops the Range header across the HF /resolve/ 302, so '-C -'
against the resolve URL pulls the 1 KB redirect body and exits 0 with
the file still 14 MiB short (measured 2026-10-05). Resolve the
effective URL first; the Xet CDN honors Range.
Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest
dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the
.hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2)
on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law).
Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images)
and the censored base (rollback = revert this commit; payload, fork
build and build-rocmfpx.sh pin stay on disk).
- serve script: podman run under the root unit, exec podman wait so
Restart=always is real; gates image digest, checkpoint byte-count,
flat regular-file tokenizer, GPU nodes; engine stdout to log.
- deploy handler: gates the artifacts, sweeps the legacy bare
llama-server + old container name by NAME (never pkill llama-server:
frank-small), retires the 2026-09 quadlet, port cutover 7770.
- fetch script: -C - resume, -f, byte count, sha256, --verify-only.
- seam rewritten (17 tests): halogen is now the deployed runtime.
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048.
Why the ubatch move is right on THIS card: the prefill curve is
card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512
and every step up costs GiB for no speed; on this CUDA 3090 it climbs
monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423
t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung.
The logical batch is free either way: two sweeps agree that at a fixed
ubatch, 2048/4096/8192 give identical prefill speed and identical peak
VRAM, because the compute buffers are sized by the ubatch alone.
The risk is all in the ubatch, and it is documented rather than assumed:
the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this
replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch
2048 failed the scratch boot" with MORE free than that. Compute buffers
are allocated at graph capture during boot, so a failure is loud and fast;
the step-down rungs (4096/1536, then 4096/1024) are named in the env.
The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and
UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test
asserts the argv follows the shipped env plus the llama.cpp invariant
ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps
the seam green, inverting the pair to 2048/4096 turns it red.
The prefix-caching OFF/ON experiment was confounded (both arms ran on
the marginal speed); the align-mode re-seed theory is unproven, not
disproven. Caching ships ON with zero gdn_hip faults under heavy
re-send traffic on stable RAM; knob stays commented. Corrects the
env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.
Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.
Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.
Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
Two bash defects in the 9d8b185 prefix-cache wiring, both measured live
on the 2026-10-05 deploy attempt (unit crash-looped 190x, exit 127,
endpoint down for the whole 3600 s health window):
1. A # comment block inside the backslash-continued podman run
terminates the continuation: the comment swallows the joined line
and the flag line then runs as its own command
('--no-enable-prefix-caching: command not found'). The comment moves
above the run block, with a no-comments-in-chain note.
2. The ${VAR:+A}${VAR:--B} double-expansion is ambiguous: when the
variable is set, ${VAR:--B} parses as a SUBSTRING OFFSET, not the
:- default — measured render was '--no-enable-prefix-caching1'.
The flag is now branched explicitly into
FRANK_PRIMARY_PREFIX_CACHE_FLAG (exactly one of the two flags
reaches the engine; unset keeps the historical flag).
Seam: the render test sources the shipped env through the branch and
asserts the exact flag word the engine receives (render-assert, the
same pattern the spec knobs use). bash -n clean; all 22 frank-primary
seams pass.
Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.
Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).
RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.
Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.
Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.
Frank