23,120 used / 1,005 free at boot, 763 settled after a vision probe
(encoder buffers resident). Rung cost 746 MiB, not the transferable
616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode
unchanged as the law says. 2048 = ceiling for this shape.
Frank's order the same night. The 1024->2048 rung measured +4.6 %
prefill for 616 MiB on the 3090 (2026-10-05 boot pair); the 112k cut
left 1,751 MiB free, so it rides with ~1.1 GiB predicted margin.
Decode is ubatch-invariant; batch 4096 >= 2048 keeps the seam happy.
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.
serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).
frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
autoresearch.sh (dev box) scp's the candidate scripts/frank-primary.env
+ the bench script to dankserver and runs it over ssh.
scripts/autoresearch-frank-primary.sh (dankserver) follows the bench
policy: stop frank-primary (relink unit: stop, not mask), boot a
scratch radiance container (arbench) on the same two cards with the
candidate argv, poll /health, run ONE self-contained greedy completion
(fixed ~140k-char real-text prompt -> ~32k tokens, 256-token decode),
parse the engine timings into METRIC lines, tear the scratch down, and
restore production via a trap. Fresh prefix-cache tier per run for cold
prefill. Baseline validated: decode_tps=31.70, prefill_tps=465.4,
ttft_ms=93745, draft_accept=0.831 at prompt_n=43627.
The writer's 31b1ec3 added LLM/frank-large-flash-server-v2-swift15.md
but did not wiki-link it from index.md, which failed the docs-index
seam (test 135). Add the link as the new ACTIVE frank-large runbook
(supersedes halogen 0.1.4).
The engine's kernel selector reads $RADIANCE_HOME/tune/<machine>.tune at
boot; $RADIANCE_HOME defaults to /opt/radiance, which lives in the image,
so a cache written by a --rm scratch container dies with it. Pre-create
$FRANK_PRIMARY_STATE_DIR/tune and bind-mount it at
/opt/radiance/share/radiance/tune so the rad-tune cache (generated in a
stopped-service window) survives and is read at every production boot.
AGENTS.md: row + section carry the measured 177,109 MiB container, floor
177000, live cutover 2026-10-06. Runbook: the three convert laws the live
runs proved (scale-name, file-shadow, symlink-through), the measured boot
(125.26 GiB page-in 38.1 s, PLE pin 47.68 GiB/43.9 s, ~2 min to health),
the MTP/vision/tool/reasoning contract results, and the SSD tier fix.
The engine's create is not recursive: booting with only the state dir
mounted left the SSD prefix tier off ('cannot create
/data/kvcache/frank-primary: No such file or directory'). The serve
script now mkdir -p's the full path before podman run.
Stop-window A/B, same 80k filler prompt, 128-token essays, runtime
timings: HALOGEN_FA_TQ=256 (upstream issue #12's carve workaround) moves
decode ±0.3% (noise, draft acceptance identical) and costs 9.5% prefill
on this topology. The depth collapse sits deeper than the FA tile knob;
the default stands. The harness's slow/advisor roles moved to
frank-primary (fp8 KV 7.32 KiB/token = 8.7x less KV read at depth).
The first deploy refused: floor 178000 > actual 177109 MiB - the plan-only
estimate (172.76 GiB) prices the data planes but not the container's
metadata tail. Floor 177000 sits just under the real payload.
Probe on the live endpoint (2026-10-06): 17.3k cold prefill 27.0 s ->
warm turn resumes at cache_n 16,384, prefill 1.7 s. The boot WARNING
'smaller than ONE full-context entry' sizes against the native 262,144
window, not our 200,700 slot — documented as expected in the checklist.
Tuning notes record the depth law (decode 5.5 t/s @ 46k -> 3.3-3.8 @ 80k
vs 30-50 shallow; KV reads ~8 GB/s effective; no supported flag touches
decode) and the AGENTS.md unit paragraph regains the unit-facts
parenthetical the reflow dropped.
Plan-only measured the .rad at 172.76 GiB (176,906 MiB) - the checkpoint's
own fp8 mass, byte-stacked experts plus the PLE re-encoded at the
checkpoint's scale - not the ~161 GiB first estimate. Floor 158000 -> 178000.
Auto cache-sizing is OFF on this box: the boot ledger refused it at
36.9 GB (2 x full-entry) floor vs 35.7 GB free, so every deep agent
request re-prefills its whole 79-81k context (84.9-179.8 s measured on
serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. Pin
HALOGEN_CACHE_MB=16384 (>= one full-slot entry at ~70 KB/token) — the
documented override; the per-store reserve re-check keeps the host safe.
Also: the serve script's 'podman run -d > LOG' captured only the
container ID — the runbook's boot-ledger grep read a one-line hex file
and the ledger existed nowhere on disk. Foreground 'podman run' makes
the ledger land in the log file as documented; exit-code and ExecStop
semantics are unchanged. Test seam updated for both.
The reader opens every index-named file in sorted order and emplaces
EVERY tensor each file physically holds, first-wins (checkpoint.cpp
add_shard + shard_list). The raw model-000NN shards carry tensors named
...ple.ple_embedding.ngram_embedding.shard_0.weight ([2500012,160] fp8)
- the identical name the plugin asks the merged table under - so
ple.merged.safetensors lost the emplace race and the plan read one raw
shard as the codes (declared [320001536,160] vs codes [2500012,160]).
Rename to 00-ple.merged.safetensors: '00-' wins the sort, the merged
bf16 tensor wins the name, table_bf16 declares and the rtn rule makes
the fp8 container form.
Runbook reshaped: 2 x 131072 no-spec identity, slot-2 arithmetic (drafter ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), ngram-mod skipped on the recurrent-rollback law, MTP references kept (44.8-52.1% acceptance, 58.9-66.9 t/s), rollback = one revert of 812d9eb. AGENTS fleet row + section, LLM/index.md intro + entry, root index entry all flipped to the 2026-10-06 shape. 360/360 suite green; e2e gate 33/33 green at deploy.
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
Run 3 rewrote the staged index THROUGH the relative symlink into the raw
checkpoint: the raw index.json was restamped with the stacked layout
(605 KB smaller), the raw verify then failed 143/144 and the overnight
run died before stacking. Two fixes:
- stack-qwen38-official-fp8.py: index/config writes are now temp+os.replace
(an atomic swap replaces the symlink itself; the raw tree stays pristine).
- build-frank-primary-rad.sh: the stage's index.json is COPIED like
config.json; only shards stay relative symlinks.
- release the numpy frombuffer view before closing the source mmap
(Python 3.14 BufferError: cannot close exported pointers exist).
- stacked scale planes rename: checkpoint.cpp scale_name() appends
_scale_inv to the STACKED tensor name verbatim (no .weight segment) -
...experts.gate_up_proj_scale_inv, not ...gate_up_proj.weight_scale_inv.
Boot ledger lines, 19,404 used / 4,721 MiB free (the q8_0 proof), decode arms (66.1 t/s @0.8, 66.9 @0, 58.9 @recipe 1.0 with acceptance 52.1%/52.2%/44.8%), vision Red probe, reasoning_effort xhigh/high contract, e2e 12-crawl transient + full-gate green rerun. Unit Description updated in-repo; the relinked unit picks it up at the next deploy/restart (no bounce for cosmetics). 360/360 suite green.
The parallel session's uncommitted swap, committed at Frank's direction
(commit whatever's in the working tree):
- llama.cpp master on the RTX 3090 now serves ukisai's
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE
131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through
the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to
rollback target.
- env/serve/download + seam twins updated together (deployer payload gates
rode the previous commit's deploy-remote.sh).
- docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology
and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135
seam green).
- comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/
workflows/ (12 KiB).
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:
- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
--quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
(experts byte-stacked into the plugin's stacked layout; PLE's 128
scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
scale; the rtn rule re-encodes at that same scale - codes round-trip
exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
TTM preflight fixed for kernels with pages_limit only (no pages_max -
an absent knob is skipped, not read as zero); pages_limit raised to
52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
+ health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
Dockerfile); payload + image stay on disk until the window closes.
deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
FreeToken has no /metrics endpoint — the job scraped 404s forever, a
structural never-green, not an outage (the Lucebox DS4.1 precedent). The
LlmEndpointDown alert regex self-adjusts (frank-.* matches existing series
only). Comments updated; seam now asserts frank-primary's absence. If
FreeToken grows /metrics, re-add the job.
FreeToken's /health returns 200 with {"status":"loading"} while the
engine loads — the FastAPI front end binds before the scheduler is up.
Measured 2026-10-06: the deploy gate went green after 6s while 131 shards
were still paging in (vLLM 503'd during load, which made the bare curl
honest there). The frank-primary gate now loops until status:ok; the
MainPID stray-port guard is unchanged.
Measured first boot: the auto attention backend is qsa_sparse, which
overrides page size to 64 and refuses --num-tokens 300000 ('nearest valid
values: 299968 or 300032'). 299,968 x 24 KiB = 7.0 GiB still clears the
177,356-token harness worst case and one native 262,144 conversation.
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201
built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201
(scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated
--no-deps install per docs/install_amd.md).
Shape (first-boot conservative): ONE R9700 (second card idle;
--tensor-parallel-size exists for future 2-card tuning), KV pool
--num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token
harness worst case and one native 262,144 conversation), MoE LRU
--moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens
36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml
flips reasoningContentField on the harness side). Experts + PLE pin
~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity.
Container identity renamed to the unit name (Frank's call): frank-primary;
vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image
gate becomes podman image exists (a local build carries no RepoDigests;
the old digest grep refused exactly this shape). Rollback tag:
frank-primary-vllm-shapeG-20261003.
- FROM pinned to sha256:a223aee1... (the pulled rocm/pytorch rocm7.14
pytorch 2.11.0 manifest); a re-pushed tag cannot change the stack.
- The pip-ROCm torch ships libhiprtc.so.7 only; the PR's _cpu_moe links
-lhiprtc and ld needs the unversioned symlink in the already-linked
_rocm_sdk_core/lib (first build died: cannot find -lhiprtc).
The parallel session's stash pop left GGUF Compatibility, Multi-Device
Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving,
vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9
unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets
that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link,
and two multi-line-wrapped link tokens). All repointed to real pages;
six rows added to the vault index; seam green.
1862dc3 repriced the Ornith rung (nine full-attn layers: f16/f16 = dead
boot) and flipped CACHE_TYPE_V to q8_0, but its note that the seam was
'retargeted in 8f53df7' covered only the payload paths — the KV test
still asserted the 2026-10-03 unquantized-both-halves contract and held
the suite red. Test now pins the shipped contract: K=f16, V=q8_0,
flash-attn on.
The default-project-set rewrite dropped non---project args (e.g. --list),
so 'run.sh --list' silently RAN the whole suite instead of listing.
Args now ride alongside the default set; an explicit --project still
takes over. Verified: default --list = 33 tests listed; opt-in --list =
34 with the frankbook walk.
The rogue session's swap left this env reprice uncommitted in the tree
(2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn
layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16
lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because
the compaction invariant needs slot >= 191,148. The only lever that
keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K
stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch
script manifest already flipped in 3ee88d7. MEASURE after boot; under
1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
The 17:47 Ornith-1.5-9B swap (77decb5) updated frank-small.env and
serve-frank-small.sh but left two consumers on the MiMo-Q6K paths:
deploy-remote.sh's LLM_SMALL_MODEL_FILE/LLM_SMALL_MMPROJ defaults (the
payload-existence gate checked files the unit no longer loads) and
frank_small_unit.bats (stubs + env assertion), which held the whole
bats suite red. Both now point at the Ornith-Q8 payload; docs updated
with the conditional frankbook e2e project.
The rogue session's Ornith commits (3ee88d7/77decb5) swapped the env and
fetch script but left the deploy handler gates and the seam asserting
the MiMo-Q6K paths — the seam failures 195-203 and a deploy that gates
a file the unit no longer reads. Flip both to Ornith-Q8 (Q8_0 GGUF +
BF16 mmproj), keeping the anti-drift twin structure intact.
Frank