Commit Graph
5 Commits
Author SHA1 Message Date
Frank 55f87fcdce llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
2026-10-10 17:03:14 -07:00
Frank 301914b1c0 feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
2026-09-11 14:48:28 -07:00
Frank 6efb0fb231 feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8
Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with
FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert
weights in host RAM with an LRU GPU cache and paged KV with radix prefix
sharing.

Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size).
Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used.

Production config:
- 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM)
- 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers)
- 4 max-running-requests (mamba slot pool: 24 slots)
- bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1
- Prefill: ~1,130 tok/s at 4k prompt

Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor
FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed).
Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart.

Old llama.cpp scripts archived to .AGENTTRASH/.
Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in
docs/LLM/freetoken-benchmark-sweep.md.
2026-08-27 02:01:16 -07:00
Frank 215d74f6ce feat: swap both LLM services + remove config-assertion tests
Dankserver (frank-default-small):
- Stock llama.cpp (latest master, CUDA build) replaces ik_llama.cpp
- Model: peculiar-ragdoll/Tiel-Coder-35B-A3B UD-Q6_K_XL (30 GB)
  Architecture: qwen35moe, 40 layers (10 attn + 30 SSM), 256 experts/8 used
- CTX_SIZE=200700 (66900/slot × 3 parallel), Q8_0 KV (~2 GB cache buffer)
- --cpu-moe for expert offload, -ngl 999 (non-expert on GPU)
- mmap enabled; removed ik_llama-specific flags

Dankclaw (frank-default-large):
- ROCmFPX fork with PR #27742 (qwen4exp arch) merged and built
  at /home/dank/GitPackages/ROCmFPX/build-qwen38
- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN (98.5 GiB)
  Architecture: qwen4exp, 48 layers (12 attn + 36 SSM), 512 experts/10 used
  QSA indexer, hyper-connections, PLE, Q4_0_ROCMFP4 quant
- CTX_SIZE=131072 (128k) × 1 parallel, f16 KV
- --fit off (auto-fit segfaults on qwen4exp hyper-connection config)
- Removed MTP draft, mmproj; kept Qwen-Sharp chat template + reasoning

Tests:
- Removed tests/frank_small_config.bats and tests/frank_large_config.bats
  (moved to .AGENTTRASH/). These were config-assertion tests that checked
  'is variable X set to Y' rather than testing the serve script's actual
  behavior. All 199 remaining tests pass.
- Updated test-changed.sh mappings and changed_tests.bats to reflect the
  removal.
2026-08-26 20:37:46 -07:00
Frank 1f565c08c3 Reorganize: tracked DFlash2 archive files -> .AGENTTRASH (repo root) 2026-08-22 15:07:31 -07:00