Raw Memories

01a042cc-e1c5-7482-87a2-14deadb82202

updated_at: 1787833610

frank-default-large: Qwen3.8-Flash-Next tuning on Dankclaw (Strix Halo gfx1151)

Model & Architecture

kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (51.2B PLE n-gram + 125.8B MoE/attention)
56B MoE: 512 experts (10 used), 48 layers (12 attn + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim
QSA indexer: 4 heads, top_k=2048; hyper-connections: count=4, low_rank=320
PLE layer 1: ngram_size=3, heads_per_ngram=8
Quant: Q4_0_ROCMFP4_STRIX (file_type=105), 113.47 GiB single GGUF
PLE table: 50.66 GiB Q8_0, streamed from page cache (NEVER in GPU VRAM)
GPU resident: ~63-69 GiB on Strix Halo unified memory pool (116 GiB SVM-backed)

Runtime

ROCmFPX llama.cpp fork at /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
PR #27742 (qwen4exp architecture) merged
Build: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie

Critical Constraints (DO NOT CHANGE)

FIT=off: auto-fit segfaults on this model's hyper-connection/indexer config
-ngl 999 places all layers on SVM-backed GPU pool
NEVER name env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE — ROCm HIP runtime reads bare env vars (2026-08-13 TEMP incident)
llama.cpp reads --host/--port/--temp as CLI flags only, never env vars
Text-only: no mmproj, no MTP draft head

Current Tunings (env file: frank-default-large.env)

BATCH_SIZE=4096, UBATCH_SIZE=4096
NUM_THREADS=16, THREAD_BATCH=32
CTX_SIZE=524288, PARALLEL=2
CACHE_TYPE_K=f16, CACHE_TYPE_V=f16
CTX_CHECKPOINTS=16, CACHE_RAM=8192
HSA_OVERRIDE_GFX_VERSION=11.5.1
Port 7770 (LISTEN_PORT)

Benchmark Status (interrupted)

Goal: tune batch/ubatch for optimal token prefill
Next planned step: threads-batch sweep
Benchmark servers targeted ports 7772 and 7773
Session interrupted during health-check polling loop; servers returned no output via curl on those ports (likely still loading 113 GiB model or not yet started)
The repeated poll pattern (5+ identical checks, ~5s apart) suggests the assistant was waiting for model load completion before starting benchmarks

Workflow Pattern

SSH into dankclaw from controller; check health via curl -s http://127.0.0.1:PORT/health
Model load on unified memory is extremely slow (113 GiB mmap); benchmark scripts must account for long startup
Use frank-default-small (the serving model for this agent) to orchestrate benchmarks against frank-default-large

01a042a4-7e85-7728-802d-c8442896f6e6

updated_at: 1787827055

frank-default-large model swap (2026-09-02)

Host: Dankclaw — AMD Radeon 8060S (Strix Halo), gfx1151 iGPU, 64 GiB unified memory. Runtime: charlie12345/ROCmFPX llama.cpp build-kairic v81. Port 7770.

Swap: STRIX_LEAN-GGUF (56B MoE / 35.76B PLE, 3 shards, 98.5 GiB) → Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B total / 51.2B PLE, single file, 113.47 GiB on disk).

File path: /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf

Key architectural fact: PLE (Projection Layer Embedding) table streams from page cache — never enters GPU VRAM regardless of tier. This is why a 114 GiB file fits in 64 GiB unified memory: only ~63-69 GiB is GPU-resident (same as LEAN variant). The PLE table occupies host RAM/page-cache.

Performance: Load time ~55 seconds. Generation speed and prompt throughput within noise of STRIX_LEAN at every context depth. 200/200 tests pass.

Config files modified:

scripts/frank-default-large.env — MODEL_DIR, MODEL_FILE, header comments
scripts/serve-frank-large.sh — header comments, default values
docs/LLM/kingjones-strix-rocmfp4.md — new 195-line runbook
docs/LLM/index.md — active model reference updated
docs/index.md — top-level index links new doc

Git: Branch frank-large-strix-model pushed to origin (Gitea on Dankclaw).

Prior history on this slot (for context): CIRU PromptForge v3 → Kairic Edge IU4 (reverted) → kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (Q6 AGENT, standalone MTP draft, Turbo4 V cache, 262K ctx/1 slot) → Q8_0 AGENT experiment (failed, set_rows bug) → STRIX MTP restored → now Flash-Next full 177B.

4.3 KiB Raw Blame History