73 lines
4.3 KiB
Markdown
73 lines
4.3 KiB
Markdown
# Raw Memories
|
|
|
|
## 01a042cc-e1c5-7482-87a2-14deadb82202
|
|
updated_at: 1787833610
|
|
## frank-default-large: Qwen3.8-Flash-Next tuning on Dankclaw (Strix Halo gfx1151)
|
|
|
|
### Model & Architecture
|
|
- kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (51.2B PLE n-gram + 125.8B MoE/attention)
|
|
- 56B MoE: 512 experts (10 used), 48 layers (12 attn + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim
|
|
- QSA indexer: 4 heads, top_k=2048; hyper-connections: count=4, low_rank=320
|
|
- PLE layer 1: ngram_size=3, heads_per_ngram=8
|
|
- Quant: Q4_0_ROCMFP4_STRIX (file_type=105), 113.47 GiB single GGUF
|
|
- PLE table: 50.66 GiB Q8_0, streamed from page cache (NEVER in GPU VRAM)
|
|
- GPU resident: ~63-69 GiB on Strix Halo unified memory pool (116 GiB SVM-backed)
|
|
|
|
### Runtime
|
|
- ROCmFPX llama.cpp fork at /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
|
|
- PR #27742 (qwen4exp architecture) merged
|
|
- Build: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie
|
|
|
|
### Critical Constraints (DO NOT CHANGE)
|
|
- FIT=off: auto-fit segfaults on this model's hyper-connection/indexer config
|
|
- -ngl 999 places all layers on SVM-backed GPU pool
|
|
- NEVER name env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE — ROCm HIP runtime reads bare env vars (2026-08-13 TEMP incident)
|
|
- llama.cpp reads --host/--port/--temp as CLI flags only, never env vars
|
|
- Text-only: no mmproj, no MTP draft head
|
|
|
|
### Current Tunings (env file: frank-default-large.env)
|
|
- BATCH_SIZE=4096, UBATCH_SIZE=4096
|
|
- NUM_THREADS=16, THREAD_BATCH=32
|
|
- CTX_SIZE=524288, PARALLEL=2
|
|
- CACHE_TYPE_K=f16, CACHE_TYPE_V=f16
|
|
- CTX_CHECKPOINTS=16, CACHE_RAM=8192
|
|
- HSA_OVERRIDE_GFX_VERSION=11.5.1
|
|
- Port 7770 (LISTEN_PORT)
|
|
|
|
### Benchmark Status (interrupted)
|
|
- Goal: tune batch/ubatch for optimal token prefill
|
|
- Next planned step: threads-batch sweep
|
|
- Benchmark servers targeted ports 7772 and 7773
|
|
- Session interrupted during health-check polling loop; servers returned no output via curl on those ports (likely still loading 113 GiB model or not yet started)
|
|
- The repeated poll pattern (5+ identical checks, ~5s apart) suggests the assistant was waiting for model load completion before starting benchmarks
|
|
|
|
### Workflow Pattern
|
|
- SSH into dankclaw from controller; check health via `curl -s http://127.0.0.1:PORT/health`
|
|
- Model load on unified memory is extremely slow (113 GiB mmap); benchmark scripts must account for long startup
|
|
- Use frank-default-small (the serving model for this agent) to orchestrate benchmarks against frank-default-large
|
|
|
|
## 01a042a4-7e85-7728-802d-c8442896f6e6
|
|
updated_at: 1787827055
|
|
## frank-default-large model swap (2026-09-02)
|
|
|
|
**Host:** Dankclaw — AMD Radeon 8060S (Strix Halo), gfx1151 iGPU, 64 GiB unified memory. Runtime: charlie12345/ROCmFPX llama.cpp build-kairic v81. Port 7770.
|
|
|
|
**Swap:** STRIX_LEAN-GGUF (56B MoE / 35.76B PLE, 3 shards, 98.5 GiB) → Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B total / 51.2B PLE, single file, 113.47 GiB on disk).
|
|
|
|
**File path:** `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf`
|
|
|
|
**Key architectural fact:** PLE (Projection Layer Embedding) table streams from page cache — never enters GPU VRAM regardless of tier. This is why a 114 GiB file fits in 64 GiB unified memory: only ~63-69 GiB is GPU-resident (same as LEAN variant). The PLE table occupies host RAM/page-cache.
|
|
|
|
**Performance:** Load time ~55 seconds. Generation speed and prompt throughput within noise of STRIX_LEAN at every context depth. 200/200 tests pass.
|
|
|
|
**Config files modified:**
|
|
- `scripts/frank-default-large.env` — MODEL_DIR, MODEL_FILE, header comments
|
|
- `scripts/serve-frank-large.sh` — header comments, default values
|
|
- `docs/LLM/kingjones-strix-rocmfp4.md` — new 195-line runbook
|
|
- `docs/LLM/index.md` — active model reference updated
|
|
- `docs/index.md` — top-level index links new doc
|
|
|
|
**Git:** Branch `frank-large-strix-model` pushed to origin (Gitea on Dankclaw).
|
|
|
|
**Prior history on this slot (for context):** CIRU PromptForge v3 → Kairic Edge IU4 (reverted) → kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (Q6 AGENT, standalone MTP draft, Turbo4 V cache, 262K ctx/1 slot) → Q8_0 AGENT experiment (failed, set_rows bug) → STRIX MTP restored → now Flash-Next full 177B.
|