50 lines
2.8 KiB
Markdown
50 lines
2.8 KiB
Markdown
# frank-default-large Model & Runtime Reference
|
|
|
|
## Host & Environment
|
|
- **Host:** Dankclaw (AMD Radeon 8060S Strix Halo, gfx1151 iGPU)
|
|
- **Memory:** 64 GiB unified memory + page cache/SVM (~116 GiB total pool)
|
|
- **Runtime:** `charlie12345/ROCmFPX` llama.cpp fork (`llama-server`)
|
|
- **Build Flags:** `-DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie`
|
|
|
|
## Model Architecture
|
|
- **Model:** `Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (177B total params)
|
|
- **Composition:** 51.2B PLE (n-gram) + 125.8B MoE/Attention (56B active MoE: 512 experts, 10 used; 48 layers: 12 attn + 36 SSM)
|
|
- **Quant:** `Q4_0_ROCMFP4_STRIX` (file_type=105), ~113.47 GiB single GGUF
|
|
- **VRAM Behavior:** PLE table streams exclusively from page cache/host RAM. GPU resident memory remains ~63-69 GiB regardless of total file size.
|
|
- **Indexing/Connections:** QSA indexer (4 heads, top_k=2048); Hyper-connections (count=4, low_rank=320); PLE layer 1 (ngram_size=3, heads_per_ngram=8)
|
|
|
|
## Critical Constraints (DO NOT OVERRIDE)
|
|
- **FIT=off**: Auto-fit crashes on this architecture's hyper-connection/indexer config.
|
|
- **Context Limit**: `-ngl 999` required to force SVM backing for all layers.
|
|
- **Env Vars**: NEVER set `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. ROCm HIP runtime consumes bare env vars. llama.cpp CLI flags (`--host`, `--port`, `--temp`) are not read from env.
|
|
- **Capabilities**: Text-only. No `mmproj`, no MTP draft head.
|
|
|
|
## Configuration & Tunings (`scripts/frank-default-large.env`)
|
|
```
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=4096
|
|
NUM_THREADS=16
|
|
THREAD_BATCH=32
|
|
CTX_SIZE=524288
|
|
PARALLEL=2
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
CTX_CHECKPOINTS=16
|
|
CACHE_RAM=8192
|
|
HSA_OVERRIDE_GFX_VERSION=11.5.1
|
|
LISTEN_PORT=7770
|
|
```
|
|
|
|
## Operational Workflow & Benchmarking
|
|
- **Startup:** Model load takes ~55s+. GPU pool handles heavy I/O via mmap.
|
|
- **Health Checks:** Use `curl -s http://127.0.0.1:7770/health` (or target benchmark ports). Always poll in loops (5+ checks, ~5s apart) before proceeding.
|
|
- **Benchmark Orchestration:** SSH from controller. Use `frank-default-small` to drive benchmark scripts against `frank-default-large`.
|
|
- **Current Status:** Batch/ubatch tuning interrupted during health-polling for benchmark servers (ports 7772/7773). Next phase: threads-batch sweep.
|
|
- **Prior History:** CIRU PromptForge v3 → Kairic Edge IU4 → Qwen3.8-27B ROCmFP4 STRIX MTP → Q8_0 AGENT (failed) → STRIX MTP → Qwen3.8-Flash-Next 177B.
|
|
|
|
## File & Git References
|
|
- **Model Path:** `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf`
|
|
- **Configs:** `scripts/frank-default-large.env`, `scripts/serve-frank-large.sh`
|
|
- **Docs:** `docs/LLM/kingjones-strix-rocmfp4.md`, `docs/LLM/index.md`, `docs/index.md`
|
|
- **Git Branch:** `frank-large-strix-model` (origin: Dankclaw Gitea)
|