Ling-3.0-flash Q6_K Trial on Dankclaw

This page records the Ling-3.0-flash trial that temporarily replaced frank-default-large on Dankclaw. It replaces the Qwen3.6-27B service on port 7770 under the same API alias.

Status

Service: frank-large-ling (systemd).
Model: inclusionAI/Ling-3.0-flash Q6_K (MoE, ~118B total params).
Runtime: llama.cpp fork aetherbird/llama.cpp branch bailingmoe3-support (commit db4480bc8, build b10411) at /home/dank/llama.cpp-ling. Stock llama.cpp cannot load the bailingmoe3 architecture.
Backend: Vulkan0 (RADV) with AMD GTT.
Spec: draft-mtp n_max 3. The MTP block is bundled in the Ling GGUF itself (block 42) - no separate drafter file.
Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s (tiny cached prompt). Concurrent streams both complete.

Why Vulkan + GTT

The Strix Halo APU has 8 x 16 GB LPDDR5 modules = 128 GB physical shared memory. The firmware carves ~96 GiB as dedicated iGPU VRAM; the kernel sees the remainder (~30 GiB, MemTotal). GRUB adds no memory options (iommu=pt only).

The ROCm runtime reports 96 GiB VRAM. Vulkan (RADV) reports a 111.2 GiB heap (113910 MiB) because it can also reach system memory through the GTT. For a model near 100 GiB, Vulkan is the only backend with room. --list-devices shows the heap:

ROCm0:   AMD Radeon Graphics (98304 MiB)
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)

Memory Fit (Q6_K = 98.26 GiB)

Item	Size
Vulkan heap (llama.cpp view)	~111.2 GiB
Model Q6_K file	98.26 GiB
Model + MTP + graphs in heap	~98.7 GiB
Free heap after load	~10 GiB

Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total context costs only ~2.5 GiB. This is why 256k context fits.

Files

scripts/frank-large-ling.service - systemd unit.
scripts/serve-ling-flash.sh - serve script (fork binary, Vulkan, draft-mtp).
scripts/frank-default-large.ling.env - tunables.

Passwordless management is installed: /usr/local/sbin/frank-restart-ling.sh and /etc/sudoers.d/frank-ling.

Switch Back to Qwen3.6-27B (MTP)

The Qwen config was not changed. It remains committed as frank-default-large.service / serve-llama-frank-large.sh / frank-default-large.env (draft-mtp, n_max 3, ubatch 8192).

Run these commands:

sudo systemctl stop frank-large-ling
sudo systemctl start frank-default-large

Caveats

The bailingmoe3 fork validates CPU and CUDA upstream. Vulkan was validated here (load + generation + concurrent streams) on 2026-08-14.
The tokenizer logs special_eos_id is not in special_eog_ids at load. It is a warning; generation stops correctly at EOS.
Physical RAM pressure stays low because the model fits in the UMA carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP config (~7.7 vs ~15-19 t/s at 4k).
The heap is effectively full (98.7 of 111.2 GiB). Do not raise the context much without checking --list-devices free MiB.

3.2 KiB Raw Blame History