91 lines
3.2 KiB
Markdown
91 lines
3.2 KiB
Markdown
# Ling-3.0-flash Q6_K Trial on Dankclaw
|
|
|
|
This page records the Ling-3.0-flash trial that temporarily replaced
|
|
`frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on
|
|
port 7770 under the same API alias.
|
|
|
|
## Status
|
|
|
|
- Service: `frank-large-ling` (systemd).
|
|
- Model: `inclusionAI/Ling-3.0-flash` Q6_K (MoE, ~118B total params).
|
|
- Runtime: llama.cpp **fork** `aetherbird/llama.cpp` branch
|
|
`bailingmoe3-support` (commit `db4480bc8`, build b10411) at
|
|
`/home/dank/llama.cpp-ling`. Stock llama.cpp cannot load the
|
|
`bailingmoe3` architecture.
|
|
- Backend: `Vulkan0` (RADV) with AMD GTT.
|
|
- Spec: `draft-mtp` `n_max 3`. The MTP block is bundled in the Ling GGUF
|
|
itself (block 42) - no separate drafter file.
|
|
- Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
|
|
- Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s
|
|
(tiny cached prompt). Concurrent streams both complete.
|
|
|
|
## Why Vulkan + GTT
|
|
|
|
The Strix Halo APU has 8 x 16 GB LPDDR5 modules = **128 GB physical
|
|
shared memory**. The firmware carves ~96 GiB as dedicated iGPU VRAM; the
|
|
kernel sees the remainder (~30 GiB, `MemTotal`). GRUB adds no memory
|
|
options (`iommu=pt` only).
|
|
|
|
The ROCm runtime reports 96 GiB VRAM. **Vulkan (RADV) reports a
|
|
111.2 GiB heap** (113910 MiB) because it can also reach system memory
|
|
through the GTT. For a model near 100 GiB, Vulkan is the only backend
|
|
with room. `--list-devices` shows the heap:
|
|
|
|
```
|
|
ROCm0: AMD Radeon Graphics (98304 MiB)
|
|
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)
|
|
```
|
|
|
|
## Memory Fit (Q6_K = 98.26 GiB)
|
|
|
|
| Item | Size |
|
|
| ---- | ---- |
|
|
| Vulkan heap (llama.cpp view) | ~111.2 GiB |
|
|
| Model Q6_K file | 98.26 GiB |
|
|
| Model + MTP + graphs in heap | ~98.7 GiB |
|
|
| Free heap after load | ~10 GiB |
|
|
|
|
Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total
|
|
context costs only ~2.5 GiB. This is why 256k context fits.
|
|
|
|
## Files
|
|
|
|
- `scripts/frank-large-ling.service` - systemd unit.
|
|
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
|
|
draft-mtp).
|
|
- `scripts/frank-default-large.ling.env` - tunables.
|
|
|
|
Passwordless management is installed:
|
|
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
|
|
|
|
## Switch Back to Qwen3.6-27B (MTP)
|
|
|
|
The Qwen config was not changed. It remains committed as
|
|
`frank-default-large.service` / `serve-llama-frank-large.sh` /
|
|
`frank-default-large.env` (draft-mtp, n_max 3, ubatch 8192).
|
|
|
|
Run these commands:
|
|
|
|
```
|
|
sudo systemctl stop frank-large-ling
|
|
sudo systemctl start frank-default-large
|
|
```
|
|
|
|
## Caveats
|
|
|
|
- The `bailingmoe3` fork validates CPU and CUDA upstream. Vulkan was
|
|
validated here (load + generation + concurrent streams) on 2026-08-14.
|
|
- The tokenizer logs `special_eos_id is not in special_eog_ids` at load.
|
|
It is a warning; generation stops correctly at EOS.
|
|
- Physical RAM pressure stays low because the model fits in the UMA
|
|
carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP
|
|
config (~7.7 vs ~15-19 t/s at 4k).
|
|
- The heap is effectively full (98.7 of 111.2 GiB). Do not raise the
|
|
context much without checking `--list-devices` free MiB.
|
|
|
|
## See Also
|
|
|
|
- [[frank-default-large-benchmark]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[deployment]]
|