docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
This commit is contained in:
Frank committed
1 parent
5a083eee6c
commit
8212d4451a
68 files changed
+1053
-1207
No files matched your search
@@ -8,15 +8,15 @@ The old scripts moved to `old/scripts/` in this repository.
|
||||
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-default-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
| frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
|
||||
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
|
||||
tensor split, no NPU.
|
||||
|
||||
## Models
|
||||
|
||||
### frank-default-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||||
### frank-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||||
|
||||
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
|
||||
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
|
||||
@@ -31,7 +31,7 @@ tensor split, no NPU.
|
||||
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
|
||||
(48/64/24).
|
||||
|
||||
### frank-default-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
### frank-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
|
||||
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
|
||||
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
|
||||
@@ -46,7 +46,7 @@ tensor split, no NPU.
|
||||
instead of the Qwen3 XML structure). The Quark quant follows the format
|
||||
cleanly; swapped in the evening of 2026-08-17.
|
||||
|
||||
## Context Math (frank-default-large, RTX 3090)
|
||||
## Context Math (frank-large, RTX 3090)
|
||||
|
||||
KV per token: 16 attention layers x 4 KV heads x 256 dim.
|
||||
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
|
||||
@@ -63,11 +63,11 @@ Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
|
||||
|
||||
Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
|
||||
`scripts/frank-default-large.env` to 122880 for ~1.8 GiB headroom.
|
||||
`scripts/frank-large.env` to 122880 for ~1.8 GiB headroom.
|
||||
|
||||
## Install Runbook
|
||||
|
||||
### Dankserver (frank-default-large)
|
||||
### Dankserver (frank-large)
|
||||
|
||||
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
|
||||
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
|
||||
@@ -87,17 +87,17 @@ Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
|
||||
4. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-default-large
|
||||
sudo systemctl restart frank-default-large
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-large.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-large
|
||||
sudo systemctl restart frank-large
|
||||
```
|
||||
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
|
||||
6. Confirm VRAM headroom with `nvidia-smi`.
|
||||
|
||||
### Dankclaw (frank-default-small)
|
||||
### Dankclaw (frank-small)
|
||||
|
||||
1. Stop the old services: `sudo systemctl stop frank-default-large
|
||||
frank-default-small` (retired PP head and NPU endpoint).
|
||||
1. Stop the old services: `sudo systemctl stop frank-large
|
||||
frank-small` (retired PP head and NPU endpoint).
|
||||
2. Install uv and create the venv:
|
||||
```
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
@@ -122,9 +122,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
vision encoder falls back to torch SDPA and engine init OOMs.
|
||||
5. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
|
||||
sudo systemctl restart frank-default-small
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-small
|
||||
sudo systemctl restart frank-small
|
||||
```
|
||||
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
|
||||
|
||||
@@ -150,7 +150,7 @@ on gfx1151; newer vLLM releases may already include it.
|
||||
|
||||
Three changes landed after the initial deployment:
|
||||
|
||||
### 1. Tool calling on frank-default-small (HTTP 400 fix)
|
||||
### 1. Tool calling on frank-small (HTTP 400 fix)
|
||||
|
||||
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
|
||||
unless the server enables tool calling. Fixes applied:
|
||||
@@ -175,7 +175,7 @@ model behavior, which motivated the swap.
|
||||
|
||||
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
|
||||
|
||||
`frank-default-small` now serves
|
||||
`frank-small` now serves
|
||||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
|
||||
vLLM-native expert layout). The Avesed compressed-tensors quant's
|
||||
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
|
||||
|
||||
Reference in new issue
Block a user