Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9cf4b1de82 | ||
|
|
d28ab8ef8c |
No files matched your search
@@ -44,13 +44,23 @@ off):
|
|||||||
|
|
||||||
```
|
```
|
||||||
MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards)
|
MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards)
|
||||||
IMAGE=docker.io/tcclaviger/vllm:latest
|
IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
||||||
|
# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
|
||||||
|
# the floating :latest tag moved once already — always launch by digest.
|
||||||
|
# repo tag of this state: qwen-fp8-tuned-20260929
|
||||||
--tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.98
|
--tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.98
|
||||||
--kv-cache-memory-bytes 2147483648 --max-num-seqs 4 --enable-prefix-caching
|
--kv-cache-memory-bytes 2147483648 --max-num-seqs 4 --enable-prefix-caching
|
||||||
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
|
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
|
||||||
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
|
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Published to the host LAN via the serve script's `PORTMAP` hook
|
||||||
|
(`PORTMAP=7773:7770`): health and completions answer on
|
||||||
|
`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
|
||||||
|
decode 36.49/34.55/35.98 through the published path). Host port 7773 was
|
||||||
|
chosen over 7770 because `frank-primary` (7770) is stopped but ENABLED —
|
||||||
|
any future start would collide with a published 7770.
|
||||||
|
|
||||||
Device notes carried from the bring-up: `--device /dev/dri` must be the
|
Device notes carried from the bring-up: `--device /dev/dri` must be the
|
||||||
WHOLE directory (individual renderD nodes enumerate but refuse contexts —
|
WHOLE directory (individual renderD nodes enumerate but refuse contexts —
|
||||||
"No CUDA GPUs are available" while device_count() reports 2); the image
|
"No CUDA GPUs are available" while device_count() reports 2); the image
|
||||||
|
|||||||
@@ -122,6 +122,21 @@
|
|||||||
(~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
|
(~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
|
||||||
GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
|
GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
|
||||||
(> 246 GiB RAM). Feasibility doc rows updated.
|
(> 246 GiB RAM). Feasibility doc rows updated.
|
||||||
|
- **Revert pins installed (Frank's request):** (1) repo tag
|
||||||
|
`qwen-fp8-tuned-20260929` at f532e03 (the campaign's docs + measured
|
||||||
|
config; `git checkout qwen-fp8-tuned-20260929` restores the state);
|
||||||
|
(2) image pinned by digest — serve script now launches
|
||||||
|
`docker.io/tcclaviger/vllm@sha256:da402966...` (ID 3322dbbbc96d) with a
|
||||||
|
local alias `qwen-fp8-runtime:tuned-20260929`; the floating `:latest`
|
||||||
|
is no longer referenced anywhere (the other image on the host,
|
||||||
|
`28.06.1-Code000` = fffca39f, is a DIFFERENT build — do not confuse).
|
||||||
|
Runbook deployment-shape section updated.
|
||||||
|
- **Endpoint published:** `PORTMAP=7773:7770` hook added to the serve
|
||||||
|
script; the 256k config relaunched with it. Host-side checks: health
|
||||||
|
200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists
|
||||||
|
qwen3.8-flash-next, decode through the published path
|
||||||
|
36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-primary
|
||||||
|
owns 7770 when it ever returns.
|
||||||
|
|
||||||
## 2026-09-28
|
## 2026-09-28
|
||||||
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
|
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
|
||||||
|
|||||||
Reference in new issue
Block a user