llama.cpp service replacing retired vllm-awq-qwen36 on port 7770 with alias
frank-default-small. Wrapper sources frank-default-small.env for tunables:
3 parallel slots at 262144 ctx, draft-dflash n-max 7, KV K=q8_0 V=f16 on GPU,
all MoE experts on CPU (--n-cpu-moe 40), non-expert tensors on GPU, mmproj
vision + jinja. Unit: frank-default-small.service (ExecStart in dankserver
repo clone /home/dank/vLLM/scripts).
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired
ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770
with MTP speculative decoding. Align Dankserver to the running
vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs
(gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4
experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
The launcher does not change directory, so process cwd becomes sys.path[0]
for the 'python -m ...registry' model-inspection subprocess. From the CIRU
package root, the top-level vllm/ repo dir shadows the editable install as a
namespace package, breaking lazy imports (SamplingParams) and failing model
architecture inspection. cd to /home/dank/ling3-ciru before exec.
- scripts/deploy.sh: target-aware deploy (dankclaw/dankserver). Pushes main,
pulls on the target, installs/enables systemd unit, restarts service, and
checks the health endpoint; --dry-run preview; pre-flight dirty-tree check
- docs/deployment.md: full deployment/update process with mermaid workflow,
target map, manual update commands, service units, and troubleshooting
- AGENTS.md: add Deployment & Update Process section (deploy.sh usage, target
map, mermaid workflow) and a Summary table row
- docs/index.md: crosslink deployment doc
- Podman container with CUDA GPU access
- DFlash speculative decoding via --spec-method dflash
- FP8 KV cache, 4 parallel streams, 262K context
- CPU expert offloading via --cpu-offload-gb
- systemd service for persistent runtime
- Replace --override-kv with --alias for model naming
- Validation script tests 1, 2, 3 parallel streams sequentially
- Model name confirmed: frank-default-large
- DFlash+ngram-mod: 17.11 tok/s vs DFlash alone: 18.34 tok/s
- ngram overhead not offset by acceptance rate at current settings
- Keeping in place for now, may tune ngram-mod params later
- Add second parallel prompt (Byzantine Empire history vs ML)
- Both hit 1024 tokens, combined 2048 in 111.6s = 18.34 tok/s
- Parallel throughput matches single-stream baseline
- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
Frank