The launcher does not change directory, so process cwd becomes sys.path[0]
for the 'python -m ...registry' model-inspection subprocess. From the CIRU
package root, the top-level vllm/ repo dir shadows the editable install as a
namespace package, breaking lazy imports (SamplingParams) and failing model
architecture inspection. cd to /home/dank/ling3-ciru before exec.
- scripts/deploy.sh: target-aware deploy (dankclaw/dankserver). Pushes main,
pulls on the target, installs/enables systemd unit, restarts service, and
checks the health endpoint; --dry-run preview; pre-flight dirty-tree check
- docs/deployment.md: full deployment/update process with mermaid workflow,
target map, manual update commands, service units, and troubleshooting
- AGENTS.md: add Deployment & Update Process section (deploy.sh usage, target
map, mermaid workflow) and a Summary table row
- docs/index.md: crosslink deployment doc
- Podman container with CUDA GPU access
- DFlash speculative decoding via --spec-method dflash
- FP8 KV cache, 4 parallel streams, 262K context
- CPU expert offloading via --cpu-offload-gb
- systemd service for persistent runtime
- Replace --override-kv with --alias for model naming
- Validation script tests 1, 2, 3 parallel streams sequentially
- Model name confirmed: frank-default-large
- DFlash+ngram-mod: 17.11 tok/s vs DFlash alone: 18.34 tok/s
- ngram overhead not offset by acceptance rate at current settings
- Keeping in place for now, may tune ngram-mod params later
- Add second parallel prompt (Byzantine Empire history vs ML)
- Both hit 1024 tokens, combined 2048 in 111.6s = 18.34 tok/s
- Parallel throughput matches single-stream baseline
- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
Frank