Frank
87721ab0fe
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
98 lines
4.4 KiB
Markdown
98 lines
4.4 KiB
Markdown
---
|
|
type: reference
|
|
title: FreeToken Builds
|
|
description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
|
|
tags: [freetoken, build, dankclaw, vllm, mtp]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# FreeToken Builds
|
|
|
|
FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
|
|
vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
|
|
different dependency traps. Both were verified end-to-end 2026-09-11/12.
|
|
|
|
## Common verification (do not skip)
|
|
|
|
**Verify the build by real completion, not by version string.** A version
|
|
bump alone does not move data through the model — a build can print a new
|
|
version and still not have completed a real token. Before declaring a
|
|
FreeToken build deployable, run an actual completion (any tiny prompt) and
|
|
read the returned content:
|
|
|
|
```python
|
|
import json, urllib.request
|
|
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
|
json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
|
|
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
|
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
|
```
|
|
|
|
A 200 with a real `choices[0].message.content` is the only build green.
|
|
|
|
## CUDA build (the torch/vLLM dependency trap)
|
|
|
|
Both paths build on **Dankclaw** with the same preinstall state: Python
|
|
3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
|
|
download.pytorch.org/whl/nightly/cu130, installed with
|
|
`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
|
|
--index-url https://download.pytorch.org/whl/nightly/cu130` — use
|
|
`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
|
|
version). Verify the exact pair first; mismatched preinstalled deps are the
|
|
#1 build-failure cause.
|
|
|
|
Then install the **matching vllm dev wheel**:
|
|
|
|
```bash
|
|
uv pip install --index-strategy unsafe-best-match \
|
|
--index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
|
|
--extra-index-url https://wheels.vllm.ai/nightly/cu130
|
|
```
|
|
|
|
The wheel URL pattern is fixed:
|
|
`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
|
|
— read it back from vLLM's CI-run page (`run details →
|
|
build_and_install_package` job, "install the vllm wheel" step). **The trap:
|
|
vLLM refuses versions < its install_requires floor at runtime, so the wheel
|
|
version must be ≥ the runtime's floor.** A wheel built at a lower base
|
|
(0.5.x) downloads but `import vllm` fails with a version constraint error —
|
|
that failure looks like a build failure and is not one; upgrade the wheel,
|
|
not the code.
|
|
|
|
If `uv pip install -e . --no-deps` refuses (some repos pin the
|
|
`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
|
|
after `python -m venv --copies .venv && source .venv/bin/activate` and
|
|
`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
|
|
re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
|
|
|
|
## ROCm build (gfx1151 via TheRock nightly)
|
|
|
|
Same venv discipline, ROCm wheels from TheRock's nightly index:
|
|
|
|
```bash
|
|
uv pip install --index-strategy unsafe-best-match \
|
|
--index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
|
|
"torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
|
|
```
|
|
|
|
Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
|
|
repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
|
|
prebuilt extension wheels into the tree (the repo's `setup.py` skips
|
|
compiling them when found). The ROCm build also needs
|
|
`install-fastrak-gfx1151.sh` from the repo's scripts and
|
|
`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
|
|
x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
|
|
`franksdank/freetoken-ops-gfx1151` — fetch it once and point
|
|
`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
|
|
the extension API changed.
|
|
|
|
`--index-strategy unsafe-best-match` is load-bearing on both paths: the
|
|
nightly indexes carry version pairs that `uv`'s strict resolver otherwise
|
|
rejects, and the error (`No candidate found for pytorch-triton`) reads like
|
|
a missing package.
|
|
|
|
## Related
|
|
|
|
- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
|
|
- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
|