serverAppConfigs/serverAppConfigs-docs/LLM/FreeToken Builds.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	FreeToken Builds	Building the FreeToken runtime here: the CUDA and ROCm builds and the dependency trap each one hit.	freetoken build dankclaw vllm mtp	2026-10-05T00:00:00-07:00

FreeToken Builds

FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime. Two build paths exist, both on Dankclaw, with different dependency traps. Both were verified end-to-end 2026-09-11/12.

Common verification (do not skip)

Verify the build by real completion, not by version string. A version bump alone does not move data through the model — a build can print a new version and still not have completed a real token. Before declaring a FreeToken build deployable, run an actual completion (any tiny prompt) and read the returned content:

import json, urllib.request
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
    json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
    "content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req, timeout=600)))

A 200 with a real choices[0].message.content is the only build green.

CUDA build (the torch/vLLM dependency trap)

Both paths build on Dankclaw with the same preinstall state: Python 3.12.10 in the system venv and Torch 2.10.0.dev (nightly, repo download.pytorch.org/whl/nightly/cu130, installed with uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130" --index-url https://download.pytorch.org/whl/nightly/cu130 — use --index-strategy unsafe-best-match, uv otherwise refuses the dev version). Verify the exact pair first; mismatched preinstalled deps are the #1 build-failure cause.

Then install the matching vllm dev wheel:

uv pip install --index-strategy unsafe-best-match \
  --index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130

The wheel URL pattern is fixed: https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl — read it back from vLLM's CI-run page (run details → build_and_install_package job, "install the vllm wheel" step). The trap: vLLM refuses versions < its install_requires floor at runtime, so the wheel version must be ≥ the runtime's floor. A wheel built at a lower base (0.5.x) downloads but import vllm fails with a version constraint error — that failure looks like a build failure and is not one; upgrade the wheel, not the code.

If uv pip install -e . --no-deps refuses (some repos pin the --no-deps branch off), fall back to python3 setup.py inside the venv, after python -m venv --copies .venv && source .venv/bin/activate and uv pip install --no-build-isolation "torch==…" — the venv must NOT re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).

ROCm build (gfx1151 via TheRock nightly)

Same venv discipline, ROCm wheels from TheRock's nightly index:

uv pip install --index-strategy unsafe-best-match \
  --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
  "torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"

Then vllm itself (uv pip install -e . --no-build-isolation --no-deps, repo branch specs-mtp-dev, commit 2ee440d), after copying the two prebuilt extension wheels into the tree (the repo's setup.py skips compiling them when found). The ROCm build also needs install-fastrak-gfx1151.sh from the repo's scripts and FASTRACK_HOME=/home/dank/FastTrack-runtime in the unit env. FreeToken's x86_64-gfx1151 prebuilt extension (freetoken-ops) ships on HF hub franksdank/freetoken-ops-gfx1151 — fetch it once and point FREETOKEN_OPS_HOME at it; never rebuild the extension from source unless the extension API changed.

--index-strategy unsafe-best-match is load-bearing on both paths: the nightly indexes carry version pairs that uv's strict resolver otherwise rejects, and the error (No candidate found for pytorch-triton) reads like a missing package.

Related

• Model Swap and Rollback — the swap/deploy story these builds serve
• Llama Cpp Serving Sizing — the harness measurement discipline