Frank
bd5c28c4e9
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201 built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated --no-deps install per docs/install_amd.md). Shape (first-boot conservative): ONE R9700 (second card idle; --tensor-parallel-size exists for future 2-card tuning), KV pool --num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token harness worst case and one native 262,144 conversation), MoE LRU --moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens 36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml flips reasoningContentField on the harness side). Experts + PLE pin ~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity. Container identity renamed to the unit name (Frank's call): frank-primary; vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image gate becomes podman image exists (a local build carries no RepoDigests; the old digest grep refused exactly this shape). Rollback tag: frank-primary-vllm-shapeG-20261003.
64 lines
3.4 KiB
Docker
64 lines
3.4 KiB
Docker
# frank-primary-freetoken.Dockerfile — FreeToken (PR #587 ROCm) serving image
|
|
# for the frank-primary endpoint (dankserver :7770, 1x R9700 gfx1201).
|
|
#
|
|
# Built on dankserver with rootless podman (dank user), same storage the
|
|
# service's `podman run` reads:
|
|
# podman build -t frank-primary-freetoken:pr587-77a7201 \
|
|
# -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
|
|
#
|
|
# Base: the PR's own docs/install_amd.md names this exact image as the
|
|
# known-good ROCm 7.14 stack for gfx1201 (torch 2.11.0+rocm7.14, triton-rocm
|
|
# 3.7 — the original pin; FreeToken main+PR587 accepts torch>=2.11,<2.16 and
|
|
# triton>=3.6,<3.9). There is no rocm7.14+torch2.14 image on Docker Hub (the
|
|
# 2.14 images are rocm10.1.0 — an untested pair for this PR), so the PR doc's
|
|
# verbatim recommendation is the deployed stack.
|
|
#
|
|
# FreeToken source: FlashML-org/FreeToken PR #587 at pinned head
|
|
# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214, base
|
|
# main 581fdca3). The PR is ROCm-gated end to end; the NVIDIA/CUDA path is
|
|
# byte-identical to main.
|
|
#
|
|
# Build law from docs/install_amd.md:
|
|
# - `pip install --no-build-isolation --no-deps -e .` — the image's ROCm
|
|
# torch is preserved and used to compile the C++/HIP extensions.
|
|
# - NEVER a plain `pip install -e .` on ROCm: the dependency set is
|
|
# authored for NVIDIA (flashlib pulls numba + nvidia-cutlass-dsl; torch
|
|
# metadata pulls cuda-bindings) and the resolve can REPLACE the ROCm
|
|
# torch with a CUDA build.
|
|
# - Runtime deps installed explicitly, curated by the same doc. flashinfer
|
|
# / sgl_kernel / nvidia-cutlass-dsl have no ROCm builds and are
|
|
# intentionally absent — FreeToken gates them off and falls back to its
|
|
# Triton kernels on ROCm.
|
|
#
|
|
# Base is digest-pinned (2026-10-05): sha256:a223aee1… is the manifest this
|
|
# build was qualified against; a re-pushed tag cannot change the stack. The
|
|
# readable tag rides along for humans (podman verifies the digest matches).
|
|
FROM rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
|
|
|
|
# Pin the PR head, not the branch: a future push to rocm714_torch214 would
|
|
# otherwise silently change the deployed engine.
|
|
# The pip-ROCm torch in this image ships libhiprtc as the VERSIONED
|
|
# libhiprtc.so.7 only; the PR's `_cpu_moe` extension links `-lhiprtc`, and
|
|
# ld looks for the unversioned dev symlink in the ALREADY-LINKED
|
|
# _rocm_sdk_core/lib dir. Create it there (first build failure 2026-10-05:
|
|
# `ld: cannot find -lhiprtc`).
|
|
RUN ln -s libhiprtc.so.7 /opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/libhiprtc.so
|
|
RUN git clone https://github.com/FlashML-org/FreeToken.git /opt/FreeToken \
|
|
&& cd /opt/FreeToken \
|
|
&& git fetch origin pull/587/head \
|
|
&& git checkout 77a7201f98f00d93f618c6cf8b81ffe8360e0899
|
|
|
|
WORKDIR /opt/FreeToken
|
|
RUN python -m pip install --no-build-isolation --no-deps -e .
|
|
|
|
# Curated runtime dependencies (docs/install_amd.md). transformers is pinned
|
|
# to the pyproject window (>=5.16,<5.17); the rest are the doc's portable set.
|
|
# torch / torchvision / triton-rocm come from the base image, 1:1 aligned.
|
|
RUN python -m pip install \
|
|
"transformers>=5.16,<5.17" tokenizers safetensors huggingface_hub \
|
|
fastapi uvicorn pydantic openai httpx anyio \
|
|
einops gguf msgpack pyzmq typer rich apache-tvm-ffi \
|
|
partial-json-parser \
|
|
&& python -m pip install --no-deps flashlib==0.3.0 \
|
|
&& python -m pip install ninja
|