diff --git a/serverAppConfigs-docs/LLM/freetoken-frank-small.md b/serverAppConfigs-docs/LLM/freetoken-frank-small.md index 3473796..b326e41 100644 --- a/serverAppConfigs-docs/LLM/freetoken-frank-small.md +++ b/serverAppConfigs-docs/LLM/freetoken-frank-small.md @@ -125,6 +125,58 @@ tensor b (...)` in `_dequant_fp8_weight`, which multiplies `weight × weight_scale` directly instead of expanding the block scales. modelopt block-fp8 remains the only FP8 path that loads. +## Sampling + +**frank-small samples by default, and our config never said so.** +`scripts/serve-frank-small.sh` passes no sampling arguments at all — only +`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`, +`--num-tokens` and `--max-running-requests`. The behaviour comes from the +runtime's own default: + +```python +# freetoken/server/args.py:31 +sampling_defaults: str = "model" +``` + +`--sampling-defaults model` fills `temperature` / `top_k` / `top_p` from the +**checkpoint's** `generation_config.json` for any request that omits them. +This checkpoint ships: + +```json +{ "do_sample": true, "temperature": 1.0, "top_k": 20, "top_p": 0.95 } +``` + +Verified behaviourally 2026-09-12 — three identical requests with **no +sampling parameters sent at all** produced three different reasoning traces +(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding +would be byte-identical. + +### This is the opposite of frank-large + +| | frank-small (FreeToken) | frank-large (halogen) | +|---|---|---| +| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults | +| Deterministic across runs | No | Yes | +| Where the default comes from | checkpoint `generation_config.json` | none; the request decides | + +A client that talks to both endpoints and sends no sampler gets +non-deterministic answers from :7770 on Dankserver and reproducible ones from +:7770 on Dankclaw. If a workflow needs determinism from frank-small it must +send `temperature: 0` explicitly, or the service must be relaunched with +`--sampling-defaults none` (framework defaults only). + +### Related request behaviour + +- **Thinking consumes the budget.** At `max_tokens: 200` all three probe runs + returned `finish_reason: "length"` with **empty `content`** and the whole + budget spent in `reasoning_content`. Budget for thinking, or the answer + never appears. +- **The API does not validate unknown fields** — a request carrying + `totally_bogus_field: 123` returns 200, as does + `chat_template_kwargs: {"preserve_thinking": true}`. A 200 is not evidence + that a key was understood. + + ## Files | File | Purpose |