docs(frank-small): document that FreeToken samples by default from the checkpoint
Answering "does the FreeToken runtime use sampling arguments?" -- yes, and
nothing in our config revealed it. serve-frank-small.sh passes no sampling
flags; the behaviour is the runtime default:
freetoken/server/args.py:31 sampling_defaults: str = "model"
which fills temperature/top_k/top_p from the checkpoint's
generation_config.json -- this one ships do_sample:true, temp 1.0, top_k 20,
top_p 0.95. Proved live: three identical requests with no sampling params
produced three different reasoning traces (487/540/577 chars).
The asymmetry with frank-large is the part that will bite someone: a silent
client samples on frank-small and decodes greedy on frank-large, so the same
request is reproducible on one endpoint and not the other. Documented with the
determinism options (explicit temperature:0, or --sampling-defaults none).
Also recorded two request-level traps found during the probe: thinking eats the
whole max_tokens budget and returns empty content with finish_reason length,
and the API accepts unknown fields with a 200, so a 200 proves nothing about
whether a key was understood.
This commit is contained in:
@@ -125,6 +125,58 @@ tensor b (...)` in `_dequant_fp8_weight`, which multiplies
|
||||
`weight × weight_scale` directly instead of expanding the block scales.
|
||||
modelopt block-fp8 remains the only FP8 path that loads.
|
||||
|
||||
## Sampling
|
||||
|
||||
**frank-small samples by default, and our config never said so.**
|
||||
`scripts/serve-frank-small.sh` passes no sampling arguments at all — only
|
||||
`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
|
||||
`--num-tokens` and `--max-running-requests`. The behaviour comes from the
|
||||
runtime's own default:
|
||||
|
||||
```python
|
||||
# freetoken/server/args.py:31
|
||||
sampling_defaults: str = "model"
|
||||
```
|
||||
|
||||
`--sampling-defaults model` fills `temperature` / `top_k` / `top_p` from the
|
||||
**checkpoint's** `generation_config.json` for any request that omits them.
|
||||
This checkpoint ships:
|
||||
|
||||
```json
|
||||
{ "do_sample": true, "temperature": 1.0, "top_k": 20, "top_p": 0.95 }
|
||||
```
|
||||
|
||||
Verified behaviourally 2026-09-12 — three identical requests with **no
|
||||
sampling parameters sent at all** produced three different reasoning traces
|
||||
(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding
|
||||
would be byte-identical.
|
||||
|
||||
### This is the opposite of frank-large
|
||||
|
||||
| | frank-small (FreeToken) | frank-large (halogen) |
|
||||
|---|---|---|
|
||||
| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
|
||||
| Deterministic across runs | No | Yes |
|
||||
| Where the default comes from | checkpoint `generation_config.json` | none; the request decides |
|
||||
|
||||
A client that talks to both endpoints and sends no sampler gets
|
||||
non-deterministic answers from :7770 on Dankserver and reproducible ones from
|
||||
:7770 on Dankclaw. If a workflow needs determinism from frank-small it must
|
||||
send `temperature: 0` explicitly, or the service must be relaunched with
|
||||
`--sampling-defaults none` (framework defaults only).
|
||||
|
||||
### Related request behaviour
|
||||
|
||||
- **Thinking consumes the budget.** At `max_tokens: 200` all three probe runs
|
||||
returned `finish_reason: "length"` with **empty `content`** and the whole
|
||||
budget spent in `reasoning_content`. Budget for thinking, or the answer
|
||||
never appears.
|
||||
- **The API does not validate unknown fields** — a request carrying
|
||||
`totally_bogus_field: 123` returns 200, as does
|
||||
`chat_template_kwargs: {"preserve_thinking": true}`. A 200 is not evidence
|
||||
that a key was understood.
|
||||
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|
||||
Reference in New Issue
Block a user