docs(radiance): correct the tokeniser corpus name and statsdump auth note
This commit is contained in:
1 parent
6e80cd44f0
commit
bdc0613739
2 files changed
+3
-2
No files matched your search
@@ -181,7 +181,8 @@ refusal**, not an engine fault: read the journal line and the argv, not the GPU.
|
|||||||
proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
|
proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
|
||||||
evictions, preemptions and queue depth. `/stats`,
|
evictions, preemptions and queue depth. `/stats`,
|
||||||
`/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
|
`/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
|
||||||
time and gap, for a machine with no scraper.
|
time and gap, for a machine with no scraper (its key comes from `--api-key` or
|
||||||
|
`RADIANCE_API_KEY`).
|
||||||
- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
|
- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
|
||||||
`RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
|
`RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
|
||||||
|
|
||||||
|
|||||||
@@ -255,7 +255,7 @@ eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at
|
|||||||
```
|
```
|
||||||
rad-tokcheck [options] <tokenizer.json> <corpus-file>...
|
rad-tokcheck [options] <tokenizer.json> <corpus-file>...
|
||||||
|
|
||||||
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus>.ids
|
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus basename>.ids
|
||||||
(little-endian int32, encode(text, add_special_tokens=False))
|
(little-endian int32, encode(text, add_special_tokens=False))
|
||||||
--no-ref skip the reference encoder (slow)
|
--no-ref skip the reference encoder (slow)
|
||||||
--user encode with parse_special off
|
--user encode with parse_special off
|
||||||
|
|||||||
Reference in new issue
Block a user