docs(radiance): correct the tokeniser corpus name and statsdump auth note

This commit is contained in:
Frank committed 2026-10-30 15:03:31 -07:00
1 parent 6e80cd44f0
commit bdc0613739
2 files changed
+3 -2

No files matched your search

+1 -1
View File
@@ -255,7 +255,7 @@ eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at
```
rad-tokcheck [options] <tokenizer.json> <corpus-file>...
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus>.ids
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus basename>.ids
(little-endian int32, encode(text, add_special_tokens=False))
--no-ref skip the reference encoder (slow)
--user encode with parse_special off