docs(radiance): correct the tokeniser corpus name and statsdump auth note
This commit is contained in:
1 parent
6e80cd44f0
commit
bdc0613739
2 files changed
+3
-2
No files matched your search
@@ -255,7 +255,7 @@ eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at
|
||||
```
|
||||
rad-tokcheck [options] <tokenizer.json> <corpus-file>...
|
||||
|
||||
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus>.ids
|
||||
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus basename>.ids
|
||||
(little-endian int32, encode(text, add_special_tokens=False))
|
||||
--no-ref skip the reference encoder (slow)
|
||||
--user encode with parse_special off
|
||||
|
||||
Reference in new issue
Block a user