| Name : python312-tokenizers
| |
| Version : 0.21.0
| Vendor : openSUSE
|
| Release : 2.1
| Date : 2025-01-09 16:52:37
|
| Group : Unspecified
| Source RPM : python-tokenizers-0.21.0-2.1.src.rpm
|
| Size : 6.94 MB
| |
| Packager : https://bugs_opensuse_org
| |
| Summary : Provides an implementation of today\'s most used tokenizers
|
Description :
Provides an implementation of today\'s most used tokenizers, with a focus on
performance and versatility.
* Train new vocabularies and tokenize, using today\'s most used tokenizers.
* Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text on a
server\'s CPU.
* Easy to use, but also extremely versatile.
* Designed for research and production.
* Normalization comes with alignments tracking. It\'s always possible to get the
part of the original sentence that corresponds to a given token.
* Does all the pre-processing: Truncate, Pad, add the special tokens your model
needs.
|
RPM found in directory: /vol/rzm3/linux-opensuse/slowroll/repo/oss/x86_64 |
Hmm ... It's impossible ;-) This RPM doesn't exist on any FTP server
Provides :
python3.12dist(tokenizers)
python312-tokenizers
python312-tokenizers(x86-64)
python3dist(tokenizers)
Requires :
