| Name : python311-tokenizers
| |
| Version : 0.20.0
| Vendor : openSUSE
|
| Release : lp160.1.3
| Date : 2024-09-23 09:19:52
|
| Group : Unspecified
| Source RPM : python-tokenizers-0.20.0-lp160.1.3.src.rpm
|
| Size : 6.90 MB
| |
| Packager : https://bugs_opensuse_org
| |
| Summary : Provides an implementation of today\'s most used tokenizers
|
Description :
Provides an implementation of today\'s most used tokenizers, with a focus on
performance and versatility.
* Train new vocabularies and tokenize, using today\'s most used tokenizers.
* Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text on a
server\'s CPU.
* Easy to use, but also extremely versatile.
* Designed for research and production.
* Normalization comes with alignments tracking. It\'s always possible to get the
part of the original sentence that corresponds to a given token.
* Does all the pre-processing: Truncate, Pad, add the special tokens your model
needs.
|
RPM found in directory: /vol/rzm3/linux-opensuse/distribution/leap/16.0/repo/oss/x86_64 |
Hmm ... It's impossible ;-) This RPM doesn't exist on any FTP server
Provides :
python3-tokenizers
python3.11dist(tokenizers)
python311-tokenizers
python311-tokenizers(x86-64)
python3dist(tokenizers)
Requires :
