| Name : python312-tokenizers
| |
| Version : 0.19.1
| Vendor : obs://build_opensuse_org/system:homeautomation
|
| Release : 2.1
| Date : 2024-07-03 16:55:36
|
| Group : Unspecified
| Source RPM : python-tokenizers-0.19.1-2.1.src.rpm
|
| Size : 8.31 MB
| |
| Packager : (none)
| |
| Summary : Provides an implementation of today\'s most used tokenizers
|
Description :
Provides an implementation of today\'s most used tokenizers, with a focus on
performance and versatility.
* Train new vocabularies and tokenize, using today\'s most used tokenizers.
* Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text on a
server\'s CPU.
* Easy to use, but also extremely versatile.
* Designed for research and production.
* Normalization comes with alignments tracking. It\'s always possible to get the
part of the original sentence that corresponds to a given token.
* Does all the pre-processing: Truncate, Pad, add the special tokens your model
needs.
|
RPM found in directory: /packages/linux-pbone/ftp5.gwdg.de/pub/opensuse/repositories/system:/homeautomation:/home-assistant/openSUSE_Factory/x86_64 |
