Update documentation of Rust feature (#1711)

* Update documentation of Rust feature * Synchronize README.md and src/lib.rs
huggingface · Jan 9, 2025 · c04b97a · c04b97a
1 parent bdfc38b
commit c04b97a
Show file tree

Hide file tree

Showing 2 changed files with 4 additions and 0 deletions.
diff --git a/tokenizers/README.md b/tokenizers/README.md
@@ -137,3 +137,5 @@ fn main() -> Result<()> {
 **progressbar**: The progress bar visualization is enabled by default. It might be disabled if
   compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios)
   dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar.
+**http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default.
+  With this feature enabled, `Tokenizer::from_pretrained` becomes accessible.
diff --git a/tokenizers/src/lib.rs b/tokenizers/src/lib.rs
@@ -125,6 +125,8 @@
 //! **progressbar**: The progress bar visualization is enabled by default. It might be disabled if
 //!   compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios)
 //!   dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar.
+//! **http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default.
+//!   With this feature enabled, `Tokenizer::from_pretrained` becomes accessible.
 
 #[macro_use]
 extern crate log;