Code

Custom ICU build#367

Code

Open

mryzhov wants to merge 31 commits intoopenvinotoolkit:masteropenvinotoolkit/openvino_tokenizers:masterfrom mryzhov:icu_buildmryzhov/openvino_tokenizers_public:icu_buildCopy head branch name to clipboard

+403-677

Commits

Commits on Nov 26, 2024

• Add unicode normalization layer tests

  apaniukov

  committed
  1dd6b66

  Copy full SHA for 1dd6b66

Commits on Nov 29, 2024

• WiP

  apaniukov

  committed
  afd4a60

  Copy full SHA for afd4a60

Commits on Dec 20, 2024

• WiP

  apaniukov

  committed
  2f24fec

  Copy full SHA for 2f24fec

Commits on Jan 8, 2025

• Switch Casefold and UnicodeNormalization to CharsMap

  apaniukov

  committed
  08052c2

  Copy full SHA for 08052c2

Commits on Jan 9, 2025

• Add unicode normalization layer tests

  apaniukov

  committed
  f6c001b

  Copy full SHA for f6c001b

• WiP

  apaniukov

  committed
  472b163

  Copy full SHA for 472b163

• WiP

  apaniukov

  committed
  04fb20c

  Copy full SHA for 04fb20c

• Switch Casefold and UnicodeNormalization to CharsMap

  apaniukov

  committed
  ed1203f

  Copy full SHA for ed1203f

• Update tests and fix custom charsmap support

  apaniukov

  committed
  012fb8e

  Copy full SHA for 012fb8e

• Merge remote-tracking branch 'origin/update-normalization' into update-normalization

  Show description for e3831ec
  apaniukov

  committed
  e3831ec

  Copy full SHA for e3831ec

• Ruff checks

  apaniukov

  committed
  8092720

  Copy full SHA for 8092720

• Merge branch 'master' into update-normalization

  apaniukov

  authored
  df34dee

  Copy full SHA for df34dee

• wip

  apaniukov

  committed
  6a611f3

  Copy full SHA for 6a611f3

• wip

  apaniukov

  committed
  258f0f4

  Copy full SHA for 258f0f4

• wip

  apaniukov

  committed
  baf0e70

  Copy full SHA for baf0e70

Commits on Jan 10, 2025

• Switch Off FastTokenizer

  Show description for 6177b81
  apaniukov

  committed
  6177b81

  Copy full SHA for 6177b81

• Delete torch from dependencies

  apaniukov

  committed
  68b7e4e

  Copy full SHA for 68b7e4e

• Delete FastTokenizer from cmake and readme

  apaniukov

  committed
  7244191

  Copy full SHA for 7244191

• Delete FastTokenizer related patches

  apaniukov

  committed
  082064c

  Copy full SHA for 082064c

• Delete FastTokenizer build form CI

  apaniukov

  committed
  7380898

  Copy full SHA for 7380898

• Delete FastTokenizer build form CI

  apaniukov

  committed
  68d0300

  Copy full SHA for 68d0300

• Delete FastTokenizer from Cmake

  apaniukov

  committed
  fc094a0

  Copy full SHA for fc094a0

• Delete FastTokenizer from Cmake

  apaniukov

  committed
  72b0646

  Copy full SHA for 72b0646

Commits on Jan 13, 2025

• use custom icu

  mryzhov

  committed
  4eb7dd0

  Copy full SHA for 4eb7dd0

• Merge branch 'master' into icu_build

  mryzhov

  authored
  9a85097

  Copy full SHA for 9a85097

• filter supported targets

  mryzhov

  committed
  cafaf03

  Copy full SHA for cafaf03

• removed tmp solution

  mryzhov

  committed
  deb6873

  Copy full SHA for deb6873

• brew icu4c

  mryzhov

  committed
  0e13658

  Copy full SHA for 0e13658

Commits on Jan 14, 2025

• install icu4c

  mryzhov

  committed
  ac21acd

  Copy full SHA for ac21acd

• fixed arch detection

  mryzhov

  committed
  a9c5b38

  Copy full SHA for a9c5b38

• fixed win subpath

  mryzhov

  committed
  e3eb2fd

  Copy full SHA for e3eb2fd