freeze tokenizer version for py3.8 (#3779)

The python package https://pypi.org/project/tokenizers dropped prebuilt binaries for python3.8 in the latest version. Freezing our Bert example's requirements file to a known working version
ROCm · Jan 24, 2025 · 0bfc8e0 · 0bfc8e0
1 parent 1c10b4d
commit 0bfc8e0
Showing 1 changed file with 3 additions and 2 deletions.
diff --git a/examples/nlp/python_bert_squad/requirements_bertsquad.txt b/examples/nlp/python_bert_squad/requirements_bertsquad.txt
@@ -1,7 +1,7 @@
 #####################################################################################
 # The MIT License (MIT)
 #
-# Copyright (c) 2015-2024 Advanced Micro Devices, Inc. All rights reserved.
+# Copyright (c) 2015-2025 Advanced Micro Devices, Inc. All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -24,4 +24,5 @@
 tensorflow
 onnxruntime
 tokenizers==0.11.1; python_version == "3.6"
-tokenizers; python_version > "3.6"
+tokenizers==0.20.3; python_version == "3.8"
+tokenizers; python_version > "3.8"