[feature](analysis) add new chinese tokenizer IK #269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zzzxl1993
reviewed
Jan 17, 2025
|
|
||
| CL_NS_DEF(analysis) | ||
|
|
||
| enum class AnalyzerMode { | ||
| Default, | ||
| All, | ||
| Search | ||
| Search, | ||
| IK_Smart, |
There was a problem hiding this comment.
Suggest separating the IK and Jieba enums for better clarity.
| if (store_size_ >= children_array_.size()) { | ||
| children_array_.resize(store_size_ + 1); | ||
| } | ||
| // 插入并保持有序 |
There was a problem hiding this comment.
Suggest using English comments.
Ryan19929
force-pushed
the
clucene-ik-20250102
branch
3 times, most recently
from
f538355 to
e45e3f3
Compare
Ryan19929
force-pushed
the
clucene-ik-20250102
branch
from
e45e3f3 to
35881b0
Compare
Migrate analysis-ik from Java to C++, implement basic tokenization functionality, and integrate it into CLucene.
Ryan19929
force-pushed
the
clucene-ik-20250102
branch
from
35881b0 to
eecf99b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support IK tokenizer for inverted index:
Migrate analysis-ik from Java to C++, Implement basic tokenization functionality.
The major differences from the original Java code are as follows:
Major changes to the original code:
/src/test/data/contribs-lib/analysis/chinese/speed-test-text.txt(红楼梦) for testing.Add IK tokenizer configuration, initialization entry, and dictionary loading logic.
Add the IK tokenization mode entry (temporary mode entry) in
AnalyzerMode.