8000 ArangoSearch: Upgrade to Snowball 2 (additional stemmer languages) by Simran-B · Pull Request #10973 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

ArangoSearch: Upgrade to Snowball 2 (additional stemmer languages) #10973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Mar 18, 2020
Merged

ArangoSearch: Upgrade to Snowball 2 (additional stemmer languages) #10973

merged 12 commits into from
Mar 18, 2020

Conversation

Simran-B
Copy link
Contributor
@Simran-B Simran-B commented Jan 24, 2020 •
edited by Dronplane
Loading

Scope & Purpose

Upgrade Snowball dependency and add/enable stemming support for additional languages:

  • Arabic (ar)
  • Basque (eu)
  • Catalan (ca)
  • Danish (da)
  • Greek (el)
  • Hindi (hi)
  • Hungarian (hu)
  • Indonesian (id)
  • Irish (ga)
  • Lithuanian (lt)
  • Nepali (ne)
  • Romanian (ro)
  • Serbian (sr)
  • Tamil (ta)
  • Turkish (tr)

Example:

RETURN TOKENS("αυτοκινητουσ πρωταγωνιστούσαν", "text_el")
// [ [ "αυτοκινητ", "πρωταγωνιστ" ] ]

Related Information

Testing & Verification

Added js test to verify new stemmer is available and working

@Simran-B
Upgrade Snowball to 2.0
53c1df6 
.gitignore not added
@Simran-B
Update static file modules.h in iresearch.build folder
f400804 
Note: This gets copied over to build dir, replacing the otherwise perl-generated file
@Simran-B
Update IResearch locales with additional language codes
9c91e84 
This makes the new languages available as Analyzers, e.g. 'text_el' for Greek
@Simran-B
Update CMakeLists
9c2154c 
Fix/add generators, add algorithms, remove modules_utf8 stuff (Snowball 2 has a single modules.txt only)

TODO: remove fallback in gen_stem macro?
@Simran-B
Simplify gen_stem macro
96cad06
@Simran-B
Snowball upstream changes since November
04b99cf
@Simran-B Simran-B added 1 Feature 9 WIP 3 Search IResearch / Fulltext index / Analyzers 3 ArangoSearch Views labels Jan 24, 2020
@Simran-B Simran-B added this to the devel milestone Jan 24, 2020
@Simran-B Simran-B self-assigned this Jan 24, 2020
@Simran-B Simran-B mentioned this pull request Jan 24, 2020
Closed
13 tasks
@Dronplane Dronplane requested a review from gnusi March 18, 2020 07:35
@Dronplane
Copy link
Contributor

https://jenkins.arangodb.biz/view/PR/job/arangodb-matrix-pr/9080/

@Dronplane Dronplane marked this pull request as ready for review March 18, 2020 07:35
@Dronplane
Copy link
Contributor

Pulled latest devel
https://jenkins.arangodb.biz/view/PR/job/arangodb-matrix-pr/9083/

@gnusi gnusi merged commit b14a7c8 into devel Mar 18, 2020
@gnusi gnusi deleted the feature/snowball-2 branch March 18, 2020 11:27
@Simran-B
Copy link
Contributor Author

Docs PR: arangodb/docs#374

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gnusi gnusi gnusi approved these changes

Assignees

@Simran-B Simran-B

@Dronplane Dronplane

Labels
1 Feature 3 ArangoSearch Views 3 Search IResearch / Fulltext index / Analyzers 9 WIP
Projects
None yet
Milestone
devel
Development

Successfully merging this pull request may close these issues.
3 participants
@Simran-B @Dronplane @gnusi
0