Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add CISTEM and Hunspell benchmarks and refresh results

Browse Source
This commit is contained in:
Leo Galambos
2026-07-06 01:51:33 +02:00
parent a52e82933f
commit 049f44e697
33 changed files with 1410 additions and 173 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/benchmarks/reference/candidates.md
View File

@@ -4,20 +4,28 @@ Implemented benchmark methods are documented on the per-language pages under [La
## Included Candidate Families
The current benchmark pages include Radixor baselines, Lucene language filters where the language matches a bundled Radixor resource, Lucene Stempel and Morfologik paths where applicable, official Snowball Java stemmers where same-language comparison is available, and selected English-specific non-Lucene baselines such as OpenNLP Porter and Paice/Husk Lancaster.
The current benchmark pages include Radixor baselines, Lucene language filters where the language matches a bundled Radixor resource, Lucene Stempel and Morfologik paths where applicable, official Snowball Java stemmers where same-language comparison is available, benchmark-only CISTEM German stemmer evaluation, benchmark-only Hunspell comparisons, and selected English-specific non-Lucene baselines such as OpenNLP Porter and Paice/Husk Lancaster.
Benchmark-only Hunspell comparisons use bundled benchmark dictionaries and the Lucene HunspellStemFilter adapter over the selected language token streams.
The CISTEM candidate is implemented in `src/jmh/java/org/egothor/stemmer/benchmark/Cistem.java` and follows the original MIT-licensed upstream implementation from Leonie Weissweiler's CISTEM project.
CISTEM German gold-standard files are not vendored in this repository. The Gradle JMH resource preparation tasks download `goldstandard1.txt` and `goldstandard2.txt` from the upstream CISTEM repository into generated build resources.
Direct stemmer APIs and Lucene TokenFilter paths are documented separately on language pages. TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
For the benchmark refresh used in this documentation build:
- Hunspell families are included in `HunspellStemmerComparisonBenchmark` (speed) and `HunspellStemmerComparisonBenchmarkQuality` (quality for all benchmark languages in this corpus). The legacy
`EnglishHunspellStemmerComparisonBenchmarkQuality` result is retained for continuity.
- CISTEM quality is present in the published per-language results under `GERMAN_CISTEM`. CISTEM speed is present as `germanCistem` in `MultiLanguageStemmerComparisonBenchmark`.
## Evaluated But Skipped Candidates
| Candidate | Language | Link/source | Reason skipped |
| --- | --- | --- | --- |
| Lucene Arabic, Bulgarian, Bengali, Sorani, Greek, Galician, Hindi, Indonesian, Latvian, Telugu filters | Various | `lucene-analysis-common` | No bundled same-language Radixor resource in this repository snapshot. |
| Lucene analyzer-only paths | Multiple | Lucene analyzers | Full analyzers mix tokenization, stop-word handling, and other behavior; direct filters are used where available. |
| Lucene HunspellStemFilter | Multiple | `lucene-analysis-common` | Requires external Hunspell dictionaries not resolved as benchmark-only resources here. |
| Lucene StemmerOverrideFilter | Multiple | `lucene-analysis-common` | Override map facility, not a stemmer algorithm. |
| Additional Snowball Lovins | English | Official Snowball Java distribution | No Lovins Java stemmer was present in the selected Snowball Java distribution. |
| Lemur Project Krovetz Stemmer | English | Lemur Project | Lucene KStem represents the Krovetz-style path without adding separate dependency and license risk. |
| Smile Lancaster / Paice-Husk | English | Smile NLP | Smile is large for one stemmer; Paice/Husk is included through a smaller benchmark-only generated path. |
| CISTEM German stemmer | German | `https://github.com/LeonieWeissweiler/CISTEM` | Clean benchmark-only Java integration was not completed in this phase. |
| `stemmerEval` reference repository | Multiple | `https://github.com/endredy/stemmerEval` | Used only as a candidate reference; no code or data copied. |

20
docs/benchmarks/reference/english-coverage.md
View File

@@ -6,16 +6,16 @@ This benchmark is the clearest demonstration of the Radixor quality/speed envelo
| Used rows | Actual row ratio | All exact | Changed exact | Root preserved | Speed ms/op | Error ms | ns/token |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 100% | 100.000% | 97.478% | 97.197% | 97.552% | 23.113 | 7.065 | 109.8 |
| 90% | 90.000% | 97.047% | 94.913% | 97.613% | 21.270 | 9.914 | 101.0 |
| 80% | 80.000% | 96.635% | 92.768% | 97.661% | 19.170 | 6.609 | 91.1 |
| 70% | 70.000% | 96.209% | 90.565% | 97.705% | 20.857 | 6.734 | 99.1 |
| 60% | 60.000% | 95.750% | 88.384% | 97.703% | 14.975 | 1.215 | 71.1 |
| 50% | 50.000% | 95.262% | 86.107% | 97.690% | 15.249 | 1.078 | 72.4 |
| 40% | 40.000% | 94.753% | 83.855% | 97.643% | 15.323 | 2.340 | 72.8 |
| 30% | 30.000% | 94.208% | 81.651% | 97.537% | 16.778 | 2.643 | 79.7 |
| 20% | 20.000% | 93.633% | 79.366% | 97.416% | 18.929 | 3.241 | 89.9 |
| 10% | 10.000% | 92.868% | 76.516% | 97.204% | 19.124 | 1.883 | 90.9 |
| 100% | 100.000% | 97.478% | 97.197% | 97.552% | 28.578 | 7.571 | 135.8 |
| 90% | 90.000% | 97.047% | 94.913% | 97.613% | 26.612 | 9.227 | 126.4 |
| 80% | 80.000% | 96.635% | 92.768% | 97.661% | 23.331 | 8.106 | 110.8 |
| 70% | 70.000% | 96.209% | 90.565% | 97.705% | 22.362 | 1.957 | 106.2 |
| 60% | 60.000% | 95.750% | 88.384% | 97.703% | 16.497 | 2.026 | 78.4 |
| 50% | 50.000% | 95.262% | 86.107% | 97.690% | 16.035 | 0.986 | 76.2 |
| 40% | 40.000% | 94.753% | 83.855% | 97.643% | 16.459 | 0.664 | 78.2 |
| 30% | 30.000% | 94.208% | 81.651% | 97.537% | 19.566 | 0.758 | 92.9 |
| 20% | 20.000% | 93.633% | 79.366% | 97.416% | 14.616 | 0.487 | 69.4 |
| 10% | 10.000% | 92.868% | 76.516% | 97.204% | 18.093 | 3.147 | 86.0 |
## Column Meanings

32
docs/benchmarks/reference/environment.md
View File

@@ -4,11 +4,11 @@ The values below are environment-specific and must not be read as universal perf
| Item | Value |
| --- | --- |
| Benchmark date | 2026-07-03 |
| Focused comparison command family | JMH jar runs limited to `EnglishStemmerComparisonBenchmark`, `MultiLanguageStemmerComparisonBenchmark`, and `SnowballLanguageStemmerComparisonBenchmark`; Radixor exact-root metrics were recomputed deterministically against the same contracted loaders |
| Benchmark date | 2026-07-06 (Europe/Prague) |
| Focused comparison command family | `./gradlew jmh -Pjmh.includes='.*StemmerComparisonBenchmark.*' --no-daemon` |
| English coverage command | `./gradlew jmh -Pjmh.includes='.*EnglishRadixorDictionaryCoverageBenchmark.*' --no-daemon` |
| Speed result reports | `build/reports/jmh/contracted/english-comparison.csv`, `multilanguage-speed.csv`, `snowball-language-speed.csv` |
| Accuracy result reports | Deterministic Radixor exact-root pass over bundled dictionaries; non-Radixor quality rows retained from the existing published quality suite |
| Speed result reports | `build/reports/jmh/stemmer-comparison-2026-07-06.csv`, `build/reports/jmh/stemmer-comparison-2026-07-06.txt`, `build/reports/jmh/english-coverage-2026-07-06.csv`, and `build/reports/jmh/english-coverage-2026-07-06.txt` |
| Accuracy result reports | `build/reports/jmh/stemmer-comparison-2026-07-06.csv`, `build/reports/jmh/english-coverage-2026-07-06.csv`, and deterministic Radixor exact-root accounting over the same bundled language corpora |
| Final comparison JMH scope | Stemmer comparison benchmarks only; internal `FrequencyTrie*` microbenchmarks were not run |
| Coverage JMH scope | English Radixor dictionary coverage benchmark only |
| JMH version | 1.37 |
@@ -16,15 +16,19 @@ The values below are environment-specific and must not be read as universal perf
| Score unit | `ns/op` |
| Speed warmup | 3 iterations, 1 s each |
| Speed measurement | 5 iterations, 1 s each |
| Accuracy warmup | none for deterministic exact-root accounting |
| Accuracy measurement | 1 deterministic measurement iteration; counters only, not speed interpretation |
| Accuracy warmup | 3 JMH warmup iterations were applied by the Gradle invocation; timing scores from quality methods are not interpreted |
| Accuracy measurement | 5 JMH measurement samples; documentation uses deterministic auxiliary counter ratios from the same report |
| Fork count in generated report files | 1 |
| Default fork policy for accuracy-only benchmark classes | `@Fork(0)` for future default runs because accuracy counters are deterministic and not interpreted as speed |
| Thread count | 1 |
| JVM reported by JMH | OpenJDK 64-Bit Server VM, 25.0.3+9 |
| JVM reported by JMH | JDK 25.0.3, OpenJDK 64-Bit Server VM, 25.0.3+9 |
| Java runtime | OpenJDK Runtime Environment, Red Hat build 25.0.3+9 |
| JVM invoker | `/usr/lib/jvm/java-25-openjdk/bin/java` |
| Operating system | Linux 7.0.13-200.fc44.x86_64 |
| CPU | AMD Ryzen 5 7600 6-Core Processor |
| Operating system | Fedora Linux 44 (MATE-Compiz) |
| Kernel | Linux 7.0.12-201.fc44.x86_64 |
| Architecture | x86_64 |
| CPU | AMD Ryzen 5 8600G w/ Radeon 760M Graphics |
| Physical cores | 6 |
| Logical CPUs | 12 |
## Contracted Trie Baseline
@@ -35,12 +39,10 @@ All Radixor rows in the refreshed benchmark tables use contracted compiled patch
Generated local report files for this benchmark update:
- `build/reports/jmh/contracted/english-comparison.csv`
- `build/reports/jmh/contracted/english-comparison.txt`
- `build/reports/jmh/contracted/multilanguage-speed.csv`
- `build/reports/jmh/contracted/multilanguage-speed.txt`
- `build/reports/jmh/contracted/snowball-language-speed.csv`
- `build/reports/jmh/contracted/snowball-language-speed.txt`
- `build/reports/jmh/stemmer-comparison-2026-07-06.csv`
- `build/reports/jmh/stemmer-comparison-2026-07-06.txt`
- `build/reports/jmh/english-coverage-2026-07-06.csv`
- `build/reports/jmh/english-coverage-2026-07-06.txt`
JMH TXT and CSV reports are still published as benchmark artifacts. They are not converted into a Porter speed badge.

2
docs/benchmarks/reference/methodology.md
View File

@@ -56,4 +56,4 @@ rootPreservedPercent = rootPreservedMatches / rootEvaluatedTokens * 100
Morfologik can emit multiple terms for one input token. The quality benchmark uses the first emitted term for exact-root accounting when no ranking weight is exposed. Throughput benchmarks for Morfologik TokenFilter paths consume all emitted terms.
Quality reports intentionally use one deterministic measurement iteration without warmup, because exact-root agreement is not a timing metric and repeated precision passes would only duplicate the same counters.
Quality reports use JMH auxiliary counter rows. Exact-root accounting is deterministic for a fixed corpus and stemmer, so repeated measurement samples duplicate the same counters; documentation uses the counter ratios and does not interpret quality benchmark timing scores.