Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: prepare Radixor 3.0.0 with contracted tries and compiled patch commands

Browse Source 
Introduce contracted compiled patch tries for faster lookup, make compiled
patch commands the primary runtime path, refresh stemmer benchmarks and
documentation, and restructure the documentation for 3.0.0 onboarding.

BREAKING CHANGE: Radixor 3.0.0 promotes compiled patch-command APIs and
new compiled trie artifacts as the primary runtime integration model.
This commit is contained in:
Leo Galambos
2026-07-03 18:44:39 +02:00
parent df4552b113
commit 38620d7e71
101 changed files with 11235 additions and 727 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/benchmarks/reference/candidates.md Normal file
View File

@@ -0,0 +1,23 @@
# Benchmark Candidate Evaluation
Implemented benchmark methods are documented on the per-language pages under [Language Benchmark Pages](../languages/index.md). This keeps the exact method names, implementation descriptions, accuracy tables, and speed tables close to the language where they are valid.
## Included Candidate Families
The current benchmark pages include Radixor baselines, Lucene language filters where the language matches a bundled Radixor resource, Lucene Stempel and Morfologik paths where applicable, official Snowball Java stemmers where same-language comparison is available, and selected English-specific non-Lucene baselines such as OpenNLP Porter and Paice/Husk Lancaster.
Direct stemmer APIs and Lucene TokenFilter paths are documented separately on language pages. TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
## Evaluated But Skipped Candidates
| Candidate | Language | Link/source | Reason skipped |
| --- | --- | --- | --- |
| Lucene Arabic, Bulgarian, Bengali, Sorani, Greek, Galician, Hindi, Indonesian, Latvian, Telugu filters | Various | `lucene-analysis-common` | No bundled same-language Radixor resource in this repository snapshot. |
| Lucene analyzer-only paths | Multiple | Lucene analyzers | Full analyzers mix tokenization, stop-word handling, and other behavior; direct filters are used where available. |
| Lucene HunspellStemFilter | Multiple | `lucene-analysis-common` | Requires external Hunspell dictionaries not resolved as benchmark-only resources here. |
| Lucene StemmerOverrideFilter | Multiple | `lucene-analysis-common` | Override map facility, not a stemmer algorithm. |
| Additional Snowball Lovins | English | Official Snowball Java distribution | No Lovins Java stemmer was present in the selected Snowball Java distribution. |
| Lemur Project Krovetz Stemmer | English | Lemur Project | Lucene KStem represents the Krovetz-style path without adding separate dependency and license risk. |
| Smile Lancaster / Paice-Husk | English | Smile NLP | Smile is large for one stemmer; Paice/Husk is included through a smaller benchmark-only generated path. |
| CISTEM German stemmer | German | `https://github.com/LeonieWeissweiler/CISTEM` | Clean benchmark-only Java integration was not completed in this phase. |
| `stemmerEval` reference repository | Multiple | `https://github.com/endredy/stemmerEval` | Used only as a candidate reference; no code or data copied. |

28
docs/benchmarks/reference/corpora.md Normal file
View File

@@ -0,0 +1,28 @@
# Benchmark Corpora
The table below describes the Radixor resources used to build speed and quality corpora. `Total tokens` is the complete dictionary token count used by quality benchmarks. `Already-root tokens` counts fields where the token is already equal to the line root. `Changed tokens` is the speed workload before the minimum-size repeat rule.
| Language resource | Dictionary rows | Total tokens | Already-root tokens | Changed tokens | Speed timing tokens |
| --- | ---: | ---: | ---: | ---: | ---: |
| `cs_cz` | 5,113 | 56,612 | 10,049 | 46,563 | 46,563 |
| `da_dk` | 4,179 | 32,256 | 8,356 | 23,900 | 23,900 |
| `de_de` | 39,315 | 213,440 | 73,799 | 139,641 | 139,641 |
| `es_es` | 65,059 | 926,393 | 120,121 | 806,272 | 806,272 |
| `fa_ir` | 69 | 3,770 | 138 | 3,632 | 5,000 |
| `fi_fi` | 57,027 | 1,865,215 | 110,525 | 1,754,690 | 1,754,690 |
| `fr_fr` | 59,240 | 474,110 | 108,141 | 365,969 | 365,969 |
| `he_il` | 2,358 | 61,071 | 4,715 | 56,356 | 56,356 |
| `hu_hu` | 19,406 | 935,713 | 38,775 | 896,938 | 896,938 |
| `it_it` | 10,009 | 337,546 | 20,004 | 317,542 | 317,542 |
| `nb_no` | 17,929 | 90,757 | 33,376 | 57,381 | 57,381 |
| `nl_nl` | 4,992 | 31,466 | 9,981 | 21,485 | 21,485 |
| `nn_no` | 4,688 | 19,651 | 6,089 | 13,562 | 13,562 |
| `pl_pl` | 9,990 | 132,308 | 19,957 | 112,351 | 112,351 |
| `pt_pt` | 4,001 | 215,490 | 8,002 | 207,488 | 207,488 |
| `ru_ru` | 37,410 | 806,279 | 74,808 | 731,471 | 731,471 |
| `sv_se` | 12,371 | 110,468 | 24,731 | 85,737 | 85,737 |
| `uk_ua` | 1,493 | 15,737 | 2,985 | 12,752 | 12,752 |
| `us_uk` | 396,939 | 1,004,374 | 793,874 | 210,500 | 210,500 |
| `yi` | 802 | 4,300 | 1,524 | 2,776 | 5,000 |
Speed benchmarks process the complete changed-token dictionary sequence for the language. Only resources with fewer than 5,000 changed tokens are repeated to reach the minimum timing size; larger resources are not sampled or truncated.

43
docs/benchmarks/reference/english-coverage.md Normal file
View File

@@ -0,0 +1,43 @@
# English Dictionary Coverage Benchmark
`EnglishRadixorDictionaryCoverageBenchmark` builds Radixor from deterministic slices of the English dictionary rows and evaluates accuracy against the complete dictionary. The speed method then stems the full changed-token English timing corpus.
This benchmark is the clearest demonstration of the Radixor quality/speed envelope after contracted-trie compilation. More dictionary knowledge still gives the strongest changed-form precision, but uniform-subtree contraction removes much of the historical lookup-depth penalty. The table should therefore be read as a measured operating curve rather than as a strictly monotonic function of dictionary size.
| Used rows | Actual row ratio | All exact | Changed exact | Root preserved | Speed ms/op | Error ms | ns/token |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 100% | 100.000% | 97.478% | 97.197% | 97.552% | 23.113 | 7.065 | 109.8 |
| 90% | 90.000% | 97.047% | 94.913% | 97.613% | 21.270 | 9.914 | 101.0 |
| 80% | 80.000% | 96.635% | 92.768% | 97.661% | 19.170 | 6.609 | 91.1 |
| 70% | 70.000% | 96.209% | 90.565% | 97.705% | 20.857 | 6.734 | 99.1 |
| 60% | 60.000% | 95.750% | 88.384% | 97.703% | 14.975 | 1.215 | 71.1 |
| 50% | 50.000% | 95.262% | 86.107% | 97.690% | 15.249 | 1.078 | 72.4 |
| 40% | 40.000% | 94.753% | 83.855% | 97.643% | 15.323 | 2.340 | 72.8 |
| 30% | 30.000% | 94.208% | 81.651% | 97.537% | 16.778 | 2.643 | 79.7 |
| 20% | 20.000% | 93.633% | 79.366% | 97.416% | 18.929 | 3.241 | 89.9 |
| 10% | 10.000% | 92.868% | 76.516% | 97.204% | 19.124 | 1.883 | 90.9 |
## Column Meanings
- `Used rows`: requested deterministic percentage of English dictionary rows used to build the trie.
- `Actual row ratio`: selected rows divided by all parsed English dictionary rows.
- `All exact`: exact agreement over the complete dictionary.
- `Changed exact`: exact agreement over dictionary tokens where `token != expectedRoot`.
- `Root preserved`: percentage of already-root dictionary tokens that are left unchanged.
- `Speed ms/op`: JMH average time for one full changed-token English operation.
- `Error ms`: JMH score error converted to milliseconds.
- `ns/token`: `Speed ms/op` divided by 210,500 changed English tokens.
For non-English languages, the same principle applies: dictionary-driven Radixor quality depends on the amount and consistency of the language resource, while contracted tries reduce the cost of uniform regions in the compiled lookup graph. The English table is the clearest because the English resource is large and the benchmark can show gradual deterministic reductions from 100% to 10%.
## Why The Historical Porter Ratio Changed
The historical English benchmark in `HEAD` used synthetic lexical families. Its `familyCount=5000` parameter generated roughly 70,000 artificial tokens rather than measuring the complete real English dictionary resource. That older workload was useful as a low-level stress test, but it was not a dictionary-quality comparison. Many synthetic tokens were not present in the Radixor dictionary, so Radixor often executed a fast miss path where lookup returned `null` and no patch command was applied.
The current benchmark is intentionally based on real Radixor dictionary data. For English, the speed workload processes 210,500 changed token/root pairs where the dictionary token differs from the expected root, and the quality workload evaluates the complete 1,004,374-token dictionary. This is a hit-heavy workload that measures real lookup plus compiled patch-command application against known expected roots. It is therefore a different and more linguistically meaningful workload than the historical synthetic benchmark.
The result must be interpreted in Radixor's favor through both speed and exact-root quality. Non-Radixor stemmers can look faster because many of them perform narrower or more aggressive transformations and do not attempt to match the dictionary root with the same precision. The English result table shows that this speed often comes with substantially lower `All exact` and `Changed exact` accuracy.
Radixor uses the dictionary as training data for transformation rules. With the full English dictionary, it reaches much higher exact-root agreement than the Porter-family and other narrow baselines. Higher speed is still possible by reducing the amount or complexity of the input dictionary used to build the stemmer, but that is an explicit quality/speed trade-off rather than an accidental benchmark artifact.
The coverage table shows that contracted tries substantially improve the operating point. Reducing dictionary knowledge still primarily damages changed-form exactness, while root preservation remains high. Even when Radixor is trained from only 10% of the English dictionary rows, the complete-dictionary `All exact` score remains above 92%. This is why Radixor performance should be discussed as a configurable quality/speed point, not as a single fixed ratio against Porter.

54
docs/benchmarks/reference/environment.md Normal file
View File

@@ -0,0 +1,54 @@
# Benchmark Environment And Reports
The values below are environment-specific and must not be read as universal performance claims.
| Item | Value |
| --- | --- |
| Benchmark date | 2026-07-03 |
| Focused comparison command family | JMH jar runs limited to `EnglishStemmerComparisonBenchmark`, `MultiLanguageStemmerComparisonBenchmark`, and `SnowballLanguageStemmerComparisonBenchmark`; Radixor exact-root metrics were recomputed deterministically against the same contracted loaders |
| English coverage command | `./gradlew jmh -Pjmh.includes='.*EnglishRadixorDictionaryCoverageBenchmark.*' --no-daemon` |
| Speed result reports | `build/reports/jmh/contracted/english-comparison.csv`, `multilanguage-speed.csv`, `snowball-language-speed.csv` |
| Accuracy result reports | Deterministic Radixor exact-root pass over bundled dictionaries; non-Radixor quality rows retained from the existing published quality suite |
| Final comparison JMH scope | Stemmer comparison benchmarks only; internal `FrequencyTrie*` microbenchmarks were not run |
| Coverage JMH scope | English Radixor dictionary coverage benchmark only |
| JMH version | 1.37 |
| Speed benchmark mode | Average time, `time/op` |
| Score unit | `ns/op` |
| Speed warmup | 3 iterations, 1 s each |
| Speed measurement | 5 iterations, 1 s each |
| Accuracy warmup | none for deterministic exact-root accounting |
| Accuracy measurement | 1 deterministic measurement iteration; counters only, not speed interpretation |
| Fork count in generated report files | 1 |
| Default fork policy for accuracy-only benchmark classes | `@Fork(0)` for future default runs because accuracy counters are deterministic and not interpreted as speed |
| Thread count | 1 |
| JVM reported by JMH | OpenJDK 64-Bit Server VM, 25.0.3+9 |
| JVM invoker | `/usr/lib/jvm/java-25-openjdk/bin/java` |
| Operating system | Linux 7.0.13-200.fc44.x86_64 |
| CPU | AMD Ryzen 5 7600 6-Core Processor |
| Logical CPUs | 12 |
## Contracted Trie Baseline
All Radixor rows in the refreshed benchmark tables use contracted compiled patch tries. During compilation, a subtree whose reachable entries all resolve to the same preferred patch command is represented as an accepting leaf. Runtime lookup can therefore stop as soon as that leaf is reached, which reduces depth in uniform regions while preserving the preferred result used by `get()`.
## Report Files
Generated local report files for this benchmark update:
- `build/reports/jmh/contracted/english-comparison.csv`
- `build/reports/jmh/contracted/english-comparison.txt`
- `build/reports/jmh/contracted/multilanguage-speed.csv`
- `build/reports/jmh/contracted/multilanguage-speed.txt`
- `build/reports/jmh/contracted/snowball-language-speed.csv`
- `build/reports/jmh/contracted/snowball-language-speed.txt`
JMH TXT and CSV reports are still published as benchmark artifacts. They are not converted into a Porter speed badge.
## Published Metrics
The historical English Radixor versus Porter performance badge is no longer generated. `tools/generate-pages-badges.py` now produces only coverage and mutation badge endpoint JSON files:
- `coverage-badge.json`
- `pitest-badge.json`
The README therefore no longer presents a single Porter speed ratio. Benchmark interpretation now uses both speed and quality, because a narrow or aggressive stemmer can be fast while disagreeing with the dictionary root much more often than Radixor.

59
docs/benchmarks/reference/methodology.md Normal file
View File

@@ -0,0 +1,59 @@
# Benchmark Methodology
The stemmer comparison suite measures Radixor and Java stemmers on the same language and deterministic Radixor dictionary-derived data. Published Radixor rows in this refresh use contracted compiled patch tries, where uniform preferred-command subtrees are collapsed into accepting leaves before the trie is frozen for lookup. For each language, the bundled dictionary resource stores the expected root as the first tab-separated field on a line and its surface forms on the same line. Every single-token field on that line can therefore be paired with the same expected root.
Published stemmer comparison results must come only from benchmark classes matching `.*StemmerComparisonBenchmark.*`. Internal `FrequencyTrie*` microbenchmarks are not part of those results.
## Benchmark Passes
There are two distinct benchmark passes:
- Speed benchmarks process only changed dictionary pairs where `token != expectedRoot`. This removes already-root tokens from timing so a stemmer is measured on words that actually require a transformation. If a language has fewer than 5,000 changed pairs, the complete changed-pair sequence is repeated in stable order until the timing corpus has at least 5,000 tokens. Larger changed-pair corpora are not sampled or truncated.
- Quality benchmarks process the complete dictionary for the language. They report exact agreement over all tokens, exact agreement over changed tokens only, and preservation of tokens that are already roots.
Timing corpora are generated once per JMH JVM and kept in memory as shared `{token, expectedRoot}` arrays. Corpus construction, dictionary loading, trie loading, table loading, and analyzer construction are setup work and are not included in measured benchmark methods.
Performance is interpreted as average time per input token:
```text
timePerChangedTokenNs = JMH score ns/op / changedTimingTokenCount
```
This is necessary because Radixor dictionaries have different token counts by language.
## Quality And Search Interpretation
Radixor speed must be interpreted together with exact-root quality. A slower Radixor row must not be read as a simple performance weakness when Radixor is also the row with accuracy close to 100% and competing stemmers are much lower.
Many fast light, minimal, possessive, or aggressive rule-based stemmers are fast because they do much less linguistic work. The measured Radixor cost buys dictionary-trained precision, and that precision is what improves search quality when queries and indexed text are reduced to the same intended roots.
The [English dictionary coverage benchmark](english-coverage.md) shows this operating curve explicitly: contracted tries reduce lookup cost in uniform regions, while reduced dictionary coverage still lowers changed-form precision.
## Normalization Policy
Radixor is measured over dictionary tokens from its own resources: lower-case with `Locale.ROOT`, diacritics preserved. The corpus is normalized during setup, so the Radixor benchmark path uses `FrequencyTrie.getNormalized(CharSequence)` and does not measure redundant lookup-time lowercasing or diacritic normalization.
Lucene TokenFilter paths include required normalization in the measured pipeline. Examples include lower-case normalization for filters requiring lower-case input, German normalization before German light/minimal stemming, and Persian decimal, Arabic, and Persian normalization before Persian stemming. No ASCII folding is applied to Czech or Polish paths, because those Lucene stemmers are diacritic-aware or dictionary/table-backed for those languages. TokenFilter throughput methods materialize each emitted `CharTermAttribute` as a `String` before passing it to the JMH `Blackhole`, so output consumption is easier to inspect and closer to the direct stemmer methods.
For right-to-left Radixor languages, patch application uses the traversal direction stored in trie metadata. This is required because static backward patch application is not correct for all bundled languages.
## Quality Metric
The quality pass reports exact-root agreement against the expected root from the Radixor dictionary line. It writes to the normal JMH report files:
- `build/reports/jmh/jmh-results.csv`
- `build/reports/jmh/jmh-results.txt`
Accuracy is computed from standard JMH secondary rows:
```text
allExactPercent = correctMatches / evaluatedTokens * 100
changedExactPercent = changedCorrectMatches / changedEvaluatedTokens * 100
rootPreservedPercent = rootPreservedMatches / rootEvaluatedTokens * 100
```
`allExactPercent` uses the complete dictionary. `changedExactPercent` uses only tokens where `token != expectedRoot`. `rootPreservedPercent` measures whether a stemmer leaves already-root dictionary entries unchanged.
Morfologik can emit multiple terms for one input token. The quality benchmark uses the first emitted term for exact-root accounting when no ranking weight is exposed. Throughput benchmarks for Morfologik TokenFilter paths consume all emitted terms.
Quality reports intentionally use one deterministic measurement iteration without warmup, because exact-root agreement is not a timing metric and repeated precision passes would only duplicate the same counters.