Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: prepare Radixor 3.0.0 with contracted tries and compiled patch commands

Browse Source 
Introduce contracted compiled patch tries for faster lookup, make compiled
patch commands the primary runtime path, refresh stemmer benchmarks and
documentation, and restructure the documentation for 3.0.0 onboarding.

BREAKING CHANGE: Radixor 3.0.0 promotes compiled patch-command APIs and
new compiled trie artifacts as the primary runtime integration model.
This commit is contained in:
Leo Galambos
2026-07-03 18:44:39 +02:00
parent df4552b113
commit 38620d7e71
101 changed files with 11235 additions and 727 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/architecture.md
View File

@@ -53,6 +53,16 @@ Those patch-command values are inserted into a mutable trie keyed by the source
Equivalent subtrees are merged into canonical reduced nodes.
Before a selected semantic reduction mode is applied, Radixor also performs uniform-subtree
contraction. If every reachable entry below a subtree resolves to the same preferred patch
command, that subtree can be represented as an accepting leaf for that command. Runtime lookup can
then stop at that leaf even when the input word still has remaining characters.
This is a structural optimization of preferred-result lookup. It reduces trie depth in regions
where the remaining suffix cannot change the selected command, while preserving the `get()` result
used by the standard stemmer path. The benchmark tables in `docs/benchmarks/` are based on this
contracted compiled representation.
### Compilation
The reduced structure is frozen into an immutable compiled trie optimized for runtime lookup.
@@ -124,7 +134,7 @@ At runtime, lookup is conceptually simple:
3. retrieve one or more stored patch commands,
4. apply the chosen patch command to the original word.
The trie itself does not create the final stem string. It selects the stored transformation command. `PatchCommandEncoder.apply(...)` then performs the actual transformation.
The trie itself does not create the final stem string. It selects the stored transformation command. Runtime code should use `CompiledPatchCommand.apply(...)` so the serialized command is compiled once and reused.
That separation is architecturally important:
@@ -170,6 +180,7 @@ The final compiled trie can be much smaller than the original dictionary for sev
- patch commands are compact,
- trie paths reuse shared structure,
- uniform preferred-command subtrees can be contracted into accepting leaves,
- reduction merges equivalent subtrees,
- binary persistence stores the already reduced form,
- GZip compression is applied on top of the binary format.

170
docs/benchmarking.md
View File

@@ -1,159 +1,39 @@
# Benchmarking
Radixor includes a JMH benchmark suite for both the internal algorithmic core and a side-by-side English comparison against the Snowball Porter stemmer family.
Radixor contains internal trie microbenchmarks, a separate stemmer comparison suite, and a dictionary coverage benchmark for Radixor itself. Published stemmer comparison results must come only from benchmark classes matching `.*StemmerComparisonBenchmark.*`; internal `FrequencyTrie*` microbenchmarks are not part of those results.
This document explains what is benchmarked, how to run the suite, and how benchmark results should be interpreted.
This page is the entry point for benchmark interpretation. Detailed tables and long reference material are split into focused subpages so that important points do not get buried.
## Scope
## Key Takeaways
The benchmark suite currently covers two categories:
- Speed and accuracy must be read together. A faster row is not necessarily a better stemmer.
- Radixor is the quality-oriented baseline in same-language comparisons. Its exact-root accuracy is often close to 100%, while many faster competitors are light, minimal, possessive, or aggressive rule-based stemmers with much lower root agreement.
- The measured Radixor cost buys dictionary-trained stemming precision. That precision improves search quality by mapping inflected forms to intended dictionary roots instead of approximate or over-reduced stems.
- Speed benchmarks process changed dictionary tokens where the surface form differs from the expected root. Accuracy benchmarks process the complete dictionary.
- Accuracy-only benchmarks intentionally use one deterministic JMH measurement iteration without warmup because repeated precision passes would duplicate the same counters.
- The historical Porter performance badge is retired. Benchmark reporting now uses speed and quality tables rather than a single Porter ratio.
- Radixor core operations,
- English stemmer comparison on the same token workload.
## Benchmark Documentation Map
The comparison benchmark processes the same deterministic English token stream through:
| Page | Purpose |
| --- | --- |
| [Benchmark methodology](benchmarks/reference/methodology.md) | Workload design, speed pass, quality pass, normalization policy, and exact-root metrics. |
| [Benchmark corpora](benchmarks/reference/corpora.md) | Dictionary row counts, complete quality tokens, already-root tokens, changed speed tokens, and timing token counts. |
| [Benchmark environment and reports](benchmarks/reference/environment.md) | Hardware, OS, JVM, JMH settings, report files, and current badge/report policy. |
| [English dictionary coverage benchmark](benchmarks/reference/english-coverage.md) | The quality/speed operating curve for contracted Radixor tries built from 100% down to 10% of English dictionary rows. |
| [Candidate evaluation](benchmarks/reference/candidates.md) | Included benchmark families and evaluated candidates that were skipped. |
| [Language benchmark pages](benchmarks/languages/index.md) | Per-language accuracy tables, speed tables, and implementation notes. |
- Radixor with bundled `US_UK` (older benchmark snapshots used the now-retired `US_UK_PROFI` resource),
- Snowball original Porter,
- Snowball English, commonly referred to as Porter2.
## How To Read Results
The purpose of the comparison is throughput measurement on identical input. It is not intended to demonstrate linguistic equivalence between the compared stemmers.
Start with the [language benchmark pages](benchmarks/languages/index.md). Each language page lists accuracy first and speed second because throughput without root agreement is not enough to interpret stemmer quality.
## How to read the published numbers
When Radixor is slower than a narrow competitor, check the accuracy table before drawing a conclusion. Many Lucene light/minimal filters and possessive filters intentionally do less work. They can be fast precisely because they are not trying to match the dictionary root with the same precision.
Two kinds of benchmark numbers are relevant in the project.
The [English dictionary coverage benchmark](benchmarks/reference/english-coverage.md) shows the central operating curve explicitly: contracted tries preserve high quality even at reduced dictionary coverage, while changed-form exactness still reflects how much language knowledge was available during training. This is why Radixor performance should be discussed as a configurable quality/speed point, not as a single fixed ratio against Porter.
### Reference measurements
## Current Result Locations
The detailed benchmark snapshot documented on this page comes from a controlled run on a Ryzen 5 system. Those numbers are the best reference point for understanding absolute throughput under a known local benchmark environment.
The current measured language results are published in [Language Benchmark Pages](benchmarks/languages/index.md). Generated local report files for this benchmark update are listed in [Benchmark environment and reports](benchmarks/reference/environment.md).
### Published badge figures
The benchmark badge metadata published through GitHub Pages is generated in the GitHub-hosted container environment. That environment is convenient for continuous publication, but it is not the right place to treat absolute throughput values as stable across time. CPU scheduling, shared-host variability, and container-level noise can materially affect raw numbers from run to run.
For that reason, the published badge values should be treated primarily as a compact status surface. They are useful for observing broad trends and relative positioning, but not as the authoritative source for precise absolute throughput claims.
## Current snapshot
A recent JMH run on JDK 21.0.10 with JMH 1.37, one thread, three warmup iterations, and five measurement iterations produced the following approximate throughput ranges:
| Workload | Radixor `US_UK` *(historical runs: `US_UK_PROFI`)* | Snowball Porter | Snowball English |
| --- | ---: | ---: | ---: |
| About 12,000 generated tokens | 30.99 M tokens/s | 8.21 M tokens/s | 5.46 M tokens/s |
| About 60,000 generated tokens | 32.25 M tokens/s | 8.02 M tokens/s | 5.11 M tokens/s |
On that workload, Radixor measured approximately:
- 4 times the throughput of Snowball original Porter,
- 6 times the throughput of Snowball English.
These values are workload-dependent and environment-dependent. They should be read as measured results for the documented setup, not as universal constants.
## Interpreting the relative result
Although the absolute numbers can move across environments, the throughput relationship between Radixor and the compared Porter-family stemmers has remained broadly stable in practical measurements. In particular, the comparison against Snowball original Porter is consistently in the rough range of about four to one in Radixor’s favor.
That relative behavior is more informative than any single absolute figure. It reflects a real architectural difference rather than a cosmetic benchmark artifact.
Radixor is built around a compiled patch-command trie that resolves the result through a direct lookup and patch application path. In contrast, classic rule-based stemmers such as the Porter family follow a different operational model. The result is that Radixor combines two properties that do not often appear together:
- dictionary-driven compiled lookup performance,
- the ability to generalize beyond explicitly listed word forms instead of behaving like a pure closed-form dictionary lookup table.
Within that design space, the measured throughput profile is strong enough to place Radixor among the fastest known practical implementations of this kind, while still supporting stemming of previously unseen forms. That should still be read as a carefully bounded engineering statement, not as an absolute claim over every possible stemmer architecture or benchmark scenario.
## Benchmark classes
The main benchmark classes are under `src/jmh/java/org/egothor/stemmer/benchmark`.
Relevant classes include:
- `FrequencyTrieLookupBenchmark`,
- `FrequencyTrieCompilationBenchmark`,
- `EnglishStemmerComparisonBenchmark`.
The English comparison benchmark uses the bundled Radixor English resource and the official Snowball Java distribution integrated into the JMH source set.
## Workload design
The English comparison benchmark uses a deterministic generated corpus rather than an uncontrolled ad hoc text sample.
The workload intentionally mixes:
- simple inflections,
- common derivational forms,
- US and UK spelling families,
- lexical forms appropriate for the current bundled `US_UK` resource (with historical continuity from earlier `US_UK_PROFI` runs).
This design keeps runs reproducible across environments and avoids accidental drift caused by changing external corpora.
## Running benchmarks
Run the full benchmark suite:
```bash
./gradlew jmh
```
Run only the English comparison benchmark:
```bash
./gradlew jmh -Pjmh.includes=EnglishStemmerComparisonBenchmark
```
## Generated reports
JMH reports are written to:
- `build/reports/jmh/jmh-results.txt`,
- `build/reports/jmh/jmh-results.csv`.
The text report is convenient for human review. The CSV report is more useful for CI archiving, historical tracking, and external processing.
## Interpreting results responsibly
Benchmark numbers should always be read with care.
Important factors include:
- CPU model and frequency behavior,
- thermal throttling,
- JVM vendor and version,
- system background load,
- operating-system scheduling noise,
- benchmark parameter changes.
For meaningful comparison, keep these stable:
- hardware or VM class,
- JDK version,
- benchmark parameters,
- thread count,
- benchmark source revision.
If a regression is suspected, repeat the run and compare against previous CSV output rather than relying on a single measurement.
## Regression tracking
The recommended regression workflow is:
1. archive `jmh-results.csv`,
2. compare the same benchmark names across runs,
3. compare only like-for-like environments,
4. investigate sustained regressions rather than one-off noise.
For public reporting, the README should keep only the condensed benchmark summary, while detailed benchmark methodology and interpretation should remain in this document.
## Notes on comparison fairness
Radixor, Snowball Porter, and Snowball English are not the same kind of stemmer.
Radixor uses a compiled patch-command trie driven by dictionary data. Snowball Porter and Snowball English are rule-based English stemmers.
Because of that, the comparison should be understood as:
- equal input workload,
- different stemming strategies,
- measured throughput rather than semantic identity.
That distinction matters whenever performance claims are discussed in documentation, release notes, or badge summaries.
JMH TXT and CSV reports are still published as benchmark artifacts. They are no longer converted into a Shields endpoint benchmark badge.

55
docs/benchmarks/index.md Normal file
View File

@@ -0,0 +1,55 @@
# Benchmark Results
This section contains the published Radixor benchmark reference set. It is intentionally split into
two layers:
- **benchmark reference pages**, which explain methodology, corpora, environment, candidate
selection, and the English dictionary coverage experiment;
- **language result pages**, which contain the actual same-language accuracy and throughput tables.
This structure keeps methodology separate from per-language result pages, while preserving all
measured data and the command-class analysis for each Radixor language resource.
## Read This First
Start with [Benchmarking](../benchmarking.md) for the high-level interpretation model. The most
important rule is that speed and exact-root quality must be read together. Many competing stemmers
are intentionally light, minimal, or aggressive; they can be fast because they are not trying to
match dictionary roots with the same precision.
Radixor rows in the refreshed tables use contracted compiled patch tries. Contraction collapses
uniform preferred-command subtrees into accepting leaves, reducing hot lookup depth while preserving
the preferred result measured by the accuracy pass.
## Reference Pages
| Page | Purpose |
| --- | --- |
| [Methodology](reference/methodology.md) | Workload design, normalization, speed metrics, quality metrics, and interpretation rules. |
| [Corpora](reference/corpora.md) | Dictionary row counts, complete quality tokens, already-root tokens, changed speed tokens, and timing token counts. |
| [Environment and reports](reference/environment.md) | Hardware, JVM, JMH settings, report files, and badge/report policy. |
| [English dictionary coverage](reference/english-coverage.md) | Quality/speed operating curve for contracted Radixor tries built from 100% down to 10% of English dictionary rows. |
| [Candidate evaluation](reference/candidates.md) | Included benchmark families and evaluated candidates that were skipped. |
## Language Results
Each language page contains:
- the dictionary corpus size,
- the Radixor patch-command distribution,
- exact-root quality metrics,
- throughput metrics,
- interpretation notes for the compared stemmers.
Open [Language Benchmark Pages](languages/index.md) for the complete language list.
## Key Published Result
The English dictionary coverage benchmark shows the current contracted-trie operating curve. With
the full English dictionary, Radixor reaches `97.478%` all-token exactness and `97.197%`
changed-token exactness at `109.8 ns/token`. Even with a deterministic 10% dictionary slice, it
keeps `92.868%` all-token exactness and `76.516%` changed-token exactness at `90.9 ns/token`.
Those figures should not be reduced to a single speed badge. The professional interpretation is a
quality/speed envelope: the amount and quality of dictionary knowledge affect stemming precision,
while contracted tries reduce lookup cost in uniform regions of the compiled graph.

51
docs/benchmarks/languages/czech.md Normal file
View File

@@ -0,0 +1,51 @@
# Czech Stemmer Benchmarks
This page reports same-language stemming benchmarks for Czech. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `CS_CZ` | 5,113 | 56,612 | 10,049 | 46,563 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **56,612**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 675 | 1.192% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 22,681 | 40.064% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 14,980 | 26.461% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 10,109 | 17.857% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 8,167 | 14.426% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.465% | 99.439% | 99.582% | Full Radixor dictionary patch-command stemmer. |
| Lucene CzechStemFilter | 16.784% | 15.538% | 22.559% | Lucene Czech suffix stemmer implemented as a TokenFilter. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `czechRadixor` | 3.117 | 0.454 | 66.9 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene CzechStemFilter | `czechLuceneCzechStemFilter` | 2.921 | 0.202 | 62.7 | 0.937 | Czech suffix stemmer implemented as a Lucene TokenFilter. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

53
docs/benchmarks/languages/danish.md Normal file
View File

@@ -0,0 +1,53 @@
# Danish Stemmer Benchmarks
This page reports same-language stemming benchmarks for Danish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `DA_DK` | 4,179 | 32,256 | 8,356 | 23,900 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **32,256**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 137 | 0.425% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 1,127 | 3.494% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 22,586 | 70.021% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 8,405 | 26.057% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 1 | 0.003% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.371% | 99.527% | 98.923% | Radixor baseline in the Snowball-language comparison family. |
| Lucene SnowballFilter | 55.509% | 54.159% | 59.371% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 55.509% | 54.159% | 59.371% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `radixor[DANISH]` | 1.065 | 0.019 | 44.6 | 1.000 | Radixor baseline for the Snowball-language comparison family. |
| Official Snowball direct | `snowballDirect[DANISH]` | 2.028 | 0.011 | 84.9 | 1.904 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[DANISH]` | 2.692 | 0.028 | 112.6 | 2.527 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

53
docs/benchmarks/languages/dutch.md Normal file
View File

@@ -0,0 +1,53 @@
# Dutch Stemmer Benchmarks
This page reports same-language stemming benchmarks for Dutch. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `NL_NL` | 4,992 | 31,466 | 9,981 | 21,485 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **31,466**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 2,107 | 6.696% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 11,484 | 36.497% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 7,732 | 24.573% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 10,127 | 32.184% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 16 | 0.051% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.120% | 98.711% | 100.000% | Radixor baseline in the Snowball-language comparison family. |
| Official Snowball direct | 15.954% | 8.992% | 30.939% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene SnowballFilter | 12.620% | 5.441% | 28.073% | Lucene TokenFilter integration path around the Snowball algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `radixor[DUTCH]` | 1.262 | 0.039 | 58.7 | 1.000 | Radixor baseline for the Snowball-language comparison family. |
| Official Snowball direct | `snowballDirect[DUTCH]` | 3.968 | 0.258 | 184.7 | 3.145 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[DUTCH]` | 6.866 | 0.337 | 319.6 | 5.441 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

67
docs/benchmarks/languages/english.md Normal file
View File

@@ -0,0 +1,67 @@
# English Stemmer Benchmarks
This page reports same-language stemming benchmarks for English. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `US_UK` | 396,939 | 1,004,374 | 793,874 | 210,500 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **1,004,374**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 28 | 0.003% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 22,493 | 2.240% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 186,764 | 18.595% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 795,024 | 79.156% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 65 | 0.006% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 97.478% | 97.197% | 97.552% | Full Radixor dictionary patch-command stemmer. |
| Lucene EnglishMinimalStemFilter | 90.981% | 65.189% | 97.820% | Minimal English plural reduction, not a full stemmer. |
| Lucene KStemFilter | 80.076% | 76.608% | 80.996% | Krovetz-style English stemming TokenFilter; broader than minimal suffix reducers. |
| Lucene EnglishPossessiveFilter | 79.032% | 0.003% | 99.987% | Possessive-ending remover only, not a full stemmer. |
| Snowball English / Porter2 | 40.342% | 46.296% | 38.763% | Porter2 rule-based suffix stemmer, distinct from original Porter. |
| Lucene PorterStemFilter | 39.538% | 46.201% | 37.772% | Lucene TokenFilter path for Porter suffix rules; not dictionary-root equivalent. |
| Lucene PorterStemmer direct copy | 39.538% | 46.201% | 37.772% | Direct Porter suffix-rule implementation generated under build for benchmark-only use. |
| OpenNLP PorterStemmer | 39.538% | 46.201% | 37.772% | Apache OpenNLP Porter suffix-rule implementation. |
| Snowball original Porter | 39.529% | 46.179% | 37.766% | Classic Porter rule-based suffix stemmer. |
| Paice/Husk Lancaster | 28.055% | 37.039% | 25.673% | Aggressive Paice/Husk rule stemmer that often produces shorter stems. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `radixorUsUkProfiPreferredStem` | 16.621 | 8.532 | 79.0 | 1.000 | Full dictionary patch-command stemmer using compiled patch commands. |
| Lucene EnglishPossessiveFilter | `luceneEnglishPossessiveFilter` | 23.845 | 0.833 | 113.3 | 1.435 | Possessive-ending remover only; not a full stemmer. |
| Lucene EnglishMinimalStemFilter | `luceneEnglishMinimalStemFilter` | 17.091 | 0.198 | 81.2 | 1.028 | Narrow plural reduction filter; not a full stemmer. |
| Lucene PorterStemmer direct copy | `lucenePorterStemmerCopied` | 18.598 | 10.954 | 88.4 | 1.119 | Benchmark-only generated copy of Lucene package-private Porter implementation. |
| OpenNLP PorterStemmer | `opennlpPorterStemmer` | 18.213 | 10.674 | 86.5 | 1.096 | Apache OpenNLP Porter implementation. |
| Snowball original Porter | `snowballOriginalPorter` | 32.921 | 11.520 | 156.4 | 1.981 | Classic Porter suffix-rule stemmer; historical English baseline, not a dictionary-equivalent stemmer. |
| Lucene PorterStemFilter | `lucenePorterStemFilter` | 42.874 | 1.321 | 203.7 | 2.579 | Lucene TokenFilter integration path for Porter; includes TokenStream overhead. |
| Lucene KStemFilter | `luceneKStemFilter` | 50.483 | 3.624 | 239.8 | 3.037 | Krovetz-style English TokenFilter; broader than minimal suffix filters. |
| Snowball English / Porter2 | `snowballEnglishPorter2` | 47.844 | 1.887 | 227.3 | 2.878 | Porter2 suffix-rule stemmer, distinct from original Porter. |
| Paice/Husk Lancaster | `paiceHuskLancaster` | 135.050 | 11.088 | 641.6 | 8.125 | Aggressive rule-based English stemmer. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

55
docs/benchmarks/languages/finnish.md Normal file
View File

@@ -0,0 +1,55 @@
# Finnish Stemmer Benchmarks
This page reports same-language stemming benchmarks for Finnish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `FI_FI` | 57,027 | 1,865,215 | 110,525 | 1,754,690 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **1,865,215**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 745 | 0.040% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 1,176,003 | 63.049% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 565,585 | 30.323% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 116,946 | 6.270% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 5,936 | 0.318% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 98.661% | 98.803% | 96.408% | Full Radixor dictionary patch-command stemmer. |
| Lucene SnowballFilter | 10.991% | 10.268% | 22.471% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 10.991% | 10.268% | 22.471% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene FinnishLightStemFilter | 4.351% | 4.294% | 5.264% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `finnishRadixor` | 228.248 | 9.245 | 130.1 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene FinnishLightStemFilter | `finnishLuceneFinnishLightStemFilter` | 175.923 | 87.177 | 100.3 | 0.771 | Light Finnish suffix stemmer. |
| Official Snowball direct | `snowballDirect[FINNISH]` | 265.579 | 95.687 | 151.4 | 1.164 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[FINNISH]` | 338.099 | 175.101 | 192.7 | 1.481 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

57
docs/benchmarks/languages/french.md Normal file
View File

@@ -0,0 +1,57 @@
# French Stemmer Benchmarks
This page reports same-language stemming benchmarks for French. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `FR_FR` | 59,240 | 474,110 | 108,141 | 365,969 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **474,110**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 5,370 | 1.133% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 185,263 | 39.076% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 153,886 | 32.458% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 116,519 | 24.576% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 13,072 | 2.757% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 94.831% | 94.859% | 94.734% | Full Radixor dictionary patch-command stemmer. |
| Lucene FrenchMinimalStemFilter | 11.472% | 6.236% | 29.192% | Minimal suffix reducer; narrow baseline, not a full stemmer. |
| Lucene SnowballFilter | 8.551% | 5.183% | 19.952% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 8.462% | 5.067% | 19.952% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene FrenchLightStemFilter | 6.377% | 3.965% | 14.540% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `frenchRadixor` | 38.598 | 5.425 | 105.5 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene FrenchMinimalStemFilter | `frenchLuceneFrenchMinimalStemFilter` | 17.657 | 1.956 | 48.2 | 0.457 | Minimal French suffix reducer; narrow baseline. |
| Lucene FrenchLightStemFilter | `frenchLuceneFrenchLightStemFilter` | 28.742 | 2.391 | 78.5 | 0.745 | Light French suffix stemmer. |
| Official Snowball direct | `snowballDirect[FRENCH]` | 104.983 | 10.116 | 286.9 | 2.720 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[FRENCH]` | 117.938 | 4.007 | 322.3 | 3.056 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

59
docs/benchmarks/languages/german.md Normal file
View File

@@ -0,0 +1,59 @@
# German Stemmer Benchmarks
This page reports same-language stemming benchmarks for German. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `DE_DE` | 39,315 | 213,440 | 73,799 | 139,641 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **213,440**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 3,627 | 1.699% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 48,605 | 22.772% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 80,443 | 37.689% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 75,717 | 35.475% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 5,048 | 2.365% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 97.455% | 97.973% | 96.476% | Full Radixor dictionary patch-command stemmer. |
| Lucene GermanLightStemFilter | 38.583% | 35.800% | 43.849% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
| Lucene GermanMinimalStemFilter | 37.492% | 38.538% | 35.513% | Minimal suffix reducer; narrow baseline, not a full stemmer. |
| Lucene SnowballFilter | 33.380% | 30.939% | 37.999% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 32.863% | 31.225% | 35.963% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene GermanStemFilter | 26.168% | 24.979% | 28.416% | German Lucene stemming TokenFilter; broader than minimal/light variants. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `germanRadixor` | 9.518 | 0.338 | 68.2 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene GermanMinimalStemFilter | `germanLuceneGermanMinimalStemFilter` | 12.020 | 0.460 | 86.1 | 1.263 | Minimal German suffix reduction; narrow baseline. |
| Lucene GermanLightStemFilter | `germanLuceneGermanLightStemFilter` | 12.413 | 1.059 | 88.9 | 1.304 | Light German suffix stemmer; narrower than a dictionary stemmer. |
| Lucene GermanStemFilter | `germanLuceneGermanStemFilter` | 36.644 | 5.667 | 262.4 | 3.850 | Older German stemming TokenFilter with normalization requirements. |
| Lucene SnowballFilter | `luceneSnowballFilter[GERMAN]` | 54.846 | 8.710 | 392.8 | 5.762 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
| Official Snowball direct | `snowballDirect[GERMAN]` | 52.974 | 7.989 | 379.4 | 5.566 | Official Snowball generated Java stemmer; direct API. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

55
docs/benchmarks/languages/hungarian.md Normal file
View File

@@ -0,0 +1,55 @@
# Hungarian Stemmer Benchmarks
This page reports same-language stemming benchmarks for Hungarian. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `HU_HU` | 19,406 | 935,713 | 38,775 | 896,938 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **935,713**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 15 | 0.002% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 149,173 | 15.942% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 746,296 | 79.757% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 40,125 | 4.288% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 104 | 0.011% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.222% | 99.537% | 91.948% | Full Radixor dictionary patch-command stemmer. |
| Lucene SnowballFilter | 66.445% | 66.938% | 55.043% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 66.445% | 66.938% | 55.043% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene HungarianLightStemFilter | 14.748% | 14.777% | 14.086% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `hungarianRadixor` | 53.844 | 5.619 | 60.0 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene HungarianLightStemFilter | `hungarianLuceneHungarianLightStemFilter` | 85.802 | 4.554 | 95.7 | 1.594 | Light Hungarian suffix stemmer. |
| Official Snowball direct | `snowballDirect[HUNGARIAN]` | 161.996 | 61.038 | 180.6 | 3.009 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[HUNGARIAN]` | 185.097 | 44.447 | 206.4 | 3.438 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

45
docs/benchmarks/languages/index.md Normal file
View File

@@ -0,0 +1,45 @@
# Language Benchmark Pages
This section splits Radixor stemmer benchmark results by language. Each language page lists accuracy first and speed second.
## Reference Pages
| Page | Purpose |
| --- | --- |
| [Methodology](../reference/methodology.md) | Workload design, normalization, speed metrics, and quality metrics. |
| [Corpora](../reference/corpora.md) | Dictionary sizes and changed-token timing workloads. |
| [Environment and reports](../reference/environment.md) | Hardware, JVM, JMH settings, report files, and badge policy. |
| [English dictionary coverage](../reference/english-coverage.md) | Quality/speed operating curve for contracted Radixor tries built from 100% down to 10% of English dictionary rows. |
| [Candidate evaluation](../reference/candidates.md) | Included and skipped stemmer candidates. |
## Languages
| Language | Resource | Benchmark page |
| --- | --- | --- |
| Czech | `CS_CZ` | [Czech](czech.md) |
| Danish | `DA_DK` | [Danish](danish.md) |
| Dutch | `NL_NL` | [Dutch](dutch.md) |
| English | `US_UK` | [English](english.md) |
| Finnish | `FI_FI` | [Finnish](finnish.md) |
| French | `FR_FR` | [French](french.md) |
| German | `DE_DE` | [German](german.md) |
| Hungarian | `HU_HU` | [Hungarian](hungarian.md) |
| Italian | `IT_IT` | [Italian](italian.md) |
| Norwegian Bokmal | `NB_NO` | [Norwegian Bokmal](norwegian-bokmal.md) |
| Norwegian Nynorsk | `NN_NO` | [Norwegian Nynorsk](norwegian-nynorsk.md) |
| Persian | `FA_IR` | [Persian](persian.md) |
| Polish | `PL_PL` | [Polish](polish.md) |
| Portuguese | `PT_PT` | [Portuguese](portuguese.md) |
| Russian | `RU_RU` | [Russian](russian.md) |
| Spanish | `ES_ES` | [Spanish](spanish.md) |
| Swedish | `SV_SE` | [Swedish](swedish.md) |
| Ukrainian | `UK_UA` | [Ukrainian](ukrainian.md) |
| Yiddish | `YI` | [Yiddish](yiddish.md) |
## Methodology Notes
- Speed benchmarks process only changed dictionary tokens where the surface form differs from the expected root.
- Accuracy benchmarks process the complete dictionary and report `All exact`, `Changed exact`, and `Root preserved`.
- Radixor speed must be interpreted together with exact-root quality. A slower Radixor row must not be read as a simple performance weakness when Radixor is also the row with accuracy close to 100% and competing stemmers are much lower. Many fast light, minimal, possessive, or aggressive rule-based stemmers are fast because they do much less linguistic work. The measured Radixor cost buys dictionary-trained precision, and that precision is what improves search quality when queries and indexed text are reduced to the same intended roots. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows this contracted-trie operating curve explicitly.
- Results are comparable only within the same language and benchmark family.
- The historical Porter badge is retired; no JMH badge JSON is generated.

54
docs/benchmarks/languages/italian.md Normal file
View File

@@ -0,0 +1,54 @@
# Italian Stemmer Benchmarks
This page reports same-language stemming benchmarks for Italian. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `IT_IT` | 10,009 | 337,546 | 20,004 | 317,542 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **337,546**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 302,171 | 89.520% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 12,348 | 3.658% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 20,013 | 5.929% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 3,014 | 0.893% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.056% | 98.997% | 100.000% | Full Radixor dictionary patch-command stemmer. |
| Lucene ItalianLightStemFilter | 0.466% | 0.479% | 0.270% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
| Lucene SnowballFilter | 0.041% | 0.043% | 0.010% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 0.041% | 0.043% | 0.010% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `italianRadixor` | 23.776 | 9.977 | 74.9 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene ItalianLightStemFilter | `italianLuceneItalianLightStemFilter` | 14.940 | 1.682 | 47.1 | 0.628 | Light Italian suffix stemmer. |
| Official Snowball direct | `snowballDirect[ITALIAN]` | 99.401 | 9.433 | 313.0 | 4.181 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[ITALIAN]` | 108.462 | 8.014 | 341.6 | 4.562 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

57
docs/benchmarks/languages/norwegian-bokmal.md Normal file
View File

@@ -0,0 +1,57 @@
# Norwegian Bokmal Stemmer Benchmarks
This page reports same-language stemming benchmarks for Norwegian Bokmal. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `NB_NO` | 17,929 | 90,757 | 33,376 | 57,381 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **90,757**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 1,500 | 1.653% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 4,296 | 4.734% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 47,619 | 52.469% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 34,420 | 37.925% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 2,922 | 3.220% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 96.852% | 97.637% | 95.503% | Full Radixor dictionary patch-command stemmer. |
| Lucene NorwegianMinimalStemFilter | 57.107% | 53.913% | 62.599% | Minimal suffix reducer; narrow baseline, not a full stemmer. |
| Official Snowball direct | 54.824% | 51.791% | 60.040% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene SnowballFilter | 54.803% | 51.780% | 60.001% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Lucene NorwegianLightStemFilter | 52.136% | 50.616% | 54.749% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `norwegianBokmalRadixor` | 3.235 | 0.147 | 56.4 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene NorwegianMinimalStemFilter | `norwegianBokmalLuceneNorwegianMinimalStemFilter` | 2.720 | 0.190 | 47.4 | 0.841 | Minimal Norwegian suffix reducer. |
| Lucene NorwegianLightStemFilter | `norwegianBokmalLuceneNorwegianLightStemFilter` | 3.189 | 0.261 | 55.6 | 0.986 | Light Norwegian suffix stemmer. |
| Official Snowball direct | `snowballDirect[NORWEGIAN_BOKMAL]` | 3.978 | 0.031 | 69.3 | 1.230 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[NORWEGIAN_BOKMAL]` | 5.526 | 0.409 | 96.3 | 1.708 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

53
docs/benchmarks/languages/norwegian-nynorsk.md Normal file
View File

@@ -0,0 +1,53 @@
# Norwegian Nynorsk Stemmer Benchmarks
This page reports same-language stemming benchmarks for Norwegian Nynorsk. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `NN_NO` | 4,688 | 19,651 | 6,089 | 13,562 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **19,651**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 224 | 1.140% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 1,505 | 7.659% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 11,017 | 56.063% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 6,427 | 32.706% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 478 | 2.432% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 93.089% | 91.395% | 96.863% | Radixor baseline in the Snowball-language comparison family. |
| Official Snowball direct | 60.974% | 60.212% | 62.670% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene SnowballFilter | 60.918% | 60.146% | 62.638% | Lucene TokenFilter integration path around the Snowball algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `radixor[NORWEGIAN_NYNORSK]` | 0.541 | 0.021 | 39.9 | 1.000 | Radixor baseline for the Snowball-language comparison family. |
| Official Snowball direct | `snowballDirect[NORWEGIAN_NYNORSK]` | 0.855 | 0.008 | 63.0 | 1.580 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[NORWEGIAN_NYNORSK]` | 1.221 | 0.025 | 90.0 | 2.258 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

49
docs/benchmarks/languages/persian.md Normal file
View File

@@ -0,0 +1,49 @@
# Persian Stemmer Benchmarks
This page reports same-language stemming benchmarks for Persian. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `FA_IR` | 69 | 3,770 | 138 | 3,632 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **3,770**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `DeletePrefixCommand` | Deletes one or more leading characters from the word form in forward traversal. | 65 | 1.724% |
| `ForwardCompoundCommand` | Applies a multi-step forward patch made from skip, delete, insert, and replace operations. | 3,567 | 94.615% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 138 | 3.660% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 95.836% | 95.677% | 100.000% | Full Radixor dictionary patch-command stemmer. |
| Lucene PersianStemFilter | 1.485% | 0.000% | 40.580% | Lucene Persian suffix stemmer with required normalization in the measured path. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `persianRadixor` | 0.231 | 0.021 | 63.6 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene PersianStemFilter | `persianLucenePersianStemFilter` | 0.443 | 0.012 | 122.0 | 1.918 | Persian suffix stemmer with Lucene normalization in the measured path. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

55
docs/benchmarks/languages/polish.md Normal file
View File

@@ -0,0 +1,55 @@
# Polish Stemmer Benchmarks
This page reports same-language stemming benchmarks for Polish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `PL_PL` | 9,990 | 132,308 | 19,957 | 112,351 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **132,308**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 1,719 | 1.299% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 53,303 | 40.287% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 37,051 | 28.004% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 20,415 | 15.430% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 19,820 | 14.980% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 98.837% | 98.744% | 99.359% | Full Radixor dictionary patch-command stemmer. |
| Lucene MorfologikFilter | 87.729% | 86.606% | 94.047% | Dictionary-based path; Morfologik can emit multiple terms. |
| Lucene StempelFilter | 70.009% | 69.262% | 74.220% | Lucene TokenFilter integration path for table-driven Polish Stempel. |
| Lucene StempelStemmer direct | 70.009% | 69.262% | 74.220% | Direct table-driven Polish Stempel stemmer API. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `polishRadixor` | 7.760 | 0.240 | 69.1 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene StempelStemmer direct | `polishLuceneStempelStemmerDirect` | 34.295 | 0.418 | 305.3 | 4.420 | Direct table-driven Polish Stempel stemmer API. |
| Lucene StempelFilter | `polishLuceneStempelFilter` | 39.116 | 1.717 | 348.2 | 5.041 | Lucene TokenFilter integration path for table-driven Polish Stempel. |
| Lucene MorfologikFilter | `polishLuceneMorfologikFilter` | 128.516 | 12.557 | 1143.9 | 16.562 | Dictionary-based Morfologik TokenFilter; may emit multiple terms. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

59
docs/benchmarks/languages/portuguese.md Normal file
View File

@@ -0,0 +1,59 @@
# Portuguese Stemmer Benchmarks
This page reports same-language stemming benchmarks for Portuguese. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `PT_PT` | 4,001 | 215,490 | 8,002 | 207,488 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **215,490**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 3,806 | 1.766% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 120,691 | 56.008% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 71,284 | 33.080% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 8,003 | 3.714% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 11,706 | 5.432% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.815% | 99.808% | 100.000% | Full Radixor dictionary patch-command stemmer. |
| Lucene PortugueseLightStemFilter | 8.966% | 5.558% | 97.326% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
| Lucene PortugueseMinimalStemFilter | 5.539% | 1.896% | 100.000% | Minimal suffix reducer; narrow baseline, not a full stemmer. |
| Lucene SnowballFilter | 0.625% | 0.558% | 2.374% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 0.625% | 0.558% | 2.374% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene PortugueseStemFilter | 0.312% | 0.308% | 0.425% | Portuguese RSLP-style Lucene TokenFilter stemmer. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `portugueseRadixor` | 10.598 | 0.273 | 51.1 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene PortugueseLightStemFilter | `portugueseLucenePortugueseLightStemFilter` | 10.101 | 0.389 | 48.7 | 0.953 | Light Portuguese suffix stemmer. |
| Lucene PortugueseMinimalStemFilter | `portugueseLucenePortugueseMinimalStemFilter` | 14.493 | 0.760 | 69.8 | 1.367 | Minimal Portuguese suffix reducer. |
| Official Snowball direct | `snowballDirect[PORTUGUESE]` | 52.508 | 6.338 | 253.1 | 4.954 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[PORTUGUESE]` | 54.048 | 1.173 | 260.5 | 5.100 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
| Lucene PortugueseStemFilter | `portugueseLucenePortugueseStemFilter` | 141.208 | 12.785 | 680.6 | 13.324 | Portuguese RSLP-style Lucene TokenFilter. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

55
docs/benchmarks/languages/russian.md Normal file
View File

@@ -0,0 +1,55 @@
# Russian Stemmer Benchmarks
This page reports same-language stemming benchmarks for Russian. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `RU_RU` | 37,410 | 806,279 | 74,808 | 731,471 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **806,279**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 9,260 | 1.148% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 584,785 | 72.529% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 82,864 | 10.277% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 75,646 | 9.382% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 53,724 | 6.663% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 98.807% | 98.696% | 99.896% | Full Radixor dictionary patch-command stemmer. |
| Lucene RussianLightStemFilter | 9.658% | 8.452% | 21.447% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
| Lucene SnowballFilter | 9.162% | 8.162% | 18.936% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 9.162% | 8.162% | 18.936% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `russianRadixor` | 72.970 | 15.642 | 99.8 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene RussianLightStemFilter | `russianLuceneRussianLightStemFilter` | 57.900 | 4.404 | 79.2 | 0.793 | Light Russian suffix stemmer. |
| Official Snowball direct | `snowballDirect[RUSSIAN]` | 99.019 | 14.688 | 135.4 | 1.357 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[RUSSIAN]` | 128.272 | 7.201 | 175.4 | 1.758 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

59
docs/benchmarks/languages/spanish.md Normal file
View File

@@ -0,0 +1,59 @@
# Spanish Stemmer Benchmarks
This page reports same-language stemming benchmarks for Spanish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `ES_ES` | 65,059 | 926,393 | 120,121 | 806,272 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **926,393**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 5,367 | 0.579% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 524,682 | 56.637% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 240,872 | 26.001% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 130,089 | 14.043% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 25,383 | 2.740% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 97.459% | 97.544% | 96.891% | Full Radixor dictionary patch-command stemmer. |
| Lucene SpanishMinimalStemFilter | 17.284% | 5.347% | 97.403% | Minimal suffix reducer; narrow baseline, not a full stemmer. |
| Lucene SpanishPluralStemFilter | 15.140% | 5.802% | 77.820% | Plural-focused suffix reducer; narrow baseline. |
| Lucene SpanishLightStemFilter | 9.577% | 7.088% | 26.279% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
| Lucene SnowballFilter | 4.889% | 4.287% | 8.932% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 4.889% | 4.287% | 8.930% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `spanishRadixor` | 64.539 | 3.448 | 80.0 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene SpanishMinimalStemFilter | `spanishLuceneSpanishMinimalStemFilter` | 38.288 | 2.382 | 47.5 | 0.593 | Minimal Spanish suffix reducer; narrow baseline. |
| Lucene SpanishLightStemFilter | `spanishLuceneSpanishLightStemFilter` | 40.855 | 2.407 | 50.7 | 0.633 | Light Spanish suffix stemmer. |
| Lucene SpanishPluralStemFilter | `spanishLuceneSpanishPluralStemFilter` | 91.054 | 8.822 | 112.9 | 1.411 | Plural-oriented Spanish suffix reducer. |
| Official Snowball direct | `snowballDirect[SPANISH]` | 168.813 | 32.860 | 209.4 | 2.616 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[SPANISH]` | 185.626 | 50.297 | 230.2 | 2.876 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

57
docs/benchmarks/languages/swedish.md Normal file
View File

@@ -0,0 +1,57 @@
# Swedish Stemmer Benchmarks
This page reports same-language stemming benchmarks for Swedish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `SV_SE` | 12,371 | 110,468 | 24,731 | 85,737 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **110,468**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 502 | 0.454% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 14,268 | 12.916% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 66,796 | 60.466% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 25,745 | 23.305% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 3,157 | 2.858% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 96.713% | 97.407% | 94.307% | Full Radixor dictionary patch-command stemmer. |
| Lucene SwedishMinimalStemFilter | 49.532% | 49.186% | 50.730% | Minimal suffix reducer; narrow baseline, not a full stemmer. |
| Lucene SwedishLightStemFilter | 45.672% | 46.383% | 43.209% | Light suffix stemmer; intentionally narrower than a dictionary-derived stemmer. |
| Official Snowball direct | 40.068% | 37.512% | 48.926% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
| Lucene SnowballFilter | 38.785% | 35.839% | 48.999% | Lucene TokenFilter integration path around the Snowball algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `swedishRadixor` | 4.916 | 0.525 | 57.3 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Lucene SwedishMinimalStemFilter | `swedishLuceneSwedishMinimalStemFilter` | 4.453 | 0.530 | 51.9 | 0.906 | Minimal Swedish suffix reducer. |
| Lucene SwedishLightStemFilter | `swedishLuceneSwedishLightStemFilter` | 4.523 | 0.151 | 52.8 | 0.920 | Light Swedish suffix stemmer. |
| Official Snowball direct | `snowballDirect[SWEDISH]` | 7.075 | 0.541 | 82.5 | 1.439 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[SWEDISH]` | 9.379 | 0.056 | 109.4 | 1.908 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

53
docs/benchmarks/languages/ukrainian.md Normal file
View File

@@ -0,0 +1,53 @@
# Ukrainian Stemmer Benchmarks
This page reports same-language stemming benchmarks for Ukrainian. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `UK_UA` | 1,493 | 15,737 | 2,985 | 12,752 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **15,737**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 249 | 1.582% |
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 4,160 | 26.435% |
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 5,859 | 37.231% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 3,004 | 19.089% |
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 2,465 | 15.664% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 99.307% | 99.365% | 99.062% | Full Radixor dictionary patch-command stemmer. |
| Lucene MorfologikFilter | 92.362% | 90.637% | 99.732% | Dictionary-based path; Morfologik can emit multiple terms. |
| Morfologik direct | 92.362% | 90.637% | 99.732% | Direct dictionary lookup; first returned stem is used for quality when no ranking weight is exposed. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `ukrainianRadixor` | 0.605 | 0.056 | 47.4 | 1.000 | Full Radixor dictionary patch-command stemmer. |
| Morfologik direct | `ukrainianMorfologikDirect` | 8.106 | 0.040 | 635.7 | 13.408 | Direct Morfologik dictionary lookup; first returned stem is used for quality. |
| Lucene MorfologikFilter | `ukrainianLuceneMorfologikFilter` | 14.684 | 5.214 | 1151.5 | 24.287 | Dictionary-based Morfologik TokenFilter; may emit multiple terms. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

52
docs/benchmarks/languages/yiddish.md Normal file
View File

@@ -0,0 +1,52 @@
# Yiddish Stemmer Benchmarks
This page reports same-language stemming benchmarks for Yiddish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
## Dictionary Corpus
| Resource | Dictionary rows | Complete quality tokens | Already-root tokens | Changed speed tokens |
| --- | ---: | ---: | ---: | ---: |
| `YI` | 802 | 4,300 | 1,524 | 2,776 |
## Radixor Patch Command Distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete language dictionary. The total number of preferred patch commands analyzed for this language is **4,300**.
| Command class | Meaning | Word forms | Share |
| --- | --- | ---: | ---: |
| `DeletePrefixCommand` | Deletes one or more leading characters from the word form in forward traversal. | 25 | 0.581% |
| `ForwardCompoundCommand` | Applies a multi-step forward patch made from skip, delete, insert, and replace operations. | 2,721 | 63.279% |
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 1,551 | 36.070% |
| `ReplaceFirstCharacterCommand` | Replaces the first character of the word form in forward traversal. | 3 | 0.070% |
## Accuracy
Accuracy is computed from one deterministic JMH measurement iteration without warmup. The benchmark may execute the full dictionary pass more than once inside that single timed iteration; percentages divide matching counters by evaluated counters from the same iteration.
| Stemmer | All exact | Changed exact | Root preserved | Note |
| --- | ---: | ---: | ---: | --- |
| Radixor | 98.930% | 98.343% | 100.000% | Radixor baseline in the Snowball-language comparison family. |
| Lucene SnowballFilter | 2.837% | 2.558% | 3.346% | Lucene TokenFilter integration path around the Snowball algorithm. |
| Official Snowball direct | 2.837% | 2.558% | 3.346% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
## Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 1 fork, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| Radixor | `radixor[YIDDISH]` | 0.236 | 0.004 | 85.1 | 1.000 | Radixor baseline for the Snowball-language comparison family. |
| Official Snowball direct | `snowballDirect[YIDDISH]` | 1.432 | 0.193 | 515.7 | 6.058 | Official Snowball generated Java stemmer; direct API. |
| Lucene SnowballFilter | `luceneSnowballFilter[YIDDISH]` | 1.595 | 0.068 | 574.6 | 6.749 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
## Interpretation Notes
- Radixor is a dictionary-derived patch-command stemmer. Its quality depends on the language resource used to train the compiled trie.
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.

23
docs/benchmarks/reference/candidates.md Normal file
View File

@@ -0,0 +1,23 @@
# Benchmark Candidate Evaluation
Implemented benchmark methods are documented on the per-language pages under [Language Benchmark Pages](../languages/index.md). This keeps the exact method names, implementation descriptions, accuracy tables, and speed tables close to the language where they are valid.
## Included Candidate Families
The current benchmark pages include Radixor baselines, Lucene language filters where the language matches a bundled Radixor resource, Lucene Stempel and Morfologik paths where applicable, official Snowball Java stemmers where same-language comparison is available, and selected English-specific non-Lucene baselines such as OpenNLP Porter and Paice/Husk Lancaster.
Direct stemmer APIs and Lucene TokenFilter paths are documented separately on language pages. TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
## Evaluated But Skipped Candidates
| Candidate | Language | Link/source | Reason skipped |
| --- | --- | --- | --- |
| Lucene Arabic, Bulgarian, Bengali, Sorani, Greek, Galician, Hindi, Indonesian, Latvian, Telugu filters | Various | `lucene-analysis-common` | No bundled same-language Radixor resource in this repository snapshot. |
| Lucene analyzer-only paths | Multiple | Lucene analyzers | Full analyzers mix tokenization, stop-word handling, and other behavior; direct filters are used where available. |
| Lucene HunspellStemFilter | Multiple | `lucene-analysis-common` | Requires external Hunspell dictionaries not resolved as benchmark-only resources here. |
| Lucene StemmerOverrideFilter | Multiple | `lucene-analysis-common` | Override map facility, not a stemmer algorithm. |
| Additional Snowball Lovins | English | Official Snowball Java distribution | No Lovins Java stemmer was present in the selected Snowball Java distribution. |
| Lemur Project Krovetz Stemmer | English | Lemur Project | Lucene KStem represents the Krovetz-style path without adding separate dependency and license risk. |
| Smile Lancaster / Paice-Husk | English | Smile NLP | Smile is large for one stemmer; Paice/Husk is included through a smaller benchmark-only generated path. |
| CISTEM German stemmer | German | `https://github.com/LeonieWeissweiler/CISTEM` | Clean benchmark-only Java integration was not completed in this phase. |
| `stemmerEval` reference repository | Multiple | `https://github.com/endredy/stemmerEval` | Used only as a candidate reference; no code or data copied. |

28
docs/benchmarks/reference/corpora.md Normal file
View File

@@ -0,0 +1,28 @@
# Benchmark Corpora
The table below describes the Radixor resources used to build speed and quality corpora. `Total tokens` is the complete dictionary token count used by quality benchmarks. `Already-root tokens` counts fields where the token is already equal to the line root. `Changed tokens` is the speed workload before the minimum-size repeat rule.
| Language resource | Dictionary rows | Total tokens | Already-root tokens | Changed tokens | Speed timing tokens |
| --- | ---: | ---: | ---: | ---: | ---: |
| `cs_cz` | 5,113 | 56,612 | 10,049 | 46,563 | 46,563 |
| `da_dk` | 4,179 | 32,256 | 8,356 | 23,900 | 23,900 |
| `de_de` | 39,315 | 213,440 | 73,799 | 139,641 | 139,641 |
| `es_es` | 65,059 | 926,393 | 120,121 | 806,272 | 806,272 |
| `fa_ir` | 69 | 3,770 | 138 | 3,632 | 5,000 |
| `fi_fi` | 57,027 | 1,865,215 | 110,525 | 1,754,690 | 1,754,690 |
| `fr_fr` | 59,240 | 474,110 | 108,141 | 365,969 | 365,969 |
| `he_il` | 2,358 | 61,071 | 4,715 | 56,356 | 56,356 |
| `hu_hu` | 19,406 | 935,713 | 38,775 | 896,938 | 896,938 |
| `it_it` | 10,009 | 337,546 | 20,004 | 317,542 | 317,542 |
| `nb_no` | 17,929 | 90,757 | 33,376 | 57,381 | 57,381 |
| `nl_nl` | 4,992 | 31,466 | 9,981 | 21,485 | 21,485 |
| `nn_no` | 4,688 | 19,651 | 6,089 | 13,562 | 13,562 |
| `pl_pl` | 9,990 | 132,308 | 19,957 | 112,351 | 112,351 |
| `pt_pt` | 4,001 | 215,490 | 8,002 | 207,488 | 207,488 |
| `ru_ru` | 37,410 | 806,279 | 74,808 | 731,471 | 731,471 |
| `sv_se` | 12,371 | 110,468 | 24,731 | 85,737 | 85,737 |
| `uk_ua` | 1,493 | 15,737 | 2,985 | 12,752 | 12,752 |
| `us_uk` | 396,939 | 1,004,374 | 793,874 | 210,500 | 210,500 |
| `yi` | 802 | 4,300 | 1,524 | 2,776 | 5,000 |
Speed benchmarks process the complete changed-token dictionary sequence for the language. Only resources with fewer than 5,000 changed tokens are repeated to reach the minimum timing size; larger resources are not sampled or truncated.

43
docs/benchmarks/reference/english-coverage.md Normal file
View File

@@ -0,0 +1,43 @@
# English Dictionary Coverage Benchmark
`EnglishRadixorDictionaryCoverageBenchmark` builds Radixor from deterministic slices of the English dictionary rows and evaluates accuracy against the complete dictionary. The speed method then stems the full changed-token English timing corpus.
This benchmark is the clearest demonstration of the Radixor quality/speed envelope after contracted-trie compilation. More dictionary knowledge still gives the strongest changed-form precision, but uniform-subtree contraction removes much of the historical lookup-depth penalty. The table should therefore be read as a measured operating curve rather than as a strictly monotonic function of dictionary size.
| Used rows | Actual row ratio | All exact | Changed exact | Root preserved | Speed ms/op | Error ms | ns/token |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 100% | 100.000% | 97.478% | 97.197% | 97.552% | 23.113 | 7.065 | 109.8 |
| 90% | 90.000% | 97.047% | 94.913% | 97.613% | 21.270 | 9.914 | 101.0 |
| 80% | 80.000% | 96.635% | 92.768% | 97.661% | 19.170 | 6.609 | 91.1 |
| 70% | 70.000% | 96.209% | 90.565% | 97.705% | 20.857 | 6.734 | 99.1 |
| 60% | 60.000% | 95.750% | 88.384% | 97.703% | 14.975 | 1.215 | 71.1 |
| 50% | 50.000% | 95.262% | 86.107% | 97.690% | 15.249 | 1.078 | 72.4 |
| 40% | 40.000% | 94.753% | 83.855% | 97.643% | 15.323 | 2.340 | 72.8 |
| 30% | 30.000% | 94.208% | 81.651% | 97.537% | 16.778 | 2.643 | 79.7 |
| 20% | 20.000% | 93.633% | 79.366% | 97.416% | 18.929 | 3.241 | 89.9 |
| 10% | 10.000% | 92.868% | 76.516% | 97.204% | 19.124 | 1.883 | 90.9 |
## Column Meanings
- `Used rows`: requested deterministic percentage of English dictionary rows used to build the trie.
- `Actual row ratio`: selected rows divided by all parsed English dictionary rows.
- `All exact`: exact agreement over the complete dictionary.
- `Changed exact`: exact agreement over dictionary tokens where `token != expectedRoot`.
- `Root preserved`: percentage of already-root dictionary tokens that are left unchanged.
- `Speed ms/op`: JMH average time for one full changed-token English operation.
- `Error ms`: JMH score error converted to milliseconds.
- `ns/token`: `Speed ms/op` divided by 210,500 changed English tokens.
For non-English languages, the same principle applies: dictionary-driven Radixor quality depends on the amount and consistency of the language resource, while contracted tries reduce the cost of uniform regions in the compiled lookup graph. The English table is the clearest because the English resource is large and the benchmark can show gradual deterministic reductions from 100% to 10%.
## Why The Historical Porter Ratio Changed
The historical English benchmark in `HEAD` used synthetic lexical families. Its `familyCount=5000` parameter generated roughly 70,000 artificial tokens rather than measuring the complete real English dictionary resource. That older workload was useful as a low-level stress test, but it was not a dictionary-quality comparison. Many synthetic tokens were not present in the Radixor dictionary, so Radixor often executed a fast miss path where lookup returned `null` and no patch command was applied.
The current benchmark is intentionally based on real Radixor dictionary data. For English, the speed workload processes 210,500 changed token/root pairs where the dictionary token differs from the expected root, and the quality workload evaluates the complete 1,004,374-token dictionary. This is a hit-heavy workload that measures real lookup plus compiled patch-command application against known expected roots. It is therefore a different and more linguistically meaningful workload than the historical synthetic benchmark.
The result must be interpreted in Radixor's favor through both speed and exact-root quality. Non-Radixor stemmers can look faster because many of them perform narrower or more aggressive transformations and do not attempt to match the dictionary root with the same precision. The English result table shows that this speed often comes with substantially lower `All exact` and `Changed exact` accuracy.
Radixor uses the dictionary as training data for transformation rules. With the full English dictionary, it reaches much higher exact-root agreement than the Porter-family and other narrow baselines. Higher speed is still possible by reducing the amount or complexity of the input dictionary used to build the stemmer, but that is an explicit quality/speed trade-off rather than an accidental benchmark artifact.
The coverage table shows that contracted tries substantially improve the operating point. Reducing dictionary knowledge still primarily damages changed-form exactness, while root preservation remains high. Even when Radixor is trained from only 10% of the English dictionary rows, the complete-dictionary `All exact` score remains above 92%. This is why Radixor performance should be discussed as a configurable quality/speed point, not as a single fixed ratio against Porter.

54
docs/benchmarks/reference/environment.md Normal file
View File

@@ -0,0 +1,54 @@
# Benchmark Environment And Reports
The values below are environment-specific and must not be read as universal performance claims.
| Item | Value |
| --- | --- |
| Benchmark date | 2026-07-03 |
| Focused comparison command family | JMH jar runs limited to `EnglishStemmerComparisonBenchmark`, `MultiLanguageStemmerComparisonBenchmark`, and `SnowballLanguageStemmerComparisonBenchmark`; Radixor exact-root metrics were recomputed deterministically against the same contracted loaders |
| English coverage command | `./gradlew jmh -Pjmh.includes='.*EnglishRadixorDictionaryCoverageBenchmark.*' --no-daemon` |
| Speed result reports | `build/reports/jmh/contracted/english-comparison.csv`, `multilanguage-speed.csv`, `snowball-language-speed.csv` |
| Accuracy result reports | Deterministic Radixor exact-root pass over bundled dictionaries; non-Radixor quality rows retained from the existing published quality suite |
| Final comparison JMH scope | Stemmer comparison benchmarks only; internal `FrequencyTrie*` microbenchmarks were not run |
| Coverage JMH scope | English Radixor dictionary coverage benchmark only |
| JMH version | 1.37 |
| Speed benchmark mode | Average time, `time/op` |
| Score unit | `ns/op` |
| Speed warmup | 3 iterations, 1 s each |
| Speed measurement | 5 iterations, 1 s each |
| Accuracy warmup | none for deterministic exact-root accounting |
| Accuracy measurement | 1 deterministic measurement iteration; counters only, not speed interpretation |
| Fork count in generated report files | 1 |
| Default fork policy for accuracy-only benchmark classes | `@Fork(0)` for future default runs because accuracy counters are deterministic and not interpreted as speed |
| Thread count | 1 |
| JVM reported by JMH | OpenJDK 64-Bit Server VM, 25.0.3+9 |
| JVM invoker | `/usr/lib/jvm/java-25-openjdk/bin/java` |
| Operating system | Linux 7.0.13-200.fc44.x86_64 |
| CPU | AMD Ryzen 5 7600 6-Core Processor |
| Logical CPUs | 12 |
## Contracted Trie Baseline
All Radixor rows in the refreshed benchmark tables use contracted compiled patch tries. During compilation, a subtree whose reachable entries all resolve to the same preferred patch command is represented as an accepting leaf. Runtime lookup can therefore stop as soon as that leaf is reached, which reduces depth in uniform regions while preserving the preferred result used by `get()`.
## Report Files
Generated local report files for this benchmark update:
- `build/reports/jmh/contracted/english-comparison.csv`
- `build/reports/jmh/contracted/english-comparison.txt`
- `build/reports/jmh/contracted/multilanguage-speed.csv`
- `build/reports/jmh/contracted/multilanguage-speed.txt`
- `build/reports/jmh/contracted/snowball-language-speed.csv`
- `build/reports/jmh/contracted/snowball-language-speed.txt`
JMH TXT and CSV reports are still published as benchmark artifacts. They are not converted into a Porter speed badge.
## Published Metrics
The historical English Radixor versus Porter performance badge is no longer generated. `tools/generate-pages-badges.py` now produces only coverage and mutation badge endpoint JSON files:
- `coverage-badge.json`
- `pitest-badge.json`
The README therefore no longer presents a single Porter speed ratio. Benchmark interpretation now uses both speed and quality, because a narrow or aggressive stemmer can be fast while disagreeing with the dictionary root much more often than Radixor.

59
docs/benchmarks/reference/methodology.md Normal file
View File

@@ -0,0 +1,59 @@
# Benchmark Methodology
The stemmer comparison suite measures Radixor and Java stemmers on the same language and deterministic Radixor dictionary-derived data. Published Radixor rows in this refresh use contracted compiled patch tries, where uniform preferred-command subtrees are collapsed into accepting leaves before the trie is frozen for lookup. For each language, the bundled dictionary resource stores the expected root as the first tab-separated field on a line and its surface forms on the same line. Every single-token field on that line can therefore be paired with the same expected root.
Published stemmer comparison results must come only from benchmark classes matching `.*StemmerComparisonBenchmark.*`. Internal `FrequencyTrie*` microbenchmarks are not part of those results.
## Benchmark Passes
There are two distinct benchmark passes:
- Speed benchmarks process only changed dictionary pairs where `token != expectedRoot`. This removes already-root tokens from timing so a stemmer is measured on words that actually require a transformation. If a language has fewer than 5,000 changed pairs, the complete changed-pair sequence is repeated in stable order until the timing corpus has at least 5,000 tokens. Larger changed-pair corpora are not sampled or truncated.
- Quality benchmarks process the complete dictionary for the language. They report exact agreement over all tokens, exact agreement over changed tokens only, and preservation of tokens that are already roots.
Timing corpora are generated once per JMH JVM and kept in memory as shared `{token, expectedRoot}` arrays. Corpus construction, dictionary loading, trie loading, table loading, and analyzer construction are setup work and are not included in measured benchmark methods.
Performance is interpreted as average time per input token:
```text
timePerChangedTokenNs = JMH score ns/op / changedTimingTokenCount
```
This is necessary because Radixor dictionaries have different token counts by language.
## Quality And Search Interpretation
Radixor speed must be interpreted together with exact-root quality. A slower Radixor row must not be read as a simple performance weakness when Radixor is also the row with accuracy close to 100% and competing stemmers are much lower.
Many fast light, minimal, possessive, or aggressive rule-based stemmers are fast because they do much less linguistic work. The measured Radixor cost buys dictionary-trained precision, and that precision is what improves search quality when queries and indexed text are reduced to the same intended roots.
The [English dictionary coverage benchmark](english-coverage.md) shows this operating curve explicitly: contracted tries reduce lookup cost in uniform regions, while reduced dictionary coverage still lowers changed-form precision.
## Normalization Policy
Radixor is measured over dictionary tokens from its own resources: lower-case with `Locale.ROOT`, diacritics preserved. The corpus is normalized during setup, so the Radixor benchmark path uses `FrequencyTrie.getNormalized(CharSequence)` and does not measure redundant lookup-time lowercasing or diacritic normalization.
Lucene TokenFilter paths include required normalization in the measured pipeline. Examples include lower-case normalization for filters requiring lower-case input, German normalization before German light/minimal stemming, and Persian decimal, Arabic, and Persian normalization before Persian stemming. No ASCII folding is applied to Czech or Polish paths, because those Lucene stemmers are diacritic-aware or dictionary/table-backed for those languages. TokenFilter throughput methods materialize each emitted `CharTermAttribute` as a `String` before passing it to the JMH `Blackhole`, so output consumption is easier to inspect and closer to the direct stemmer methods.
For right-to-left Radixor languages, patch application uses the traversal direction stored in trie metadata. This is required because static backward patch application is not correct for all bundled languages.
## Quality Metric
The quality pass reports exact-root agreement against the expected root from the Radixor dictionary line. It writes to the normal JMH report files:
- `build/reports/jmh/jmh-results.csv`
- `build/reports/jmh/jmh-results.txt`
Accuracy is computed from standard JMH secondary rows:
```text
allExactPercent = correctMatches / evaluatedTokens * 100
changedExactPercent = changedCorrectMatches / changedEvaluatedTokens * 100
rootPreservedPercent = rootPreservedMatches / rootEvaluatedTokens * 100
```
`allExactPercent` uses the complete dictionary. `changedExactPercent` uses only tokens where `token != expectedRoot`. `rootPreservedPercent` measures whether a stemmer leaves already-root dictionary entries unchanged.
Morfologik can emit multiple terms for one input token. The quality benchmark uses the first emitted term for exact-root accounting when no ranking weight is exposed. Throughput benchmarks for Morfologik TokenFilter paths consume all emitted terms.
Quality reports intentionally use one deterministic measurement iteration without warmup, because exact-root agreement is not a timing metric and repeated precision passes would only duplicate the same counters.

79
docs/built-in-languages.md
View File

@@ -10,7 +10,7 @@ Bundled dictionaries are exposed through:
org.egothor.stemmer.StemmerPatchTrieLoader.Language
```
Each bundled dictionary is packaged with the library as a compressed UTF-8 text resource. When loaded, the resource is parsed by `StemmerDictionaryParser`, transformed into patch-command mappings, and compiled into a read-only `FrequencyTrie<String>` by `StemmerPatchTrieLoader`.
Each bundled dictionary is packaged with the library as a compressed UTF-8 text resource. When loaded through the runtime API, the resource is parsed by `StemmerDictionaryParser`, transformed into patch-command mappings, and compiled into a read-only `FrequencyTrie<CompiledPatchCommand>` by `StemmerPatchTrieLoader`.
The bundled language definition also carries a language-level right-to-left flag. That flag is used by the loader to derive the `WordTraversalDirection` used for both trie-key construction and patch-command generation. In practice, left-to-right bundled languages use historical backward Egothor traversal, while right-to-left bundled languages use forward traversal over the stored form.
@@ -18,28 +18,28 @@ The bundled language definition also carries a language-level right-to-left flag
The following bundled language identifiers are currently available:
| Language | Enum constant | Writing direction | Notes |
|---|---|---:|---|
| Czech | `CS_CZ` | LTR | Bundled general-purpose dictionary |
| Danish | `DA_DK` | LTR | Bundled general-purpose dictionary |
| German | `DE_DE` | LTR | Bundled general-purpose dictionary |
| Spanish | `ES_ES` | LTR | Bundled general-purpose dictionary |
| Persian | `FA_IR` | RTL | Bundled dictionary uses forward traversal over the stored form |
| Finnish | `FI_FI` | LTR | Bundled general-purpose dictionary |
| French | `FR_FR` | LTR | Bundled general-purpose dictionary |
| Hebrew | `HE_IL` | RTL | Bundled dictionary uses forward traversal over the stored form |
| Hungarian | `HU_HU` | LTR | Bundled general-purpose dictionary |
| Italian | `IT_IT` | LTR | Bundled general-purpose dictionary |
| Norwegian Bokmål | `NB_NO` | LTR | Bundled general-purpose dictionary |
| Dutch | `NL_NL` | LTR | Bundled general-purpose dictionary |
| Norwegian Nynorsk | `NN_NO` | LTR | Bundled general-purpose dictionary |
| Polish | `PL_PL` | LTR | Bundled general-purpose dictionary |
| Portuguese | `PT_PT` | LTR | Bundled general-purpose dictionary |
| Russian | `RU_RU` | LTR | Bundled general-purpose dictionary |
| Swedish | `SV_SE` | LTR | Bundled general-purpose dictionary |
| Ukrainian | `UK_UA` | LTR | Bundled general-purpose dictionary |
| English | `US_UK` | LTR | Bundled general-purpose dictionary |
| Yiddish | `YI` | RTL | Bundled dictionary uses forward traversal over the stored form |
| Language | Enum constant | Writing direction | Notes | Benchmark page |
|---|---|---:|---|---|
| Czech | `CS_CZ` | LTR | Bundled general-purpose dictionary | [Czech](benchmarks/languages/czech.md) |
| Danish | `DA_DK` | LTR | Bundled general-purpose dictionary | [Danish](benchmarks/languages/danish.md) |
| German | `DE_DE` | LTR | Bundled general-purpose dictionary | [German](benchmarks/languages/german.md) |
| Spanish | `ES_ES` | LTR | Bundled general-purpose dictionary | [Spanish](benchmarks/languages/spanish.md) |
| Persian | `FA_IR` | RTL | Bundled dictionary uses forward traversal over the stored form | [Persian](benchmarks/languages/persian.md) |
| Finnish | `FI_FI` | LTR | Bundled general-purpose dictionary | [Finnish](benchmarks/languages/finnish.md) |
| French | `FR_FR` | LTR | Bundled general-purpose dictionary | [French](benchmarks/languages/french.md) |
| Hebrew | `HE_IL` | RTL | Bundled dictionary uses forward traversal over the stored form | No same-language external benchmark in this run |
| Hungarian | `HU_HU` | LTR | Bundled general-purpose dictionary | [Hungarian](benchmarks/languages/hungarian.md) |
| Italian | `IT_IT` | LTR | Bundled general-purpose dictionary | [Italian](benchmarks/languages/italian.md) |
| Norwegian Bokmål | `NB_NO` | LTR | Bundled general-purpose dictionary | [Norwegian Bokmal](benchmarks/languages/norwegian-bokmal.md) |
| Dutch | `NL_NL` | LTR | Bundled general-purpose dictionary | [Dutch](benchmarks/languages/dutch.md) |
| Norwegian Nynorsk | `NN_NO` | LTR | Bundled general-purpose dictionary | [Norwegian Nynorsk](benchmarks/languages/norwegian-nynorsk.md) |
| Polish | `PL_PL` | LTR | Bundled general-purpose dictionary | [Polish](benchmarks/languages/polish.md) |
| Portuguese | `PT_PT` | LTR | Bundled general-purpose dictionary | [Portuguese](benchmarks/languages/portuguese.md) |
| Russian | `RU_RU` | LTR | Bundled general-purpose dictionary | [Russian](benchmarks/languages/russian.md) |
| Swedish | `SV_SE` | LTR | Bundled general-purpose dictionary | [Swedish](benchmarks/languages/swedish.md) |
| Ukrainian | `UK_UA` | LTR | Bundled general-purpose dictionary | [Ukrainian](benchmarks/languages/ukrainian.md) |
| English | `US_UK` | LTR | Bundled general-purpose dictionary | [English](benchmarks/languages/english.md) |
| Yiddish | `YI` | RTL | Bundled dictionary uses forward traversal over the stored form | [Yiddish](benchmarks/languages/yiddish.md) |
## Basic usage
@@ -48,6 +48,7 @@ Load a bundled dictionary like this:
```java
import java.io.IOException;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -59,7 +60,7 @@ public final class BuiltInExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
StemmerPatchTrieLoader.Language.US_UK,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
@@ -76,8 +77,8 @@ This call loads the bundled dictionary resource for the selected language, parse
```java
import java.io.IOException;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -88,21 +89,21 @@ public final class EnglishExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
StemmerPatchTrieLoader.Language.US_UK,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
final String word = "running";
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch, trie.traversalDirection());
final CompiledPatchCommand patch = trie.get(word);
final String stem = patch == null ? word : patch.apply(word);
System.out.println(word + " -> " + stem);
}
}
```
Passing `trie.traversalDirection()` to `PatchCommandEncoder.apply(...)` is the correct general contract. It ensures that the patch is applied using the same logical traversal model that was used when the trie and its patch commands were produced.
`CompiledPatchCommand` values are compiled with the traversal direction used when the trie and its patch commands were produced.
## Traversal behavior and right-to-left languages
@@ -114,7 +115,7 @@ For bundled right-to-left languages such as Persian, Hebrew, and Yiddish, Radixo
- trie keys are traversed from the logical beginning of the stored form,
- patch commands are generated in that same forward direction,
- patch application must use `WordTraversalDirection.FORWARD`, which is naturally obtained from `trie.traversalDirection()`.
- compiled patch-command application uses `WordTraversalDirection.FORWARD`, which is naturally captured when `loadCompiled(...)` creates `CompiledPatchCommand` values.
This design keeps the traversal policy explicit and consistent across dictionary loading, trie lookup, binary persistence, builder reconstruction, and patch application.
@@ -124,11 +125,16 @@ Bundled dictionaries can be compiled using any supported `ReductionMode`. The re
Typical entry points are:
- `StemmerPatchTrieLoader.load(language, storeOriginal, reductionMode)`
- `StemmerPatchTrieLoader.load(language, storeOriginal, reductionSettings)`
- `StemmerPatchTrieLoader.loadCompiled(language, storeOriginal, reductionMode)`
- `StemmerPatchTrieLoader.loadCompiled(language, storeOriginal, reductionSettings)`
For most users, `ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS` is the most conservative general-purpose choice because it preserves ranked `getAll(...)` behavior.
Compiled bundled dictionaries also use internal uniform-subtree contraction. If a whole subtree
would return the same preferred patch command, Radixor stores that subtree as an accepting leaf and
removes the deeper branches. This is the contracted trie representation used by the published
benchmark tables and is independent of the public reduction mode selected by the caller.
## Intended role of bundled dictionaries
Bundled dictionaries should be understood as practical default resources.
@@ -178,7 +184,7 @@ For production systems, the most robust workflow is usually:
2. extend it with domain-specific forms if needed,
3. rebuild it into a binary artifact,
4. deploy that compiled binary artifact,
5. load it at runtime through `loadBinary(...)`.
5. load it at runtime through `loadBinaryCompiled(...)`.
This avoids repeated startup parsing and makes the deployed stemming behavior explicit, reproducible, and versionable.
@@ -190,6 +196,7 @@ import java.nio.file.Path;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.FrequencyTrieBuilders;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;
@@ -213,7 +220,11 @@ public final class BundledRefinementExample {
ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS));
builder.put("microservices", "Na");
final PatchCommandEncoder encoder = PatchCommandEncoder.builder()
.traversalDirection(base.traversalDirection())
.build();
builder.put("microservices", encoder.encode("microservices", "microservice"));
final FrequencyTrie<String> compiled = builder.build();

14
docs/cli-compilation.md
View File

@@ -171,6 +171,11 @@ The short form `-h` is also supported.
Reduction mode is not only a storage decision. It also influences what semantics are preserved when the mutable trie is compiled into its canonical read-only form.
Before the selected public reduction mode is applied, compilation performs uniform-subtree
contraction. If all reachable entries below a subtree select the same preferred patch command, the
compiler stores that subtree as an accepting leaf and removes the deeper branches. This reduces
runtime lookup depth without changing the preferred result returned by the standard stemming path.
### Ranked `getAll()` equivalence
`MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS` merges subtrees whose `getAll()` results remain equivalent for every reachable key suffix and whose local result ordering is the same.
@@ -203,9 +208,9 @@ The CLI is best used as a preparation step during packaging, deployment, or cont
### Treat compiled files as versioned assets
A `.radixor.gz` file should be handled as a versioned output artifact. It represents a specific dictionary state, a specific reduction mode, and, where relevant, specific dominant-result thresholds.
A `.radixor.gz` file should be handled as a versioned output artifact. It represents a specific dictionary state, a specific reduction mode, whether uniform-subtree contraction was used, and, where relevant, specific dominant-result thresholds.
Compiled tries also persist a human-readable metadata block (`key=value` lines) that includes format version, traversal direction, RTL indicator, reduction mode, dominant thresholds, diacritic-processing mode, and case-processing mode. After decompression, you can inspect this block directly to identify what dictionary/trie configuration the artifact contains. The current CLI uses `DiacriticProcessingMode.AS_IS`; custom diacritic stripping is available through the programmatic builder and loader APIs rather than through a CLI flag.
Compiled tries also persist a human-readable metadata block (`key=value` lines) that includes format version, traversal direction, RTL indicator, reduction mode, contraction flag, dominant thresholds, diacritic-processing mode, and case-processing mode. After decompression, you can inspect this block directly to identify what dictionary/trie configuration the artifact contains. The current CLI uses `DiacriticProcessingMode.AS_IS`; custom diacritic stripping is available through the programmatic builder and loader APIs rather than through a CLI flag.
### Choose reduction mode deliberately
@@ -237,11 +242,12 @@ java org.egothor.stemmer.Compile \
### 3. Load it in an application
```java
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
final FrequencyTrie<String> trie =
StemmerPatchTrieLoader.loadBinary("english.radixor.gz");
final FrequencyTrie<CompiledPatchCommand> trie =
StemmerPatchTrieLoader.loadBinaryCompiled("english.radixor.gz");
```
## Exit codes and error handling

11
docs/compatibility-and-guarantees.md
View File

@@ -81,7 +81,11 @@ Compiled `FrequencyTrie` instances are immutable and thread-safe for concurrent
### Stable patch application behavior
`PatchCommandEncoder.apply(...)` remains the compatibility API for string-returning patch application. Buffer-oriented `applyTo(...)` overloads are additive APIs for caller-owned output storage. They do not retain output arrays, report insufficient capacity with `APPLY_INSUFFICIENT_CAPACITY`, and preserve the existing malformed-patch compatibility behavior where `apply(...)` preserves the source.
Serialized patch-command strings remain the stable stored representation used by textual dictionaries and binary artifacts. Runtime stemming should use `CompiledPatchCommand` values produced by `StemmerPatchTrieLoader.loadCompiled(...)`, `StemmerPatchTrieLoader.loadBinaryCompiled(...)`, or `PatchCommandEncoder.compile(...)`.
The historical `PatchCommandEncoder.apply(...)` and String-based `applyTo(...)` overloads remain compatibility APIs during the 2.x transition, but they are deprecated because they reparse the patch-command string on each application. See [Migration and Backward Compatibility](migration-and-backward-compatibility.md) for old and new code examples.
Compiled buffer-oriented `CompiledPatchCommand.applyTo(...)` overloads use caller-owned output storage. They do not retain output arrays and report insufficient capacity with `CompiledPatchCommand.APPLY_INSUFFICIENT_CAPACITY`.
### Stable reduction-mode intent
@@ -89,6 +93,11 @@ Each public `ReductionMode` constant carries a semantic contract that should rem
In other words, the implementation may evolve, but the intended meaning of modes such as ranked `getAll()` equivalence, unordered `getAll()` equivalence, and dominant `get()` equivalence should not drift casually.
Internal pre-reduction optimizations may still change the physical compiled trie shape when they
preserve the documented lookup contract. Uniform-subtree contraction is one such optimization: it
can replace a subtree with an accepting leaf when all reachable entries choose the same preferred
patch command.
### Stable binary artifact purpose
Compiled `.radixor.gz` artifacts are a first-class project output. Loading and persisting compiled stemmer artifacts is part of the intended usage model, not an incidental implementation side effect.

130
docs/fast-track.md Normal file
View File

@@ -0,0 +1,130 @@
# Fast Track
This page is the shortest path from an empty Java project to a working Radixor stemmer.
It deliberately uses a bundled dictionary and the preferred compiled-command runtime API, so the
first result does not require writing a dictionary, running the CLI compiler, or understanding
reduction internals.
Use this page when the goal is:
- add the dependency,
- load a bundled language resource,
- stem a token,
- know where to go next.
For deeper production guidance, see [Integration Deep Dive](integration-deep-dive.md).
## 1. Add The Dependency
Radixor is published as:
```text
groupId: org.egothor
artifactId: radixor
```
Use the current published version from Maven Central. The snippets below use `3.0.0`; replace it
with the version you deploy if a newer release is available.
For a Gradle project:
```kotlin
dependencies {
implementation("org.egothor:radixor:3.0.0")
}
```
For a Maven project:
```xml
<dependency>
<groupId>org.egothor</groupId>
<artifactId>radixor</artifactId>
<version>3.0.0</version>
</dependency>
```
Radixor targets modern Java and has a dependency-light runtime core. The project documentation and
benchmarks assume a current JDK; Java 21 or newer is the practical baseline for current releases.
## 2. Load A Bundled Dictionary
The fastest path is to use a bundled dictionary through `StemmerPatchTrieLoader.Language`.
This example uses the bundled English resource, `US_UK`.
```java
import java.io.IOException;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class RadixorFirstStem {
private RadixorFirstStem() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<CompiledPatchCommand> stemmer = StemmerPatchTrieLoader.loadCompiled(
StemmerPatchTrieLoader.Language.US_UK,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
final String token = "running";
final CompiledPatchCommand command = stemmer.get(token);
final String stem = command == null ? token : command.apply(token);
System.out.println(token + " -> " + stem);
}
}
```
The loaded `FrequencyTrie<CompiledPatchCommand>` is immutable and can be shared across request
threads. Load it once during application startup and reuse it for indexing and query processing.
## 3. Choose A Language Resource
Bundled dictionaries are exposed as enum constants. Common examples:
| Language | Enum constant |
| --- | --- |
| English | `US_UK` |
| German | `DE_DE` |
| French | `FR_FR` |
| Spanish | `ES_ES` |
| Italian | `IT_IT` |
| Polish | `PL_PL` |
| Russian | `RU_RU` |
| Czech | `CS_CZ` |
The full list, writing-direction notes, and benchmark links are in
[Built-in Languages](built-in-languages.md).
## 4. Use The Same Stemmer On Both Sides
For search, use the same Radixor configuration during indexing and query processing. A typical
minimal integration flow is:
1. tokenize text with your application or search platform,
2. normalize tokens consistently,
3. call `stemmer.get(token)`,
4. apply the returned `CompiledPatchCommand`,
5. index or query with the resulting stem.
Do not load the trie per token. The compiled trie is the runtime artifact; per-token work should be
limited to lookup and patch application.
## 5. Next Step For Production
The fast path compiles a bundled dictionary during startup. That is convenient for evaluation and
small services. For larger deployments, compile once, persist a `.radixor.gz` artifact, and load
that binary artifact at runtime.
Continue with:
- [Integration Deep Dive](integration-deep-dive.md) for production lifecycle guidance.
- [Loading and Building Stemmers](programmatic-loading-and-building.md) for all loading APIs.
- [Built-in Languages](built-in-languages.md) for bundled resources and dictionary locations.
- [Benchmarking](benchmarking.md) for speed and quality interpretation.

8
docs/index.md
View File

@@ -30,8 +30,10 @@ Radixor is intended for teams that require consistent stemming quality at scale,
## Start here
- Read [Quick Start](quick-start.md) for immediate implementation guidance.
- Use [Programmatic Usage](programmatic-usage.md) for application integration patterns.
- Review [Benchmarking](benchmarking.md) for reproducible performance methodology.
- Read [Fast Track](fast-track.md) when you want the shortest path to a working bundled stemmer.
- Use [Integration Deep Dive](integration-deep-dive.md) when you are wiring Radixor into a real application or search pipeline.
- Read [Quick Start](quick-start.md) for the broader developer walkthrough after the first result works.
- Use [Built-in Languages](built-in-languages.md) to find the bundled dictionaries exposed by Radixor.
- Review [Benchmarking](benchmarking.md) and [Benchmark Results](benchmarks/index.md) for reproducible performance and quality methodology.
- Open [CI Reports](reports.md) to inspect published build artifacts and quality metrics.
- See the historical paper: [*Lemmatizer for Document Information Retrieval Systems in JAVA*](https://www.researchgate.net/publication/221512865_Lemmatizer_for_Document_Information_Retrieval_Systems_in_JAVA).

235
docs/integration-deep-dive.md Normal file
View File

@@ -0,0 +1,235 @@
# Integration Deep Dive
This page explains how to integrate Radixor into a real Java application after the first
fast-track experiment works. It covers dependencies, bundled dictionaries, runtime lifecycle,
deployment artifacts, and the decisions that matter in search or text-processing systems.
## Integration Model
Radixor has two separate phases:
| Phase | Work | Typical location |
| --- | --- | --- |
| Preparation | Parse dictionaries, derive patch commands, reduce and contract the trie, optionally persist a binary artifact. | Build pipeline, packaging job, admin tool, or startup for small services. |
| Runtime | Load an immutable compiled trie, look up patch commands, apply them to tokens. | Search indexing, query processing, text normalization, enrichment pipelines. |
The practical rule is simple: compile rarely, stem often.
For production systems, prefer a startup-owned or dependency-injected singleton
`FrequencyTrie<CompiledPatchCommand>` per language/configuration. The trie is immutable after
construction and is suitable for concurrent reads.
## Dependency Coordinates
The Maven coordinates are:
```text
org.egothor:radixor
```
Gradle:
```kotlin
dependencies {
implementation("org.egothor:radixor:3.0.0")
}
```
Maven:
```xml
<dependency>
<groupId>org.egothor</groupId>
<artifactId>radixor</artifactId>
<version>3.0.0</version>
</dependency>
```
Replace `3.0.0` with the current release selected for your deployment.
The core Java module is:
```java
module org.egothor.radixor;
```
A named consuming module declares:
```java
module example.search {
requires org.egothor.radixor;
}
```
## Bundled Dictionaries
Radixor ships bundled dictionaries inside the library artifact. The public API exposes them through:
```java
StemmerPatchTrieLoader.Language
```
The physical resources are packaged as compressed UTF-8 dictionaries under resource directories
such as:
```text
us_uk/stemmer.gz
de_de/stemmer.gz
fr_fr/stemmer.gz
pl_pl/stemmer.gz
```
Treat those resource paths as implementation details. Application code should load bundled
dictionaries through `StemmerPatchTrieLoader.Language`, because the enum also carries the language
metadata needed for correct traversal.
See [Built-in Languages](built-in-languages.md) for the complete language list, writing-direction
notes, and links to per-language benchmark pages.
## Minimal Service Wrapper
A small service wrapper keeps loading, null handling, and fallback behavior in one place.
```java
import java.io.IOException;
import java.util.Objects;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class RadixorStemmerService {
private final FrequencyTrie<CompiledPatchCommand> trie;
public RadixorStemmerService(final StemmerPatchTrieLoader.Language language) throws IOException {
this.trie = StemmerPatchTrieLoader.loadCompiled(
Objects.requireNonNull(language, "language"),
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
}
public String stem(final String token) {
final String checkedToken = Objects.requireNonNull(token, "token");
final CompiledPatchCommand command = trie.get(checkedToken);
return command == null ? checkedToken : command.apply(checkedToken);
}
}
```
The fallback behavior preserves the original token when the trie has no patch command for it. That
is usually the right default for search normalization, because unknown tokens should remain
searchable.
## Production Artifact Workflow
For a controlled deployment, compile once and deploy the binary artifact:
1. choose a bundled or custom dictionary,
2. optionally extend it with domain vocabulary,
3. compile a contracted trie,
4. persist it as `.radixor.gz`,
5. deploy that artifact with the application,
6. load it with `StemmerPatchTrieLoader.loadBinaryCompiled(...)`.
Runtime loading then avoids dictionary parsing and preparation-time memory pressure.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class BinaryStemmerLoader {
private BinaryStemmerLoader() {
throw new AssertionError("No instances.");
}
public static FrequencyTrie<CompiledPatchCommand> loadEnglish() throws IOException {
return StemmerPatchTrieLoader.loadBinaryCompiled(Path.of("stemmers", "english.radixor.gz"));
}
}
```
Use [CLI Compilation](cli-compilation.md) for command-line artifact creation, or
[Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md) for
programmatic artifact generation.
## Search Pipeline Guidance
Use Radixor consistently across indexing and querying:
- choose one language dictionary per field or per analysis chain,
- apply the same token normalization before stemming on both sides,
- keep the compiled trie in memory and reuse it,
- use `get(...)` for a single preferred stem,
- use `getAll(...)` when a retrieval model benefits from preserving alternatives,
- version custom `.radixor.gz` artifacts with the application or index schema.
For multilingual content, do not run every token through every language. Route text by field,
document metadata, or language detection before stemming.
## Choosing Bundled Versus Custom Dictionaries
Start with bundled dictionaries when:
- the language is supported,
- the application needs a strong baseline quickly,
- the vocabulary is general-purpose,
- the team is evaluating Radixor or building an initial integration.
Use custom or extended dictionaries when:
- domain vocabulary changes search quality,
- product names, technical terms, legal terms, or biomedical terms must be preserved consistently,
- stemming behavior must be curated and reviewed,
- a release process needs a versioned lexical artifact.
The dictionary format is intentionally simple and documented in
[Dictionary Format](dictionary-format.md). Contribution standards are described in
[Contributing Dictionaries](contributing-dictionaries.md).
## Performance Practices
The hot path should be only:
```text
token -> trie lookup -> compiled command application -> stem
```
Avoid these patterns in production request paths:
- loading or compiling dictionaries per request,
- applying serialized patch strings repeatedly instead of `CompiledPatchCommand`,
- rebuilding tries for short-lived batches,
- mixing different stemmer configurations between indexing and querying,
- interpreting speed without checking exact-root quality.
The current benchmark documentation separates methodology, corpora, environment, and language
results so performance claims remain auditable. Start with [Benchmarking](benchmarking.md), then
use [Benchmark Results](benchmarks/index.md) for the detailed reference tree.
## Operational Checklist
Before production rollout:
- dependency version is pinned,
- language resource and reduction mode are documented,
- indexing and query pipelines use the same stemming configuration,
- custom artifacts are versioned and reproducible,
- fallback behavior for unknown tokens is explicit,
- benchmark expectations are read together with quality metrics,
- CI includes at least a smoke test that stems representative project vocabulary.
## Related Pages
- [Fast Track](fast-track.md)
- [Quick Start](quick-start.md)
- [Built-in Languages](built-in-languages.md)
- [Programmatic Usage](programmatic-usage.md)
- [CLI Compilation](cli-compilation.md)
- [Benchmarking](benchmarking.md)

15
docs/lookup-edge-optimization.md
View File

@@ -6,7 +6,10 @@ Compiled trie nodes (`CompiledNode`) use three lookup strategies when resolving
2. linear scan for very small child counts,
3. binary search over sorted edge labels.
This page explains the dense path, what `maxExpandedIndex` controls, and how to tune it.
This page explains the dense path, what `maxExpandedIndex` controls, and how to tune it. These
edge lookup strategies operate after trie reduction and uniform-subtree contraction. If lookup
reaches an accepting contracted leaf, no child edge search is needed for the remaining input
characters.
## Runtime model of one node
@@ -75,7 +78,7 @@ The value never changes lookup semantics. It only changes the in-memory structur
This threshold is **not** stored in `TrieMetadata`.
- The binary format stores only trie payload and semantic metadata (`reduction`, `traversal`,
case/diacritic settings, and stream version).
case/diacritic settings, contraction settings, and stream version).
- `maxExpandedIndex` is chosen when materializing nodes in memory.
- You can therefore keep one persisted artifact and load it with different in-memory
trade-offs depending on deployment constraints.
@@ -135,15 +138,17 @@ At artifact load time, you can tune the same trade-off independently of persiste
```java
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
var defaultLookup = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> defaultLookup = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"));
var fastLookup = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> fastLookup = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"), 1024);
var compactLookup = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> compactLookup = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"), 0);
```

155
docs/migration-and-backward-compatibility.md Normal file
View File

@@ -0,0 +1,155 @@
# Migration and Backward Compatibility
This page describes the migration from repeated serialized patch-command application to compiled patch commands.
## Summary
Radixor patch commands are still encoded as compact strings when dictionaries are built and persisted. That serialized form remains the interchange format used by textual dictionaries, binary artifacts, and compilation tooling.
Runtime stemming should no longer repeatedly apply those serialized strings directly. Since 2.3.0, the String-based patch application API is deprecated. Code that stems live input should load or create `CompiledPatchCommand` values and reuse them. The deprecated API remains available for compatibility during the transition, but applications should migrate before 3.0.0.
The reason is performance. The old API parses the serialized P-command every time it is applied. `CompiledPatchCommand` parses it once and stores a concrete immutable command object, so repeated stemming avoids the same analysis work.
## Deprecated Runtime APIs
The following API family is kept for source compatibility but is no longer the preferred runtime path:
- `PatchCommandEncoder.apply(String, String)`
- `PatchCommandEncoder.apply(String, String, WordTraversalDirection)`
- `PatchCommandEncoder.applyTo(..., String, WordTraversalDirection, ...)`
- `PatchCommandEncoder.applyWithConfiguredDirection(String, String)`
- `StemmerPatchTrieLoader.load(...)` overloads returning `FrequencyTrie<String>`
- `StemmerPatchTrieLoader.loadBinary(...)` overloads returning `FrequencyTrie<String>`
Use the compiled equivalents for runtime stemming:
- `CompiledPatchCommand.compile(String, WordTraversalDirection)`
- `PatchCommandEncoder.compile(String)`
- `PatchCommandEncoder.compile(String, WordTraversalDirection)`
- `StemmerPatchTrieLoader.loadCompiled(...)`
- `StemmerPatchTrieLoader.loadBinaryCompiled(...)`
## Loading A Text Dictionary
Old runtime code:
```java
Path dictionary = Path.of("dictionary.txt");
ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(dictionary, true, settings);
String word = "running";
String patch = trie.get(word);
String stem = patch == null
? word
: PatchCommandEncoder.apply(word, patch, trie.traversalDirection());
```
New runtime code:
```java
Path dictionary = Path.of("dictionary.txt");
ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(dictionary, true, settings);
String word = "running";
CompiledPatchCommand patch = trie.get(word);
String stem = patch == null ? word : patch.apply(word);
```
## Loading A Binary Artifact
Old runtime code:
```java
FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary(Path.of("us-uk.radixor.gz"));
String word = "studies";
String patch = trie.get(word);
String stem = patch == null
? word
: PatchCommandEncoder.apply(word, patch, trie.traversalDirection());
```
New runtime code:
```java
FrequencyTrie<CompiledPatchCommand> trie =
StemmerPatchTrieLoader.loadBinaryCompiled(Path.of("us-uk.radixor.gz"));
String word = "studies";
CompiledPatchCommand patch = trie.get(word);
String stem = patch == null ? word : patch.apply(word);
```
Existing binary artifacts remain readable. `loadBinaryCompiled(...)` reads the stored serialized patch strings and compiles them during load setup, before live stemming begins.
## Manual Patch Encoding
Encoding still produces a serialized patch command because that is the compact stored representation:
```java
PatchCommandEncoder encoder = PatchCommandEncoder.builder().build();
String patch = encoder.encode("running", "run");
```
Old repeated application:
```java
String stem = PatchCommandEncoder.apply("running", patch);
```
New repeated application:
```java
CompiledPatchCommand compiled = encoder.compile(patch);
String stem = compiled.apply("running");
```
## Caller-Owned Output Buffers
Old buffer-oriented code:
```java
char[] output = new char[32];
int length = PatchCommandEncoder.applyTo(
"running",
patch,
WordTraversalDirection.BACKWARD,
output,
0,
output.length);
```
New buffer-oriented code:
```java
CompiledPatchCommand compiled = CompiledPatchCommand.compile(patch, WordTraversalDirection.BACKWARD);
char[] output = new char[32];
int length = compiled.applyTo("running", output, 0, output.length);
```
Both APIs return `CompiledPatchCommand.APPLY_INSUFFICIENT_CAPACITY` when the caller-owned output range is too small.
## Compatibility Rules
Serialized patch strings remain part of the dictionary and artifact format. The deprecation is about repeated runtime application of serialized strings, not about the stored representation itself.
Compatibility tests may continue to exercise the deprecated API to prove that old artifacts and source code still work during the transition. New production code, examples, and benchmark runtime paths should use `CompiledPatchCommand`.
The command-line compiler still writes artifacts containing serialized patch commands. Runtime loaders can expose those commands as compiled immutable objects through `loadCompiled(...)` and `loadBinaryCompiled(...)`.
## Contracted Trie Artifacts
Current compiled loaders and freshly written binary artifacts can use contracted compiled tries.
Contraction replaces a subtree with an accepting leaf when every reachable entry below that subtree
selects the same preferred patch command. This changes the physical trie shape and the binary
stream version, but it does not change the serialized patch-command language.
Existing binary artifacts remain readable through the compatibility reader. To obtain the
contracted runtime representation, rebuild the artifact with the current compiler or load the
source dictionary through the current `loadCompiled(...)` APIs. Applications that only consume
`CompiledPatchCommand` values through `get()` and `apply(...)` do not need code changes for this
optimization.

3
docs/programmatic-extending-and-persistence.md
View File

@@ -94,7 +94,8 @@ This model works especially well when domain-specific extensions are added in la
After loading a compiled artifact, applications can inspect the persisted build descriptor directly:
```java
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary("build/stemmers/cs_cz.dat.gz");
final FrequencyTrie<CompiledPatchCommand> trie =
StemmerPatchTrieLoader.loadBinaryCompiled("build/stemmers/cs_cz.dat.gz");
final TrieMetadata metadata = trie.metadata();
System.out.println(metadata.formatVersion());

30
docs/programmatic-loading-and-building.md
View File

@@ -4,11 +4,12 @@ This document explains how to acquire a compiled Radixor stemmer in Java.
## Load a bundled language dictionary
Bundled language resources are simple to use and compile directly into a `FrequencyTrie<String>` during loading.
Bundled language resources are simple to use and compile directly into a `FrequencyTrie<CompiledPatchCommand>` during loading.
```java
import java.io.IOException;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -20,7 +21,7 @@ public final class BundledLanguageExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
StemmerPatchTrieLoader.Language.US_UK,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
@@ -30,6 +31,11 @@ public final class BundledLanguageExample {
The `storeOriginal` flag controls whether the canonical stem is inserted as a no-op patch entry for the stem itself.
Bundled `loadCompiled(...)` entry points build the runtime trie with the same contracted
representation used by the published benchmarks. During compilation, uniform preferred-command
subtrees are collapsed into accepting leaves, so lookup can stop before consuming the entire input
when the remaining characters cannot change the selected patch command.
## Load a textual dictionary
Loading from a dictionary file follows the same preparation model as bundled resources, but the source comes from your own file or path. The input may be plain UTF-8 text or GZip-compressed UTF-8 text; the loader detects GZip data from the stream header. The textual format is tab-separated values, meaning that columns are separated by the tab character. Each non-empty logical line starts with the stem column and may contain zero or more variant columns. Input case normalization is controlled by `CaseProcessingMode` (default: `LOWERCASE_WITH_LOCALE_ROOT`), trailing remarks introduced by `#` or `//` are ignored, and dictionary items containing embedded whitespace are currently ignored with warning-level diagnostics.
@@ -38,6 +44,7 @@ Loading from a dictionary file follows the same preparation model as bundled res
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
@@ -50,7 +57,7 @@ public final class LoadTextDictionaryExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
Path.of("data", "stemmer.tsv"),
true,
ReductionSettings.withDefaults(
@@ -59,7 +66,12 @@ public final class LoadTextDictionaryExample {
}
```
Additional `StemmerPatchTrieLoader.load(...)` overloads let callers provide explicit `WordTraversalDirection`, `CaseProcessingMode`, `DiacriticProcessingMode`, or a complete `TrieMetadata` instance. Use those overloads when a custom dictionary must be compiled with forward traversal for right-to-left languages, case-sensitive keys, or diacritic stripping.
Additional `StemmerPatchTrieLoader.loadCompiled(...)` overloads let callers provide explicit `WordTraversalDirection`, `CaseProcessingMode`, `DiacriticProcessingMode`, or a complete `TrieMetadata` instance. Use those overloads when a custom dictionary must be compiled with forward traversal for right-to-left languages, case-sensitive keys, or diacritic stripping.
When `ReductionSettings` are supplied through these compiled loader APIs, uniform-subtree
contraction is still enabled as an internal pre-reduction step. The public `ReductionMode` remains
the semantic policy for subtree equivalence after that contraction has removed regions whose
preferred command is already uniform.
## Load a compiled binary artifact
@@ -69,6 +81,7 @@ Binary loading is typically the preferred runtime path because it avoids reparsi
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -79,7 +92,7 @@ public final class LoadBinaryExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"));
}
}
@@ -96,6 +109,7 @@ at load time. This does not change persisted metadata, only the materialized in-
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -106,14 +120,14 @@ public final class LoadBinaryWithDenseLookupExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> balanced = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> balanced = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"));
final FrequencyTrie<String> fast = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> fast = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"),
1024);
final FrequencyTrie<String> compact = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> compact = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"),
0);
}

29
docs/programmatic-querying-and-ambiguity.md
View File

@@ -10,7 +10,7 @@ This document explains how a compiled Radixor trie is queried and how ambiguity
```java
final String word = "running";
final String patch = trie.get(word);
final CompiledPatchCommand patch = trie.get(word);
```
### `getAll(...)`: ordered local values
@@ -18,7 +18,7 @@ final String patch = trie.get(word);
`FrequencyTrie.getAll(String)` returns all values stored at the addressed node, ordered by descending frequency using the same deterministic tie-breaking rules. The returned array is a defensive copy. If the key is missing or has no local values, an empty array is returned.
```java
final String[] patches = trie.getAll("axes");
final CompiledPatchCommand[] patches = trie.getAll("axes");
```
### `getEntries(...)`: values with counts
@@ -28,9 +28,10 @@ final String[] patches = trie.getAll("axes");
```java
import java.util.List;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.ValueCount;
final List<ValueCount<String>> entries = trie.getEntries("axes");
final List<ValueCount<CompiledPatchCommand>> entries = trie.getEntries("axes");
```
### Visitor lookup for hot paths
@@ -55,31 +56,29 @@ trie.getAllNormalized(token, 0, token.length, (patch, count, rank) -> {
`getAllNormalized(...)` bypasses `caseProcessingMode` and `diacriticProcessingMode`; callers are responsible for supplying canonical input. `maxResults == 0` visits nothing, negative values are rejected, and a sink returning `false` stops iteration after the current callback.
## Apply patch commands
## Apply compiled patch commands
A patch command is not the final stem. It must be applied to the original input token. `PatchCommandEncoder.apply(source, patchCommand)` performs that transformation directly on the serialized command format. If the source is `null`, the method returns `null`. If the patch is `null`, empty, or malformed in compatibility-relevant ways, the original source word is preserved. Equal source and target words are represented by the canonical no-op patch.
A patch command is not the final stem. It must be applied to the original input token. Runtime code should use `CompiledPatchCommand`, which parses the stored patch-command representation once during setup and then applies the concrete immutable command repeatedly.
```java
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.CompiledPatchCommand;
final String word = "running";
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch);
final CompiledPatchCommand patch = trie.get(word);
final String stem = patch == null ? word : patch.apply(word);
```
Hot paths can apply a patch into caller-owned character storage:
```java
final char[] output = new char[32];
final int produced = PatchCommandEncoder.applyTo(
final int produced = patch.applyTo(
word,
patch,
trie.traversalDirection(),
output,
0,
output.length);
if (produced != PatchCommandEncoder.APPLY_INSUFFICIENT_CAPACITY) {
if (produced != CompiledPatchCommand.APPLY_INSUFFICIENT_CAPACITY) {
final String stem = new String(output, 0, produced);
}
```
@@ -90,12 +89,14 @@ For multiple candidates:
```java
final String word = "axes";
for (final String patch : trie.getAll(word)) {
final String stem = PatchCommandEncoder.apply(word, patch);
for (final CompiledPatchCommand patch : trie.getAll(word)) {
final String stem = patch.apply(word);
System.out.println(word + " -> " + stem + " (" + patch + ")");
}
```
The historical `PatchCommandEncoder.apply(...)` API still exists for compatibility with code that directly handles serialized patch-command strings, but it is deprecated because it reparses the command on every call. See [Migration and Backward Compatibility](migration-and-backward-compatibility.md) for the old and new forms side by side.
## Understand reduction modes
Reduction mode determines how mutable subtrees are merged during compilation. All modes operate on full subtree semantics rather than only on local node content.

13
docs/programmatic-usage.md
View File

@@ -24,6 +24,8 @@ This is why Radixor can generalize beyond explicitly listed forms and why compil
The programmatic API is easier to understand when split by developer task:
- [Fast Track](fast-track.md) gives the shortest dependency-to-first-stem path for a new Java project.
- [Integration Deep Dive](integration-deep-dive.md) explains production integration, deployment artifacts, search-pipeline usage, and operational decisions.
- [Loading and Building Stemmers](programmatic-loading-and-building.md) explains how to acquire a compiled stemmer from bundled resources, textual dictionaries, binary artifacts, or direct builder usage.
- [Lookup Edge Optimization](lookup-edge-optimization.md) explains dense child lookup tuning and the speed/memory trade-off when materializing compiled tries.
- [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md) explains `get(...)`, `getAll(...)`, `getEntries(...)`, patch application, and the practical meaning of reduction modes.
@@ -35,7 +37,8 @@ The main types involved in programmatic usage are:
- `FrequencyTrie.Builder<V>` for mutable construction and extension,
- `FrequencyTrie<V>` for the compiled read-only trie,
- `PatchCommandEncoder` for creating and applying patch commands,
- `PatchCommandEncoder` for creating serialized patch commands,
- `CompiledPatchCommand` for repeated runtime patch application,
- `StemmerPatchTrieLoader` for loading bundled or textual dictionaries,
- `StemmerPatchTrieBinaryIO` for reading and writing compressed binary artifacts,
- `FrequencyTrieBuilders` for reconstructing a mutable builder from a compiled trie,
@@ -63,9 +66,11 @@ The core module is standalone and can be consumed directly as a normal Java modu
For most developers, the best order is:
1. [Loading and Building Stemmers](programmatic-loading-and-building.md)
2. [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md)
3. [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md)
1. [Fast Track](fast-track.md)
2. [Integration Deep Dive](integration-deep-dive.md)
3. [Loading and Building Stemmers](programmatic-loading-and-building.md)
4. [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md)
5. [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md)
## Next steps

44
docs/quick-start.md
View File

@@ -2,6 +2,10 @@
This guide introduces the fastest practical path to using **Radixor**.
If you are new to Radixor and want the shortest possible path to a first working stem, start with
[Fast Track](fast-track.md). This Quick Start is a broader developer walkthrough: it introduces the
main loading options, query methods, artifact workflow, and metadata model.
Radixor separates preparation from runtime usage. Source dictionaries are used to derive patch commands and reduce them into a compact read-only trie. Runtime stemming then operates on that compiled structure rather than on the original dictionary text. A richer dictionary usually improves the quality and coverage of inferred transformations, including transformations that are applicable to words not explicitly present in the source material. The reduction step also removes a large amount of redundant lexical information, which is why very large dictionaries can still produce compact runtime artifacts. These artifacts can be persisted and loaded directly when needed.
A practical workflow usually consists of two independent phases:
@@ -15,11 +19,12 @@ A compiled stemmer can be obtained in three common ways.
### Use a bundled language dictionary
Radixor ships with bundled dictionaries for a set of supported languages. These resources are line-oriented dictionaries stored with the library and compiled into a `FrequencyTrie<String>` when loaded. The loader can also store the canonical stem itself as a no-op patch command. Compiled trie artifacts now persist self-describing metadata, including the traversal direction and compilation reduction settings used to build the artifact.
Radixor ships with bundled dictionaries for a set of supported languages. These resources are line-oriented dictionaries stored with the library and compiled into a `FrequencyTrie<CompiledPatchCommand>` when loaded through the runtime API. The loader can also store the canonical stem itself as a no-op patch command. Compiled trie artifacts now persist self-describing metadata, including the traversal direction and compilation reduction settings used to build the artifact.
```java
import java.io.IOException;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -31,7 +36,7 @@ public final class BundledStemmerExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
StemmerPatchTrieLoader.Language.US_UK,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
@@ -49,6 +54,7 @@ Compiled stemmers can be stored as GZip-compressed binary artifacts and loaded d
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -59,7 +65,7 @@ public final class LoadBinaryStemmerExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"));
System.out.println("Canonical node count: " + trie.size());
@@ -73,6 +79,7 @@ You can tune in-memory child lookup density at load time without changing the ar
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -83,10 +90,10 @@ public final class LoadBinaryStemmerExampleTuned {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> fast = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> fast = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"),
1024);
final FrequencyTrie<String> compact = StemmerPatchTrieLoader.loadBinary(
final FrequencyTrie<CompiledPatchCommand> compact = StemmerPatchTrieLoader.loadBinaryCompiled(
Path.of("stemmers", "english.radixor.gz"),
128);
@@ -107,7 +114,7 @@ A dedicated CLI compilation workflow deserves its own focused page and should re
## 2. Use the compiled stemmer
A compiled `FrequencyTrie<String>` stores patch commands, not final stems. Querying therefore has two steps:
A compiled `FrequencyTrie<CompiledPatchCommand>` stores patch commands, not final stems. Querying therefore has two steps:
1. retrieve one or more patch commands from the trie,
2. apply each patch command to the original input word.
@@ -121,8 +128,8 @@ Use `get(...)` when the application needs a single preferred transformation.
```java
import java.io.IOException;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
@@ -133,14 +140,14 @@ public final class SingleStemExample {
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
StemmerPatchTrieLoader.Language.US_UK,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
final String word = "running";
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch);
final CompiledPatchCommand patch = trie.get(word);
final String stem = patch == null ? word : patch.apply(word);
System.out.println(word + " -> " + stem + " (" + patch + ")");
}
@@ -153,10 +160,10 @@ Use `getAll(...)` when the application should preserve ambiguity instead of coll
```java
final String word = "axes";
final String[] patches = trie.getAll(word);
final CompiledPatchCommand[] patches = trie.getAll(word);
for (final String patch : patches) {
final String stem = PatchCommandEncoder.apply(word, patch);
for (final CompiledPatchCommand patch : patches) {
final String stem = patch.apply(word);
System.out.println(word + " -> " + stem + " (" + patch + ")");
}
```
@@ -168,11 +175,12 @@ For diagnostics or advanced ranking logic, use `getEntries(...)` to obtain value
```java
import java.util.List;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.ValueCount;
final List<ValueCount<String>> entries = trie.getEntries("axes");
final List<ValueCount<CompiledPatchCommand>> entries = trie.getEntries("axes");
for (final ValueCount<String> entry : entries) {
for (final ValueCount<CompiledPatchCommand> entry : entries) {
System.out.println(entry.value() + " -> " + entry.count());
}
```
@@ -210,7 +218,11 @@ public final class ExtendCompiledStemmerExample {
String[]::new,
settings);
builder.put("microservices", "Na");
final PatchCommandEncoder encoder = PatchCommandEncoder.builder()
.traversalDirection(compiledTrie.traversalDirection())
.build();
builder.put("microservices", encoder.encode("microservices", "microservice"));
final FrequencyTrie<String> updatedTrie = builder.build();

18
docs/reduction-semantics.md
View File

@@ -103,6 +103,24 @@ This turns reduction into a canonicalization process:
That is how Radixor eliminates duplicated equivalent subtrees.
## Uniform-subtree contraction
Radixor performs one additional internal reduction before each public reduction mode is applied.
When all reachable entries below a subtree have the same preferred patch command, the subtree is
contracted into an accepting leaf for that command.
This optimization is deliberately narrower than the public reduction modes:
- it is based on preferred `get()` behavior,
- it does not depend on child edge shape once the preferred command is uniform,
- it removes lookup depth that cannot affect the selected command,
- it preserves the standard single-result stemming path used by `StemmerPatchTrieLoader.loadCompiled(...)`.
The effect is especially visible in large dictionary tries with many inflected forms that map to
the same command class, such as no-op preservation or common suffix deletion. Runtime lookup can
return the accepting leaf as soon as it is reached instead of traversing the remaining characters
only to discover the same command deeper in the trie.
## Count aggregation and compiled state
When multiple original build-time subtrees collapse into one canonical reduced node, local counts may be aggregated.

5
docs/reports.md
View File

@@ -42,15 +42,14 @@ They are useful for dependency inspection, downstream integration, compliance-or
## Benchmark outputs and badge metadata
These resources expose benchmark results and generated badge metadata derived from the latest published build:
These resources expose benchmark results and generated badge metadata derived from the latest published build. JMH benchmark reports are published as TXT and CSV files; the historical Porter comparison badge is no longer generated.
- [JMH benchmark results (TXT)](https://leogalambos.github.io/Radixor/builds/latest/jmh/jmh-results.txt)
- [JMH benchmark results (CSV)](https://leogalambos.github.io/Radixor/builds/latest/jmh/jmh-results.csv)
- [Coverage badge metadata](https://leogalambos.github.io/Radixor/builds/latest/metrics/coverage-badge.json)
- [Mutation badge metadata](https://leogalambos.github.io/Radixor/builds/latest/metrics/pitest-badge.json)
- [Benchmark badge metadata](https://leogalambos.github.io/Radixor/builds/latest/metrics/jmh-badge.json)
The benchmark outputs provide direct access to the published JMH result files, while the badge metadata endpoints are intended for status surfaces such as the project README or other generated dashboards.
The benchmark outputs provide direct access to the published JMH result files. Coverage and mutation badge metadata endpoints are intended for status surfaces such as the project README or other generated dashboards.
## Practical usage

10
docs/test-taxonomy-and-filtering.md
View File

@@ -145,6 +145,14 @@ defensive fallback in case of future tag drift.
All examples use Gradle with JUnit Platform integration:
- Default fast test run:
```
./gradlew test
```
The default `test` task excludes `slow` tests. Supplying `-DincludeTags` or `-PincludeTags` still excludes `slow` unless the include expression contains `slow`; supplying an explicit exclude expression replaces the default. Long-running bundled-dictionary compilation and full-language loading checks therefore run only through an explicit tag expression such as `-DincludeTags=slow` or a dedicated profile such as `ciSlow`.
- Only unit tests:
```
@@ -154,7 +162,7 @@ All examples use Gradle with JUnit Platform integration:
- Integration tests only:
```
./gradlew test -DincludeTags=integration
./gradlew test -DincludeTags=integration -DexcludeTags=slow
```
- Only trie subsystem tests: