Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: prepare Radixor 3.0.0 with contracted tries and compiled patch commands

Browse Source 
Introduce contracted compiled patch tries for faster lookup, make compiled
patch commands the primary runtime path, refresh stemmer benchmarks and
documentation, and restructure the documentation for 3.0.0 onboarding.

BREAKING CHANGE: Radixor 3.0.0 promotes compiled patch-command APIs and
new compiled trie artifacts as the primary runtime integration model.
This commit is contained in:
Leo Galambos
2026-07-03 18:44:39 +02:00
parent df4552b113
commit 38620d7e71
101 changed files with 11235 additions and 727 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
docs/benchmarking.md
View File

@@ -1,159 +1,39 @@
# Benchmarking
Radixor includes a JMH benchmark suite for both the internal algorithmic core and a side-by-side English comparison against the Snowball Porter stemmer family.
Radixor contains internal trie microbenchmarks, a separate stemmer comparison suite, and a dictionary coverage benchmark for Radixor itself. Published stemmer comparison results must come only from benchmark classes matching `.*StemmerComparisonBenchmark.*`; internal `FrequencyTrie*` microbenchmarks are not part of those results.
This document explains what is benchmarked, how to run the suite, and how benchmark results should be interpreted.
This page is the entry point for benchmark interpretation. Detailed tables and long reference material are split into focused subpages so that important points do not get buried.
## Scope
## Key Takeaways
The benchmark suite currently covers two categories:
- Speed and accuracy must be read together. A faster row is not necessarily a better stemmer.
- Radixor is the quality-oriented baseline in same-language comparisons. Its exact-root accuracy is often close to 100%, while many faster competitors are light, minimal, possessive, or aggressive rule-based stemmers with much lower root agreement.
- The measured Radixor cost buys dictionary-trained stemming precision. That precision improves search quality by mapping inflected forms to intended dictionary roots instead of approximate or over-reduced stems.
- Speed benchmarks process changed dictionary tokens where the surface form differs from the expected root. Accuracy benchmarks process the complete dictionary.
- Accuracy-only benchmarks intentionally use one deterministic JMH measurement iteration without warmup because repeated precision passes would duplicate the same counters.
- The historical Porter performance badge is retired. Benchmark reporting now uses speed and quality tables rather than a single Porter ratio.
- Radixor core operations,
- English stemmer comparison on the same token workload.
## Benchmark Documentation Map
The comparison benchmark processes the same deterministic English token stream through:
| Page | Purpose |
| --- | --- |
| [Benchmark methodology](benchmarks/reference/methodology.md) | Workload design, speed pass, quality pass, normalization policy, and exact-root metrics. |
| [Benchmark corpora](benchmarks/reference/corpora.md) | Dictionary row counts, complete quality tokens, already-root tokens, changed speed tokens, and timing token counts. |
| [Benchmark environment and reports](benchmarks/reference/environment.md) | Hardware, OS, JVM, JMH settings, report files, and current badge/report policy. |
| [English dictionary coverage benchmark](benchmarks/reference/english-coverage.md) | The quality/speed operating curve for contracted Radixor tries built from 100% down to 10% of English dictionary rows. |
| [Candidate evaluation](benchmarks/reference/candidates.md) | Included benchmark families and evaluated candidates that were skipped. |
| [Language benchmark pages](benchmarks/languages/index.md) | Per-language accuracy tables, speed tables, and implementation notes. |
- Radixor with bundled `US_UK` (older benchmark snapshots used the now-retired `US_UK_PROFI` resource),
- Snowball original Porter,
- Snowball English, commonly referred to as Porter2.
## How To Read Results
The purpose of the comparison is throughput measurement on identical input. It is not intended to demonstrate linguistic equivalence between the compared stemmers.
Start with the [language benchmark pages](benchmarks/languages/index.md). Each language page lists accuracy first and speed second because throughput without root agreement is not enough to interpret stemmer quality.
## How to read the published numbers
When Radixor is slower than a narrow competitor, check the accuracy table before drawing a conclusion. Many Lucene light/minimal filters and possessive filters intentionally do less work. They can be fast precisely because they are not trying to match the dictionary root with the same precision.
Two kinds of benchmark numbers are relevant in the project.
The [English dictionary coverage benchmark](benchmarks/reference/english-coverage.md) shows the central operating curve explicitly: contracted tries preserve high quality even at reduced dictionary coverage, while changed-form exactness still reflects how much language knowledge was available during training. This is why Radixor performance should be discussed as a configurable quality/speed point, not as a single fixed ratio against Porter.
### Reference measurements
## Current Result Locations
The detailed benchmark snapshot documented on this page comes from a controlled run on a Ryzen 5 system. Those numbers are the best reference point for understanding absolute throughput under a known local benchmark environment.
The current measured language results are published in [Language Benchmark Pages](benchmarks/languages/index.md). Generated local report files for this benchmark update are listed in [Benchmark environment and reports](benchmarks/reference/environment.md).
### Published badge figures
The benchmark badge metadata published through GitHub Pages is generated in the GitHub-hosted container environment. That environment is convenient for continuous publication, but it is not the right place to treat absolute throughput values as stable across time. CPU scheduling, shared-host variability, and container-level noise can materially affect raw numbers from run to run.
For that reason, the published badge values should be treated primarily as a compact status surface. They are useful for observing broad trends and relative positioning, but not as the authoritative source for precise absolute throughput claims.
## Current snapshot
A recent JMH run on JDK 21.0.10 with JMH 1.37, one thread, three warmup iterations, and five measurement iterations produced the following approximate throughput ranges:
| Workload | Radixor `US_UK` *(historical runs: `US_UK_PROFI`)* | Snowball Porter | Snowball English |
| --- | ---: | ---: | ---: |
| About 12,000 generated tokens | 30.99 M tokens/s | 8.21 M tokens/s | 5.46 M tokens/s |
| About 60,000 generated tokens | 32.25 M tokens/s | 8.02 M tokens/s | 5.11 M tokens/s |
On that workload, Radixor measured approximately:
- 4 times the throughput of Snowball original Porter,
- 6 times the throughput of Snowball English.
These values are workload-dependent and environment-dependent. They should be read as measured results for the documented setup, not as universal constants.
## Interpreting the relative result
Although the absolute numbers can move across environments, the throughput relationship between Radixor and the compared Porter-family stemmers has remained broadly stable in practical measurements. In particular, the comparison against Snowball original Porter is consistently in the rough range of about four to one in Radixor’s favor.
That relative behavior is more informative than any single absolute figure. It reflects a real architectural difference rather than a cosmetic benchmark artifact.
Radixor is built around a compiled patch-command trie that resolves the result through a direct lookup and patch application path. In contrast, classic rule-based stemmers such as the Porter family follow a different operational model. The result is that Radixor combines two properties that do not often appear together:
- dictionary-driven compiled lookup performance,
- the ability to generalize beyond explicitly listed word forms instead of behaving like a pure closed-form dictionary lookup table.
Within that design space, the measured throughput profile is strong enough to place Radixor among the fastest known practical implementations of this kind, while still supporting stemming of previously unseen forms. That should still be read as a carefully bounded engineering statement, not as an absolute claim over every possible stemmer architecture or benchmark scenario.
## Benchmark classes
The main benchmark classes are under `src/jmh/java/org/egothor/stemmer/benchmark`.
Relevant classes include:
- `FrequencyTrieLookupBenchmark`,
- `FrequencyTrieCompilationBenchmark`,
- `EnglishStemmerComparisonBenchmark`.
The English comparison benchmark uses the bundled Radixor English resource and the official Snowball Java distribution integrated into the JMH source set.
## Workload design
The English comparison benchmark uses a deterministic generated corpus rather than an uncontrolled ad hoc text sample.
The workload intentionally mixes:
- simple inflections,
- common derivational forms,
- US and UK spelling families,
- lexical forms appropriate for the current bundled `US_UK` resource (with historical continuity from earlier `US_UK_PROFI` runs).
This design keeps runs reproducible across environments and avoids accidental drift caused by changing external corpora.
## Running benchmarks
Run the full benchmark suite:
```bash
./gradlew jmh
```
Run only the English comparison benchmark:
```bash
./gradlew jmh -Pjmh.includes=EnglishStemmerComparisonBenchmark
```
## Generated reports
JMH reports are written to:
- `build/reports/jmh/jmh-results.txt`,
- `build/reports/jmh/jmh-results.csv`.
The text report is convenient for human review. The CSV report is more useful for CI archiving, historical tracking, and external processing.
## Interpreting results responsibly
Benchmark numbers should always be read with care.
Important factors include:
- CPU model and frequency behavior,
- thermal throttling,
- JVM vendor and version,
- system background load,
- operating-system scheduling noise,
- benchmark parameter changes.
For meaningful comparison, keep these stable:
- hardware or VM class,
- JDK version,
- benchmark parameters,
- thread count,
- benchmark source revision.
If a regression is suspected, repeat the run and compare against previous CSV output rather than relying on a single measurement.
## Regression tracking
The recommended regression workflow is:
1. archive `jmh-results.csv`,
2. compare the same benchmark names across runs,
3. compare only like-for-like environments,
4. investigate sustained regressions rather than one-off noise.
For public reporting, the README should keep only the condensed benchmark summary, while detailed benchmark methodology and interpretation should remain in this document.
## Notes on comparison fairness
Radixor, Snowball Porter, and Snowball English are not the same kind of stemmer.
Radixor uses a compiled patch-command trie driven by dictionary data. Snowball Porter and Snowball English are rule-based English stemmers.
Because of that, the comparison should be understood as:
- equal input workload,
- different stemming strategies,
- measured throughput rather than semantic identity.
That distinction matters whenever performance claims are discussed in documentation, release notes, or badge summaries.
JMH TXT and CSV reports are still published as benchmark artifacts. They are no longer converted into a Shields endpoint benchmark badge.