Leo Galambos
e7800b29c9
Move bundled stemmer dictionaries from the core artifact into independently versioned model modules. Add model discovery and explicit model-loading APIs, a standard model aggregate, a model BOM, and dedicated model and catalog release workflows. Add full PoliMorf integration, model provenance and licensing validation, streaming model-input verification, strict dependency verification, consumer resolution tests, Configuration Cache compatibility, and expanded JMH, quality, documentation, and release checks. Upgrade the CycloneDX and JMH Gradle plugins and remove Gradle 10 and Java compiler deprecations. BREAKING CHANGE: The core Radixor artifact no longer contains bundled stemmer dictionaries. Applications must add the required model artifacts, the standard model aggregate, or model dependencies managed through the Radixor model BOM.
4.6 KiB
Home
Radixor is a high-performance, multi-language stemmer for Java, built for production-grade search and text-processing systems.
It modernizes the proven Egothor patch-command trie approach and extends it for deployment realities that classic stemming pipelines do not handle well.
Traditional Egothor-style stemming workflows usually treat a compiled dictionary as a fixed artifact. Once built, its lexical knowledge is effectively closed unless the original source dictionary is recompiled. Radixor removes that constraint. An already compiled stemming structure can be extended with additional words and transformations, which makes it possible to evolve an existing dictionary for domain-specific, customer-specific, or deployment-specific vocabulary without rebuilding the entire lexical base from scratch.
Radixor also improves how ambiguous reductions can be handled at runtime. Instead of always forcing a single result, it can return multiple plausible stems when the input token cannot be reduced unambiguously. This allows downstream systems to preserve linguistic ambiguity where that is operationally useful, whether for retrieval quality, ranking strategies, diagnostics, or domain-specific normalization policies.
The project also has a clear research lineage. The historical idea behind this stemming family is described in Leo Galambos's paper Lemmatizer for Document Information Retrieval Systems in JAVA (SOFSEM 2001), which presents a semi-automatic stemming technique designed for Java-based information retrieval systems. In Radixor documentation, this reference serves as historical and algorithmic background rather than as technical documentation of the current implementation.
Unlike traditional Egothor-based deployments, Radixor can extend an already compiled stemmer dictionary and can return multiple stems when a word is not reducible to a single unambiguous form.
Radixor delivers:
- Fast runtime stemming with compact lookup structures
- Multi-language adaptability through dictionary-driven compilation
- Extension of compiled stemmer structures without full recompilation from source dictionaries
- Incremental vocabulary growth for deployment-specific lexical refinement
- Support for multiple stemming results when reduction is ambiguous
- Deterministic behavior suitable for reproducible processing pipelines
- Flexible integration paths, including CLI-based and programmatic workflows
- Operational transparency through continuously published quality and benchmark reports
Radixor is intended for teams that require consistent stemming quality at scale, while retaining the ability to evolve lexical resources after compilation and to handle ambiguous reductions with greater precision than traditional single-stem pipelines allow.
Add the core and model data
The core org.egothor:radixor JAR contains no language dictionary. A minimal application adds one model; broad deployments may use the optional standard pack:
dependencies {
implementation 'org.egothor:radixor:<radixor-version>'
runtimeOnly 'org.egothor:radixor-model-pl-pl-unimorph:1.0.0'
}
StemmerPatchTrieLoader.loadCompiled(Language.PL_PL, ...) resolves the default pl-pl-unimorph. pl-pl-polimorf is a separate optional model selected by stable model ID. Follow Model Selection and Loading for runnable examples or choose artifacts from the generated model catalog.
Start here
- Read Fast Track when you want the shortest path to a working bundled stemmer.
- Use Model Selection and Loading for default, explicit, dual-model, and ClassLoader examples.
- Use Integration Deep Dive when you are wiring Radixor into a real application or search pipeline.
- Read Quick Start for the broader developer walkthrough after the first result works.
- Use Built-in Languages to interpret language defaults and optional model variants.
- Review Benchmarking and Benchmark Results for reproducible performance and quality methodology.
- Open CI Reports to inspect published build artifacts and quality metrics.
- See the historical paper: Lemmatizer for Document Information Retrieval Systems in JAVA.