Leo Galambos
038514bad0
feat: implement Compile CLI for building binary stemmer tables from source dictionaries feat: add loading support for persisted compiled tries, including GZip-compressed binaries feat: add a builder path for recreating a writable trie from a compiled trie feat: expose read-only value/count access for compiled trie entries feat: support deterministic NOOP patch encoding for identical source and target words fix: make value selection deterministic for equal frequencies using length and lexical tie-breakers fix: preserve valid alternative reductions during trie optimization and reduction fix: correct patch command edge cases discovered in round-trip and malformed-input tests fix: address persistence and compiled-trie handling defects found during implementation review fix: resolve test failures and behavioral regressions uncovered by PMD and JUnit runs refactor: reorganize trie-related support types into dedicated packages and classes refactor: simplify the core FrequencyTrie design toward a cleaner practical architecture refactor: improve compiled/read-only trie boundaries without restoring mutability refactor: clean up internal reduction, serialization, and helper structure test: add professional JUnit coverage for stemmer core classes test: split trie tests into dedicated test classes per production type test: improve parameterized tests for readability, diagnostics, and edge-case traceability test: cover positive, negative, malformed, persistence, and round-trip scenarios test: verify compiled dictionaries against source inputs using getAll semantics docs: write public README and supplementary Markdown documentation for project publishing docs: document architecture, reduction model, built-in languages, and operational guidance docs: clarify reverse-word storage, mutable construction, and compiled-trie runtime behavior docs: remove placeholders, vague buzzwords, and unexplained terminology from the documentation docs: improve examples and wording for professional reader-facing project guidance chore: align project materials with the practical Radix scope and Egothor/Stempel lineage chore: raise overall project quality through documentation review and test hardening
5.6 KiB
Radixor
Fast algorithmic stemming with compact patch-command tries.
Radixor is a fast, algorithmic stemming toolkit for Java, built around compact patch-command tries in the tradition of the original Egothor stemmer.
It is designed for production search and text-processing systems that need stemming which is:
- fast at runtime
- compact in memory and on disk
- deterministic in behavior
- driven by dictionary data rather than hardcoded language rules
- practical to maintain, extend, and test
Radixor keeps the valuable core of the original Egothor idea, modernizes the implementation, and adds capabilities that make it more useful in real software systems today.
Table of Contents
- Why Radixor
- Heritage
- What Radixor adds
- Key features
- Documentation
- Project philosophy
- Historical note
Why Radixor
The central idea behind Radixor is simple: learn how to transform a word form into its stem, encode that transformation as a compact patch command, store it in a trie, and make runtime lookup extremely fast.
This gives you a stemmer that is:
- data-driven rather than rule-hardcoded
- reusable across languages
- compact enough for deployment-friendly binary artifacts
- suitable for both offline compilation and runtime loading
Radixor is especially attractive when you want something more adaptable than simple suffix stripping, but much smaller and easier to operate than a full morphological analyzer.
Heritage
Radixor stands in the line of the original Egothor stemming work and its later Stempel packaging.
Historical Stempel documentation describes the stemmer code as taken virtually unchanged from the Egothor project, and Elasticsearch still documents the Stempel analysis plugin as integrating Lucene’s Stempel module for Polish.
Useful historical references:
Radixor is not just a repackaging of legacy code. It is a practical modernization of the approach for current Java development and long-term maintainability.
What Radixor adds
Radixor keeps the patch-command trie model, but improves the engineering around it.
Compared with the historical baseline, Radixor emphasizes:
-
simplification to the most practical core
The implementation focuses on the parts of the original approach that are most useful in production. -
immutable compiled tries
Runtime lookup uses compact read-only structures optimized for efficient access. -
support for more than one stemming result
Radixor can expose both a preferred result and multiple candidate results where the data is ambiguous. -
frequency-aware deterministic ordering
Candidate results are ordered consistently and reproducibly. -
practical subtree reduction modes
Reduction can be tuned toward stronger compression or more conservative behavioral preservation. -
reconstruction of writable builders from compiled tables
Existing compiled stemmer tables can be reopened, modified, and compiled again. -
better tests and implementation stability
Stronger coverage improves confidence during refactoring and further development.
Key features
- Fast algorithmic stemming
- Compact compiled binary artifacts
- Patch-command based transformation model
- Dictionary-driven language adaptation
- Single-result and multi-result lookup
- Deterministic result ordering
- Compressed binary persistence
- Programmatic compilation and loading
- CLI compilation tool
- Bundled language resources
- Support for extending compiled stemmer tables
Documentation
The repository keeps the front page concise and places detailed documentation under docs/.
Start here:
-
Quick Start
A practical first guide to loading, compiling, and using Radixor. -
Dictionary Format
How to write stemming dictionaries. -
Compilation (CLI tool)
How to compile dictionaries with theCompileCLI. -
Programmatic Usage
How to build, load, modify, and query Radixor from Java code. -
Built-in Languages
How to use integrated language resources such asUS_UK_PROFI. -
Architecture and Reduction
Internal model, compiled trie design, and reduction strategies. -
Quality and Operations
Testing, persistence, deployment, and operational guidance.
Project philosophy
Radixor does not preserve historical complexity for its own sake.
It preserves the valuable idea:
- compact learned transformations
- trie-based lookup
- language-data driven stemming
- practical runtime speed
Then it improves the parts modern users care about:
- maintainability
- testability
- modification workflows
- persistence
- determinism
- clearer APIs
The goal is to keep the Egothor/Stempel lineage useful as a serious contemporary software component.
Historical note
Egothor showed that stemming could be both algorithmic and compact. Stempel proved that the approach was practical enough to survive inside major search ecosystems. Radixor continues that tradition with a modernized implementation focused on production use, maintainability, and controlled evolution.