Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: improve README, MkDocs content, branding assets, and site polish

Browse Source
This commit is contained in:
Leo Galambos
2026-04-19 00:18:42 +02:00
parent db79dd2d4f
commit 0b674a39a8
19 changed files with 1836 additions and 1698 deletions
Download Patch File Download Diff File Expand all files Collapse all files

182
README.md
View File

@@ -11,53 +11,61 @@
[![License](https://img.shields.io/github/license/leogalambos/Radixor)](LICENSE)
[![Java](https://img.shields.io/badge/Java-21%2B-brightgreen)](#)
*Fast algorithmic stemming with compact patch-command tries — measured at about 4× to 6× the throughput of the Snowball Porter stemmer family on the current English benchmark workload.*
*Fast, deterministic, multi-language stemming for Java, built around compact patch-command tries and measured at roughly 4× to 6× the throughput of the Snowball Porter stemmer family on the current English benchmark workload.*
**Radixor** is a fast, algorithmic stemming toolkit for Java, built around compact **patch-command tries** in the tradition of the original **Egothor** stemmer.
**Radixor** is a modern multi-language stemming toolkit for Java in the tradition of the original **Egothor** approach. It learns compact word-to-stem transformations from dictionary data, stores them in compiled patch-command tries, and exposes a runtime model designed for speed, determinism, and operational simplicity. Unlike a closed-form dictionary lookup stemmer, Radixor can also generalize beyond explicitly listed word forms.
On the current JMH English comparison benchmark, Radixor with bundled `US_UK_PROFI`
reaches approximately **31 to 32 million tokens per second**, compared with about
**8 million tokens per second** for Snowball original Porter and about
**5 to 5.5 million tokens per second** for Snowball English (Porter2).
It is particularly well suited to systems that need stemming which is:
That means the current Radixor implementation is approximately:
- fast at runtime,
- compact in memory and on disk,
- deterministic in behavior,
- adaptable through dictionary data rather than hardcoded language rules,
- practical to compile, persist, version, extend, and deploy.
- **4× faster** than Snowball original Porter
- **6× faster** than Snowball English (Porter2)
It is designed for production search and text-processing systems that need stemming which is:
- fast at runtime
- compact in memory and on disk
- deterministic in behavior
- driven by dictionary data rather than hardcoded language rules
- practical to maintain, extend, and test
Radixor keeps the valuable core of the original Egothor idea, modernizes the implementation, and adds capabilities that make it more useful in real software systems today.
It also retains the operational advantages of a compiled artifact model: predictable runtime behavior, direct binary loading, and clear separation between preparation-time compilation and live request processing.
## Table of Contents
- [Why Radixor](#why-radixor)
- [Performance](#performance)
- [Heritage](#heritage)
- [What Radixor adds](#what-radixor-adds)
- [Key features](#key-features)
- [Performance](#performance)
- [Documentation](#documentation)
- [Project philosophy](#project-philosophy)
- [Historical note](#historical-note)
## Why Radixor
The central idea behind Radixor is simple: learn how to transform a word form into its stem, encode that transformation as a compact patch command, store it in a trie, and make runtime lookup extremely fast.
The central idea behind Radixor is simple: learn how to transform a word form into its stem, encode that transformation as a compact patch command, store it in a trie, and make the runtime path as small and direct as possible.
This gives you a stemmer that is:
That produces a stemmer that is:
- data-driven rather than rule-hardcoded
- reusable across languages
- compact enough for deployment-friendly binary artifacts
- suitable for both offline compilation and runtime loading
- data-driven rather than rule-hardcoded,
- applicable across languages through compiled transformation models learned from dictionary data,
- compact enough for deployment-friendly binary artifacts,
- suitable for both offline compilation and direct runtime loading,
- capable of exposing either a preferred result or multiple candidate results when ambiguity matters.
Radixor is especially attractive when you want something more adaptable than simple suffix stripping, but much smaller and easier to operate than a full morphological analyzer. In the current English benchmark comparison against the Snowball Porter stemmer family, it also delivers a substantial throughput advantage.
Radixor is especially attractive when you want something more adaptable than simple suffix stripping, but much smaller and easier to operate than a full morphological analyzer.
## Performance
Radixor includes a JMH benchmark suite for both its own algorithmic core and a side-by-side English comparison against the Snowball Porter stemmer family.
On the current English comparison workload, Radixor with bundled `US_UK_PROFI` reaches approximately **31 to 32 million tokens per second**. Snowball original Porter reaches approximately **8 million tokens per second**, and Snowball English (Porter2) approximately **5 to 5.5 million tokens per second**.
That places Radixor at approximately:
- **4× the throughput of Snowball original Porter**
- **6× the throughput of Snowball English (Porter2)**
on the current benchmark workload.
This is a throughput comparison on the same deterministic token stream. It is **not** a claim that the compared stemmers are linguistically equivalent or interchangeable.
For benchmark scope, workload design, environment, commands, report locations, and interpretation guidance, see [Benchmarking](docs/benchmarking.md).
## Heritage
@@ -69,44 +77,47 @@ Useful historical references:
- [Egothor project](http://www.egothor.org/)
- [Stempel overview](https://www.getopt.org/stempel/)
- [Leo Galambos, *Lemmatizer for Document Information Retrieval Systems in JAVA* (SOFSEM 2001)](https://www.researchgate.net/publication/221512865_Lemmatizer_for_Document_Information_Retrieval_Systems_in_JAVA)
- [Lucene Stempel overview](https://lucene.apache.org/core/5_3_0/analyzers-stempel/index.html)
- [Elasticsearch Stempel plugin](https://www.elastic.co/docs/reference/elasticsearch/plugins/analysis-stempel)
Radixor is not just a repackaging of legacy code. It is a practical modernization of the approach for current Java development and long-term maintainability.
The Galambos paper is a useful historical reference for the semi-automatic, transformation-based stemming idea that later informed the Egothor lineage and, in turn, the conceptual background of Radixor. It should be read as research and heritage context rather than as a description of Radixor's present-day implementation.
Radixor is not a repackaging of legacy code. It is a modern implementation that preserves the valuable core idea while reworking the engineering around maintainability, testing, persistence, and long-term operational use.
## What Radixor adds
Radixor keeps the patch-command trie model, but improves the engineering around it.
Radixor keeps the patch-command trie model, but improves the engineering around it in ways that matter in real software systems.
Compared with the historical baseline, Radixor emphasizes:
- **simplification to the most practical core**
The implementation focuses on the parts of the original approach that are most useful in production.
- **a focused practical core**
The implementation concentrates on the parts of the original approach that are most useful in production.
- **immutable compiled tries**
Runtime lookup uses compact read-only structures optimized for efficient access.
- **support for more than one stemming result**
Radixor can expose both a preferred result and multiple candidate results where the data is ambiguous.
Radixor can expose both a preferred result and multiple candidate results when the underlying data is ambiguous.
- **frequency-aware deterministic ordering**
Candidate results are ordered consistently and reproducibly.
- **practical subtree reduction modes**
Reduction can be tuned toward stronger compression or more conservative behavioral preservation.
Reduction can be tuned toward stronger compression or more conservative semantic preservation.
- **reconstruction of writable builders from compiled tables**
- **reconstruction of writable builders from compiled artifacts**
Existing compiled stemmer tables can be reopened, modified, and compiled again.
- **better tests and implementation stability**
Stronger coverage improves confidence during refactoring and further development.
- **strong validation discipline**
Coverage, mutation testing, benchmark visibility, and published reports are treated as part of the engineering standard rather than optional project decoration.
## Key features
- Fast algorithmic stemming
- Compact compiled binary artifacts
- Patch-command based transformation model
- Dictionary-driven language adaptation
- Multi-language stemming through compiled transformation models
- Single-result and multi-result lookup
- Deterministic result ordering
- Compressed binary persistence
@@ -114,57 +125,69 @@ Compared with the historical baseline, Radixor emphasizes:
- CLI compilation tool
- Bundled language resources
- Support for extending compiled stemmer tables
## Performance
Radixor includes a JMH benchmark suite for both its own algorithmic core and a
side-by-side comparison against the Snowball Porter stemmer family.
On the current English comparison workload, Radixor with bundled `US_UK_PROFI`
reaches approximately **31 to 32 million tokens per second**. Snowball original
Porter reaches approximately **8 million tokens per second**, and Snowball
English (Porter2) approximately **5 to 5.5 million tokens per second**.
That places Radixor at approximately **4× the throughput of Snowball original Porter**
and approximately **6× the throughput of Snowball English (Porter2)**
on the current benchmark workload.
This is a throughput comparison on the same deterministic token stream. It is
not a claim that the compared stemmers are linguistically equivalent or
interchangeable.
For benchmark scope, workload design, environment, commands, report locations,
and interpretation guidance, see [Benchmarking](docs/benchmarking.md).
- Reproducible and auditable engineering posture
## Documentation
The repository keeps the front page concise and places detailed documentation under `docs/`.
Start here:
### Getting Started
- [Quick Start](docs/quick-start.md)
A practical first guide to loading, compiling, and using Radixor.
- [Built-in Languages](docs/built-in-languages.md)
Overview of bundled language resources such as `US_UK` and `US_UK_PROFI`.
- [Dictionary Format](docs/dictionary-format.md)
How to write stemming dictionaries.
How to write and normalize stemming dictionaries.
- [Compilation (CLI tool)](docs/cli-compilation.md)
How to compile dictionaries with the `Compile` CLI.
How to compile dictionaries into deployable binary artifacts.
- [Programmatic Usage](docs/programmatic-usage.md)
How to build, load, modify, and query Radixor from Java code.
### Programmatic Usage
- [Built-in Languages](docs/built-in-languages.md)
How to use integrated language resources such as `US_UK_PROFI`.
- [Programmatic Usage Overview](docs/programmatic-usage.md)
Entry point to the Java API and the overall usage model.
- [Architecture and Reduction](docs/architecture-and-reduction.md)
Internal model, compiled trie design, and reduction strategies.
- [Loading and Building Stemmers](docs/programmatic-loading-and-building.md)
Loading bundled resources, textual dictionaries, binary artifacts, and direct builder usage.
- [Querying and Ambiguity Handling](docs/programmatic-querying-and-ambiguity.md)
`get()`, `getAll()`, `getEntries()`, patch application, and ambiguity behavior.
- [Extending and Persisting Compiled Tries](docs/programmatic-extending-and-persistence.md)
Reopening compiled tries, rebuilding them, and writing binary artifacts.
### Concepts and Internals
- [Architecture and Reduction Overview](docs/architecture-and-reduction.md)
High-level explanation of the build pipeline and compiled trie model.
- [Architecture](docs/architecture.md)
Structural model, data flow, and runtime lookup behavior.
- [Reduction Semantics](docs/reduction-semantics.md)
Ranked, unordered, and dominant reduction behavior.
- [Compatibility and Guarantees](docs/compatibility-and-guarantees.md)
Supported public API, internal API boundaries, and compatibility expectations.
### Dictionaries and Language Resources
- [Contributing Dictionaries](docs/contributing-dictionaries.md)
Guidance for high-quality lexical resource contributions.
### Quality and Operations
- [Quality and Operations](docs/quality-and-operations.md)
Testing, persistence, deployment, and operational guidance.
Engineering standards, validation posture, auditability, and operational model.
- [Benchmarking](docs/benchmarking.md)
JMH benchmark design, Snowball comparison, execution, and interpretation.
JMH benchmark methodology, Porter comparison, and result interpretation.
- [Published Reports](docs/reports.md)
Entry points to CI-published reports and GitHub Pages artifacts.
## Project philosophy
@@ -172,19 +195,20 @@ Radixor does not preserve historical complexity for its own sake.
It preserves the valuable idea:
- compact learned transformations
- trie-based lookup
- language-data driven stemming
- practical runtime speed
- compact learned transformations,
- trie-based lookup,
- language-data driven stemming,
- practical runtime speed.
Then it improves the parts modern users care about:
- maintainability
- testability
- modification workflows
- persistence
- determinism
- clearer APIs
- maintainability,
- testability,
- modification workflows,
- persistence,
- determinism,
- clearer APIs,
- explicit quality evidence.
The goal is to keep the Egothor/Stempel lineage useful as a serious contemporary software component.

9
build.gradle
View File

@@ -207,14 +207,7 @@ distributions {
from('docs') {
into 'docs'
include 'quick-start.md'
include 'cli-compilation.md'
include 'dictionary-format.md'
include 'built-in-languages.md'
include 'programmatic-usage.md'
include 'architecture-and-reduction.md'
include 'quality-and-operations.md'
include 'benchmarking.md'
include '**/*.md'
}
from(layout.buildDirectory.dir('generated/release-notes')) {

470
docs/architecture-and-reduction.md
View File

@@ -1,468 +1,52 @@
# Architecture and Reduction
This document describes the internal architecture of **Radixor** and the principles behind its **trie compilation and reduction model**.
This section explains how **Radixor** turns textual dictionary input into a compact compiled stemmer and how reduction affects the semantics preserved in the final runtime artifact.
It explains:
Radixor is easiest to understand when separated into two related concerns:
- how data flows from dictionary input to compiled trie
- how patch-command tries are structured
- how subtree reduction works
- how reduction modes affect behavior and size
- **architecture**: what structures exist, how data moves through them, and what runtime lookup actually does,
- **reduction semantics**: what it means for two subtrees to be considered equivalent and how that choice affects `get()` and `getAll()` behavior.
## The short version
Radixor does not keep a large flat table of final stems. Instead, it converts dictionary entries into **patch commands**, stores them in a trie, reduces equivalent subtrees, and freezes the result into an immutable compiled structure.
## Overview
The build-time flow is:
Radixor transforms dictionary data into an optimized runtime structure through three stages:
1. **Mutable construction**
2. **Reduction (canonicalization)**
3. **Compilation (freezing)**
```
Dictionary → Mutable trie → Reduced trie → Compiled trie
```text
Dictionary -> Mutable trie -> Reduced trie -> Compiled trie
```
Each stage has a distinct purpose:
At runtime, the compiled trie does not directly return the final stem string. It returns one or more stored patch commands for the addressed key, and those commands are then applied to the original input word.
| Stage | Purpose | Structure |
|------------|----------------------------------|-------------------------|
| Build | Collect mappings | `MutableNode` |
| Reduction | Merge equivalent subtrees | `ReducedNode` |
| Compilation | Optimize for runtime lookup | `CompiledNode` |
## Why this matters
This design gives Radixor several practical properties at once:
- compact deployable artifacts,
- deterministic runtime behavior,
- support for both preferred and multiple candidate results,
- separation of preparation-time complexity from runtime lookup.
## Core data model
It also explains why a large source dictionary can be transformed into a much smaller compiled artifact without discarding the operational behavior that matters to the caller.
### Patch-command trie
## Reading guide
Radixor stores **patch commands** instead of stems directly.
Use the following pages depending on what you need to understand:
- keys: word forms
- values: transformation commands
- structure: trie (prefix tree)
- [Architecture](architecture.md) explains the data flow, core structures, patch-command lookup model, and why the compiled trie is efficient at runtime.
- [Reduction Semantics](reduction-semantics.md) explains how subtree equivalence is defined, what ranked, unordered, and dominant reduction preserve, and how those choices affect observable lookup behavior.
At runtime:
## Recommended reading order
1. the word is traversed through the trie
2. a patch command is retrieved
3. the patch is applied to reconstruct the stem
For most readers, the best order is:
1. [Architecture](architecture.md)
2. [Reduction Semantics](reduction-semantics.md)
## Related documentation
## Stage 1: Mutable construction
The builder (`FrequencyTrie.Builder`) constructs a trie using:
- `MutableNode`
- maps of children (`char → node`)
- maps of value counts (`value → frequency`)
Characteristics:
- insertion-order preserving
- mutable
- optimized for building, not querying
Example structure:
```
g
└─ n
└─ i
└─ n
└─ n
└─ u
└─ r
└─ (values: {
"<patch-command-1>": 3,
"<patch-command-2>": 1
})
```
This example represents the word "running", stored in reversed form.
- each edge corresponds to one character of the word
- the path is traversed from the end of the word toward the beginning
- the terminal node stores one or more patch commands together with their local frequencies
The values represent transformations from the word form to candidate stems, and the counts indicate how often each mapping was observed during construction.
Note: Radixor stores word forms in reversed order so that suffix-based transformations can be matched efficiently in a trie.
## Local value summary
Before reduction, each node is summarized using `LocalValueSummary`.
It computes:
- ordered values (by frequency)
- aligned counts
- total frequency
- dominant value (if any)
- second-best value
This summary is critical for:
- deterministic ordering
- reduction decisions
- dominance evaluation
## Stage 2: Reduction (canonicalization)
Reduction is the process of merging **semantically equivalent subtrees**.
### Why reduction exists
Without reduction:
- trie size grows linearly with input data
- repeated patterns are duplicated
With reduction:
- identical subtrees are shared
- memory footprint is reduced
- binary output becomes smaller
## Reduction signature
Each subtree is represented by a **ReductionSignature**.
A signature consists of:
1. **local descriptor** (node semantics)
2. **child descriptors** (structure)
```
Signature = (LocalDescriptor, SortedChildDescriptors)
```
Two subtrees are merged if their signatures are equal.
## Local descriptors
The local descriptor encodes how values at a node are interpreted.
Radixor supports three descriptor types:
### 1. Ranked descriptor
Preserves:
- full ordering of values (`getAll()`)
Uses:
- ordered value list
Best for:
- correctness
- deterministic multi-result behavior
### 2. Unordered descriptor
Preserves:
- only membership (set of values)
Ignores:
- ordering differences
Best for:
- higher compression
- use cases where ordering is irrelevant
### 3. Dominant descriptor
Preserves:
- only the dominant value (`get()`)
Condition:
- dominant value must satisfy thresholds:
- minimum percentage
- ratio over second-best
Fallback:
- if dominance is not strong enough → ranked descriptor is used
Best for:
- maximum compression
- single-result workflows
## Child descriptors
Each child is represented as:
```
(edge character, child signature)
```
Children are sorted by edge character to ensure:
- deterministic signatures
- stable equality comparisons
## Reduction context
`ReductionContext` maintains:
- mapping: `ReductionSignature → ReducedNode`
- canonical instances of subtrees
Workflow:
1. compute signature
2. check if already exists
3. reuse existing node or create new one
This ensures:
- structural sharing
- no duplicate equivalent subtrees
## Reduced nodes
`ReducedNode` represents:
- canonical subtree
- aggregated value counts
- canonical children
It supports:
- merging local counts
- verifying structural consistency
At this stage:
- structure is canonical
- still mutable (internally)
## Stage 3: Compilation (freezing)
The reduced trie is converted into a **CompiledNode** structure.
### CompiledNode characteristics
- immutable
- array-based storage
- optimized for fast lookup
Fields:
- `char[] edgeLabels`
- `CompiledNode[] children`
- `V[] orderedValues`
- `int[] orderedCounts`
## Lookup algorithm
Runtime lookup:
1. traverse trie using `edgeLabels` (matching characters from the end of the word toward the beginning)
2. binary search per node
3. retrieve values
4. apply patch command
Properties:
- O(length of word)
- low memory overhead
- minimal memory allocation during lookup; patch application produces the resulting string
## Deterministic ordering
Value ordering is deterministic and stable:
1. higher frequency first
2. shorter string first
3. lexicographically smaller
4. insertion order
This guarantees:
- reproducible builds
- stable query results
- predictable ranking
## Reduction modes
Reduction modes control how local descriptors are chosen.
### Ranked mode
```
MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
```
- preserves full semantics
- safest option
- recommended default
### Unordered mode
```
MERGE_SUBTREES_WITH_EQUIVALENT_UNORDERED_GET_ALL_RESULTS
```
- ignores ordering
- higher compression
- slightly weaker semantics
### Dominant mode
```
MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS
```
- keeps only dominant result
- highest compression
- may lose alternative candidates
## Trade-offs
| Aspect | Ranked | Unordered | Dominant |
|---------------|--------|----------|----------|
| Compression | Medium | High | Highest |
| Accuracy | High | Medium | Lower |
| getAll() | Full | Partial | Limited |
| get() | Exact | Exact | Heuristic|
## Deserialization model
Binary loading uses:
- `NodeData` as intermediate representation
- reconstruction of `CompiledNode`
This separates:
- I/O format
- in-memory structure
## Why this architecture works
Radixor achieves:
### Compactness
- subtree sharing
- efficient encoding
- compressed binary output
### Performance
- array-based lookup
- no runtime reduction
- minimal branching
### Flexibility
- configurable reduction strategies
- multiple result support
- dictionary-driven behavior
### Determinism
- stable ordering
- canonical signatures
- reproducible builds
## Design philosophy
The architecture reflects a few key principles:
- separate build-time complexity from runtime simplicity
- encode semantics explicitly (not implicitly in code)
- favor deterministic behavior over heuristic shortcuts
- allow controlled trade-offs between size and fidelity
## When to tune reduction
You should consider changing reduction mode when:
- binary size is too large
- memory footprint must be minimized
- only single-result stemming is needed
Otherwise:
**use ranked mode by default**
## Next steps
- [Quick start](quick-start.md)
- [Programmatic usage](programmatic-usage.md)
- [CLI compilation](cli-compilation.md)
- [Dictionary format](dictionary-format.md)
## Summary
Radixor’s architecture is built around:
- patch-command tries
- canonical subtree reduction
- immutable compiled structures
This design allows the system to remain:
- fast
- compact
- deterministic
- adaptable
while still supporting advanced use cases such as:
- ambiguity-aware stemming
- dictionary evolution
- controlled trade-offs between size and behavior

209
docs/architecture.md Normal file
View File

@@ -0,0 +1,209 @@
# Architecture
This document explains the structural architecture of **Radixor**: what data is stored, how it flows through the build pipeline, and how runtime lookup works once a compiled trie has been produced.
## The central idea
Radixor does not store final stems directly as a large flat lookup table. Instead, it stores **patch commands** that describe how a word form should be transformed into a canonical stem.
For example, if a dictionary states that `running` should reduce to `run`, the final runtime artifact does not need to store a full redundant `running -> run` output string entry in the simplest possible form. It can store a compact transformation command that expresses how to turn the source form into the target form.
That matters because many words share similar transformation patterns. Once those mappings are organized in a trie and compiled into a canonical structure, the result is much smaller and more reusable than a naive direct-output table.
## End-to-end build flow
The full build-time flow is:
```text
Dictionary -> Mutable trie -> Reduced trie -> Compiled trie
```
Each stage has a different purpose.
### Dictionary input
The textual dictionary groups known word forms under a canonical stem:
```text
run running runs ran
connect connected connecting connection
```
The first token is the canonical stem. The following tokens are known variants.
### Patch-command generation
Each variant is converted into a patch command that transforms the variant into the stem.
Conceptually:
```text
running -> <patch> -> run
runs -> <patch> -> run
ran -> <patch> -> run
```
If `storeOriginal` is enabled, the stem itself is also inserted using a canonical no-op patch.
### Mutable trie construction
Those patch-command values are inserted into a mutable trie keyed by the source surface form.
### Reduction
Equivalent subtrees are merged into canonical reduced nodes.
### Compilation
The reduced structure is frozen into an immutable compiled trie optimized for runtime lookup.
## Why a trie is used
A trie is useful because many word forms share structural fragments. Instead of storing each word independently, the trie reuses paths and organizes lookup by character traversal.
A trie node can contain:
- outgoing edges,
- one or more ordered values,
- counts aligned with those values.
This is why the structure can represent both:
- a single preferred result,
- multiple competing results for the same key.
## Stage 1: Mutable construction
The mutable build-time structure is created by `FrequencyTrie.Builder`.
This stage is optimized for insertion rather than runtime lookup. As dictionary data is added, the builder accumulates:
- child edges,
- local values,
- local frequencies of those values.
Those frequencies are not incidental metadata. They later influence both result ordering and, depending on reduction mode, the semantic identity of subtrees during reduction.
### Why the build-time form is mutable
The builder must be easy to extend and easy to aggregate into. That is the opposite of what a runtime lookup structure needs.
Build-time priorities are:
- flexibility,
- accumulation of counts,
- structural growth.
Runtime priorities are:
- compactness,
- immutability,
- fast lookup.
Radixor therefore keeps construction and runtime representation strictly separate.
## What a compiled node contains
After reduction and freezing, the runtime structure uses immutable compiled nodes.
A compiled node stores:
- `char[] edgeLabels`
- child-node references aligned with those labels
- ordered value arrays
- aligned count arrays
This array-based form is compact and efficient for lookup.
## Runtime lookup model
At runtime, lookup is conceptually simple:
1. traverse the compiled trie by the input key,
2. reach the node addressed by that key,
3. retrieve one or more stored patch commands,
4. apply the chosen patch command to the original word.
The trie itself does not create the final stem string. It selects the stored transformation command. `PatchCommandEncoder.apply(...)` then performs the actual transformation.
That separation is architecturally important:
- the trie is responsible for **selection**,
- patch application is responsible for **transformation**.
## `get()` and `getAll()`
The runtime API exposes two complementary views of the addressed node.
### `get()`
`get()` returns the locally preferred value stored at that node.
Preference is deterministic:
1. higher local frequency wins,
2. shorter textual representation wins,
3. lexicographically lower textual representation wins,
4. stable first-seen order acts as the final tie-breaker.
### `getAll()`
`getAll()` returns all locally stored values in deterministic ranked order.
This is what allows Radixor to preserve ambiguity explicitly instead of forcing every key into a single answer.
## Why multiple results can exist
Some stemming systems discard ambiguity early because they insist on returning exactly one answer.
Radixor does not require that simplification. If multiple plausible patch commands exist for a key, the compiled trie can preserve them and the runtime API can expose them.
That is useful when downstream logic wants to:
- inspect ambiguity,
- preserve alternatives for retrieval,
- apply later ranking or domain-specific selection.
## Why compiled artifacts are compact
The final compiled trie can be much smaller than the original dictionary for several reasons working together:
- patch commands are compact,
- trie paths reuse shared structure,
- reduction merges equivalent subtrees,
- binary persistence stores the already reduced form,
- GZip compression is applied on top of the binary format.
This is why a very large dictionary can still produce a manageable deployable runtime artifact.
## Why preparation can still use more memory
The compactness of the final artifact should not be confused with the memory usage of preparation.
Before reduction has completed, the mutable build-time structure must exist in memory. For large dictionaries, that temporary preparation cost can be noticeably higher than the size of the final persisted artifact or the loaded compiled trie.
That is why the preferred operational model is usually:
- compile offline,
- persist the compiled artifact,
- load the finished artifact in runtime services.
## Determinism as a design principle
Radixor favors deterministic behavior throughout the pipeline.
This appears in:
- lowercased dictionary parsing,
- stable value ordering,
- sorted child descriptors,
- canonical reduction signatures,
- reproducible compiled lookup behavior.
Determinism matters not only for tests, but also for operational trust. It makes stemming behavior explainable and reproducible across builds and environments.
## Continue with
- [Reduction Semantics](reduction-semantics.md)
- [Programmatic usage](programmatic-usage.md)
- [CLI compilation](cli-compilation.md)

109
docs/assets/stylesheets/extra.css Normal file
View File

@@ -0,0 +1,109 @@
/* Compact technical typography for Radixor */
:root {
--md-text-font: "Inter", "Segoe UI", "Roboto", "Helvetica Neue", Arial, sans-serif;
}
/* Hide page title only on the landing page */
.visually-hidden {
display: none;
}
/* Main article text */
.md-typeset {
font-size: 0.78rem;
line-height: 1.3;
}
/* Paragraph spacing */
.md-typeset p,
.md-typeset ul,
.md-typeset ol,
.md-typeset dl,
.md-typeset blockquote {
margin-top: 0.45em;
margin-bottom: 0.45em;
}
/* Headings */
.md-typeset h1 {
margin: 0 0 0.7rem;
font-size: 1.8rem;
line-height: 1.15;
}
.md-typeset h2 {
margin: 1.2rem 0 0.55rem;
font-size: 1.3rem;
line-height: 1.2;
}
.md-typeset h3 {
margin: 1rem 0 0.45rem;
font-size: 1.05rem;
line-height: 1.25;
}
.md-typeset h4,
.md-typeset h5,
.md-typeset h6 {
margin: 0.85rem 0 0.35rem;
line-height: 1.25;
}
/* Lists */
.md-typeset li {
margin-bottom: 0.15em;
}
.md-typeset ul,
.md-typeset ol {
padding-left: 1.1rem;
}
/* Tables */
.md-typeset table:not([class]) td,
.md-typeset table:not([class]) th {
padding: 0.45rem 0.7rem;
}
/* Code blocks */
.md-typeset pre > code {
font-size: 0.72rem;
line-height: 1.4;
}
/* Inline code */
.md-typeset code {
font-size: 0.72rem;
}
/* Navigation density */
.md-nav__item .md-nav__link {
margin-top: 0.12rem;
margin-bottom: 0.12rem;
}
.md-sidebar__scrollwrap {
padding-top: 0.3rem;
padding-bottom: 0.3rem;
}
/* Slightly narrower content rhythm */
.md-content__inner {
margin-top: 0.6rem;
padding-bottom: 1.2rem;
}
/* Admonitions more compact */
.md-typeset .admonition,
.md-typeset details {
margin: 0.8rem 0;
}
/* Optional: use a bit wider content area on large screens */
@media screen and (min-width: 76.25em) {
.md-grid {
max-width: 68rem;
}
}

111
docs/benchmarking.md
View File

@@ -2,22 +2,36 @@
Radixor includes a JMH benchmark suite for both the internal algorithmic core and a side-by-side English comparison against the Snowball Porter stemmer family.
This document explains what is benchmarked, how to run it, and how to interpret the results responsibly.
This document explains what is benchmarked, how to run the suite, and how benchmark results should be interpreted.
## Scope
The benchmark suite currently covers two categories:
- Radixor core operations
- English stemmer comparison on the same token workload
- Radixor core operations,
- English stemmer comparison on the same token workload.
The comparison benchmark processes the same deterministic English token stream through:
- Radixor with bundled `US_UK_PROFI`
- Snowball original Porter
- Snowball English, commonly referred to as Porter2
- Radixor with bundled `US_UK_PROFI`,
- Snowball original Porter,
- Snowball English, commonly referred to as Porter2.
The purpose of the comparison is throughput measurement on identical input. It is not intended to prove linguistic equivalence between the compared stemmers.
The purpose of the comparison is throughput measurement on identical input. It is not intended to demonstrate linguistic equivalence between the compared stemmers.
## How to read the published numbers
Two kinds of benchmark numbers are relevant in the project.
### Reference measurements
The detailed benchmark snapshot documented on this page comes from a controlled run on a Ryzen 5 system. Those numbers are the best reference point for understanding absolute throughput under a known local benchmark environment.
### Published badge figures
The benchmark badge metadata published through GitHub Pages is generated in the GitHub-hosted container environment. That environment is convenient for continuous publication, but it is not the right place to treat absolute throughput values as stable across time. CPU scheduling, shared-host variability, and container-level noise can materially affect raw numbers from run to run.
For that reason, the published badge values should be treated primarily as a compact status surface. They are useful for observing broad trends and relative positioning, but not as the authoritative source for precise absolute throughput claims.
## Current snapshot
@@ -28,12 +42,25 @@ A recent JMH run on JDK 21.0.10 with JMH 1.37, one thread, three warmup iteratio
| About 12,000 generated tokens | 30.99 M tokens/s | 8.21 M tokens/s | 5.46 M tokens/s |
| About 60,000 generated tokens | 32.25 M tokens/s | 8.02 M tokens/s | 5.11 M tokens/s |
On that workload, Radixor is approximately:
On that workload, Radixor measured approximately:
- 4 times faster than Snowball original Porter
- 6 times faster than Snowball English
- 4 times the throughput of Snowball original Porter,
- 6 times the throughput of Snowball English.
These values are workload- and environment-dependent. Treat them as measured results for the documented benchmark setup, not as universal constants.
These values are workload-dependent and environment-dependent. They should be read as measured results for the documented setup, not as universal constants.
## Interpreting the relative result
Although the absolute numbers can move across environments, the throughput relationship between Radixor and the compared Porter-family stemmers has remained broadly stable in practical measurements. In particular, the comparison against Snowball original Porter is consistently in the rough range of about four to one in Radixor’s favor.
That relative behavior is more informative than any single absolute figure. It reflects a real architectural difference rather than a cosmetic benchmark artifact.
Radixor is built around a compiled patch-command trie that resolves the result through a direct lookup and patch application path. In contrast, classic rule-based stemmers such as the Porter family follow a different operational model. The result is that Radixor combines two properties that do not often appear together:
- dictionary-driven compiled lookup performance,
- the ability to generalize beyond explicitly listed word forms instead of behaving like a pure closed-form dictionary lookup table.
Within that design space, the measured throughput profile is strong enough to place Radixor among the fastest known practical implementations of this kind, while still supporting stemming of previously unseen forms. That should still be read as a carefully bounded engineering statement, not as an absolute claim over every possible stemmer architecture or benchmark scenario.
## Benchmark classes
@@ -41,9 +68,9 @@ The main benchmark classes are under `src/jmh/java/org/egothor/stemmer/benchmark
Relevant classes include:
- `FrequencyTrieLookupBenchmark`
- `FrequencyTrieCompilationBenchmark`
- `EnglishStemmerComparisonBenchmark`
- `FrequencyTrieLookupBenchmark`,
- `FrequencyTrieCompilationBenchmark`,
- `EnglishStemmerComparisonBenchmark`.
The English comparison benchmark uses the bundled Radixor English resource and the official Snowball Java distribution integrated into the JMH source set.
@@ -53,10 +80,10 @@ The English comparison benchmark uses a deterministic generated corpus rather th
The workload intentionally mixes:
- simple inflections
- common derivational forms
- US and UK spelling families
- lexical forms appropriate for `US_UK_PROFI`
- simple inflections,
- common derivational forms,
- US and UK spelling families,
- lexical forms appropriate for `US_UK_PROFI`.
This design keeps runs reproducible across environments and avoids accidental drift caused by changing external corpora.
@@ -78,42 +105,42 @@ Run only the English comparison benchmark:
JMH reports are written to:
- `build/reports/jmh/jmh-results.txt`
- `build/reports/jmh/jmh-results.csv`
- `build/reports/jmh/jmh-results.txt`,
- `build/reports/jmh/jmh-results.csv`.
The text report is convenient for human review. The CSV report is more useful for CI archiving, historical tracking, and external processing.
## Interpreting results
## Interpreting results responsibly
Benchmark numbers should be read with care.
Benchmark numbers should always be read with care.
Important factors include:
- CPU model and frequency behavior
- thermal throttling
- JVM vendor and version
- system background load
- operating-system scheduling noise
- benchmark parameter changes
- CPU model and frequency behavior,
- thermal throttling,
- JVM vendor and version,
- system background load,
- operating-system scheduling noise,
- benchmark parameter changes.
For meaningful comparison, keep these stable:
- hardware or VM class
- JDK version
- benchmark parameters
- thread count
- benchmark source revision
- hardware or VM class,
- JDK version,
- benchmark parameters,
- thread count,
- benchmark source revision.
If a regression is suspected, repeat the run and compare against the previous CSV output rather than relying on a single measurement.
If a regression is suspected, repeat the run and compare against previous CSV output rather than relying on a single measurement.
## Regression tracking
The recommended regression workflow is:
1. archive `jmh-results.csv`
2. compare the same benchmark names across runs
3. compare only like-for-like environments
4. investigate sustained regressions rather than one-off noise
1. archive `jmh-results.csv`,
2. compare the same benchmark names across runs,
3. compare only like-for-like environments,
4. investigate sustained regressions rather than one-off noise.
For public reporting, the README should keep only the condensed benchmark summary, while detailed benchmark methodology and interpretation should remain in this document.
@@ -125,8 +152,8 @@ Radixor uses a compiled patch-command trie driven by dictionary data. Snowball P
Because of that, the comparison should be understood as:
- equal input workload
- different stemming strategies
- measured throughput, not semantic identity
- equal input workload,
- different stemming strategies,
- measured throughput rather than semantic identity.
That distinction matters whenever performance claims are discussed in documentation or release notes.
That distinction matters whenever performance claims are discussed in documentation, release notes, or badge summaries.

260
docs/built-in-languages.md
View File

@@ -1,15 +1,8 @@
# Built-in Languages
Radixor provides a set of **bundled stemmer dictionaries** that can be loaded directly without preparing custom data.
These built-in resources are useful for:
- quick integration
- testing and evaluation
- reference behavior
- prototyping search pipelines
Radixor provides a set of bundled stemmer dictionaries that can be loaded directly without preparing custom lexical data first.
These resources are intended as practical default dictionaries for common use. They provide a solid starting point for evaluation, integration, and general-purpose stemming workloads, while still fitting naturally into workflows where the bundled baseline is later refined, extended, or replaced by a custom dictionary.
## Overview
@@ -19,34 +12,30 @@ Bundled dictionaries are exposed through:
StemmerPatchTrieLoader.Language
```
They are packaged with the library and loaded from the classpath.
They are packaged with the library as text resources and compiled into a `FrequencyTrie<String>` when loaded.
## Supported languages
The following language identifiers are currently available:
| Language | Enum constant | Description |
|----------|------------------|------------------------------|
| Danish | `DA_DK` | Danish |
| German | `DE_DE` | German |
| Spanish | `ES_ES` | Spanish |
| French | `FR_FR` | French |
| Italian | `IT_IT` | Italian |
| Dutch | `NL_NL` | Dutch |
| Norwegian| `NO_NO` | Norwegian |
| Portuguese| `PT_PT` | Portuguese |
| Russian | `RU_RU` | Russian |
| Swedish | `SV_SE` | Swedish |
| English | `US_UK` | Standard English |
| English | `US_UK_PROFI` | Extended English dictionary |
The following bundled language identifiers are currently available:
| Language | Enum constant | Notes |
|---|---|---|
| Danish | `DA_DK` | Bundled general-purpose dictionary |
| German | `DE_DE` | Bundled general-purpose dictionary |
| Spanish | `ES_ES` | Bundled general-purpose dictionary |
| French | `FR_FR` | Bundled general-purpose dictionary |
| Italian | `IT_IT` | Bundled general-purpose dictionary |
| Dutch | `NL_NL` | Bundled general-purpose dictionary |
| Norwegian | `NO_NO` | Bundled general-purpose dictionary |
| Portuguese | `PT_PT` | Bundled general-purpose dictionary |
| Russian | `RU_RU` | Currently supplied in normalized transliterated form |
| Swedish | `SV_SE` | Bundled general-purpose dictionary |
| English | `US_UK` | Standard English dictionary |
| English | `US_UK_PROFI` | Extended English dictionary |
## Basic usage
Load a bundled stemmer:
Load a bundled stemmer like this:
```java
import java.io.IOException;
@@ -57,194 +46,177 @@ import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class BuiltInExample {
public static void main(String[] args) throws IOException {
FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
private BuiltInExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
);
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
}
}
```
The loader reads the bundled dictionary resource, parses the textual entries, derives patch-command mappings, and compiles the result into a read-only trie.
## Example: stemming with `US_UK_PROFI`
```java
import java.io.IOException;
import org.egothor.stemmer.*;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class EnglishExample {
public static void main(String[] args) throws IOException {
FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
private EnglishExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
);
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
String word = "running";
String patch = trie.get(word);
String stem = PatchCommandEncoder.apply(word, patch);
final String word = "running";
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch);
System.out.println(word + " -> " + stem);
}
}
```
## `US_UK` and `US_UK_PROFI`
## `US_UK` vs `US_UK_PROFI`
Radixor currently provides two bundled English variants.
### `US_UK`
* smaller dictionary
* faster load time
* suitable for lightweight use cases
`US_UK` is the lighter-weight bundled English resource. It is suitable where a smaller default dictionary is preferred and maximal lexical coverage is not the primary goal.
### `US_UK_PROFI`
* larger and more complete dataset
* better coverage of word forms
* improved stemming quality
* slightly larger memory footprint
`US_UK_PROFI` is the more extensive bundled English resource. It offers broader lexical coverage and is the better default for most applications that want stronger out-of-the-box behavior.
### Recommendation
Use:
For most English-language deployments, prefer:
```
```text
US_UK_PROFI
```
for most applications unless memory constraints are strict.
Use `US_UK` when a smaller bundled baseline is more appropriate.
## Intended role of bundled dictionaries
Bundled dictionaries should be understood as **general-purpose default resources**.
## How bundled dictionaries are loaded
They are a good fit when:
Internally:
- a supported language is already available,
- immediate usability matters,
- a reasonable baseline is sufficient,
- the goal is evaluation, prototyping, or straightforward integration.
- dictionaries are stored as text resources
- parsed using `StemmerDictionaryParser`
- compiled into a trie at load time
They are also well suited to staged refinement workflows in which the bundled base is loaded first, then extended with domain-specific vocabulary, and finally persisted as a custom binary artifact.
This means:
## Character representation
- first load includes parsing + compilation cost
- subsequent usage is fast
The current bundled resources follow a pragmatic normalization convention.
At present, bundled dictionaries are supplied in normalized plain-ASCII form. For some languages, this is simply a lightweight maintenance convention. For others, especially languages commonly written in another script, it reflects a transliterated lexical resource. Russian is the clearest example in the current bundled set.
This convention belongs to the supplied dictionary resources, not to the core stemming model. The parser reads UTF-8 text, the dictionary model works with ordinary Java strings, and the trie and patch-command mechanism operate on general character sequences. In practical terms, the architecture is compatible with native-script dictionaries when suitable lexical resources are available.
## When to use bundled languages
## When to prefer custom dictionaries
Bundled dictionaries are suitable when:
A custom dictionary is usually the better choice when:
- you need quick results without preparing custom data
- you are prototyping or experimenting
- your language requirements match the provided datasets
## When to use custom dictionaries
You should prefer custom dictionaries when:
- domain-specific vocabulary is important
- accuracy requirements are high
- you need full control over stemming behavior
Typical examples:
- technical terminology
- product catalogs
- biomedical text
- legal or financial language
- domain-specific vocabulary materially affects stemming quality,
- lexical coverage must be controlled more precisely,
- a stronger language resource is available than the bundled baseline,
- native-script support is needed beyond the currently bundled resources.
Typical examples include:
- technical terminology,
- biomedical language,
- legal or financial vocabulary,
- organization-specific product and process names,
- language resources maintained in native scripts.
## Production recommendation
For production systems:
For production systems, the most robust workflow is usually:
1. Load a bundled dictionary
2. Extend it with domain-specific terms (optional)
3. Compile it into a binary `.radixor.gz` file
4. Deploy the compiled artifact
5. Load it using `loadBinary(...)`
1. start from a bundled dictionary when it is suitable,
2. extend it with domain-specific forms if needed,
3. compile or rebuild it into a binary `.radixor.gz` artifact,
4. deploy that compiled artifact,
5. load it at runtime using `loadBinary(...)`.
This avoids:
This avoids repeated startup parsing and makes the deployed stemming behavior explicit and versionable.
- runtime parsing overhead
- repeated compilation
- startup latency
## Example workflow
## Example refinement workflow
```java
// 1. Load bundled dictionary
FrequencyTrie<String> base = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
);
import java.io.IOException;
import java.nio.file.Path;
// 2. Modify (optional)
FrequencyTrie.Builder<String> builder =
FrequencyTrieBuilders.copyOf(
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.FrequencyTrieBuilders;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class BundledRefinementExample {
private BundledRefinementExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> base = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
final FrequencyTrie.Builder<String> builder = FrequencyTrieBuilders.copyOf(
base,
String[]::new,
ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
)
);
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS));
builder.put("microservices", PatchCommandEncoder.NOOP_PATCH);
builder.put("microservices", "Na");
// 3. Compile
FrequencyTrie<String> compiled = builder.build();
final FrequencyTrie<String> compiled = builder.build();
// 4. Save
StemmerPatchTrieBinaryIO.write(compiled, Path.of("english-custom.radixor.gz"));
StemmerPatchTrieBinaryIO.write(compiled, Path.of("english-custom.radixor.gz"));
}
}
```
## Extending language support
The built-in set is intentionally a practical baseline rather than a closed catalog. High-quality dictionaries for additional languages, improved language coverage, and stronger native-script resources are all natural extension paths for the project.
## Limitations
* bundled dictionaries are **general-purpose**
* they may not reflect:
* domain-specific usage
* rare or specialized vocabulary
* organization-specific terminology
What matters most is not only the number of entries, but the quality, consistency, and operational usefulness of the lexical resource being added.
## Next steps
* [Quick start](quick-start.md)
* [Dictionary format](dictionary-format.md)
* [CLI compilation](cli-compilation.md)
* [Programmatic usage](programmatic-usage.md)
- [Quick start](quick-start.md)
- [Dictionary format](dictionary-format.md)
- [CLI compilation](cli-compilation.md)
- [Programmatic usage](programmatic-usage.md)
## Summary
Radixor’s built-in language support provides:
* immediate usability
* reference datasets
* a starting point for customization
For production systems, they are best used as:
* a baseline
* a seed for further extension
* a source for compiled deployment artifacts
Radixor’s built-in language support provides immediate usability, practical default dictionaries, and a strong starting point for custom refinement. The current bundled resources follow a pragmatic normalization convention, while the underlying architecture remains well suited to richer language resources and future extensions.

359
docs/cli-compilation.md
View File

@@ -1,303 +1,248 @@
# CLI Compilation
Radixor provides a command-line tool for compiling dictionary files into compact, production-ready binary stemmer tables.
Radixor provides a command-line compiler for turning line-oriented dictionary files into compact binary stemmer artifacts.
This is the recommended workflow for deployment environments, as it separates:
This is the preferred preparation workflow when stemming should run against an already compiled artifact rather than against raw dictionary input. The CLI reads the dictionary, derives patch commands, builds a mutable trie, applies the selected subtree reduction strategy, and writes the final compiled trie in the project binary format under GZip compression. The result is a deployment-ready `.radixor.gz` file that can be loaded directly by application code.
- dictionary preparation (offline)
- stemming execution (runtime)
## What the CLI does
The `Compile` tool performs the following steps:
1. reads the input dictionary in the standard Radixor stemmer format,
2. parses each line into a canonical stem and its known variants,
3. converts variants into patch commands,
4. builds a mutable trie of patch-command values,
5. applies the configured reduction mode,
6. writes the compiled trie as a GZip-compressed binary artifact.
## Overview
The `Compile` tool:
1. reads a line-oriented dictionary file
2. converts word–stem pairs into patch commands
3. builds a trie structure
4. applies subtree reduction
5. writes a compressed binary artifact
The output is a `.radixor.gz` file suitable for fast runtime loading.
This workflow is intentionally aligned with the same dictionary semantics used elsewhere in the library. Remarks introduced by `#` or `//` are supported through the shared dictionary parser.
## Basic usage
```bash
java org.egothor.stemmer.Compile \
--input ./data/stemmer.txt \
--output ./build/english.radixor.gz \
--reduction-mode MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS \
--store-original \
--overwrite
--input ./data/stemmer.txt \
--output ./build/english.radixor.gz \
--reduction-mode MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS \
--store-original \
--overwrite
```
## Supported arguments
The CLI supports the following arguments:
## Required arguments
```text
--input <file>
--output <file>
--reduction-mode <mode>
[--store-original]
[--dominant-winner-min-percent <1..100>]
[--dominant-winner-over-second-ratio <1..n>]
[--overwrite]
[--help]
```
### `--input`
### `--input <file>`
Path to the source dictionary file.
* must be in the [dictionary format](dictionary-format.md)
* must be readable
* UTF-8 encoding is expected
```
--input ./data/stemmer.txt
```
### `--output`
Path to the output binary file.
* parent directories are created automatically
* output is written as **GZip-compressed binary**
```
--output ./build/english.radixor.gz
```
## Optional arguments
### `--reduction-mode`
Controls how aggressively the trie is reduced during compilation.
Available values:
* `MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS`
* `MERGE_SUBTREES_WITH_EQUIVALENT_UNORDERED_GET_ALL_RESULTS`
* `MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`
The file must use the standard line-oriented dictionary format. Each non-empty logical line starts with the canonical stem and may contain zero or more variants. The parser expects UTF-8 input, lowercases it using `Locale.ROOT`, and ignores trailing remarks introduced by `#` or `//`.
Example:
```text
--input ./data/stemmer.txt
```
### `--output <file>`
Path to the output binary artifact.
The output file is written as a GZip-compressed binary trie. Parent directories are created automatically when needed.
Example:
```text
--output ./build/english.radixor.gz
```
### `--reduction-mode <mode>`
Selects the subtree reduction strategy used during compilation.
Supported values are:
- `MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS`
- `MERGE_SUBTREES_WITH_EQUIVALENT_UNORDERED_GET_ALL_RESULTS`
- `MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`
Example:
```text
--reduction-mode MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
```
#### Recommendation
Use:
```
MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
```
This provides:
* safe behavior
* deterministic ordering
* good compression
This argument is required.
### `--store-original`
Stores the stem itself as a no-op mapping.
When this flag is present, the canonical stem itself is inserted using the no-op patch command.
```
```text
--store-original
```
Effect:
This is usually a sensible default for real dictionaries because it ensures that canonical forms are directly representable in the compiled trie rather than relying only on their variants.
* ensures that canonical forms are always resolvable
* improves robustness in real-world inputs
### `--dominant-winner-min-percent <1..100>`
Recommended for most use cases.
Sets the minimum winner percentage used by dominant-result reduction settings.
Example:
```text
--dominant-winner-min-percent 75
```
This option matters primarily when `--reduction-mode` is `MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`. The default value is `75`.
### `--dominant-winner-over-second-ratio <1..n>`
Sets the minimum winner-over-second ratio used by dominant-result reduction settings.
Example:
```text
--dominant-winner-over-second-ratio 3
```
This option also matters primarily for `MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`. The default value is `3`.
### `--overwrite`
Allows overwriting an existing output file.
Allows the CLI to replace an already existing output file.
```
```text
--overwrite
```
Without this flag:
Without this flag, compilation fails when the output path already exists.
* compilation fails if the output file already exists
### `--help`
Prints usage help and exits successfully.
```text
--help
```
## Reduction strategy explained
The short form `-h` is also supported.
Reduction merges semantically equivalent subtrees to reduce memory and file size.
## Reduction modes in practice
Trade-offs:
Reduction mode is not only a storage decision. It also influences what semantics are preserved when the mutable trie is compiled into its canonical read-only form.
| Mode | Compression | Behavioral fidelity |
| --------- | ----------- | ------------------- |
| Ranked | Medium | High |
| Unordered | High | Medium |
| Dominant | Highest | Lower (heuristic) |
### Ranked `getAll()` equivalence
### Ranked (recommended)
`MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS` merges subtrees whose `getAll()` results remain equivalent for every reachable key suffix and whose local result ordering is the same.
* preserves full `getAll()` ordering
* safest and most predictable
This is the best general-purpose choice when result ordering and ambiguity handling matter. It preserves ranked multi-result semantics while still achieving useful structural reduction.
### Unordered
This is the recommended default for most users.
* ignores ordering differences
* higher compression, but less precise semantics
### Unordered `getAll()` equivalence
### Dominant
`MERGE_SUBTREES_WITH_EQUIVALENT_UNORDERED_GET_ALL_RESULTS` also uses `getAll()`-level equivalence, but it ignores local ordering differences in addition to absolute frequencies.
* focuses on the most frequent result
* useful when only `get()` is relevant
* may lose secondary candidates
This can yield stronger reduction, but it also weakens the precision of ordered multi-result semantics.
Choose this mode only when the application does not depend on the ordering of alternative results.
### Dominant `get()` equivalence
## Output format
`MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS` focuses on preserving preferred-result semantics for `get()`, subject to dominance thresholds.
The compiled file:
If a node does not satisfy the configured dominance constraints, compilation falls back to ranked `getAll()` semantics for that node to avoid unsafe over-reduction.
* is a binary representation of the trie
* uses **GZip compression**
* is optimized for:
This mode is most suitable when the application primarily consumes the preferred result and does not rely on preserving richer ambiguity information.
* fast loading
* minimal memory footprint
## Recommended usage patterns
Typical properties:
### Use offline preparation
* small file size
* fast deserialization
* no runtime preprocessing required
The CLI is best used as a preparation step during packaging, deployment, or controlled artifact generation. This keeps compilation outside the runtime startup path and allows services to load only the finished binary trie.
### Treat compiled files as versioned assets
A `.radixor.gz` file should be handled as a versioned output artifact. It represents a specific dictionary state, a specific reduction mode, and, where relevant, specific dominant-result thresholds.
### Choose reduction mode deliberately
The ranked `getAll()` mode is the safest default. The unordered and dominant modes should be chosen only when their trade-offs are acceptable for the consuming application.
### Expect memory pressure during preparation, not runtime
Compilation is usually a one-time step and is generally fast. The more important operational consideration is memory usage during preparation, because the dictionary-derived mutable structure exists before reduction compacts it into the final read-only trie. This is especially relevant for very large source dictionaries.
## Example workflow
### 1. Prepare dictionary
### 1. Prepare a dictionary
```
```text
run running runs ran
connect connected connecting
```
### 2. Compile
### 2. Compile it
```bash
java org.egothor.stemmer.Compile \
--input ./data/stemmer.txt \
--output ./build/english.radixor.gz \
--reduction-mode MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS \
--store-original
--input ./data/stemmer.txt \
--output ./build/english.radixor.gz \
--reduction-mode MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS \
--store-original
```
### 3. Use in application
### 3. Load it in an application
```java
FrequencyTrie<String> trie =
StemmerPatchTrieLoader.loadBinary("english.radixor.gz");
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
final FrequencyTrie<String> trie =
StemmerPatchTrieLoader.loadBinary("english.radixor.gz");
```
## Exit codes and error handling
The CLI uses three exit outcomes:
## Error handling
- `0` for success,
- `1` for processing failures such as I/O or compilation errors,
- `2` for invalid command-line usage.
The CLI reports:
When argument parsing fails, the CLI prints the error message, prints the usage summary, and exits with usage error status.
* missing input file
* invalid arguments
* I/O failures
* parsing errors
When compilation fails during processing, the CLI prints a `Compilation failed: ...` message to standard error and exits with processing error status.
Typical exit codes:
Examples of failure conditions include:
* `0` – success
* non-zero – failure
Error details are printed to standard error.
## Performance considerations
### Compilation
* typically CPU-bound
* depends on dictionary size and reduction mode
### Output size
* depends on:
* dictionary completeness
* reduction strategy
* can vary significantly between modes
### Runtime impact
* compiled tries are optimized for:
* fast lookup
* low allocation
* predictable latency
## Best practices
### Use offline compilation
* compile dictionaries during build or deployment
* do not compile on application startup
### Version your artifacts
* treat `.radixor.gz` files as versioned assets
* store them alongside application releases
### Choose reduction mode deliberately
* use **ranked** for correctness
* use **dominant** only if you fully understand the trade-offs
### Keep dictionaries clean
* better input → better compiled output
* avoid noise and inconsistencies
## Integration tips
* store compiled files under `resources/` or a dedicated directory
* load them once and reuse the trie instance
* avoid repeated loading in frequently executed code paths (for example, per-request processing)
- missing required arguments,
- unknown arguments,
- invalid integer values for dominant thresholds,
- missing input files,
- unreadable input,
- existing output file without `--overwrite`,
- general I/O failures during reading or writing.
## Relation to programmatic usage
The CLI and the programmatic API implement the same conceptual preparation step. The CLI is the operationally convenient choice when you want a ready-made binary artifact. The programmatic API is the better fit when compilation must be integrated directly into custom Java workflows.
## Next steps
* [Dictionary format](dictionary-format.md)
* [Programmatic usage](programmatic-usage.md)
* [Quick start](quick-start.md)
## Summary
The `Compile` CLI is the bridge between:
* human-readable dictionary data
* optimized runtime stemmer tables
It enables a clean separation between:
* data preparation
* runtime execution
and is the preferred way to prepare Radixor for production use.
- [Dictionary format](dictionary-format.md)
- [Quick start](quick-start.md)
- [Programmatic usage](programmatic-usage.md)
- [Architecture and reduction](architecture-and-reduction.md)

308
docs/dictionary-format.md
View File

@@ -1,253 +1,229 @@
# Dictionary Format
Radixor uses a simple, line-oriented dictionary format to define mappings between **word forms** and their **canonical stems**.
Radixor uses a simple line-oriented dictionary format designed for practical stemming workflows.
This format is intentionally minimal, language-agnostic, and easy to generate from existing linguistic resources or corpora.
Each logical line describes one canonical stem and zero or more known word variants that should reduce to that stem. The format is intentionally lightweight, easy to maintain in source control, and directly consumable both by the programmatic loader and by the CLI compiler.
## Overview
## Core structure
Each logical line defines:
Each non-empty logical line has the following shape:
- one **canonical stem**
- zero or more **word variants** belonging to that stem
```
stem variant1 variant2 variant3 ...
```text
<stem> <variant1> <variant2> <variant3> ...
```
At compile time:
The first token is interpreted as the **canonical stem**. Every following token on the same line is interpreted as a **known variant** belonging to that stem.
- each variant is converted into a **patch command** transforming the variant into the stem
- the stem itself may optionally be stored as a **no-op mapping**
Example:
## Basic example
```
```text
run running runs ran
connect connected connecting connection
analyze analyzing analysed analyses
```
This defines:
In this example:
| Stem | Variants |
|----------|----------------------------------------|
| run | running, runs, ran |
| connect | connected, connecting, connection |
| analyze | analyzing, analysed, analyses |
- `run` is the canonical stem for `running`, `runs`, and `ran`,
- `connect` is the canonical stem for `connected`, `connecting`, and `connection`.
## Syntax rules
## How the loader interprets a line
### 1. Tokenization
When a dictionary is loaded through `StemmerPatchTrieLoader`, the loader processes each parsed line as follows:
- Tokens are separated by **whitespace**
- Multiple spaces and tabs are treated as a single separator
- Leading and trailing whitespace is ignored
1. the first token becomes the canonical stem,
2. every following token is treated as a variant,
3. each variant is converted into a patch command that transforms the variant into the stem,
4. if `storeOriginal` is enabled, the stem itself is also inserted using the canonical no-op patch command.
### 2. First token is the stem
This means the textual dictionary is not used directly at runtime. Instead, it is transformed into patch-command data and compiled into a reduced read-only trie.
- The **first token** on each line is always the canonical stem
- All following tokens are treated as variants of that stem
## Minimal valid lines
### 3. Case normalization
A line may consist of the stem only:
- All input is normalized to **lowercase using `Locale.ROOT`**
- Dictionaries should ideally already be lowercase to avoid ambiguity
```text
run
```
### 4. Empty lines
This is syntactically valid. It defines a stem entry with no explicit variants on that line.
- Empty lines are ignored
Whether such a line is operationally useful depends on how the dictionary is loaded:
### 5. Duplicate variants
- if `storeOriginal` is enabled, the stem itself is inserted as a no-op mapping,
- if `storeOriginal` is disabled, the line contributes no explicit variant mappings.
- Duplicate variants are allowed but have no additional effect
- Frequency is determined by occurrence across the entire dataset
## Whitespace rules
## Remarks (comments)
Tokens are separated by whitespace. Leading and trailing whitespace is ignored.
These lines are equivalent:
```text
run running runs ran
```
```text
run running runs ran
```
Tabs and repeated spaces are both accepted because tokenization is whitespace-based.
## Empty lines
Empty lines are ignored.
Example:
```text
run running runs ran
connect connected connecting
```
The blank line between entries has no effect.
## Remarks and comments
The parser supports both full-line and trailing remarks.
### Supported remark markers
Two remark markers are recognized:
- `#`
- `//`
### Examples
The earliest occurrence of either marker terminates the logical content of the line, and the remainder of that line is ignored.
```
run running runs ran # English verb forms
connect connected connecting // basic forms
Examples:
```text
run running runs ran # English verb forms
connect connected connecting // Common derived forms
```
Everything after the first occurrence of a remark marker is ignored.
This is also valid:
### Important note
Remark markers are not escaped. If `#` or `//` appear in a token, they will terminate the line.
## Storing the original form
When compiling, you may enable:
```
--store-original
```text
# This line is ignored completely
// This line is also ignored completely
```
This causes the stem itself to be stored using a **no-op patch command**.
## Case normalization
Input lines are normalized to lower case using `Locale.ROOT` before tokenization is processed into dictionary entries.
That means dictionary authors should treat the format as **case-insensitive at load time**. If a file contains uppercase or mixed-case tokens, they will be normalized during parsing.
Example:
```
run running runs
```text
Run Running Runs Ran
```
With `--store-original`, this implicitly includes:
is processed the same way as:
```
run -> run
```text
run running runs ran
```
This is useful when:
## Character set and practical convention
- the input may already be normalized
- you want stable identity mappings
- you want to avoid missing entries for canonical forms
Dictionary files are read as UTF-8 text.
## Frequency and ordering
From the perspective of the parser and the stemming algorithm, the format is not restricted to plain ASCII tokens. The parser accepts ordinary Java `String` data, and the trie itself works with general character sequences rather than with an ASCII-only internal model. In principle, this means the system could process diacritic and non-diacritic forms alike, and it could also store forms with inconsistently used diacritics.
Radixor tracks **local frequencies** of values.
In practice, however, the format is currently best understood as **primarily intended for classical basic ASCII lexical input**, especially in the traditional stemming style where language data is normalized into plain characters in the ASCII range up to character code 127. This convention is particularly relevant for languages whose original orthography includes diacritics but whose stemming dictionaries are commonly maintained in normalized non-diacritic form.
Frequency is determined by:
Future versions may expand the documentation and operational guidance for dictionaries that intentionally preserve diacritics. At present, that workflow is not the primary documented use case, not because the algorithm fundamentally forbids it, but because a concrete project requirement for such support has not yet emerged.
- how many times a mapping appears during construction
- merging behavior during reduction
## Distinct stem and variant semantics
When multiple stems exist for a word:
The format expresses a one-line grouping of forms under a canonical stem. It does not encode linguistic metadata, part-of-speech information, weights, or explicit ambiguity markers.
- results are ordered by **descending frequency**
- ties are resolved deterministically:
1. shorter textual representation wins
2. lexicographically smaller value wins
3. earlier insertion order wins
For example:
This guarantees **stable and reproducible results**.
## Ambiguity and multiple stems
A word may legitimately map to more than one stem:
```
axes ax axe
```text
axis axes
axe axes
```
This allows Radixor to represent ambiguity explicitly.
These are simply two independent lines. If both contribute mappings for the same surface form, the compiled trie may later expose one or more candidate patch commands depending on the accumulated local counts and the selected reduction mode.
At runtime:
In other words, the dictionary format itself is deliberately simple. Richer behavior such as preferred-result ranking or multiple candidate results emerges during trie construction and reduction rather than through extra syntax in the dictionary file.
- `get(word)` returns the **preferred result**
- `getAll(word)` returns **all candidates**
## Duplicate forms and repeated entries
## Design guidelines
The format does not reserve any special syntax for duplicates. If the same mapping is inserted multiple times through repeated dictionary content, the builder accumulates local counts for the stored value at the addressed key.
### Keep stems consistent
This matters because compiled tries preserve local value frequencies and use them to determine preferred ordering for `get(...)`, `getAll(...)`, and `getEntries(...)`.
Use a single canonical form:
As a result, repeating the same mapping is not just redundant text. It can influence the ranking behavior of the compiled trie.
- `run` instead of mixing `run` / `running`
- `analyze` vs `analyse` — pick one convention
## Practical examples
### Avoid noise
### Simple English example
Do not include:
- typos
- extremely rare forms (unless required)
- inconsistent normalization
### Prefer completeness over clever rules
Radixor is data-driven:
- more complete dictionaries → better results
- no hidden rule system compensates for missing entries
### Handle domain-specific vocabulary
You can extend dictionaries with:
- product names
- technical terms
- organization-specific terminology
## Example: minimal dictionary
```
go goes going went
be is are was were being
have has having had
```text
run running runs ran
connect connected connecting connection
build building builds built
```
## Example: domain-specific extension
### Dictionary with remarks
```
microservice microservices
container containers containerized
kubernetes kubernetes
```text
run running runs ran # canonical verb family
connect connected connecting // derived forms
build building builds built
```
## Common pitfalls
### Stem-only entries
### Mixing cases
```
Run running Runs ❌
```text
run
connect connected connecting
build
```
→ normalized to lowercase, but inconsistent input is error-prone
### Mixed case input
### Multiple stems on one line
```
run running connect ❌
```text
Run Running Runs Ran
CONNECT Connected Connecting
```
→ `connect` becomes a variant of `run`, which is incorrect
This is accepted, but it is normalized to lower case during parsing.
### Hidden comments
## Format limitations
```
run running //comment runs ❌
```
The current dictionary format intentionally stays minimal:
→ everything after `//` is ignored
- no quoted tokens,
- no escaping rules,
- no multi-word entries,
- no inline weighting syntax,
- no explicit ambiguity syntax,
- no sectioning or nested structure.
## When to use this format
Each token is simply a whitespace-delimited word form after remark stripping and lowercasing.
This format is suitable for:
## Authoring guidance
- curated linguistic datasets
- exported morphological dictionaries
- domain-specific vocabularies
- generated `(word, stem)` pairs from corpora
For reliable results, keep dictionaries:
## Next steps
- consistent in normalization,
- free of accidental duplicates unless repeated weighting is intentional,
- focused on meaningful stem-to-variant groupings,
- encoded in UTF-8,
- easy to audit in plain text form.
For most current deployments, it is sensible to keep dictionary content in normalized basic ASCII form unless there is a clear requirement to preserve diacritics end-to-end.
## Relationship to other documentation
This page describes only the textual source format.
To understand how those dictionary lines are transformed into compiled runtime artifacts, continue with:
- [CLI compilation](cli-compilation.md)
- [Programmatic usage](programmatic-usage.md)
- [Quick start](quick-start.md)
## Summary
Radixor dictionaries are intentionally simple:
- one line per stem
- whitespace-separated tokens
- optional remarks
- no embedded rules
This simplicity enables:
- easy generation
- fast parsing
- deterministic behavior
- efficient compilation into compact patch-command tries
- [Architecture and reduction](architecture-and-reduction.md)

26
docs/index.md
View File

@@ -1,23 +1,37 @@
# Radixor
<h1 class="visually-hidden">Home</h1>
<p align="center">
<img src="assets/images/banner.jpg" alt="Radixor banner" style="width: 100%; max-width: 1100px;">
</p>
**Radixor** is a high-performance, multi-language stemmer for Java, designed for production-grade search and text-processing systems.
**Radixor** is a high-performance, multi-language stemmer for Java, built for production-grade search and text-processing systems.
It modernizes the proven Egothor patch-command trie approach while introducing an important practical enhancement: compiled dictionaries are no longer treated as a final, closed artifact. Instead, they can be extended through additional transformation layers, allowing existing lexical assets to be refined and evolved without full recompilation from source dictionaries.
It modernizes the proven Egothor patch-command trie approach and extends it for deployment realities that classic stemming pipelines do not handle well.
Traditional Egothor-style stemming workflows usually treat a compiled dictionary as a fixed artifact. Once built, its lexical knowledge is effectively closed unless the original source dictionary is recompiled. Radixor removes that constraint. An already compiled stemming structure can be extended with additional words and transformations, which makes it possible to evolve an existing dictionary for domain-specific, customer-specific, or deployment-specific vocabulary without rebuilding the entire lexical base from scratch.
Radixor also improves how ambiguous reductions can be handled at runtime. Instead of always forcing a single result, it can return multiple plausible stems when the input token cannot be reduced unambiguously. This allows downstream systems to preserve linguistic ambiguity where that is operationally useful, whether for retrieval quality, ranking strategies, diagnostics, or domain-specific normalization policies.
The project also has a clear research lineage. The historical idea behind this stemming family is described in Leo Galambos's paper *Lemmatizer for Document Information Retrieval Systems in JAVA* (SOFSEM 2001), which presents a semi-automatic stemming technique designed for Java-based information retrieval systems. In Radixor documentation, this reference serves as historical and algorithmic background rather than as technical documentation of the current implementation.
> Unlike traditional Egothor-based deployments, Radixor can extend an already compiled stemmer dictionary and can return multiple stems when a word is not reducible to a single unambiguous form.
Radixor delivers:
- **Fast runtime stemming** with compact lookup structures
- **Multi-language adaptability** through dictionary-driven compilation
- **Incremental extensibility of compiled dictionaries** through additional transformation layers
- **Extension of compiled stemmer structures** without full recompilation from source dictionaries
- **Incremental vocabulary growth** for deployment-specific lexical refinement
- **Support for multiple stemming results** when reduction is ambiguous
- **Deterministic behavior** suitable for reproducible processing pipelines
- **Flexible integration paths**, including CLI-based and programmatic workflows
- **Operational transparency** through continuously published quality and benchmark reports
Radixor is intended for teams that require consistent stemming quality at scale without compromising maintainability, deployment efficiency, or the long-term evolvability of compiled lexical resources.
Radixor is intended for teams that require consistent stemming quality at scale, while retaining the ability to evolve lexical resources after compilation and to handle ambiguous reductions with greater precision than traditional single-stem pipelines allow.
## Start here
- Read [Quick Start](quick-start.md) for immediate implementation guidance.
- Use [Programmatic Usage](programmatic-usage.md) for application integration patterns.
- Review [Benchmarking](benchmarking.md) for reproducible performance methodology.
- Open [CI Reports](reports.md) to inspect published build artifacts and quality metrics.
- Open [CI Reports](reports.md) to inspect published build artifacts and quality metrics.
- See the historical paper: [*Lemmatizer for Document Information Retrieval Systems in JAVA*](https://www.researchgate.net/publication/221512865_Lemmatizer_for_Document_Information_Retrieval_Systems_in_JAVA).

89
docs/programmatic-extending-and-persistence.md Normal file
View File

@@ -0,0 +1,89 @@
# Extending and Persisting Compiled Tries
This document explains how compiled Radixor tries can be reopened, extended, rebuilt, and stored for deployment.
## Reopen and extend a compiled trie
`FrequencyTrieBuilders.copyOf(...)` reconstructs a mutable builder from a compiled trie. The reconstructed builder preserves the key-local value counts of the compiled trie as currently stored, making it suitable for subsequent modification and recompilation. Reconstruction is performed from the compiled state, not from the original unreduced insertion history.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.FrequencyTrieBuilders;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;
public final class ExtendCompiledStemmerExample {
private ExtendCompiledStemmerExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> compiledTrie = StemmerPatchTrieBinaryIO.read(
Path.of("stemmers", "english.radixor.gz"));
final ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
final FrequencyTrie.Builder<String> builder = FrequencyTrieBuilders.copyOf(
compiledTrie,
String[]::new,
settings);
builder.put("microservices", "Na");
final FrequencyTrie<String> updatedTrie = builder.build();
StemmerPatchTrieBinaryIO.write(
updatedTrie,
Path.of("stemmers", "english-custom.radixor.gz"));
}
}
```
This enables a layered workflow:
1. start from a bundled or already compiled stemmer,
2. reconstruct a builder,
3. add custom lexical data,
4. compile and persist a new binary artifact.
## Persist and deploy compiled tries
`StemmerPatchTrieBinaryIO` reads and writes patch-command tries as GZip-compressed binary files. `StemmerPatchTrieLoader` exposes convenience methods around the same persistence functionality.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;
StemmerPatchTrieBinaryIO.write(trie, Path.of("stemmers", "english.radixor.gz"));
```
In deployment terms, the cleanest model is usually:
- compile once,
- persist the binary artifact,
- load the artifact directly in runtime services.
## Binary-first operational model
For larger dictionaries or controlled deployment environments, a binary-first workflow is usually the most robust choice:
- prepare the compiled trie offline,
- keep the preparation step outside the runtime startup path,
- version and distribute the binary artifact,
- load the finished trie directly in production.
This model works especially well when domain-specific extensions are added in layers and then recompiled into a new read-only artifact.
## Continue with
- [Loading and Building Stemmers](programmatic-loading-and-building.md)
- [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md)

133
docs/programmatic-loading-and-building.md Normal file
View File

@@ -0,0 +1,133 @@
# Loading and Building Stemmers
This document explains how to acquire a compiled Radixor stemmer in Java.
## Load a bundled language dictionary
Bundled language resources are simple to use and compile directly into a `FrequencyTrie<String>` during loading.
```java
import java.io.IOException;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class BundledLanguageExample {
private BundledLanguageExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
}
}
```
The `storeOriginal` flag controls whether the canonical stem is inserted as a no-op patch entry for the stem itself.
## Load a textual dictionary
Loading from a dictionary file follows the same preparation model as bundled resources, but the source comes from your own file or path. Each non-empty logical line starts with the stem and may contain zero or more variants. Input is normalized to lower case using `Locale.ROOT`, and trailing remarks introduced by `#` or `//` are ignored.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class LoadTextDictionaryExample {
private LoadTextDictionaryExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
Path.of("data", "stemmer.txt"),
true,
ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS));
}
}
```
## Load a compiled binary artifact
Binary loading is typically the preferred runtime path because it avoids reparsing the textual source and skips the preparation step entirely.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class LoadBinaryExample {
private LoadBinaryExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary(
Path.of("stemmers", "english.radixor.gz"));
}
}
```
The binary format is the native `FrequencyTrie` serialization wrapped in GZip compression.
## Build directly with a mutable builder
A `FrequencyTrie.Builder<V>` accepts repeated `put(key, value)` calls and compiles the final read-only trie through `build()`. Compilation performs bottom-up reduction and produces the compact immutable runtime representation.
```java
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
public final class BuilderExample {
private BuilderExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) {
final ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
final FrequencyTrie.Builder<String> builder =
new FrequencyTrie.Builder<>(String[]::new, settings);
final PatchCommandEncoder encoder = new PatchCommandEncoder();
builder.put("running", encoder.encode("running", "run"));
builder.put("runs", encoder.encode("runs", "run"));
builder.put("ran", encoder.encode("ran", "run"));
builder.put("runner", encoder.encode("runner", "run"));
final FrequencyTrie<String> trie = builder.build();
System.out.println("Canonical node count: " + trie.size());
}
}
```
## Preparation-time memory characteristics
Compilation is commonly a one-time preparation activity and is generally fast enough not to be the main operational concern. The more important constraint is memory usage while building from textual dictionary data. Before reduction produces the compact immutable structure, the mutable build-time representation keeps the inserted data in memory. This is precisely why very large source dictionaries may require noticeably more memory during preparation than after compilation. The resulting compiled trie, by contrast, is designed as the compact runtime form.
This makes offline preparation especially attractive for large dictionaries.
## Continue with
- [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md)
- [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md)

83
docs/programmatic-querying-and-ambiguity.md Normal file
View File

@@ -0,0 +1,83 @@
# Querying and Ambiguity Handling
This document explains how a compiled Radixor trie is queried and how ambiguity is represented.
## Query a compiled trie
### `get(...)`: preferred local value
`FrequencyTrie.get(String)` returns the most frequent value stored at the node addressed by the supplied key. If several values have the same local frequency, the winner is chosen deterministically by shorter `toString()` value first, then by lexicographically lower `toString()`, and finally by stable first-seen order. If the key does not exist or no value is stored at the addressed node, `null` is returned.
```java
final String word = "running";
final String patch = trie.get(word);
```
### `getAll(...)`: ordered local values
`FrequencyTrie.getAll(String)` returns all values stored at the addressed node, ordered by descending frequency using the same deterministic tie-breaking rules. The returned array is a defensive copy. If the key is missing or has no local values, an empty array is returned.
```java
final String[] patches = trie.getAll("axes");
```
### `getEntries(...)`: values with counts
`FrequencyTrie.getEntries(String)` returns immutable `ValueCount<V>` objects aligned with the same ordering used by `getAll(...)`.
```java
import java.util.List;
import org.egothor.stemmer.ValueCount;
final List<ValueCount<String>> entries = trie.getEntries("axes");
```
## Apply patch commands
A patch command is not the final stem. It must be applied to the original input token. `PatchCommandEncoder.apply(source, patchCommand)` performs that transformation directly on the serialized command format. If the source is `null`, the method returns `null`. If the patch is `null`, empty, or malformed in compatibility-relevant ways, the original source word is preserved. Equal source and target words are represented by the canonical no-op patch.
```java
import org.egothor.stemmer.PatchCommandEncoder;
final String word = "running";
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch);
```
For multiple candidates:
```java
final String word = "axes";
for (final String patch : trie.getAll(word)) {
final String stem = PatchCommandEncoder.apply(word, patch);
System.out.println(word + " -> " + stem + " (" + patch + ")");
}
```
## Understand reduction modes
Reduction mode determines how mutable subtrees are merged during compilation. All modes operate on full subtree semantics rather than only on local node content.
### `MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS`
This mode merges subtrees whose `getAll()` results are equivalent for every reachable key suffix and whose local result ordering is the same. It ignores absolute frequencies when comparing subtree signatures, but it preserves ranked multi-result ordering semantics.
### `MERGE_SUBTREES_WITH_EQUIVALENT_UNORDERED_GET_ALL_RESULTS`
This mode also merges according to `getAll()` equivalence for every reachable key suffix, but it ignores local result ordering in addition to absolute frequencies. It is therefore more aggressive in what it considers equivalent.
### `MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`
This mode focuses on `get()` equivalence for every reachable key suffix, subject to dominance constraints. If a node does not satisfy the configured dominance thresholds, the implementation falls back to ranked `getAll()` semantics for that node to avoid unsafe over-reduction. The thresholds are configured through `ReductionSettings`. Defaults are 75 percent minimum winner share and a winner-over-second ratio of 3.
## Practical guidance
- choose a ranked `getAll()` mode when downstream ambiguity handling matters,
- choose the dominant `get()` mode when the primary operational concern is the preferred result,
- treat reduction mode as part of observable lookup semantics, not merely as an internal compression setting.
## Continue with
- [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md)
- [Loading and Building Stemmers](programmatic-loading-and-building.md)

334
docs/programmatic-usage.md
View File

@@ -1,320 +1,56 @@
# Programmatic Usage
This document describes how to use **Radixor** programmatically from Java.
This document provides the programmatic entry point to **Radixor**.
It covers:
Radixor follows a clear lifecycle:
- building a trie from dictionary data
- compiling it into an immutable structure
- loading compiled stemmers
- querying for stems
- working with multiple candidates
- modifying existing compiled stemmers
1. acquire a compiled stemmer,
2. query it for patch commands,
3. apply those commands to produce stems,
4. reopen and extend the compiled structure when needed.
## Conceptual model
Radixor is dictionary-driven, but runtime stemming does not operate by scanning raw dictionary files. A source dictionary is parsed as a sequence of canonical stems and their known variants. Each variant is converted into a compact patch command that transforms the variant into the stem, while the stem itself may optionally be stored as a canonical no-op patch. The mutable trie is then reduced into a compiled read-only structure that stores ordered values and their counts at addressed nodes.
## Overview
Two consequences matter for developers:
Radixor separates the stemming lifecycle into three stages:
- the quality and coverage of stemming behavior depend on dictionary richness,
- runtime usage is based on compiled patch-command lookup rather than on direct dictionary traversal.
1. **Build** – collect word–stem mappings in a mutable structure
2. **Compile** – reduce and convert to an immutable trie
3. **Query** – perform fast runtime lookups
This is why Radixor can generalize beyond explicitly listed forms and why compiled artifacts are well suited for deployment.
These stages are represented by:
## Documentation map
- `FrequencyTrie.Builder` (mutable)
- `FrequencyTrie` (immutable, compiled)
- `StemmerPatchTrieLoader` / `StemmerPatchTrieBinaryIO` (I/O)
The programmatic API is easier to understand when split by developer task:
- [Loading and Building Stemmers](programmatic-loading-and-building.md) explains how to acquire a compiled stemmer from bundled resources, textual dictionaries, binary artifacts, or direct builder usage.
- [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md) explains `get(...)`, `getAll(...)`, `getEntries(...)`, patch application, and the practical meaning of reduction modes.
- [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md) explains how to reopen compiled tries, add new lexical data, rebuild them, and store them as binary artifacts.
## Core types
## Building a trie programmatically
The main types involved in programmatic usage are:
You can construct a trie directly without using the CLI.
- `FrequencyTrie.Builder<V>` for mutable construction and extension,
- `FrequencyTrie<V>` for the compiled read-only trie,
- `PatchCommandEncoder` for creating and applying patch commands,
- `StemmerPatchTrieLoader` for loading bundled or textual dictionaries,
- `StemmerPatchTrieBinaryIO` for reading and writing compressed binary artifacts,
- `FrequencyTrieBuilders` for reconstructing a mutable builder from a compiled trie,
- `ReductionMode` and `ReductionSettings` for controlling compilation semantics.
```java
import org.egothor.stemmer.*;
public final class BuildExample {
public static void main(String[] args) {
ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
);
FrequencyTrie.Builder<String> builder =
new FrequencyTrie.Builder<>(String[]::new, settings);
PatchCommandEncoder encoder = new PatchCommandEncoder();
builder.put("running", encoder.encode("running", "run"));
builder.put("runs", encoder.encode("runs", "run"));
builder.put("ran", encoder.encode("ran", "run"));
FrequencyTrie<String> trie = builder.build();
}
}
```
## Loading from dictionary files
To parse dictionary files directly:
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.*;
public final class LoadFromDictionaryExample {
public static void main(String[] args) throws IOException {
FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
Path.of("data/stemmer.txt"),
true,
ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
)
);
}
}
```
## Loading a compiled binary trie
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.*;
public final class LoadBinaryExample {
public static void main(String[] args) throws IOException {
FrequencyTrie<String> trie =
StemmerPatchTrieLoader.loadBinary(Path.of("english.radixor.gz"));
}
}
```
This is the **preferred production approach**.
## Querying for stems
### Preferred result
```java
String word = "running";
String patch = trie.get(word);
String stem = PatchCommandEncoder.apply(word, patch);
```
### All candidates
```java
String[] patches = trie.getAll(word);
for (String patch : patches) {
String stem = PatchCommandEncoder.apply(word, patch);
}
```
## Accessing value frequencies
For diagnostic or advanced use cases:
```java
import org.egothor.stemmer.ValueCount;
java.util.List<ValueCount<String>> entries = trie.getEntries("axes");
for (ValueCount<String> entry : entries) {
String patch = entry.value();
int count = entry.count();
}
```
This allows:
* inspecting ambiguity
* understanding ranking decisions
* debugging dictionary quality
## Using bundled language resources
```java
FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
);
```
Bundled dictionaries are useful for:
* quick integration
* testing
* reference behavior
## Persisting a compiled trie
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.*;
public final class SaveExample {
public static void main(String[] args) throws IOException {
StemmerPatchTrieBinaryIO.write(trie, Path.of("english.radixor.gz"));
}
}
```
## Modifying an existing trie
A compiled trie can be reopened into a builder, extended, and rebuilt.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.*;
public final class ModifyExample {
public static void main(String[] args) throws IOException {
FrequencyTrie<String> compiled =
StemmerPatchTrieBinaryIO.read(Path.of("english.radixor.gz"));
ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS
);
FrequencyTrie.Builder<String> builder =
FrequencyTrieBuilders.copyOf(compiled, String[]::new, settings);
builder.put("microservices", PatchCommandEncoder.NOOP_PATCH);
FrequencyTrie<String> updated = builder.build();
StemmerPatchTrieBinaryIO.write(updated,
Path.of("english-custom.radixor.gz"));
}
}
```
## Thread safety
* `FrequencyTrie` (compiled):
* **thread-safe**
* safe for concurrent reads
* `FrequencyTrie.Builder`:
* **not thread-safe**
* intended for single-threaded construction
## Performance characteristics
### Querying
* O(length of word)
* minimal allocations
* suitable for high-throughput pipelines
### Loading
* binary loading is fast
* no preprocessing required
### Building
* depends on dictionary size
* reduction phase may be CPU-intensive
## Best practices
### Reuse compiled trie instances
* load once
* share across threads
### Prefer binary loading in production
* avoid rebuilding at runtime
* treat compiled files as deployable artifacts
### Use `getAll()` only when needed
* `get()` is faster and sufficient for most use cases
### Keep builders short-lived
* build → compile → discard
## Integration patterns
### Search systems
* apply stemming during indexing and querying
* ensure consistent dictionary usage
### Text normalization pipelines
* integrate as a transformation step
* combine with tokenization and filtering
### Domain adaptation
* extend dictionaries with domain-specific vocabulary
* rebuild compiled artifacts
## Recommended reading order
For most developers, the best order is:
1. [Loading and Building Stemmers](programmatic-loading-and-building.md)
2. [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md)
3. [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md)
## Next steps
* [Dictionary format](dictionary-format.md)
* [CLI compilation](cli-compilation.md)
* [Architecture and reduction](architecture-and-reduction.md)
## Summary
Programmatic usage of Radixor follows a clear pattern:
* build or load a trie
* query using patch commands
* apply transformations
The API is intentionally simple at the surface, while providing deeper control when needed for:
* ambiguity handling
* diagnostics
* dictionary evolution
- [Quick Start](quick-start.md)
- [CLI compilation](cli-compilation.md)
- [Dictionary format](dictionary-format.md)
- [Architecture and reduction](architecture-and-reduction.md)

380
docs/quality-and-operations.md
View File

@@ -1,315 +1,239 @@
# Quality and Operations
This document describes quality, testing, and operational practices for **Radixor**.
This document describes the engineering standards, quality posture, and operational model of **Radixor**.
It focuses on:
It is intentionally broader than a test checklist. The purpose of the project is not only to provide a fast stemmer, but to provide one whose behavior is explainable, measurable, reproducible, and straightforward to audit. That objective influences both the implementation style and the surrounding operational practices.
- reliability and determinism
- testing strategies
- deployment patterns
- performance considerations
- lifecycle management of stemmer data
## Engineering position
Radixor is developed with a strong preference for objective quality signals over informal confidence.
In practical terms, that means the project emphasizes:
## Overview
- deterministic behavior,
- reproducible compiled artifacts,
- very high structural test coverage,
- very high mutation resistance,
- explicit benchmark methodology,
- minimal operational ambiguity in deployment.
Radixor is designed to separate:
This is not treated as a cosmetic quality layer added after the implementation. It is part of the design goal of the project itself.
- **data preparation** (dictionary construction and compilation)
- **runtime execution** (lookup and patch application)
## Why quality discipline matters here
This separation enables:
A stemmer can appear deceptively simple from the outside. In practice, however, correctness depends on several interacting layers:
- predictable runtime behavior
- reproducible builds
- controlled evolution of stemming data
- dictionary parsing,
- patch-command generation,
- trie construction,
- reduction semantics,
- binary persistence,
- runtime lookup behavior.
A defect in any one of these layers can produce subtle and difficult-to-detect errors, including silent ranking drift, loss of ambiguity information, reconstruction inconsistencies, or incorrect stemming outcomes under only a narrow subset of inputs.
For that reason, Radixor aims to be validated not only by example-based tests, but by a broader quality model that combines functional testing, mutation testing, coverage analysis, benchmark visibility, and artifact publication.
## Determinism and reproducibility
Radixor emphasizes deterministic behavior.
Determinism is a foundational property of the project.
### Deterministic outputs
Given the same dictionary input and the same reduction settings, the project aims to produce:
Given:
- the same compiled trie semantics,
- the same local value ordering,
- the same observable `get()` and `getAll()` behavior,
- the same persisted binary output structure in semantic terms.
- the same dictionary input
- the same reduction settings
This matters for more than technical elegance. It enables:
Radixor guarantees:
- stable search behavior across deployments,
- reproducible build outputs,
- reliable regression analysis,
- explainable differences when a dictionary or reduction setting changes.
- identical compiled trie structure
- identical value ordering
- identical lookup results
A deterministic system is easier to test, easier to reason about, and safer to integrate into production pipelines.
### Why this matters
## Test strategy
- stable search behavior across deployments
- reproducible builds
- easier debugging and regression analysis
The project is intended to maintain very high confidence in both core correctness and behavioral stability.
### Structural coverage
High code coverage is treated as a useful signal, but not as a sufficient goal on its own. Coverage is valuable only when the covered scenarios actually pressure the implementation in meaningful ways.
## Testing strategy
In Radixor, strong coverage is expected across areas such as:
### Unit testing
- patch encoding and application,
- mutable trie construction,
- subtree reduction,
- compiled trie lookup,
- binary serialization and deserialization,
- reconstruction from compiled state,
- dictionary parsing and CLI behavior.
Core components should be tested independently:
### Mutation resistance
- patch encoding and decoding
- trie construction
- reduction behavior
- binary serialization and deserialization
Mutation testing is especially important for this project because it helps distinguish superficial test execution from genuinely discriminating tests.
### Dictionary validation tests
A project can report high line or branch coverage while still failing to detect semantically dangerous implementation drift. Mutation testing provides a stronger objective signal: whether the test suite actually notices meaningful behavioral changes.
A recommended pattern:
For Radixor, very high mutation scores are therefore part of the intended engineering standard, not an optional vanity metric.
1. load dictionary input
2. compile trie
3. re-apply all word → stem mappings
4. verify that:
### Boundary and negative-path validation
- expected stem is present in `getAll()`
- preferred result (`get()`) is correct when deterministic
The project also benefits from extensive negative and edge-case testing, for example around:
This ensures:
- malformed patch commands,
- missing or corrupt binary data,
- invalid CLI arguments,
- ambiguous mappings,
- dominance-threshold edge conditions,
- reconstruction of reduced compiled tries,
- empty inputs and short words.
- no data loss during reduction
- correctness of patch encoding
These cases are important because many real integration failures occur at the boundary conditions, not in the central happy path.
## Quality signals and published evidence
The project publishes durable quality artifacts through GitHub Pages so that important signals remain externally inspectable rather than existing only as transient CI output.
## Regression testing
Those published surfaces include:
Maintain a stable test dataset:
- unit test results,
- coverage reports,
- mutation testing reports,
- static analysis reports,
- benchmark outputs,
- software composition artifacts.
- representative vocabulary
- edge cases (short words, long words, ambiguous forms)
This publication model improves transparency and makes it easier to inspect the project’s quality posture without having to reconstruct the CI environment locally.
Use it to:
## Operational model
- detect unintended changes
- verify behavior after refactoring
- validate reduction mode changes
Radixor is designed around a clean separation between preparation-time work and runtime execution.
### Preparation phase
Preparation includes:
## Performance testing
- creating or refining dictionary data,
- compiling the dictionary into a reduced read-only trie,
- validating the resulting artifact,
- persisting it as a deployable binary stemmer.
Performance should be evaluated in terms of:
### Runtime phase
### Throughput
Runtime usage is intentionally simpler:
- words processed per second
- load the compiled artifact,
- reuse the resulting trie,
- perform fast lookups and patch application,
- avoid rebuilding or reparsing during live request handling.
### Latency
This separation reduces startup unpredictability, keeps runtime behavior stable, and makes deployment artifacts explicit.
- time per lookup
## Production posture
### Memory footprint
For production use, the preferred model is straightforward:
- size of compiled trie
- runtime memory usage
1. prepare or refine the lexical resource,
2. compile it offline,
3. validate the resulting artifact,
4. deploy the compiled binary,
5. load it once and reuse it.
Benchmark with:
This model has several advantages:
- realistic token streams
- production-like dictionaries
- no runtime compilation cost,
- no repeated parsing overhead,
- clear versioning of stemming behavior,
- better reproducibility across environments,
- simpler operational diagnosis when results change.
## Auditability and dependency posture
Radixor deliberately avoids external runtime dependencies.
## Deployment model
That choice serves a practical engineering goal: the project should be easy to audit from both a correctness and a security perspective, without forcing downstream users to reason through a large dependency graph or a complex software supply chain for core functionality.
### Recommended workflow
A dependency-free core does not make a project automatically secure, but it does simplify several important activities:
1. prepare dictionary data
2. compile using CLI
3. store `.radixor.gz` artifact
4. deploy artifact with application
5. load using `loadBinary(...)`
- source review,
- behavioral auditing,
- release inspection,
- software composition analysis,
- long-term maintenance.
### Why this model
In operational terms, this means there is less hidden behavior outside the project’s own codebase and less need to evaluate third-party runtime libraries for the core implementation path.
- avoids runtime compilation overhead
- reduces startup latency
- ensures consistent behavior across environments
## Security-minded operational guidance
The project’s operational simplicity should be preserved in deployment practice.
Recommended principles include:
## Artifact management
- treat source dictionaries as controlled inputs,
- generate compiled artifacts in known build environments,
- version compiled artifacts explicitly,
- avoid loading untrusted binary stemmer files,
- keep benchmark, test, and quality outputs attached to the same revision that produced the artifact.
Compiled stemmers should be treated as versioned assets.
These practices support traceability and reduce ambiguity about what exactly is running in production.
### Versioning
## Performance as a quality concern
- include version in filename or metadata
- track dictionary source and reduction settings
Performance is not isolated from quality; for Radixor, it is part of the project’s engineering contract.
Example:
The benchmark suite exists to make throughput behavior measurable and historically visible. At the same time, benchmark interpretation must remain disciplined. Absolute numbers can vary by environment, especially when published through shared CI infrastructure. Sustained relative behavior and reproducible local benchmark methodology are more meaningful than one-off raw figures.
```
english-v1.2-ranked.radixor.gz
```
This is why benchmarking belongs alongside testing and reporting rather than outside the quality discussion altogether.
### Storage
## Operational observability
- store in repository or artifact storage
- ensure consistent distribution across environments
Radixor itself is intentionally small and does not attempt to become an observability framework. Instead, integrations should provide the surrounding operational visibility that production systems require.
Typical integration-level observability includes:
- reporting load failures,
- monitoring startup artifact loading,
- measuring lookup throughput in the host application,
- tracking memory usage of loaded compiled tries,
- optionally sampling ambiguity-heavy cases when `getAll()` is part of the application logic.
## Runtime usage
The project’s role is to remain deterministic and inspectable enough that such operational signals are meaningful.
### Loading
## What feedback is most valuable
- load once during application startup
- reuse `FrequencyTrie` instance
Feedback is especially valuable when it improves the objectivity or professional rigor of the project.
### Thread safety
That includes, for example:
- compiled trie is safe for concurrent access
- no synchronization required for reads
- defects in behavioral correctness,
- weaknesses in reduction semantics or edge-case handling,
- benchmark methodology issues,
- gaps in tests or mutation resistance,
- ambiguities in published reports,
- opportunities to improve auditability, reproducibility, or operational clarity.
### Avoid repeated loading
Project feedback is most useful when it helps strengthen the project as an implementation that can be trusted, reviewed, and maintained at a professional standard.
Do not:
## Practical summary
- load trie per request
- rebuild trie at runtime
Radixor aims to combine:
- strong algorithmic performance,
- deterministic behavior,
- very high validation standards,
- transparent published quality evidence,
- low operational ambiguity,
- easy auditability of the core implementation.
That combination is central to the identity of the project. The goal is not merely to be fast, but to be fast in a way that remains explainable, testable, reproducible, and professionally defensible.
## Memory considerations
## Related documentation
- compiled tries are compact but not negligible
- size depends on:
- dictionary size
- reduction mode
Recommendations:
- monitor memory usage in production
- choose reduction mode appropriately
## Reduction mode in production
Default recommendation:
- use **ranked mode**
Switch to other modes only when:
- memory constraints are strict
- multiple candidate results are not required
Always validate behavior after changing reduction mode.
## Dictionary lifecycle
### Updating dictionaries
When dictionary data changes:
1. update source file
2. recompile
3. run validation tests
4. deploy new artifact
### Backward compatibility
- changes in dictionary may affect stemming results
- evaluate impact on search relevance
## Observability
Radixor itself does not provide observability features; integration should provide:
- logging for loading failures
- metrics for lookup throughput
- monitoring of memory usage
Optional:
- sampling of ambiguous results (`getAll()`)
## Error handling
### During compilation
Handle:
- invalid dictionary format
- I/O failures
- invalid arguments
### During runtime
Handle:
- missing dictionary files
- corrupted binary artifacts
Fail fast on initialization errors.
## Operational best practices
- compile dictionaries offline
- version compiled artifacts
- test before deployment
- load once and reuse
- monitor performance and memory
- document reduction settings used
## Security considerations
- treat dictionary input as trusted data
- validate external sources before compilation
- avoid loading unverified binary artifacts
## Integration checklist
Before production deployment:
- dictionary validated
- compiled artifact generated
- reduction mode documented
- performance tested
- memory usage verified
- regression tests passing
## Next steps
- [Quick start](quick-start.md)
- [Benchmarking](benchmarking.md)
- [Reports](reports.md)
- [CLI compilation](cli-compilation.md)
- [Programmatic usage](programmatic-usage.md)
## Summary
Radixor is designed for:
- deterministic behavior
- efficient runtime execution
- controlled data-driven evolution
By separating compilation from runtime and following proper operational practices, it can be reliably integrated into production-grade systems.

180
docs/quick-start.md
View File

@@ -1,8 +1,92 @@
# Quick Start
This guide shows the fastest way to start using **Radixor** and the most common next steps.
This guide introduces the fastest practical path to using **Radixor**.
## Hello world
Radixor separates preparation from runtime usage. Source dictionaries are used to derive patch commands and reduce them into a compact read-only trie. Runtime stemming then operates on that compiled structure rather than on the original dictionary text. A richer dictionary usually improves the quality and coverage of inferred transformations, including transformations that are applicable to words not explicitly present in the source material. The reduction step also removes a large amount of redundant lexical information, which is why very large dictionaries can still produce compact runtime artifacts. These artifacts can be persisted and loaded directly when needed.
A practical workflow usually consists of two independent phases:
1. obtain a compiled stemmer,
2. use the compiled stemmer.
## 1. Obtain a compiled stemmer
A compiled stemmer can be obtained in three common ways.
### Use a bundled language dictionary
Radixor ships with bundled dictionaries for a set of supported languages. These resources are line-oriented dictionaries stored with the library and compiled into a `FrequencyTrie<String>` when loaded. The loader can also store the canonical stem itself as a no-op patch command.
```java
import java.io.IOException;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class BundledStemmerExample {
private BundledStemmerExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.load(
StemmerPatchTrieLoader.Language.US_UK_PROFI,
true,
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
System.out.println("Canonical node count: " + trie.size());
}
}
```
### Load a previously compiled binary stemmer
Compiled stemmers can be stored as GZip-compressed binary artifacts and loaded directly. This is usually the most convenient production path because no dictionary parsing or recompilation is needed during application startup.
```java
import java.io.IOException;
import java.nio.file.Path;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class LoadBinaryStemmerExample {
private LoadBinaryStemmerExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary(
Path.of("stemmers", "english.radixor.gz"));
System.out.println("Canonical node count: " + trie.size());
}
}
```
### Build or extend a stemmer from dictionary data
Radixor can also build a compiled trie from a custom dictionary. Dictionary lines consist of a canonical stem followed by zero or more variants. The parser lowercases input with `Locale.ROOT`, ignores leading and trailing whitespace, and supports line remarks introduced by `#` or `//`.
This path is also relevant when you extend an existing compiled stemmer with additional domain-specific entries and rebuild a new compact artifact.
A dedicated CLI compilation workflow deserves its own focused page and should remain separate from Quick Start, but conceptually it is simply another way to prepare the compiled artifact before runtime use.
## 2. Use the compiled stemmer
A compiled `FrequencyTrie<String>` stores patch commands, not final stems. Querying therefore has two steps:
1. retrieve one or more patch commands from the trie,
2. apply each patch command to the original input word.
The trie returns values associated with the exact addressed node. `get(...)` returns the locally preferred value, while `getAll(...)` returns all locally stored values ordered by descending frequency with deterministic tie-breaking.
### Get the preferred result
Use `get(...)` when the application needs a single preferred transformation.
```java
import java.io.IOException;
@@ -12,9 +96,9 @@ import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class HelloRadixor {
public final class SingleStemExample {
private HelloRadixor() {
private SingleStemExample() {
throw new AssertionError("No instances.");
}
@@ -28,75 +112,44 @@ public final class HelloRadixor {
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch);
System.out.println(word + " -> " + stem);
System.out.println(word + " -> " + stem + " (" + patch + ")");
}
}
```
This example shows the core workflow:
### Get all candidate results
1. load a trie
2. get a patch command for a word
3. apply the patch
4. obtain the stem
## Retrieve multiple candidate stems
If you need more than one candidate result, use `getAll(...)` instead of `get(...)`.
Use `getAll(...)` when the application should preserve ambiguity instead of collapsing everything into one result. The method is available on every compiled trie. What changes across reduction modes is the semantic strength with which multi-result behavior is preserved during reduction, not whether the method exists.
```java
final String word = "axes";
final String[] patches = trie.getAll(word);
for (String patch : patches) {
for (final String patch : patches) {
final String stem = PatchCommandEncoder.apply(word, patch);
System.out.println(word + " -> " + stem + " (" + patch + ")");
}
```
## Load a compiled binary stemmer
### Inspect ranked values and counts
For production systems, the preferred approach is usually to precompile the dictionary and load the compressed binary artifact at runtime.
For diagnostics or advanced ranking logic, use `getEntries(...)` to obtain value-count pairs in the same deterministic order as `getAll(...)`.
```java
import java.io.IOException;
import java.nio.file.Path;
import java.util.List;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.StemmerPatchTrieLoader;
import org.egothor.stemmer.ValueCount;
public final class BinaryStemmerExample {
final List<ValueCount<String>> entries = trie.getEntries("axes");
private BinaryStemmerExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final Path path = Path.of("stemmers", "english.radixor.gz");
final FrequencyTrie<String> trie = StemmerPatchTrieLoader.loadBinary(path);
final String word = "connected";
final String patch = trie.get(word);
final String stem = PatchCommandEncoder.apply(word, patch);
System.out.println(word + " -> " + stem);
}
for (final ValueCount<String> entry : entries) {
System.out.println(entry.value() + " -> " + entry.count());
}
```
## Compile a dictionary from the command line
## Extend an existing compiled stemmer
```bash
java org.egothor.stemmer.Compile \
--input ./data/stemmer.txt \
--output ./build/english.radixor.gz \
--reduction-mode MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS \
--store-original \
--overwrite
```
## Modify an existing compiled stemmer
A compiled trie is read-only, but it is not permanently closed. Radixor can reconstruct a mutable builder from a compiled trie, preserve the currently stored local counts, accept additional insertions, and then compile a new read-only trie. Reconstruction operates on the compiled form, so if the source trie was already reduced by subtree merging, the reopened builder reflects that compiled state rather than the original unreduced insertion history.
```java
import java.io.IOException;
@@ -109,17 +162,15 @@ import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;
public final class ModifyCompiledExample {
public final class ExtendCompiledStemmerExample {
private ModifyCompiledExample() {
private ExtendCompiledStemmerExample() {
throw new AssertionError("No instances.");
}
public static void main(final String[] arguments) throws IOException {
final Path input = Path.of("stemmers", "english.radixor.gz");
final Path output = Path.of("stemmers", "english-custom.radixor.gz");
final FrequencyTrie<String> compiledTrie = StemmerPatchTrieBinaryIO.read(input);
final FrequencyTrie<String> compiledTrie = StemmerPatchTrieBinaryIO.read(
Path.of("stemmers", "english.radixor.gz"));
final ReductionSettings settings = ReductionSettings.withDefaults(
ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
@@ -129,18 +180,25 @@ public final class ModifyCompiledExample {
String[]::new,
settings);
builder.put("microservices", PatchCommandEncoder.NOOP_PATCH);
builder.put("microservices", "Na");
final FrequencyTrie<String> updatedTrie = builder.build();
StemmerPatchTrieBinaryIO.write(updatedTrie, output);
StemmerPatchTrieBinaryIO.write(
updatedTrie,
Path.of("stemmers", "english-custom.radixor.gz"));
}
}
```
## Operational note on memory and preparation
Dictionary compilation is usually a one-time preparation step and is generally fast. The more relevant operational constraint is memory consumption during preparation: before reduction, the mutable build-time structure keeps the full dictionary-derived content in RAM. Reduction then compacts it substantially, but very large source dictionaries can still require significant memory during the initial build phase. The best operational model is therefore to compile once, persist the resulting binary artifact, and load that artifact directly in runtime environments.
## Where to continue
* [Dictionary format](dictionary-format.md)
* [CLI compilation](cli-compilation.md)
* [Programmatic usage](programmatic-usage.md)
* [Built-in languages](built-in-languages.md)
* [Architecture and reduction](architecture-and-reduction.md)
- [Programmatic Usage](programmatic-usage.md)
- [Dictionary format](dictionary-format.md)
- [CLI compilation](cli-compilation.md)
- [Built-in languages](built-in-languages.md)
- [Architecture and reduction](architecture-and-reduction.md)

208
docs/reduction-semantics.md Normal file
View File

@@ -0,0 +1,208 @@
# Reduction Semantics
This document explains how **Radixor** decides that two subtrees are equivalent, how the different reduction modes work, and how those choices affect observable runtime behavior.
## Why reduction exists
Without reduction, the trie would still work, but many subtrees that mean the same thing would remain duplicated. The result would be a much larger runtime artifact than necessary.
Reduction solves that by merging semantically equivalent subtrees into one canonical representative.
The key idea is simple:
> if two subtrees behave the same way under the semantic contract chosen for compilation, only one physical copy is needed.
## Reduction is semantic, not merely structural
Radixor does not reduce nodes merely because they look similar locally. It reduces subtrees only when their **meaning** matches according to the selected mode.
That is why reduction is based on a **signature** that captures both:
1. the local semantics of the current node,
2. the structure and semantics of all descendant edges.
Conceptually:
```text
Signature = (LocalDescriptor, SortedChildDescriptors)
```
Two subtrees are merged only if their signatures are equal.
## Local descriptors
The local descriptor defines what “equivalent” means for the values stored at one node.
Radixor supports three semantic views.
### Ranked descriptor
The ranked descriptor preserves the full ordered result semantics of `getAll()`.
That means:
- candidate membership is preserved,
- local ordering is preserved,
- observable ranked multi-result behavior remains stable.
This is the most semantically faithful mode when ambiguity handling matters.
### Unordered descriptor
The unordered descriptor preserves the set of reachable results, but not their local ordering.
That means:
- candidate membership is preserved,
- ordering differences may be ignored,
- more subtrees can be merged than in ranked mode.
This mode is useful when alternative candidates matter but exact ranking does not.
### Dominant descriptor
The dominant descriptor focuses on the preferred result returned by `get()`.
This mode is used only when the dominant local candidate is strong enough according to configured thresholds:
- minimum winner percentage,
- winner-over-second ratio.
If that local dominance is not strong enough, Radixor does not force dominant semantics anyway. It falls back to ranked semantics for that node to avoid unsafe over-reduction.
That fallback is one of the most important safeguards in the design.
## Child descriptors
A subtree is not defined only by the values stored at the current node. It is also defined by what behavior is reachable through its children.
Each child contributes:
```text
(edge character, child signature)
```
Children are sorted by edge character so that signatures remain deterministic and stable.
This matters because reduction must not depend on incidental map iteration order or other non-semantic implementation details.
## Canonicalization
Once a subtree signature is computed, the reduction process checks whether an equivalent canonical subtree already exists.
If yes, the existing reduced node is reused.
If no, a new canonical reduced node is created and registered.
This turns reduction into a canonicalization process:
- compute semantic identity,
- find canonical representative,
- reuse or create,
- continue bottom-up.
That is how Radixor eliminates duplicated equivalent subtrees.
## Count aggregation and compiled state
When multiple original build-time subtrees collapse into one canonical reduced node, local counts may be aggregated.
This is an important point for understanding compiled artifacts.
A compiled trie is not always a verbatim replay of original insertion history. It is a canonical runtime structure that preserves the semantics guaranteed by the chosen reduction mode.
This explains two things:
- why compiled artifacts can become dramatically smaller,
- why reconstructing a builder from a compiled trie reflects the compiled state rather than the full original unreduced history.
## Reduction modes
### `MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS`
This mode merges subtrees only when their `getAll()` results are equivalent for every reachable key suffix and when local ordering is preserved.
Use this mode when:
- ambiguity handling matters,
- `getAll()` ordering should remain meaningful,
- behavioral fidelity is more important than maximum compression.
This is the safest and most generally recommended mode.
### `MERGE_SUBTREES_WITH_EQUIVALENT_UNORDERED_GET_ALL_RESULTS`
This mode also preserves `getAll()`-level membership equivalence for every reachable key suffix, but it ignores local ordering differences.
Use this mode when:
- alternative candidates still matter,
- exact ordering is less important,
- stronger reduction is acceptable.
This mode is more aggressive than ranked mode, but less semantically rich.
### `MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`
This mode focuses on preserving dominant `get()` semantics for every reachable key suffix, subject to dominance thresholds.
Use this mode when:
- the main operational concern is the preferred result,
- richer alternative-result behavior is less important,
- stronger reduction is desirable.
Because non-dominant nodes fall back to ranked semantics, this mode is not simply “discard everything except the winner”. It is a controlled reduction strategy with a built-in safety condition.
## Practical effect on runtime behavior
Reduction mode is not just a storage optimization setting. It affects what distinctions remain visible after compilation.
### When ranked mode is used
You can rely on full ranked `getAll()` semantics being preserved.
### When unordered mode is used
You can rely on candidate membership, but not necessarily on preserving the same local ranking distinctions.
### When dominant mode is used
You optimize primarily for preferred-result semantics. Alternative-result behavior may still exist, but it is no longer the primary semantic contract of the reduction.
## Choosing a mode
A practical rule of thumb is:
- choose **ranked** if you are unsure,
- choose **unordered** if alternative membership matters but ranking does not,
- choose **dominant** only when your application is fundamentally driven by `get()` and you understand the trade-off.
## Why this design works well
The reduction model succeeds because it does not confuse “smaller” with “acceptable”.
Instead, it makes the semantic contract explicit:
- what exactly must be preserved,
- what differences may be ignored,
- when a more aggressive mode is safe,
- when the system must fall back to a stricter interpretation.
That explicitness is what makes the compression trustworthy.
## Mental model to keep
If you want one concise mental model for reduction, use this one:
- build-time insertion collects examples,
- reduction asks which subtrees mean the same thing,
- the answer depends on the chosen semantic contract,
- canonical representatives are shared,
- the compiled trie preserves the behavior promised by that contract.
## Continue with
- [Architecture](architecture.md)
- [Programmatic usage](programmatic-usage.md)
- [CLI compilation](cli-compilation.md)

46
docs/reports.md
View File

@@ -1,20 +1,48 @@
# CI Reports
# Reports and Published Build Artifacts
Radixor publishes durable CI artifacts to GitHub Pages on every qualifying run of `.github/workflows/pages.yml`.
Radixor publishes durable build outputs to GitHub Pages from qualifying runs of `.github/workflows/pages.yml`.
## Primary report entry points
This page is the central entry point for published project artifacts, including build summaries, API documentation, test and quality reports, benchmark outputs, and software composition materials. It is intended both for routine project inspection and for linking stable report surfaces from external references such as the README, release notes, or development workflows.
## Stable entry points
The following links are the primary stable locations for the most recent published build outputs:
- [Latest build summary](https://leogalambos.github.io/Radixor/builds/latest/)
- [Browse historical build reports](https://leogalambos.github.io/Radixor/builds/)
Use `builds/latest/` when you want the current published report surface. Use `builds/` when you need to inspect or compare retained historical runs.
## API and developer documentation
These reports are primarily useful when reviewing the published API surface and generated developer-facing documentation:
- [Javadoc](https://leogalambos.github.io/Radixor/builds/latest/javadoc/)
## Verification and code quality reports
These reports describe the outcome of core verification and static-analysis stages for the latest published build:
- [Unit test report](https://leogalambos.github.io/Radixor/builds/latest/test/)
- [PMD report](https://leogalambos.github.io/Radixor/builds/latest/pmd/main.html)
- [JaCoCo coverage report](https://leogalambos.github.io/Radixor/builds/latest/coverage/)
- [PIT mutation testing report](https://leogalambos.github.io/Radixor/builds/latest/pitest/)
- [Dependency vulnerability report](https://leogalambos.github.io/Radixor/builds/latest/dependency-check/dependency-check-report.html)
Together, these reports provide the most direct published view of functional correctness, static quality signals, coverage, mutation resistance, and dependency-level security review outputs.
## Software composition artifacts
These artifacts expose the published software bill of materials for the latest build:
- [SBOM (JSON)](https://leogalambos.github.io/Radixor/builds/latest/sbom/radixor-sbom.json)
- [SBOM (XML)](https://leogalambos.github.io/Radixor/builds/latest/sbom/radixor-sbom.xml)
## Benchmark and badge metadata
They are useful for dependency inspection, downstream integration, compliance-oriented workflows, and artifact traceability.
## Benchmark outputs and badge metadata
These resources expose benchmark results and generated badge metadata derived from the latest published build:
- [JMH benchmark results (TXT)](https://leogalambos.github.io/Radixor/builds/latest/jmh/jmh-results.txt)
- [JMH benchmark results (CSV)](https://leogalambos.github.io/Radixor/builds/latest/jmh/jmh-results.csv)
@@ -22,6 +50,12 @@ Radixor publishes durable CI artifacts to GitHub Pages on every qualifying run o
- [Mutation badge metadata](https://leogalambos.github.io/Radixor/builds/latest/metrics/pitest-badge.json)
- [Benchmark badge metadata](https://leogalambos.github.io/Radixor/builds/latest/metrics/jmh-badge.json)
## Historical runs
The benchmark outputs provide direct access to the published JMH result files, while the badge metadata endpoints are intended for status surfaces such as the project README or other generated dashboards.
- [Browse historical build reports](https://leogalambos.github.io/Radixor/builds/)
## Practical usage
In most cases, the recommended entry path is:
1. start with the [Latest build summary](https://leogalambos.github.io/Radixor/builds/latest/),
2. open the specific report category relevant to your task,
3. use [Browse historical build reports](https://leogalambos.github.io/Radixor/builds/) when historical inspection is needed.

38
mkdocs.yml
View File

@@ -23,6 +23,9 @@ theme:
extra:
generator: false
extra_css:
- assets/stylesheets/extra.css
markdown_extensions:
- admonition
- attr_list
@@ -34,12 +37,29 @@ markdown_extensions:
nav:
- Home: index.md
- Quick Start: quick-start.md
- Dictionary Format: dictionary-format.md
- CLI Compilation: cli-compilation.md
- Programmatic Usage: programmatic-usage.md
- Built-in Languages: built-in-languages.md
- Architecture and Reduction: architecture-and-reduction.md
- Quality and Operations: quality-and-operations.md
- Benchmarking: benchmarking.md
- CI Reports: reports.md
- Getting Started:
- Quick Start: quick-start.md
- Built-in Languages: built-in-languages.md
- Dictionary Format: dictionary-format.md
- CLI Compilation: cli-compilation.md
- Programmatic Usage:
- Overview: programmatic-usage.md
- Loading and Building Stemmers: programmatic-loading-and-building.md
- Querying and Ambiguity Handling: programmatic-querying-and-ambiguity.md
- Extending and Persisting Compiled Tries: programmatic-extending-and-persistence.md
- Architecture and Semantics:
- Overview: architecture-and-reduction.md
- Architecture: architecture.md
- Reduction Semantics: reduction-semantics.md
- Compatibility and Guarantees: compatibility-and-guarantees.md
- Dictionaries:
- Contributing Dictionaries: contributing-dictionaries.md
- Quality and Operations:
- Quality and Operations: quality-and-operations.md
- Benchmarking: benchmarking.md
- Reports: reports.md