Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat!: modularize stemmer models and release infrastructure

Browse Source 
Move bundled stemmer dictionaries from the core artifact into independently
versioned model modules. Add model discovery and explicit model-loading APIs,
a standard model aggregate, a model BOM, and dedicated model and catalog
release workflows.

Add full PoliMorf integration, model provenance and licensing validation,
streaming model-input verification, strict dependency verification, consumer
resolution tests, Configuration Cache compatibility, and expanded JMH,
quality, documentation, and release checks.

Upgrade the CycloneDX and JMH Gradle plugins and remove Gradle 10 and Java
compiler deprecations.

BREAKING CHANGE: The core Radixor artifact no longer contains bundled stemmer
dictionaries. Applications must add the required model artifacts, the standard
model aggregate, or model dependencies managed through the Radixor model BOM.
This commit is contained in:
Leo Galambos
2026-07-22 23:33:28 +02:00
parent 9c5b9e331b
commit e7800b29c9
250 changed files with 7297 additions and 905 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/programmatic-loading-and-building.md
View File

@@ -2,9 +2,9 @@
This document explains how to acquire a compiled Radixor stemmer in Java.
## Load a bundled language dictionary
## Load a registered default model
Bundled language resources are simple to use and compile directly into a `FrequencyTrie<CompiledPatchCommand>` during loading.
Language-oriented entry points resolve a registered default model and compile its GZip textual dictionary into a `FrequencyTrie<CompiledPatchCommand>`. The corresponding model JAR must be on the runtime classpath; the core contains no dictionary.
```java
import java.io.IOException;
@@ -14,9 +14,9 @@ import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;
public final class BundledLanguageExample {
public final class RegisteredLanguageModelExample {
private BundledLanguageExample() {
private RegisteredLanguageModelExample() {
throw new AssertionError("No instances.");
}
@@ -31,14 +31,16 @@ public final class BundledLanguageExample {
The `storeOriginal` flag controls whether the canonical stem is inserted as a no-op patch entry for the stem itself.
Bundled `loadCompiled(...)` entry points build the runtime trie with the same contracted
Language-oriented `loadCompiled(...)` entry points build the runtime trie with the same contracted
representation used by the published benchmarks. During compilation, uniform preferred-command
subtrees are collapsed into accepting leaves, so lookup can stop before consuming the entire input
when the remaining characters cannot change the selected patch command.
## Load a textual dictionary
Loading from a dictionary file follows the same preparation model as bundled resources, but the source comes from your own file or path. The input may be plain UTF-8 text or GZip-compressed UTF-8 text; the loader detects GZip data from the stream header. The textual format is tab-separated values, meaning that columns are separated by the tab character. Each non-empty logical line starts with the stem column and may contain zero or more variant columns. Input case normalization is controlled by `CaseProcessingMode` (default: `LOWERCASE_WITH_LOCALE_ROOT`), trailing remarks introduced by `#` or `//` are ignored, and dictionary items containing embedded whitespace are currently ignored with warning-level diagnostics.
Loading from a dictionary file follows the same trie preparation model as registered model resources, but the source comes from your own file or path and bypasses registry metadata. The input may be plain UTF-8 text or GZip-compressed UTF-8 text; the loader detects GZip data from the stream header. The textual format is tab-separated values, meaning that columns are separated by the tab character. Each non-empty logical line starts with the stem column and may contain zero or more variant columns. Input case normalization is controlled by `CaseProcessingMode` (default: `LOWERCASE_WITH_LOCALE_ROOT`), trailing remarks introduced by `#` or `//` are ignored, and dictionary items containing embedded whitespace are currently ignored with warning-level diagnostics.
For explicit model IDs, multiple variants, and ClassLoader control, see [Model Selection and Loading](model-selection-and-loading.md).
```java
import java.io.IOException;