feat!: modularize stemmer models and release infrastructure

Move bundled stemmer dictionaries from the core artifact into independently
versioned model modules. Add model discovery and explicit model-loading APIs,
a standard model aggregate, a model BOM, and dedicated model and catalog
release workflows.

Add full PoliMorf integration, model provenance and licensing validation,
streaming model-input verification, strict dependency verification, consumer
resolution tests, Configuration Cache compatibility, and expanded JMH,
quality, documentation, and release checks.

Upgrade the CycloneDX and JMH Gradle plugins and remove Gradle 10 and Java
compiler deprecations.

BREAKING CHANGE: The core Radixor artifact no longer contains bundled stemmer
dictionaries. Applications must add the required model artifacts, the standard
model aggregate, or model dependencies managed through the Radixor model BOM.
This commit is contained in:
2026-07-22 23:33:28 +02:00
parent 9c5b9e331b
commit e7800b29c9
250 changed files with 7297 additions and 905 deletions

View File

@@ -37,7 +37,7 @@ This API is expected to remain supportable across future versions. The preferred
Examples of likely additive evolution include:
- additional bundled language resources,
- additional independently versioned language models,
- fuller support for diacritics or native-script language resources,
- expanded documentation and operational tooling,
- new convenience methods that do not break existing code.
@@ -83,6 +83,8 @@ Compiled `FrequencyTrie` instances are immutable and thread-safe for concurrent
Serialized patch-command strings remain the stable stored representation used by textual dictionaries and binary artifacts. Runtime stemming should use `CompiledPatchCommand` values produced by `StemmerPatchTrieLoader.loadCompiled(...)`, `StemmerPatchTrieLoader.loadBinaryCompiled(...)`, or `PatchCommandEncoder.compile(...)`.
Language-default, descriptor, and stable model-ID `loadCompiled` entry points share the same compiled-value conversion. Explicit model IDs never fall back to a language default. Model loading is not cached, and construction-memory requirements are model-dependent; the unusually large PoliMorf input is verified separately with a 6 GiB maximum heap.
The historical `PatchCommandEncoder.apply(...)` and String-based `applyTo(...)` overloads remain compatibility APIs during the 2.x transition, but they are deprecated because they reparse the patch-command string on each application. See [Migration and Backward Compatibility](migration-and-backward-compatibility.md) for old and new code examples.
Compiled buffer-oriented `CompiledPatchCommand.applyTo(...)` overloads use caller-owned output storage. They do not retain output arrays and report insufficient capacity with `CompiledPatchCommand.APPLY_INSUFFICIENT_CAPACITY`.
@@ -110,7 +112,7 @@ The following kinds of change are generally compatible with the projects dire
- improved internal data structures,
- changes inside `org.egothor.stemmer.trie`,
- expanded bundled dictionaries,
- expanded model dictionaries,
- additional supported languages,
- improved native-script handling,
- better benchmarks, tests, and reports,
@@ -122,11 +124,11 @@ The project should be able to improve substantially while keeping the main user-
Some areas should be treated as stable in intent but still approached carefully when changed.
### Bundled dictionary contents
### Independently versioned model contents
Bundled resources are versioned project data, not immutable language standards. Their contents may improve over time.
Model resources are independently versioned project data, not immutable language standards. Their contents may improve over time.
That means stemming outcomes can legitimately change when bundled dictionaries are refined or expanded. Such changes are compatible with the projects direction, but they should still be understood as behavior changes at the lexical-resource level.
That means stemming outcomes can legitimately change when a model artifact is updated. Such changes are separate from core compatibility and should be reviewed as lexical-resource behavior changes.
### Binary format evolution
@@ -159,7 +161,7 @@ Users should avoid depending on:
- internal trie package details,
- undocumented internal classes or intermediate representations,
- incidental internal ordering outside documented lookup semantics,
- assumptions that bundled dictionary contents will never evolve,
- assumptions that a model's dictionary contents will never evolve across model versions,
- assumptions that internal binary-format details are frozen forever.
If a behavior is important to your integration, it should ideally be documented at the public API or project-documentation level rather than inferred from internal implementation details.