feat!: modularize stemmer models and release infrastructure
Move bundled stemmer dictionaries from the core artifact into independently versioned model modules. Add model discovery and explicit model-loading APIs, a standard model aggregate, a model BOM, and dedicated model and catalog release workflows. Add full PoliMorf integration, model provenance and licensing validation, streaming model-input verification, strict dependency verification, consumer resolution tests, Configuration Cache compatibility, and expanded JMH, quality, documentation, and release checks. Upgrade the CycloneDX and JMH Gradle plugins and remove Gradle 10 and Java compiler deprecations. BREAKING CHANGE: The core Radixor artifact no longer contains bundled stemmer dictionaries. Applications must add the required model artifacts, the standard model aggregate, or model dependencies managed through the Radixor model BOM.
This commit is contained in:
@@ -17,6 +17,10 @@ The build-time flow is:
|
||||
Dictionary -> Mutable trie -> Reduced trie -> Compiled trie
|
||||
```
|
||||
|
||||
For registered models, the dictionary is an independently versioned GZip resource discovered through a descriptor and verified before this flow begins. The model resource is input to trie construction, not a precompiled trie. See [Model Selection and Loading](model-selection-and-loading.md) for discovery and [Architecture](architecture.md) for component and release boundaries.
|
||||
|
||||
Explicit descriptors and stable model IDs now use the same compiled-value path as language defaults. `loadCompiled(descriptor, ...)` and `loadCompiled(modelId, ...)` first build with serialized patch commands and then map those values to `CompiledPatchCommand` while preserving metadata, reduction semantics, and ranked `getAll` order. Very large inputs can have a high temporary construction peak; PoliMorf is verified in an isolated 6 GiB JVM rather than increasing ordinary test or Gradle daemon heaps.
|
||||
|
||||
At runtime, the compiled trie does not directly return the final stem string. It returns one or more stored patch commands for the addressed key, and those commands are then applied to the original input word.
|
||||
|
||||
## Why this matters
|
||||
@@ -50,3 +54,5 @@ For most readers, the best order is:
|
||||
- [Programmatic usage](programmatic-usage.md)
|
||||
- [CLI compilation](cli-compilation.md)
|
||||
- [Dictionary format](dictionary-format.md)
|
||||
- [Model selection and loading](model-selection-and-loading.md)
|
||||
- [Stemmer models](stemmer-models.md)
|
||||
|
||||
Reference in New Issue
Block a user