feat!: modularize stemmer models and release infrastructure

Move bundled stemmer dictionaries from the core artifact into independently
versioned model modules. Add model discovery and explicit model-loading APIs,
a standard model aggregate, a model BOM, and dedicated model and catalog
release workflows.

Add full PoliMorf integration, model provenance and licensing validation,
streaming model-input verification, strict dependency verification, consumer
resolution tests, Configuration Cache compatibility, and expanded JMH,
quality, documentation, and release checks.

Upgrade the CycloneDX and JMH Gradle plugins and remove Gradle 10 and Java
compiler deprecations.

BREAKING CHANGE: The core Radixor artifact no longer contains bundled stemmer
dictionaries. Applications must add the required model artifacts, the standard
model aggregate, or model dependencies managed through the Radixor model BOM.
This commit is contained in:
2026-07-22 23:33:28 +02:00
parent 9c5b9e331b
commit e7800b29c9
250 changed files with 7297 additions and 905 deletions

View File

@@ -20,7 +20,8 @@ The CSV contains raw TP, FP, FN, and TN counts; raw over/under numerators and de
./gradlew publishStemmingQualityDocumentation
./gradlew verifyStemmingQualityDocumentation
./gradlew test
mkdocs build --strict
./gradlew prepareMkDocsSource
mkdocs build --strict --config-file build/mkdocs/mkdocs.yml
```
`stemmingQuality` performs the expensive complete evaluation and is intentionally not attached to `test` or `check`. It prepares JMH third-party dependencies automatically and writes:
@@ -34,6 +35,26 @@ Audit mode is enabled with `-PstemmingQualityAudit=true`. Language, stemmer, dic
`publishStemmingQualityDocumentation` validates the complete build CSV, copies a versioned documentation snapshot, and replaces only marked generated sections. `verifyStemmingQualityDocumentation` re-renders from the checked-in snapshot and fails on changed values, ordering, missing pages, duplicate keys, arithmetic inconsistencies, policy violations, or stale sections.
The model catalog and rendered site are build outputs under `build/`. They are generated for publication and are never maintained in Git.
For new measurements, record language, stable model ID, model artifact version, descriptor checksum, source dictionary identity/version, core revision, and benchmark configuration. JMH resolves the required default models and optional PoliMorf directly from their individual model JARs; these benchmark-only dependencies are not transitive to ordinary users.
Current model descriptors also record the official repository, dataset, license, attribution,
verification date, transformations, and source-revision status. Exact historical revisions were
not recorded for the legacy UniMorph imports; that limitation is disclosed with
`not-recorded-in-legacy-import` rather than reconstructed. Future imports must record the exact
upstream revision and source-archive checksum. This reproducibility limitation does not replace or
weaken the packaged license and attribution requirements.
Each UniMorph-derived model artifact carries its own notice with the canonical CC BY-SA 3.0 URI,
upstream attribution, transformations, ShareAlike statement, and Leo Galambos contribution notice.
The full CC legal text is not duplicated or presented as a root-project license. PoliMorf retains
its separately packaged BSD-2-Clause license.
For a future full PoliMorf measurement, also record the startup heap separately from benchmark parameters. Complete runtime construction is currently verified with a dedicated 6 GiB maximum heap; this limit is neither a retained-trie measurement nor a setting applied to ordinary JMH runs.
The Pages workflow publishes that staged documentation together with Javadoc, JUnit, PMD, JaCoCo, PIT, representative JMH, SBOM, optional dependency-check output, badge metadata, and retained build history. Its filesystem merge explicitly preserves the `builds/` tree in the separate `gh-pages` publication branch, so documentation regeneration cannot erase durable report URLs.
## Performance benchmark reproduction
The JMH comparison command family is:
@@ -46,7 +67,7 @@ The exact JMH configuration, hardware, operating system, and JDK captured for th
## Recorded and unavailable provenance
The performance documentation records its 2026-07-06 environment, JDK 25.0.3, operating system, and hardware. The quality CSV records the evaluated identifiers and counts but does not embed the Radixor Git revision, generation date, JDK, operating system, dictionary content hash, or immutable upstream revisions for every downloaded source. These fields are explicitly unavailable for this snapshot and are not reconstructed from filesystem timestamps.
The performance documentation records its 2026-07-06 environment, JDK 25.0.3, operating system, and hardware. The quality CSV records the evaluated identifiers and counts but does not embed the Radixor Git revision, generation date, JDK, operating system, model ID, dictionary content hash, or immutable upstream revisions for every downloaded source. These fields are explicitly unavailable for this historical snapshot and are not reconstructed from filesystem timestamps. In particular, the snapshot predates the optional PoliMorf integration and must not be relabeled as `pl-pl-polimorf`.
Dependency versions that are reproducible from repository configuration include Apache Lucene 10.5.0, Morfologik 2.1.9, the Ukrainian dictionary artifact 4.9.1, and JMH 1.37. Other upstream branches or downloaded dictionary revisions should be pinned and embedded in a future result schema.
@@ -59,3 +80,4 @@ Audit reports preserve original multilingual forms and identify high-contributin
## JMH badge compatibility
The quality documentation generator does not invoke JMH, change JMH result formats, or modify badge tooling. Existing JMH result paths and historical badge-compatible inputs remain independent. The repository currently publishes coverage and mutation badge metadata and retains JMH TXT/CSV artifacts as documented in [Environment and reports](environment.md).
See [Model Selection and Loading](../../model-selection-and-loading.md), [Stemmer Models](../../stemmer-models.md), and the generated [model catalog](../../stemmer-model-catalog.md) for current model identities.