Radixor/docs/stemming-quality.md

Stemming quality evaluation

The explicit stemmingQuality analysis measures agreement between stemmer outputs and gold-standard equivalence classes represented by registered multilingual model dictionary rows. Dictionary text remains unchanged; reports and diagnostics use English.

JMH adapters, registries, third-party versions, language mappings, and preparation remain in src/jmh. The evaluator, reports, audits, and tests reside in the standard src/test source set. The former src/stemmingQualityTest source set was removed, and neither analytical nor JMH classes enter the production JAR.

Language and adapter coverage

The authoritative Radixor universe is the validated one-to-one reconciliation of every StemmerPatchTrieLoader.Language value with its registered default model descriptor. All 20 current values have exactly one documented default. Optional comparison models, including pl-pl-polimorf, are identified separately and never replace default benchmark rows. Third-party combinations come only from explicit JMH adapter metadata.

Default Polish evaluation is therefore Radixor with model pl-pl-unimorph. A future PoliMorf evaluation is a distinct Radixor / pl-pl-polimorf row. Evaluation classpaths receive individual models through direct non-production Gradle dependencies; ordinary applications inherit none of them from the core.

Complete PoliMorf trie construction and deterministic stemming smoke fixtures are runtime-verified separately. That functional verification is not a linguistic-quality measurement and does not enter the current default-model quality snapshot.

The expected matrix is constructed before evaluation from stemmer, language, dictionary mode, and supported output policy. Generation fails on missing, duplicate, unexpected, or stale keys.

Dictionary groups and modes

Every usable parsed row contributes one gold-standard group. A distinct surface form is one evaluated item and may belong to several groups. ALL_WORDS preserves every valid form. LOWERCASE_GROUPS_ONLY excludes a complete group containing an uppercase or titlecase Unicode code point. Retained words are not lowercased or normalized by the evaluator.

Output policies

PRIMARY_OUTPUT uses the deterministic JMH output and defines a strict partition.

For multi-output adapters, C(w) is the immutable, sorted, exactly deduplicated candidate set. It is non-null, non-empty, contains no null, and contains the primary output. Radixor obtains alternatives through getAll. The repository's Morphologik lookups can return distinct lemma strings and are multi-output. Configured Hunspell filters can emit several stems at one token position. Other adapters emit only primary rows.

ANY_CANDIDATE is an optimistic oracle-assisted pairwise upper bound. A gold-related pair succeeds when its sets intersect. A gold-negative pair is an error only when both sets are the same singleton; otherwise unequal candidates can be selected for that pair. Choices may vary between pairs and need not form one realizable global assignment.

ALL_CANDIDATES activates every candidate. Two forms are predicted as related when their sets intersect, for both gold-related and gold-negative pairs. This relation can overlap and need not be transitive. A pair sharing several candidates is counted once.

The evaluator verifies:

ANY under <= PRIMARY under
ALL under <= PRIMARY under
ANY under = ALL under
ANY over <= PRIMARY over
ALL over >= PRIMARY over

Pair definitions and efficient counting

For the unique form population W:

underPossible = |{{u,v} subset W : G(u) intersection G(v) is not empty}|
overPossible = C2(|W|) - underPossible

For each distinct form w, let G(w) be its set of included dictionary groups. Two forms are gold-related exactly when their membership sets intersect. Under-stemming counts gold-related pairs that the output relation separates. Over-stemming counts gold-negative pairs that the output relation conflates. A form is processed once, and a pair sharing several groups is counted once. Primary output uses global and per-group stem frequencies with explicit overlap corrections. Candidate sets are canonical signatures counted globally and per group. An inverted candidate-to-signature index discovers intersections, and signature pairs shared through several candidates are deduplicated. ANY_CANDIDATE over-stemming uses only equal singleton signatures. All pair arithmetic uses checked long operations; complete production word pairs are never enumerated.

Confusion and aggregate metrics

TP = underPossible - underError
FN = underError
FP = overError
TN = overPossible - overError

Under-stemming is Paice UI FN/(TP+FN) and over-stemming is Paice OI FP/(TN+FP); their denominators differ. This is an explicit pairwise generalization of Paice's disjoint lemma groups to the overlapping gold relation above. For PRIMARY_OUTPUT and ALL_CANDIDATES, the CSV also publishes TP/FP/FN/TN, precision, recall, specificity, accuracy, balanced accuracy, F0.5, F1, F2, Jaccard, Fowlkes-Mallows, Matthews correlation coefficient, and pairwise error rate. ANY_CANDIDATE uses different optimistic oracle conditions for positive and negative pairs and therefore has no single confusion matrix; its TP/FP/FN/TN and aggregate classification fields are empty, while its explicit under/over numerators and denominators remain available. Metrics use raw counts, not rounded rates. Zero denominators produce n/a in Markdown and empty CSV fields.

Standard Adjusted Rand Index, homogeneity, completeness, V-measure, and normalized mutual information are not calculated. Their ordinary contingency-table definitions require an exclusive gold partition, while these dictionary groups form an overlapping cover.

Micro summaries sum confusion counts before calculation. Macro summaries average defined language values and retain coverage counts. Common-language comparisons use the exact language intersection and never score unsupported languages as zero. Rankings are separated by policy and metric; the default F0.5 choice is navigation, not a universal scientific preference.

Pearson and average-tie-rank Spearman reports use unrounded values and separate dictionary-mode and output-policy cohorts. Fewer than three observations, undefined inputs, and zero variance produce documented missing values. The reports provide reproducible data and make no automatic scientific conclusion.

Exact accuracy and pairwise under-stemming

Exact textual accuracy and pairwise grouping use different denominators. One erroneous form in a 12-form group creates 11 erroneous pairs: with 88 singleton groups, exact accuracy can be 99% while pairwise under-stemming is 11/C2(12) = 16.666667%. Singleton groups affect word accuracy but add no within-group pairs.

Running the analysis

./gradlew stemmingQuality
./gradlew stemmingQuality -PstemmingQualityStemmer=Radixor -PstemmingQualityLanguage=DE_DE -PstemmingQualityMode=ALL_WORDS -PstemmingQualityAudit=true

Optional properties are stemmingQualityLanguage, stemmingQualityStemmer, stemmingQualityMode, stemmingQualityOutputPolicy, stemmingQualityRankMetric, stemmingQualityAudit, and stemmingQualityAuditLimit. Policies are PRIMARY_OUTPUT, ANY_CANDIDATE, and ALL_CANDIDATES. Filtered reports carry -filtered and cannot overwrite complete output.

Generated files under build/reports/stemming-quality/ include stemming-quality.md, stemming-quality.csv, metric-correlations-pearson.csv, metric-correlations-spearman.csv, and optional audit Markdown. Every CSV scenario records the exact dictionary model ID, independent model version, and descriptor SHA-256.

Limitations

These measurements evaluate agreement with the available dictionary grouping. They do not capture every semantic, morphological, downstream, or dataset-specific property. ANY_CANDIDATE is optimistic and may not be globally realizable. ALL_CANDIDATES measures an overlap graph rather than a partition. Language coverage must remain visible in cross-stemmer comparisons. No single published metric establishes universal superiority; multiple metrics and their correlations are provided for transparent scientific assessment. The checked-in quality snapshot is regenerated from all 20 current default models. The optional PoliMorf model is not part of it and must not be attributed to the default Polish results. See Model Selection and Loading and the generated model catalog.