- add the Rust-backed Python API with PyStemmer compatibility - distribute standard compiled models as a separate Python package - generate model artifacts during builds instead of storing them in Git - add GitHub release and Pages-backed package index workflows - add Python tests, benchmarks, documentation, and Gradle integration - refresh the documentation site, branding, and language benchmarks
328 lines
22 KiB
Markdown
328 lines
22 KiB
Markdown
# Finnish Stemmer Benchmarks
|
||
|
||
This page reports same-language stemming benchmarks for Finnish. Accuracy is listed first because speed without root agreement is not enough to interpret stemmer quality.
|
||
|
||
All speed values are environment-specific and were measured on the hardware and JVM listed in the [benchmark overview](../index.md). Speed benchmark operations process changed dictionary tokens only. Accuracy uses the complete Radixor dictionary for the language.
|
||
|
||
Radixor must not be read as simply "slower" when a narrow competitor has a lower timing row. In these tables Radixor is the quality-oriented baseline: its exact-root accuracy is typically close to 100%, while many faster rule-based, light, minimal, or possessive filters reach that speed by doing much less linguistic work and often score far lower in `All exact` and `Changed exact`. The Radixor rows in this benchmark refresh use the contracted compiled patch trie: compilation collapses uniform patch-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred stemming result measured by the accuracy pass. The [EnglishRadixorDictionaryCoverageBenchmark](../reference/english-coverage.md) table shows the resulting quality/speed envelope explicitly. The same interpretation applies to this language page: speed rows must be read together with the accuracy table above them.
|
||
|
||
## Dictionary Corpus
|
||
|
||
| Model ID | Model version | Language | Dictionary rows | Complete quality tokens | Already-root tokens | Changed tokens | JMH timing tokens |
|
||
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |
|
||
| `fi-fi-default` | `1.0.0` | `FI_FI` | 57,027 | 1,865,215 | 110,525 | 1,754,690 | 1,754,690 |
|
||
|
||
## Radixor Patch Command Distribution
|
||
|
||
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete default-model dictionary. The total number of preferred patch commands analyzed for this language is **1,865,215**.
|
||
|
||
| Command class | Meaning | Word forms | Share |
|
||
| --- | --- | ---: | ---: |
|
||
| `AppendCharacterCommand` | Appends one character to the end of the word form. | 1,117 | 0.060% |
|
||
| `BackwardCompoundCommand` | Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 1,175,880 | 63.043% |
|
||
| `DeleteSuffixCommand` | Deletes one or more trailing characters from the word form. | 570,130 | 30.566% |
|
||
| `PreserveCommand` | Returns the word form unchanged because it already matches the preferred root. | 112,029 | 6.006% |
|
||
| `ReplaceLastCharacterCommand` | Replaces the final character of the word form. | 6,059 | 0.325% |
|
||
|
||
## Accuracy
|
||
|
||
Accuracy is computed from JMH auxiliary counters in the current report. The counters are deterministic for a fixed corpus and stemmer; percentages divide matching counters by evaluated counters from the same report and are not timing metrics.
|
||
|
||
| Stemmer | All exact | Changed exact | Root preserved | Note |
|
||
| --- | ---: | ---: | ---: | --- |
|
||
| Radixor | 98.661% | 98.803% | 96.408% | Radixor dictionary-trained patch-command stemmer. |
|
||
| Lucene SnowballFilter | 10.991% | 10.268% | 22.471% | Lucene TokenFilter integration path around the Snowball algorithm. |
|
||
| Official Snowball direct | 10.995% | 10.272% | 22.462% | Official Snowball generated Java stemmer; rule-based suffix algorithm. |
|
||
| Lucene FinnishLightStemFilter | 4.351% | 4.294% | 5.264% | Light suffix stemmer; intentionally narrower than Radixor's dictionary-trained transformation model. |
|
||
|
||
## Speed
|
||
|
||
Speed uses JMH average time, 5 warmup iterations, 10 measurement iterations, 3 independent forks, and 1 thread. Relative factor is computed against the single Radixor row on this language page. Values below 1.000 are faster than that Radixor baseline; values above 1.000 are slower.
|
||
|
||
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
|
||
| --- | --- | ---: | ---: | ---: | ---: | --- |
|
||
| Radixor | `finnishRadixor` | 225.954 | 2.940 | 128.8 | 1.000 | Radixor dictionary-trained patch-command stemmer. |
|
||
| Lucene FinnishLightStemFilter | `finnishLuceneFinnishLightStemFilter` | 168.756 | 6.027 | 96.2 | 0.747 | Light Finnish suffix stemmer. |
|
||
| Official Snowball direct | `snowballDirect[FINNISH]` | 247.984 | 16.083 | 141.3 | 1.097 | Official Snowball generated Java stemmer; direct API. |
|
||
| Lucene SnowballFilter | `luceneSnowballFilter[FINNISH]` | 321.331 | 9.790 | 183.1 | 1.422 | Lucene TokenFilter path around Snowball; includes TokenStream overhead. |
|
||
|
||
## Interpretation Notes
|
||
|
||
- Radixor is a dictionary-trained patch-command stemmer. Its learned transformations can generalize beyond the word forms listed in the training resource.
|
||
- Light, minimal, plural, and possessive filters are narrow baselines. They can be fast because they intentionally perform less linguistic work.
|
||
- Lucene TokenFilter rows include TokenStream, attribute, and required normalization overhead. Direct rows measure exposed direct APIs.
|
||
- Morfologik rows are dictionary-based and can emit multiple terms for one input token. Quality rows use the first returned term when no ranking weight is available.
|
||
- Snowball rows are rule-based generated suffix stemmers; they are useful algorithmic baselines, not dictionary-root equivalence guarantees.
|
||
|
||
<!-- STEMMING-QUALITY:START -->
|
||
|
||
## Stemming Quality
|
||
|
||
Runtime performance and linguistic grouping quality are independent dimensions. This section evaluates language `FI_FI` using the complete validated stemming-quality result matrix. Every distinct surface form is one evaluated item and can belong to several dictionary groups. Two forms are a positive pair when their group-membership sets intersect and a negative pair when those sets are disjoint. A pair shared through several groups is counted once. Exact equality with a predetermined lemma is not required.
|
||
|
||
`ALL_WORDS` includes every valid group and its original forms. `LOWERCASE_GROUPS_ONLY` excludes an entire group when any Unicode code point is uppercase or titlecase; retained words are not lowercased or otherwise rewritten. This isolates case-handling effects without changing retained inputs. [Download the complete machine-readable result snapshot](../data/stemming-quality.csv).
|
||
|
||
### Evaluation Scope and Key Findings
|
||
|
||
The default model is `fi-fi-default`, loaded from classpath resource `org/egothor/stemmer/models/fi-fi-default/stemmer.gz`. The following findings compare only deterministic `PRIMARY_OUTPUT` rows over identical included groups; candidate policies are reported separately as capability analyses.
|
||
|
||
- **ALL_WORDS:** `Radixor` ranks first by balanced accuracy at **0.984838** among 4 deterministic stemmers. The runner-up is `SNOWBALL FINNISH LUCENE FILTER` at 0.740279, a difference of 0.244559. This rank does not imply leadership in throughput or every secondary metric.
|
||
- **LOWERCASE_GROUPS_ONLY:** `Radixor` ranks first by balanced accuracy at **0.988242** among 4 deterministic stemmers. The runner-up is `SNOWBALL FINNISH DIRECT` at 0.738543, a difference of 0.249699. This rank does not imply leadership in throughput or every secondary metric.
|
||
### `ALL_WORDS`
|
||
|
||
This mode contains **6 result rows**, **4 evaluated stemmers**, and **3 output policies**. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. `PRIMARY_OUTPUT` and `ALL_CANDIDATES` rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. `ANY_CANDIDATE` has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
|
||
|
||
#### `PRIMARY_OUTPUT` ranking
|
||
|
||
<div class="quality-summary" markdown="1">
|
||
|
||
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|
||
|---:|---|---:|---:|---:|
|
||
|1|Radixor|0.984838|<0.000001%|3.032474%|
|
||
|2|SNOWBALL FINNISH LUCENE FILTER|0.740279|0.000081%|51.944179%|
|
||
|3|SNOWBALL FINNISH DIRECT|0.739870|0.000060%|52.025976%|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|0.695725|0.000094%|60.854936%|
|
||
|
||
</div>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Classification metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|PRIMARY_OUTPUT|0.999974|0.969675|1.000000|0.984838|0.999999|0.000001|
|
||
|2|SNOWBALL FINNISH LUCENE FILTER|PRIMARY_OUTPUT|0.921471|0.480558|0.999999|0.740279|0.999989|0.000011|
|
||
|3|SNOWBALL FINNISH DIRECT|PRIMARY_OUTPUT|0.940647|0.479740|0.999999|0.739870|0.999989|0.000011|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|PRIMARY_OUTPUT|0.890914|0.391451|0.999999|0.695725|0.999987|0.000013|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Pair-relation metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|PRIMARY_OUTPUT|0.993763|0.984591|0.975587|0.969650|0.984708|0.984708|
|
||
|2|SNOWBALL FINNISH LUCENE FILTER|PRIMARY_OUTPUT|0.778598|0.631685|0.531413|0.461652|0.665448|0.665443|
|
||
|3|SNOWBALL FINNISH DIRECT|PRIMARY_OUTPUT|0.789035|0.635413|0.531862|0.465645|0.671764|0.671760|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|PRIMARY_OUTPUT|0.709787|0.543915|0.440884|0.373546|0.590550|0.590545|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Raw pair counts</summary>
|
||
|
||
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|PRIMARY_OUTPUT|30511413|804|954186|1599841738533|804 / 1599841739337|954186 / 31465599|
|
||
|2|SNOWBALL FINNISH LUCENE FILTER|PRIMARY_OUTPUT|15121052|1288634|16344547|1599840450703|1288634 / 1599841739337|16344547 / 31465599|
|
||
|3|SNOWBALL FINNISH DIRECT|PRIMARY_OUTPUT|15095314|952479|16370285|1599840786858|952479 / 1599841739337|16370285 / 31465599|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|PRIMARY_OUTPUT|12317229|1508153|19148370|1599840231184|1508153 / 1599841739337|19148370 / 31465599|
|
||
|
||
</details>
|
||
|
||
#### `ANY_CANDIDATE` oracle bounds
|
||
|
||
These results are measured, not missing. `ANY_CANDIDATE` answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically **not applicable**, rather than unknown.
|
||
|
||
<div class="quality-summary quality-summary--oracle" markdown="1">
|
||
|
||
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|
||
|---|---:|---:|
|
||
|Radixor|0.000000%|0.000000%|
|
||
|
||
</div>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Oracle-bound pair counts</summary>
|
||
|
||
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|
||
|---|---:|---:|
|
||
|Radixor|0 / 1599841739337|0 / 31465599|
|
||
|
||
</details>
|
||
|
||
#### `ALL_CANDIDATES` ranking
|
||
|
||
<div class="quality-summary" markdown="1">
|
||
|
||
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|
||
|---:|---|---:|---:|---:|
|
||
|1|Radixor|1.000000|<0.000001%|0.000000%|
|
||
|
||
</div>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Classification metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|ALL_CANDIDATES|0.999926|1.000000|1.000000|1.000000|1.000000|0.000000|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Pair-relation metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|ALL_CANDIDATES|0.999941|0.999963|0.999985|0.999926|0.999963|0.999963|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Raw pair counts</summary>
|
||
|
||
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|ALL_CANDIDATES|31465599|2327|0|1599841737010|2327 / 1599841739337|0 / 31465599|
|
||
|
||
</details>
|
||
|
||
#### Multi-output analysis
|
||
|
||
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
|
||
|
||
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|
||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||
|Radixor|954186|804|1523|34395|1.922815%|6|1826768|
|
||
|
||
### `LOWERCASE_GROUPS_ONLY`
|
||
|
||
This mode contains **6 result rows**, **4 evaluated stemmers**, and **3 output policies**. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. `PRIMARY_OUTPUT` and `ALL_CANDIDATES` rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. `ANY_CANDIDATE` has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
|
||
|
||
#### `PRIMARY_OUTPUT` ranking
|
||
|
||
<div class="quality-summary" markdown="1">
|
||
|
||
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|
||
|---:|---|---:|---:|---:|
|
||
|1|Radixor|0.988242|<0.000001%|2.351587%|
|
||
|2|SNOWBALL FINNISH DIRECT|0.738543|0.000062%|52.291340%|
|
||
|3|SNOWBALL FINNISH LUCENE FILTER|0.738344|0.000062%|52.331112%|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|0.694308|0.000077%|61.138333%|
|
||
|
||
</div>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Classification metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|PRIMARY_OUTPUT|0.999973|0.976484|1.000000|0.988242|1.000000|0.000000|
|
||
|2|SNOWBALL FINNISH DIRECT|PRIMARY_OUTPUT|0.939988|0.477087|0.999999|0.738543|0.999989|0.000011|
|
||
|3|SNOWBALL FINNISH LUCENE FILTER|PRIMARY_OUTPUT|0.939951|0.476689|0.999999|0.738344|0.999989|0.000011|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|PRIMARY_OUTPUT|0.911893|0.388617|0.999999|0.694308|0.999987|0.000013|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Pair-relation metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|PRIMARY_OUTPUT|0.995185|0.988089|0.981093|0.976459|0.988159|0.988159|
|
||
|2|SNOWBALL FINNISH DIRECT|PRIMARY_OUTPUT|0.787224|0.632932|0.529209|0.462985|0.669668|0.669664|
|
||
|3|SNOWBALL FINNISH LUCENE FILTER|PRIMARY_OUTPUT|0.786987|0.632573|0.528815|0.462601|0.669376|0.669372|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|PRIMARY_OUTPUT|0.718421|0.544981|0.438999|0.374553|0.595296|0.595291|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Raw pair counts</summary>
|
||
|
||
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|PRIMARY_OUTPUT|30037514|804|723369|1504706134249|804 / 1504706135053|723369 / 30760883|
|
||
|2|SNOWBALL FINNISH DIRECT|PRIMARY_OUTPUT|14675605|936938|16085278|1504705198115|936938 / 1504706135053|16085278 / 30760883|
|
||
|3|SNOWBALL FINNISH LUCENE FILTER|PRIMARY_OUTPUT|14663371|936765|16097512|1504705198288|936765 / 1504706135053|16097512 / 30760883|
|
||
|4|FINNISH LUCENE FINNISH LIGHT STEM FILTER|PRIMARY_OUTPUT|11954192|1155011|18806691|1504704980042|1155011 / 1504706135053|18806691 / 30760883|
|
||
|
||
</details>
|
||
|
||
#### `ANY_CANDIDATE` oracle bounds
|
||
|
||
These results are measured, not missing. `ANY_CANDIDATE` answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically **not applicable**, rather than unknown.
|
||
|
||
<div class="quality-summary quality-summary--oracle" markdown="1">
|
||
|
||
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|
||
|---|---:|---:|
|
||
|Radixor|0.000000%|0.000000%|
|
||
|
||
</div>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Oracle-bound pair counts</summary>
|
||
|
||
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|
||
|---|---:|---:|
|
||
|Radixor|0 / 1504706135053|0 / 30760883|
|
||
|
||
</details>
|
||
|
||
#### `ALL_CANDIDATES` ranking
|
||
|
||
<div class="quality-summary" markdown="1">
|
||
|
||
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|
||
|---:|---|---:|---:|---:|
|
||
|1|Radixor|1.000000|<0.000001%|0.000000%|
|
||
|
||
</div>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Classification metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|ALL_CANDIDATES|0.999927|1.000000|1.000000|1.000000|1.000000|0.000000|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Pair-relation metrics</summary>
|
||
|
||
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|ALL_CANDIDATES|0.999942|0.999964|0.999985|0.999927|0.999964|0.999964|
|
||
|
||
</details>
|
||
|
||
<details class="quality-details" markdown="1"><summary>Raw pair counts</summary>
|
||
|
||
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|
||
|---:|---|---|---:|---:|---:|---:|---:|---:|
|
||
|1|Radixor|ALL_CANDIDATES|30760883|2235|0|1504706132818|2235 / 1504706135053|0 / 30760883|
|
||
|
||
</details>
|
||
|
||
#### Multi-output analysis
|
||
|
||
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
|
||
|
||
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|
||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||
|Radixor|723369|804|1431|22060|1.271628%|6|1758300|
|
||
|
||
### Output Policies and Metric Definitions
|
||
|
||
Each distinct surface form is one item and may belong to several gold groups. Two forms are gold-related when their membership sets intersect; a relation shared by several groups is counted once. `PRIMARY_OUTPUT` uses one deterministic stem per form. `ANY_CANDIDATE` is an optimistic oracle-assisted pairwise upper bound: a gold-related pair succeeds when candidates intersect, while a gold-negative pair succeeds when a non-colliding selection exists. Candidate choices may differ between pairs, so this is not deterministic runtime behaviour and does not define one confusion matrix. `ALL_CANDIDATES` activates every returned candidate; forms are related when candidate sets intersect.
|
||
|
||
For `PRIMARY_OUTPUT` and `ALL_CANDIDATES`, `TP = underPossiblePairs - underErrorPairs`, `FN = underErrorPairs`, `FP = overErrorPairs`, and `TN = overPossiblePairs - overErrorPairs`. `ANY_CANDIDATE` publishes only its separate oracle-assisted under/over bounds; confusion-derived metrics are mathematically inapplicable and are not presented in its language-page section. Their machine-readable CSV fields remain empty. Undefined metric denominators in otherwise applicable policies are rendered as `n/a`.
|
||
|
||
- Under-stemming rate (Paice UI): `FN / (TP + FN)`, the false-negative rate over gold-related pairs.
|
||
- Over-stemming rate (Paice OI): `FP / (TN + FP)`, the false-positive rate over gold-negative pairs.
|
||
- Pairwise precision: `TP / (TP + FP)`, the fraction of predicted conflations that are gold-standard positive pairs.
|
||
- Pairwise recall: `TP / (TP + FN)`, the fraction of gold-standard positive pairs successfully connected.
|
||
- Pairwise specificity: `TN / (TN + FP)`, the fraction of gold-negative pairs correctly separated.
|
||
- Balanced accuracy: `(recall + specificity) / 2`. It gives equal weight to positive and negative pair classes and is less dominated by the large true-negative class than ordinary accuracy. It does not replace the raw errors or other metrics.
|
||
- Pairwise F-beta: `((1 + betaSquared) * TP) / (((1 + betaSquared) * TP) + (betaSquared * FN) + FP)`. F0.5 emphasizes precision and penalizes over-stemming more; F1 weights precision and recall equally; F2 emphasizes recall and penalizes under-stemming more.
|
||
- MCC: `(TP * TN - FP * FN) / sqrt((TP + FP) * (TP + FN) * (TN + FP) * (TN + FN))`. It uses all confusion counts and remains useful under class imbalance, except when its denominator is degenerate.
|
||
- Jaccard index: `TP / (TP + FP + FN)`.
|
||
- Fowlkes–Mallows index: `sqrt(precision * recall)`.
|
||
- Pairwise accuracy: `(TP + TN) / (TP + TN + FP + FN)`. It can be dominated by true-negative cross-group pairs.
|
||
- Pairwise error rate: `(FP + FN) / (TP + TN + FP + FN)`.
|
||
|
||
Standard ARI, homogeneity, completeness, V-measure, and NMI are not calculated: their usual contingency-table definitions require an exclusive gold partition, while this gold standard is an overlapping cover.
|
||
|
||
### Provenance
|
||
|
||
- Authoritative source: `docs/benchmarks/data/stemming-quality.csv`
|
||
- Source SHA-256: `d34f325da320a2e040b54d8d8b5c216d70448f08cfb8659a423e99882aa1afb5`
|
||
- Evaluation command: `./gradlew stemmingQuality --no-daemon`
|
||
- Dictionary language: `FI_FI`
|
||
- Processing modes: `ALL_WORDS`, `LOWERCASE_GROUPS_ONLY`
|
||
- Stemmer versions and transitive artifacts: resolved by the repository's JMH Gradle configuration and `gradle.lockfile`
|
||
- Model ID, version, and SHA-256: recorded in every CSV row
|
||
- Run date, core source state, JDK, operating system, and hardware: recorded on the [benchmark environment page](../reference/environment.md)
|
||
|
||
<!-- STEMMING-QUALITY:END -->
|