Leo Galambos
5e3d3c7c7d
- add the Rust-backed Python API with PyStemmer compatibility - distribute standard compiled models as a separate Python package - generate model artifacts during builds instead of storing them in Git - add GitHub release and Pages-backed package index workflows - add Python tests, benchmarks, documentation, and Gradle integration - refresh the documentation site, branding, and language benchmarks
6.9 KiB
Benchmark Environment And Reports
The values below are environment-specific and must not be read as universal performance claims.
| Item | Value |
|---|---|
| Benchmark date | 2026-08-10 (Europe/Prague) |
| Corpus command | ./gradlew benchmarkCorpusReport --no-daemon |
| Exact-root accuracy command | tools/run-published-accuracy-benchmarks.sh 2026-08-10; all four *BenchmarkQuality classes are selected and timing scores are discarded |
| Stemming-quality command | ./gradlew stemmingQuality --no-daemon |
| Published speed command | tools/run-published-speed-benchmarks.sh 2026-08-10 |
| Published speed run interval | 2026-08-10 16:22:33 to 18:36:41 Europe/Prague (2 h 14 min 8 s, including idle intervals and both JMH suites) |
| Stabilization intervals | 120 s before the main speed matrix; 60 s between the main matrix and coverage-speed suite |
| Corpus and command report | build/reports/jmh/benchmark-corpora.csv |
| Exact-root reports | build/reports/jmh/stemmer-accuracy-2026-08-10.csv and .txt |
| Speed reports | build/reports/jmh/stemmer-speed-2026-08-10.csv and .txt |
| English coverage accuracy reports | build/reports/jmh/english-coverage-accuracy-2026-08-10.csv and .txt |
| English coverage speed reports | build/reports/jmh/english-coverage-speed-2026-08-10.csv and .txt |
| Stemming-quality reports | build/reports/stemming-quality/stemming-quality.csv and .md |
| Environment report | build/reports/jmh/performance-environment-2026-08-10.txt |
| Selected speed methods | build/reports/jmh/published-speed-benchmarks-2026-08-10.txt |
| Comparison scope | Same-language methods used by the 20 language pages; PolishPolimorfStemmerComparisonBenchmark, all quality methods, the separate CISTEM gold-standard experiment, and internal trie microbenchmarks are excluded |
| Model scope | Exactly the 20 IDs declared by Language.defaultModelId(); Polish uses pl-pl-unimorph, and pl-pl-polimorf is not measured |
| Core base commit | b45e143c8484c2ae1d6e31069a2e67232c4f7f29 |
| Measured source state | release@4.1.0-dirty; exact tracked changes and untracked-source checksums are retained as measured-source-2026-08-10.patch and measured-untracked-2026-08-10.sha256 |
| JMH version | 1.37 |
| Speed benchmark mode | Average time, time/op |
| Score unit | ns/op; language pages additionally derive ms/op and ns/token |
| Speed warmup | 5 iterations, 1 s each, independently in every fork |
| Speed measurement | 10 iterations, 1 s each, independently in every fork |
| Speed forks | 3 independent JVM forks |
| Speed threads | 1 |
| Speed fork heap | Fixed -Xms6g -Xmx6g |
| Reported uncertainty | JMH Score Error (99.9%) over 30 measured samples |
| Observed relative uncertainty | Main speed matrix: maximum 10.607%, with 2 of 105 rows above 10%; coverage-speed curve: maximum 12.142%, with 2 of 10 rows above 10%; no published row exceeded 20% |
| Deterministic measurements | Corpus, patch-command distribution, exact-root counters, coverage accuracy, and pairwise stemming quality are evaluated without interpreting runtime scores; no warmup is required |
| JVM reported by JMH | JDK 25.0.4, OpenJDK 64-Bit Server VM, 25.0.4+7 |
| Java runtime | OpenJDK Runtime Environment, Red Hat build 25.0.4+7 |
| JVM invoker | /usr/lib/jvm/java-25-openjdk/bin/java |
| Operating system | Fedora Linux 44 (MATE-Compiz) |
| Kernel | Linux 7.1.7-200.fc44.x86_64 |
| Architecture | x86_64 |
| CPU | AMD Ryzen 5 7600 6-Core Processor |
| Physical / logical CPUs | 6 / 12 |
| CPU frequency policy | amd-pstate-epp; governor performance on every logical CPU; EPP performance; boost enabled |
| CPU affinity | Scheduler default; no explicit pinning |
| Installed memory | 61 GiB reported by the operating system |
| Pre-run idle state | Load average 0.16 / 0.42 / 0.88 after the 120 s idle interval; CPU Tctl 57.4 degrees Celsius; swap unused |
| End-of-run state | Load average 1.59 / 1.49 / 1.36; CPU Tctl 78.1 degrees Celsius |
| Power and idle policy | Developer workstation on stable power; screensaver, suspend, and hibernation disabled |
| Concurrent project work | None during the published speed and coverage-speed run |
The workstation is not a hard real-time system. Normal kernel and desktop background activity was not removed, so the three independent forks and the published 99.9% error interval remain essential parts of result interpretation. Initial/final load and temperature sensor readings are stored in the environment report.
Contracted Trie Baseline
All Radixor rows use contracted compiled patch tries. During compilation, a subtree whose reachable entries all resolve to the same preferred patch command is represented as an accepting leaf. Runtime lookup can therefore stop as soon as that leaf is reached while preserving the preferred result used by get().
Model And Source Identity
benchmark-corpora.csv records the model ID, independent artifact version, and descriptor SHA-256 for every language. Every stemming-quality CSV row repeats the same three fields. The performance environment report additionally records checksums of the executable JMH JAR, runtime classpath manifest, corpus report, quality report, measured source patch, and untracked-source manifest.
The JMH runtime classpath contains the optional model artifact because it is a separately testable project dependency. It is not selected by any published benchmark. The selected-method manifest rejects PolishPolimorf, and the corpus/quality publication validators reject any non-default Polish model.
Report Files
Generated local report files for this benchmark update:
build/reports/jmh/benchmark-corpora.csvbuild/reports/jmh/stemmer-accuracy-2026-08-10.csvbuild/reports/jmh/stemmer-accuracy-2026-08-10.txtbuild/reports/jmh/stemmer-speed-2026-08-10.csvbuild/reports/jmh/stemmer-speed-2026-08-10.txtbuild/reports/jmh/english-coverage-accuracy-2026-08-10.csvbuild/reports/jmh/english-coverage-accuracy-2026-08-10.txtbuild/reports/jmh/english-coverage-speed-2026-08-10.csvbuild/reports/jmh/english-coverage-speed-2026-08-10.txtbuild/reports/jmh/performance-environment-2026-08-10.txtbuild/reports/stemming-quality/stemming-quality.csvbuild/reports/stemming-quality/stemming-quality.mdbuild/reports/stemming-quality/metric-correlations-pearson.csvbuild/reports/stemming-quality/metric-correlations-spearman.csv
The versioned documentation snapshot under docs/benchmarks/data/ preserves the complete stemming-quality matrix. Machine-specific JMH reports remain build artifacts.
Published Metrics
The historical English Radixor versus Porter performance badge is retired. tools/generate-pages-badges.py produces only coverage and mutation badge endpoint JSON files. Benchmark interpretation uses both speed and quality because a narrow or aggressive stemmer can be fast while disagreeing with the dictionary root much more often than Radixor.