Leo Galambos
5e3d3c7c7d
- add the Rust-backed Python API with PyStemmer compatibility - distribute standard compiled models as a separate Python package - generate model artifacts during builds instead of storing them in Git - add GitHub release and Pages-backed package index workflows - add Python tests, benchmarks, documentation, and Gradle integration - refresh the documentation site, branding, and language benchmarks
7.9 KiB
Performance (Python)
This page reports runtime stemming throughput of the Python implementation against
common Python stemmers, and — crucially — documents exactly how the comparison
is made fair. The scripts are in the repository (python/benchmarks/); anyone
can reproduce the numbers.
!!! info "Published single-machine measurement"
These results were regenerated on 2026-08-08 on the current benchmark
workstation: Fedora Linux 44 (7.1.6-201.fc44.x86_64), AMD Ryzen 5 5625U
(6 cores / 12 threads), CPython 3.14.6, Rust 1.97.1, and a release wheel.
All logical CPUs used the schedutil governor. Absolute timings remain
machine-specific; compare ratios only within this run.
What is measured
- Runtime stemming only. Model construction / dictionary compilation happens once in setup and is excluded from every timing.
- Workload = the Java JMH corpus. The changed-token corpus derived from
the bundled UniMorph gold-standard dictionaries: every dictionary field paired
with its line's root, normalized
trim().lower(), keeping only tokens that differ from their root (the forms a stemmer must actually rewrite), padded to ≥ 5 000 tokens. This is identical to the JavaLanguageBenchmarkCorpus. - Batch sizes 10/20/50/100 are swept and a line is fit to
per_call(N) = intercept + N · slopeas a descriptive scaling summary. This is an unconstrained OLS fit, so noise may produce a negative intercept; it is not a physical decomposition of runtime. The best of many repeats is reported.
Fairness: making the comparison apples-to-apples
Three asymmetries silently distort stemmer comparisons. Each is neutralized, and where it cannot be neutralized the effect is described.
- Result caching — neutralized. PyStemmer caches results by default
(
maxCacheSize=10000). Since a benchmark stems the same corpus repeatedly, that cache would turn measured passes into dictionary lookups rather than stemming. The harness explicitly disables both PyStemmer's cache (maxCacheSize=0) and radixor's default cache (cache_size=0). The other engines have no cache. - Lowercasing — neutralized. Snowball (PyStemmer, snowballstemmer) and
CISTEM differ in whether they case-fold. Snowball does no case handling;
it assumes pre-lowercased input. The corpus is pre-lowercased for every
engine, and radixor is therefore run with
lowercase=Falseso it does the same work. On already-lowercased input radixor returns identical results either way. Exception — CISTEM: it always performs its own lowercasing and German umlaut normalization internally and cannot be told to skip it, so CISTEM does slightly more normalization work than the others. This unavoidable extra work biases the comparison modestly in radixor's favour, not CISTEM's. - Hidden delegation — neutralized.
snowballstemmerdelegates to PyStemmer when PyStemmer is installed (they become the same C code). The harness bypasses that and uses snowballstemmer's genuine pure-Python backend, and records each engine's backing module + whether it is a compiled extension so the provenance is verifiable.
Environment and parameters
| Item | Published value |
|---|---|
| CPU | AMD Ryzen 5 5625U with Radeon Graphics |
| CPU topology | 6 physical cores / 12 logical CPUs |
| OS | Fedora Linux 44, kernel 7.1.6-201.fc44.x86_64 |
| CPU governor | schedutil on all 12 logical CPUs; boost enabled |
| Python | CPython 3.14.6 |
| Radixor | 4.1.0, release-mode ABI3 wheel, cache disabled |
| PyStemmer | 3.1.0 (libstemmer_c 3.1.0), cache disabled |
| snowballstemmer | 3.1.1, forced pure-Python backend |
| NLTK | 3.10.2 |
| Workload | 5,000 changed tokens per language and measurement |
| Batch sizes | 10, 20, 50, 100 |
| Timing | best of 15 measured passes after 3 warm-up passes |
The authoritative command was:
./gradlew pythonBenchmarkAllLanguagesBatch --rerun-tasks
It completed successfully in 3 minutes 33 seconds and emitted
the full per-size CSV and JSON reports under
build/reports/python-benchmarks/.
Results — batch size 100, cache disabled
The table reports nanoseconds per word at N=100 (lower is better). A dash
means that the engine has no implementation for that language. Every available
competitor was measured in the same process, with the same corpus and batch
partitioning.
| Language | Radixor | PyStemmer (Snowball C) | CISTEM (pure Py) | snowballstemmer (pure Py) | NLTK Porter (pure Py) |
|---|---|---|---|---|---|
Czech (cs) |
224.3 | 236.6 | — | 4,835.2 | — |
Danish (da) |
178.3 | 267.6 | — | 8,568.9 | — |
German (de) |
230.9 | 635.5 | 3,341.9 | 33,654.1 | — |
English (en) |
180.5 | 331.9 | — | 20,195.0 | 7,740.3 |
Spanish (es) |
184.2 | 316.6 | — | 19,640.1 | — |
Persian (fa) |
210.1 | 497.1 | — | 32,732.3 | — |
Finnish (fi) |
227.8 | 258.8 | — | 12,339.5 | — |
French (fr) |
234.2 | 503.7 | — | 36,161.9 | — |
Hebrew (he) |
228.6 | — | — | — | — |
Hungarian (hu) |
198.2 | 264.7 | — | 13,694.3 | — |
Italian (it) |
170.8 | 517.0 | — | 34,504.6 | — |
Norwegian Bokmål (nb) |
187.1 | 239.7 | — | 7,457.6 | — |
Dutch (nl) |
187.1 | 354.8 | — | 18,148.2 | — |
Norwegian Nynorsk (nn) |
168.7 | 231.2 | — | 7,489.4 | — |
Polish (pl) |
194.6 | 214.5 | — | 5,282.9 | — |
Portuguese (pt) |
166.9 | 293.2 | — | 21,157.2 | — |
Russian (ru) |
273.4 | 414.4 | — | 15,703.8 | — |
Swedish (sv) |
189.3 | 212.5 | — | 5,351.4 | — |
Ukrainian (uk) |
221.5 | — | — | — | — |
Yiddish (yi) |
227.5 | 624.2 | — | 33,251.6 | — |
Radixor won all 18 / 18 direct PyStemmer comparisons. At N=100, its
geometric-mean speedup was 1.67×; the largest direct advantage was 3.03×
for Italian. Across all 20 Radixor languages, throughput ranged from 3.66 to
5.99 million words/s.
CISTEM comparison for German
The German row also provides a direct comparison with CISTEM:
| Engine | Implementation | N=100 | vs radixor |
|---|---|---|---|
| radixor | Rust trie | 230.9 ns/word | — |
| PyStemmer (de) | Snowball C | 635.5 ns/word | 2.75× slower |
| CISTEM | pure Python (nltk) |
3,341.9 ns/word | 14.47× slower |
CISTEM has no batch entry point (it is a per-word Python loop), so its per-word cost is flat across batch sizes and batching cannot amortize it. It is a compact ~40-rule German heuristic with no dictionary — a different design point that trades coverage for simplicity. Because CISTEM's unavoidable normalization work modestly biases the measurement in radixor's favour (point 2 above), the 14.47× result is not a perfectly normalization-matched ratio.
The all-language Gradle task does not measure stage-level profiling or cached lookup performance. This page therefore does not mix such figures from an older workstation into the published run.
A note on comparability of quality
These are speed comparisons. Radixor is a lexicon-trained transformation stemmer: it learns patch commands from UniMorph-grounded word–stem evidence and can generalize those transformations beyond exact training entries. Snowball, Porter, and CISTEM use hand-written rule systems. They produce different stems and are not directly comparable on output; see the shared linguistic quality methodology for how stemming quality is assessed separately from throughput.
Reproduce
pip install -r python/benchmarks/requirements-bench.txt
./gradlew pythonBenchmarkAllLanguagesBatch --rerun-tasks
The run prints the machine/Python/engine versions and each engine's backing
module (provenance), and writes per-point rows (CSV) plus the full report
including environment (JSON). Methodology and fairness notes live in
python/benchmarks/README.md.