feat(python): add native distribution and release infrastructure

- add the Rust-backed Python API with PyStemmer compatibility
- distribute standard compiled models as a separate Python package
- generate model artifacts during builds instead of storing them in Git
- add GitHub release and Pages-backed package index workflows
- add Python tests, benchmarks, documentation, and Gradle integration
- refresh the documentation site, branding, and language benchmarks
This commit is contained in:
2026-08-10 22:34:32 +02:00
parent b45e143c84
commit 5e3d3c7c7d
139 changed files with 11420 additions and 747 deletions

111
python/benchmarks/README.md Normal file
View File

@@ -0,0 +1,111 @@
# Runtime stemming benchmarks
These scripts measure **runtime stemming throughput only** — model construction
and dictionary compilation happen once during setup and are **excluded** from
every timing. Anyone can reproduce the numbers on their own machine.
## What is measured, and why batch sizes
Each engine is driven through its batch entry point over a fixed word budget
(default 5 000 tokens), split into batches of **10, 20, 50, 100** words. The
harness fits the descriptive line `per_call(N) ≈ intercept + N · slope` across
those sizes. The fit is unconstrained and timing noise can make its intercept
negative, so it describes observed scaling rather than physically separating
overhead from word work. We report the *best* (minimum) of many repeats — the
microbenchmark convention that suppresses OS/GC noise.
## Data — identical to the Java JMH benchmarks
The workload is the **changed-token corpus** built from the repository's
canonical dictionaries (`models/<model>/src/modelInput/stemmer.gz`),
mirroring `LanguageBenchmarkCorpus` in the Java project: each dictionary field
is paired with its line's root, normalized `trim().lower()`, and only tokens
that **differ** from their root are kept, in dictionary order, padded to ≥ 5 000
tokens. See `corpus.py`.
## Fairness — three things that quietly break stemmer comparisons
Getting a *fair* comparison turned out to matter more than any micro-optimization.
Three asymmetries, if left in, make the numbers meaningless:
1. **Result caching.** PyStemmer caches results by default (`maxCacheSize=10000`).
Because a benchmark stems the same corpus every repeat, that cache turns
measured passes into dict lookups rather than stemming. **The harness
explicitly disables both caches**: PyStemmer uses `maxCacheSize=0` and
radixor uses `cache_size=0`, so both engines do real stemming.
snowballstemmer-pure, nltk-porter, and cistem have no cache.
2. **Lowercasing.** Snowball/PyStemmer do no case handling — they assume the
caller pre-lowercased the input (our corpus is pre-lowercased for everyone).
radixor normally lowercases internally; for a same-work comparison the
harness runs radixor with **`lowercase=False`** (assume-already-lowercased),
so Snowball and radixor do identical normalization work on identical input.
CISTEM always performs its own lowercasing and German umlaut normalization;
that unavoidable extra work modestly biases its comparison in radixor's
favour.
3. **Delegation.** `snowballstemmer` delegates to PyStemmer when it is installed
(they become the same C code). The harness bypasses that and uses
snowballstemmer's genuine pure-Python backend, and records each engine's
backing module + whether it is a compiled extension (`--json`) as proof.
## Engines compared
| Engine | Implementation | Batch API |
|---|---|---|
| `radixor` | Rust patch-command trie (cache disabled) | `stem_batch` — one FFI call per batch |
| `PyStemmer` | Snowball C `libstemmer` (cache disabled) | `stemWords(list)` — one C call per batch |
| `snowballstemmer-pure` | Official **pure-Python** Snowball | `stemWords(list)` — Python loop |
| `nltk-porter` | Porter (English), pure Python | scalar loop |
| `cistem` | CISTEM (German), pure Python (`nltk`) | scalar loop |
## Reproduce
```bash
cd python/
maturin develop --release # build the radixor extension into your env
cd ..
./gradlew pythonBuildStandardModels # generate and package standard models
pip install --no-deps build/python/dist/standard/radixor_models_standard-0.0.0-py3-none-any.whl
cd python/
pip install -r benchmarks/requirements-bench.txt
python benchmarks/run_benchmark.py --language en de fr ru fi \
--sizes 10 20 50 100 --repeats 21 \
--json benchmarks/results.json --csv benchmarks/results.csv
```
From the repository root, the Gradle integration builds an isolated host wheel
and benchmarks Radixor plus every available comparison engine over all
supported languages with the fixed 10/20/50/100 size sweep:
```bash
./gradlew pythonBenchmarkAllLanguagesBatch
```
The generated CSV and JSON reports are placed in
`build/reports/python-benchmarks/`.
Comparison engines are auto-detected in the environment selected by the Gradle
`pythonExecutable` property. Install `requirements-bench.txt` in that environment
to enable the complete comparison set.
The run prints machine/Python/engine versions and each engine's backing module,
and writes per-point rows (CSV) plus the full report incl. environment and
provenance (JSON), so results are self-describing and verifiable.
## Published results
The canonical, current single-machine results and complete environment metadata
are published on the documentation site's [Python performance
page](../../docs/python/performance.md). Keeping the measured table in one place
prevents results from different CPUs or benchmark runs from being mixed.
## Interpretation
- In the published 2026-08-08 run, **radixor is the fastest stemmer measured in
Python** in all 18 languages directly shared with PyStemmer.
- The benchmark intentionally disables caches. Cached-operation performance is
outside this suite and must not be inferred from its results.
- radixor and Snowball remain different *classes* of stemmer: radixor is
dictionary-based (UniMorph gold coverage), Snowball is rule-based. radixor
gives dictionary-quality stems *and* the best measured throughput.