feat(python): add native distribution and release infrastructure

- add the Rust-backed Python API with PyStemmer compatibility - distribute standard compiled models as a separate Python package - generate model artifacts during builds instead of storing them in Git - add GitHub release and Pages-backed package index workflows - add Python tests, benchmarks, documentation, and Gradle integration - refresh the documentation site, branding, and language benchmarks
2026-08-10 22:34:32 +02:00
parent b45e143c84
commit 5e3d3c7c7d
139 changed files with 11420 additions and 747 deletions
--- a/docs/python/fast-track.md
+++ b/docs/python/fast-track.md
@@ -0,0 +1,66 @@
+# Python Fast Track
+
+This is the shortest path from an empty Python environment to a working
+Radixor stemmer. The installation includes the native runtime and the separate
+standard-model package with 20 precompiled language models.
+
+## 1. Install
+
+=== "PyPI"
+
+    ```bash
+    python -m pip install --only-binary=:all: radixor
+    ```
+
+=== "GitHub Releases"
+
+    ```bash
+    python -m pip install --only-binary=:all: \
+      --index-url https://leogalambos.github.io/Radixor/python/simple/ radixor
+    ```
+
+PyPI publication is pending, and the GitHub index becomes live with the first
+Python releases. Until then, follow the source-checkout procedure on
+[Installation and Builds](installation.md).
+
+Radixor supports CPython 3.9 and newer. A JVM, Java dependency, and source
+dictionary are not required.
+
+## 2. Stem words
+
+```python
+from radixor import Stemmer
+
+stemmer = Stemmer("en")
+
+print(stemmer.stem("running"))
+print(stemmer.stem_batch(["running", "studies", "cars"]))
+```
+
+Expected first output:
+
+```text
+run
+```
+
+`stem()` and `stem_batch()` preserve Radixor's original API: a word for which
+the trie finds no patch command produces `None`.
+
+## 3. Use PyStemmer-compatible fallback semantics
+
+For a low-friction migration from PyStemmer, use the compatible method names:
+
+```python
+stemmer.stemWord("running")
+stemmer.stemWords(["running", "unknown_word"])
+```
+
+These methods return the original input whenever no patch command is found, so
+their results are always strings rather than `None`.
+
+## Next
+
+- Continue with the [Python Quick Start](quick-start.md) for model selection,
+  batch processing, custom compiled models, and deployment guidance.
+- Use [Python Usage and API](usage.md) as the method reference.
+- Review the reproducible [Python performance results](performance.md).
--- a/docs/python/index.md
+++ b/docs/python/index.md
@@ -0,0 +1,96 @@
+# Radixor for Python
+
+The **`radixor`** package is Radixor's native Python implementation. It is not
+a wrapper around the Java library and does not require a JVM: it is a
+compiled extension (Rust, via [PyO3](https://pyo3.rs/) and
+[maturin](https://www.maturin.rs/)) that loads precompiled patch-command tries
+derived from the same canonical UniMorph data as the Java models.
+
+```python
+from radixor import Stemmer
+
+s = Stemmer("en")
+s.stem("running")               # 'run'
+s.stem_batch(["cats", "ran"])   # ['cat', 'run']
+```
+
+- [Fast Track](fast-track.md) — install and produce the first stem.
+- [Quick Start](quick-start.md) — the complete application-oriented learning path.
+- [Installation and building](installation.md) — Linux, Windows, macOS.
+- [Usage and examples](usage.md) — batch API, caching, and custom models.
+- [Dictionary compilation](model-compilation.md) — prepare a version 7 binary
+  once and share it with Python or Java.
+- [Performance](performance.md) — fair, reproducible comparisons vs PyStemmer,
+  snowballstemmer, NLTK Porter, and CISTEM.
+
+The language and model-ID mapping is shared with Java and maintained on the
+[Built-in Languages](../built-in-languages.md) page. Installing `radixor`
+also resolves the separate pure `radixor-models-standard` distribution containing
+the 20 default compiled models; Java applications select independently
+versioned model JARs.
+
+!!! note "Same models, same results, different runtime"
+    The standard Python models are compiled from the identical canonical
+    dictionaries with the identical production reduction configuration
+    (`MERGE_SUBTREES_WITH_EQUIVALENT_DOMINANT_GET_RESULTS`, 75 % / 3×,
+    uniform-subtree contraction, `LOWERCASE_WITH_LOCALE_ROOT`, `AS_IS`
+    diacritics, `storeOriginal=true`). For a word present in a model, both
+    implementations return the same dominant stem. The compiled **binary format
+    is shared** (see below), so a model compiled by one side loads in the other.
+
+## Java vs. Python: read this first
+
+The two implementations solve the same problem but make different runtime
+trade-offs. Mixing their mental models causes confusion, so the differences are
+stated explicitly. **Neither is “better”** — they target different runtimes.
+
+| Aspect | Java (`org.egothor:radixor`) | Python (`radixor`) |
+|---|---|---|
+| Runtime | JVM library | Compiled extension (Rust/PyO3), no JVM |
+| Distribution | Maven JAR + model JARs | `abi3` wheel (one wheel per OS/arch, Python ≥ 3.9) |
+| Hot-path data structure | `CompiledNode` graph; routines operate on caller-owned **`char[]`** with zero-copy normalized lookups and visitor sinks (`EntrySink`) | Flat **CSR arrays** (no per-node objects); reused UTF‑16 scratch buffers |
+| Result cache | **None** — `get()` is stateless and re-stems every call | **Bounded**, 10,000 entries by default (matching PyStemmer); `Stemmer(cache_size=0)` disables it |
+| Batch API | Not a batch call; you loop and reuse `char[]`/visitors to avoid allocation | **`stem_batch()` / `stem_all_batch()`** — one Python↔Rust crossing amortized over the whole list |
+| Reduction modes | All three modes selectable at compile time | Fixed to the production `DOMINANT` mode |
+| Extending a compiled trie | **Supported** — add words/transformations to an already-compiled trie without recompiling | **Not exposed** — compile from a dictionary (or load a compiled binary) |
+| Model resolution | `ServiceLoader` registry, descriptors, SHA‑256 integrity checks | Separate standard data package; catalog/format/SHA‑256 validation before synchronous native load |
+| Normalization control | Case and diacritic modes fully configurable | `lowercase` toggle; diacritics `AS_IS` (models are built this way) |
+| Binary format | `StemmerPatchTrieBinaryIO` v7 read/write (versioned, fingerprinted) | v7 read/write, **inner stream byte-identical** to Java; **v7 only** (no legacy v1–v6) |
+| Multiple stems | `getAll(...)` | `stem_all()` / `stem_all_batch()` |
+
+### Runtime capabilities that differ
+
+To avoid surprises, these Java capabilities are **not** in the Python package:
+
+- **Extending / incrementally growing a compiled trie.** Python compiles from a
+  source dictionary (or loads a compiled binary); it does not add words to an
+  existing compiled trie at runtime.
+- **Selectable reduction modes.** Only the production `DOMINANT` mode is used.
+- **Pluggable provider discovery.** Python currently resolves one known
+  standard provider directly; entry-point plugins are not yet exposed.
+- **Legacy binary versions.** Only stream version 7 is read/written.
+- **Diacritic-removal modes** beyond `AS_IS` (the bundled models are `AS_IS`).
+
+### Python-specific capabilities
+
+- A **batch API** (`stem_batch`) that amortizes the Python↔native boundary — the
+  single most important call for throughput from Python.
+- A **bounded result cache** (`cache_size=10_000` by default) for workloads with
+  repeated tokens. It is shared by `stem()`, `stemWord()`, `stem_batch()`, and
+  `stemWords()`; pass `cache_size=0` to disable it. The `stem_all*()` methods are
+  not cached.
+- A `lowercase=False` mode to skip per-lookup lowercasing when the caller
+  guarantees already-lowercased input.
+
+## Interoperability
+
+The compiled binary is Radixor's **v7 trie stream**, and the Python runtime writes
+the *inner stream byte-for-byte identically to the Java*
+`StemmerPatchTrieBinaryIO`. Consequently:
+
+- a model compiled by **Java** (`org.egothor.stemmer.Compile` /
+  `StemmerPatchTrieBinaryIO.write`) loads in **Python**, and
+- a model compiled by **Python** (`radixor.compile(...)`) loads in **Java**.
+
+(The outer gzip wrapper bytes differ between the two gzip implementations; this
+is irrelevant — both sides decompress to the same v7 stream.)
--- a/docs/python/installation.md
+++ b/docs/python/installation.md
@@ -0,0 +1,215 @@
+# Installing and building (Linux, Windows, macOS)
+
+The Python package ships as an **`abi3` wheel** — a single binary per
+OS/architecture that works on CPython ≥ 3.9 (including 3.14) through the stable
+ABI. Most users just `pip install`; building from source is only needed for
+development or unsupported platforms.
+
+## Install from PyPI
+
+PyPI is the intended primary index once the Radixor projects are approved and
+published there:
+
+```bash
+python -m pip install --only-binary=:all: radixor
+```
+
+PyPI publication is not live yet. Until the `radixor` and
+`radixor-models-standard` project pages exist, this command cannot install the
+project.
+
+## Install compiled packages from GitHub
+
+Python releases are published as immutable GitHub Release assets. A small
+PEP 503 index on GitHub Pages exposes both packages to `pip`:
+
+```bash
+python -m pip install --only-binary=:all: \
+  --index-url https://leogalambos.github.io/Radixor/python/simple/ radixor
+```
+
+The index links directly to checksummed wheel assets in GitHub Releases; Pages
+does not duplicate the package files. It is not live until the first Python
+model and native releases have been published. This was verified before the
+initial release: the URL returned HTTP 404 and the repository contained no
+Python Release assets.
+
+Do not configure the GitHub index as an `--extra-index-url`: `pip` does not
+prioritize one index over another. Use it as the sole `--index-url`, as shown
+above. The binary-only constraint also prevents an accidental source build
+with an unprepared toolchain.
+
+Wheels are provided for Linux (`manylinux`), Windows, and macOS
+(x86‑64 and Apple Silicon). A source distribution is also published; installing
+it triggers a source build, which needs the toolchain described below.
+
+## Install or build from the GitHub source repository
+
+Building requires the **Rust toolchain**, a linker for the target platform, and
+**maturin**. The crate and its dependencies contain no project C/C++ sources,
+but the selected Rust target still needs its normal platform linker and SDK.
+
+```bash
+git clone https://github.com/leogalambos/Radixor
+cd Radixor
+python -m venv python/.venv
+# activate the venv (see per-OS note below)
+pip install maturin build setuptools wheel pytest
+./gradlew pythonBuildStandardModels
+pip install --no-deps build/python/dist/standard/radixor_models_standard-0.0.0-py3-none-any.whl
+cd python
+maturin develop --release        # compile + install into the venv
+pytest -q                        # run the test suite
+```
+
+For a reproducible application build, check out a release tag or exact commit
+instead of a moving branch. Repository descriptors deliberately use the
+non-release placeholder `0.0.0`; release workflows inject the version from the
+Git tag into isolated staging trees. Consequently, source-checkout development
+installs use `--no-deps` for the generated development model wheel, while
+published packages carry normal release versions and dependency resolution
+works automatically.
+
+The native distribution requires
+`radixor-models-standard>=1.0,<2.0`; an installation of `radixor`
+resolves it automatically. The local `--no-deps` command installs the generated
+data wheel for development without contacting a package index.
+
+The installed package source (package index, environment, and `sys.path`) is
+the model-provider trust boundary. Manifest SHA-256 checks detect accidental
+corruption after installation; they do not authenticate a malicious provider.
+
+## Integrity and provenance
+
+Every GitHub Release contains `SHA256SUMS` for its wheel and source archives.
+The release workflows also create GitHub artifact attestations for those
+archives. After downloading a release, maintainers and users can verify it with:
+
+```bash
+sha256sum --check SHA256SUMS
+gh attestation verify radixor-<version>-<wheel-tags>.whl \
+  --repo leogalambos/Radixor
+```
+
+Python packages do **not** reuse the OpenPGP key configured for Java/Maven
+Central publications. Java's `SIGNING_KEY` and `SIGNING_PASSWORD` produce Maven
+signatures; Python currently uses release checksums plus GitHub's
+identity-bound build-provenance attestation. A future PyPI publication should
+use PyPI Trusted Publishing and its supported attestations rather than copying
+the Java signing mechanism.
+
+## Build through Gradle
+
+From the repository root, the supported build entry point creates the native
+wheel/sdist and pure standard-model wheel/sdist:
+
+```bash
+./gradlew pythonBuild
+```
+
+Artifacts are written below `build/python/dist/`; they are not installed into
+the invoking interpreter. `./gradlew pythonVerifyDistributions` also validates
+archive contents, dependency metadata, checksums, v7 headers, and a fresh
+offline wheel-only installation. Platform convenience tasks are also available:
+
+```bash
+./gradlew pythonBuildLinux
+./gradlew pythonBuildWindows
+./gradlew pythonBuildMacos
+```
+
+The task matching the current host delegates to `pythonBuild`. A non-host task
+uses the default Rust target for that operating system and therefore succeeds
+only when its Rust target, linker, and platform SDK are installed. Override a
+default with `pythonLinuxTarget`, `pythonWindowsTarget`, or `pythonMacosTarget`.
+For example:
+
+```bash
+./gradlew pythonBuildWindows -PpythonWindowsTarget=x86_64-pc-windows-gnu
+```
+
+Use `-PpythonExecutable=/path/to/python` or
+`-PmaturinExecutable=/path/to/maturin` when those tools are not on `PATH`.
+These Gradle tasks are the repository integration; direct `maturin` commands
+remain useful while developing inside `python/`.
+
+### Prerequisites per platform
+
+=== "Linux"
+
+    ```bash
+    # Rust (rustup); most distros already ship Python 3.9+
+    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+    python3 -m venv .venv && source .venv/bin/activate
+    pip install maturin pytest
+    ```
+    Producing distributable `manylinux` wheels is easiest with
+    `maturin build --release` inside the official maturin/`manylinux` container.
+
+=== "macOS"
+
+    ```bash
+    brew install rustup-init && rustup-init -y      # or: curl https://sh.rustup.rs | sh
+    python3 -m venv .venv && source .venv/bin/activate
+    pip install maturin pytest
+    maturin develop --release
+    ```
+    Both Apple Silicon (`aarch64-apple-darwin`) and Intel
+    (`x86_64-apple-darwin`) are supported; `maturin build --release --target
+    universal2-apple-darwin` produces a universal wheel.
+
+=== "Windows"
+
+    ```powershell
+    winget install -e --id Rustlang.Rustup
+    py -m venv .venv
+    .\.venv\Scripts\Activate.ps1
+    pip install maturin pytest
+    maturin develop --release
+    ```
+    The self-contained GNU toolchain avoids needing Visual Studio Build Tools:
+    ```powershell
+    rustup toolchain install stable-x86_64-pc-windows-gnu
+    rustup default stable-x86_64-pc-windows-gnu
+    ```
+    (The MSVC toolchain also works if you already have the C++ Build Tools.)
+
+### Python 3.14 (and newer than your PyO3 knows about)
+
+Because the extension targets the stable ABI, it links against interpreters
+newer than the PyO3 version was released for. If a build against a very new
+CPython refuses, set the forward-compatibility flag once in the build shell:
+
+=== "Linux / macOS"
+
+    ```bash
+    export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
+    maturin develop --release
+    ```
+
+=== "Windows (PowerShell)"
+
+    ```powershell
+    $env:PYO3_USE_ABI3_FORWARD_COMPATIBILITY = "1"
+    maturin develop --release
+    ```
+
+## Verifying the build
+
+```bash
+python -c "from radixor import Stemmer; print(Stemmer('en').stem('running'))"   # -> run
+pytest -q
+```
+
+## Notes and caveats
+
+- **Model packaging.** Neither runtime distribution contains textual
+  dictionaries. `radixor-models-standard` ships 20 compiled gzip v7 resources,
+  the checksum/provenance manifest, and CC BY-SA 3.0 notices; optional
+  `pl-pl-polimorf` is excluded.
+- **Catalog compatibility.** Radixor 4.1 accepts model-distribution major 1
+  (`>=1.0,<2.0`) carrying the independent 2026.1 catalog identity. Missing,
+  incompatible, or corrupt data produces an
+  actionable error before native loading.
+- **Toolchain PATH.** After installing rustup, open a fresh shell (or ensure
+  `~/.cargo/bin` is on `PATH`) so `maturin` can find `cargo`/`rustc`.
--- a/docs/python/model-compilation.md
+++ b/docs/python/model-compilation.md
@@ -0,0 +1,105 @@
+# Compiling Dictionaries in Python
+
+The Python package can compile a textual Radixor dictionary into the shared
+version 7 binary trie format. This moves dictionary parsing, patch-command
+generation, trie construction, reduction, and serialization out of application
+startup.
+
+Use this workflow when the application owns its model file. Standard language
+aliases already load validated, precompiled `.rxc` resources from
+`radixor-models-standard`; they do not parse or compile textual dictionaries
+when a `Stemmer` is constructed.
+
+## Source format
+
+The input is a plain UTF-8 or GZip-compressed UTF-8 tab-separated dictionary.
+The first column is the canonical stem and the remaining columns are its known
+surface forms:
+
+```text
+run	running	runs	ran
+cat	cats
+```
+
+Remarks beginning with `#` or `//` are accepted. The complete syntax and
+normalization rules are documented on the shared [Dictionary Format](../dictionary-format.md)
+page.
+
+## Compile a model
+
+```python
+import radixor
+
+radixor.compile(
+    "stemmer.tsv.gz",
+    "english.rxc",
+    language="en",
+)
+```
+
+`language` is used to choose traversal direction when `backward` is omitted.
+Persian (`fa`), Hebrew (`he`), and Yiddish (`yi`) use forward traversal; the
+other bundled languages use backward traversal. For a custom language, select
+the direction explicitly:
+
+```python
+radixor.compile(
+    "custom.tsv",
+    "custom.rxc",
+    backward=True,
+    store_original=True,
+    lowercase=True,
+)
+```
+
+The arguments are:
+
+| Argument | Meaning |
+|---|---|
+| `source` | Plain or GZip-compressed textual dictionary. |
+| `out_path` | Destination for the GZip-compressed version 7 trie. |
+| `language` | Optional alias or model ID used only to infer traversal direction. |
+| `backward` | Explicit traversal direction; overrides inference from `language`. |
+| `store_original` | Include a no-op mapping for every canonical stem. Defaults to `True`. |
+| `lowercase` | Record lowercase lookup normalization in the compiled metadata. Defaults to `True`. |
+
+Compilation refuses an input that is already a compiled trie. The destination
+is written by the native extension; the caller is responsible for choosing its
+location and for replacing an existing file only when that is intended.
+
+## Load the compiled model
+
+```python
+from radixor import Stemmer
+
+stemmer = Stemmer(compiled="english.rxc")
+print(stemmer.stem("running"))
+```
+
+`Stemmer(path=...)` also auto-detects textual dictionaries and compiled version
+7 streams, but `compiled=` communicates the deployment intent more clearly.
+Traversal direction, `store_original`, and lookup normalization are already
+stored in a compiled artifact; constructor build options do not rewrite them.
+
+## Java interoperability
+
+Python and Java share the inner version 7 trie stream. A binary produced by
+`radixor.compile(...)` can be loaded by Java's
+`StemmerPatchTrieLoader.loadBinaryCompiled(...)`, and Python can load a version
+7 artifact written by `StemmerPatchTrieBinaryIO`.
+
+The outer GZip bytes need not be identical because compressor implementations
+may differ. Interoperability applies to the decompressed version 7 stream and
+its persisted metadata.
+
+## Differences from the Java compiler
+
+Python compilation intentionally exposes the production dominant-result
+configuration used by the Python runtime. Java additionally offers three selectable
+reduction modes, more normalization controls, incremental extension, and a CLI
+with explicit overwrite handling. Use [Java CLI Compilation](../cli-compilation.md)
+when those controls are required.
+
+For normal Python use, compile once during preparation, deploy the resulting
+`.rxc` file as an application-owned asset, and reuse one loaded `Stemmer` at
+runtime.
--- a/docs/python/performance.md
+++ b/docs/python/performance.md
@@ -0,0 +1,160 @@
+# Performance (Python)
+
+This page reports **runtime stemming throughput** of the Python implementation against
+common Python stemmers, and — crucially — documents exactly how the comparison
+is made fair. The scripts are in the repository (`python/benchmarks/`); anyone
+can reproduce the numbers.
+
+!!! info "Published single-machine measurement"
+    These results were regenerated on 2026-08-08 on the current benchmark
+    workstation: Fedora Linux 44 (`7.1.6-201.fc44.x86_64`), AMD Ryzen 5 5625U
+    (6 cores / 12 threads), CPython 3.14.6, Rust 1.97.1, and a release wheel.
+    All logical CPUs used the `schedutil` governor. Absolute timings remain
+    machine-specific; compare ratios only within this run.
+
+## What is measured
+
+- **Runtime stemming only.** Model construction / dictionary compilation happens
+  once in setup and is excluded from every timing.
+- **Workload = the Java JMH corpus.** The *changed-token* corpus derived from
+  the bundled UniMorph gold-standard dictionaries: every dictionary field paired
+  with its line's root, normalized `trim().lower()`, keeping only tokens that
+  differ from their root (the forms a stemmer must actually rewrite), padded to
+  ≥ 5 000 tokens. This is identical to the Java `LanguageBenchmarkCorpus`.
+- **Batch sizes 10/20/50/100** are swept and a line is fit to `per_call(N) =
+  intercept + N · slope` as a descriptive scaling summary. This is an
+  unconstrained OLS fit, so noise may produce a negative intercept; it is not a
+  physical decomposition of runtime. The *best* of many repeats is reported.
+
+## Fairness: making the comparison apples-to-apples
+
+Three asymmetries silently distort stemmer comparisons. Each is neutralized, and
+where it **cannot** be neutralized the effect is described.
+
+1. **Result caching — neutralized.** PyStemmer caches results by default
+   (`maxCacheSize=10000`). Since a benchmark stems the same corpus repeatedly,
+   that cache would turn measured passes into dictionary lookups rather than
+   stemming. The harness explicitly disables **both** PyStemmer's cache
+   (`maxCacheSize=0`) and radixor's default cache (`cache_size=0`). The other
+   engines have no cache.
+2. **Lowercasing — neutralized.** Snowball (PyStemmer, snowballstemmer) and
+   CISTEM differ in whether they case-fold. Snowball does **no** case handling;
+   it assumes pre-lowercased input. The corpus is pre-lowercased for every
+   engine, and radixor is therefore run with **`lowercase=False`** so it does
+   the same work. On already-lowercased input radixor returns identical results
+   either way. **Exception — CISTEM:** it always performs its own lowercasing
+   and German umlaut normalization internally and cannot be told to skip it, so
+   CISTEM does *slightly more* normalization work than the others. This
+   unavoidable extra work biases the comparison modestly **in radixor's
+   favour**, not CISTEM's.
+3. **Hidden delegation — neutralized.** `snowballstemmer` delegates to PyStemmer
+   when PyStemmer is installed (they become the same C code). The harness
+   bypasses that and uses snowballstemmer's genuine pure-Python backend, and
+   records each engine's backing module + whether it is a compiled extension so
+   the provenance is verifiable.
+
+## Environment and parameters
+
+| Item | Published value |
+|---|---|
+| CPU | AMD Ryzen 5 5625U with Radeon Graphics |
+| CPU topology | 6 physical cores / 12 logical CPUs |
+| OS | Fedora Linux 44, kernel `7.1.6-201.fc44.x86_64` |
+| CPU governor | `schedutil` on all 12 logical CPUs; boost enabled |
+| Python | CPython 3.14.6 |
+| Radixor | 4.1.0, release-mode ABI3 wheel, cache disabled |
+| PyStemmer | 3.1.0 (`libstemmer_c` 3.1.0), cache disabled |
+| snowballstemmer | 3.1.1, forced pure-Python backend |
+| NLTK | 3.10.2 |
+| Workload | 5,000 changed tokens per language and measurement |
+| Batch sizes | 10, 20, 50, 100 |
+| Timing | best of 15 measured passes after 3 warm-up passes |
+
+The authoritative command was:
+
+```bash
+./gradlew pythonBenchmarkAllLanguagesBatch --rerun-tasks
+```
+
+It completed successfully in 3 minutes 33 seconds and emitted
+the full per-size CSV and JSON reports under
+`build/reports/python-benchmarks/`.
+
+## Results — batch size 100, cache disabled
+
+The table reports nanoseconds per word at `N=100` (lower is better). A dash
+means that the engine has no implementation for that language. Every available
+competitor was measured in the same process, with the same corpus and batch
+partitioning.
+
+| Language | Radixor | PyStemmer (Snowball C) | CISTEM (pure Py) | snowballstemmer (pure Py) | NLTK Porter (pure Py) |
+|---|---:|---:|---:|---:|---:|
+| Czech (`cs`) | **224.3** | 236.6 | — | 4,835.2 | — |
+| Danish (`da`) | **178.3** | 267.6 | — | 8,568.9 | — |
+| German (`de`) | **230.9** | 635.5 | 3,341.9 | 33,654.1 | — |
+| English (`en`) | **180.5** | 331.9 | — | 20,195.0 | 7,740.3 |
+| Spanish (`es`) | **184.2** | 316.6 | — | 19,640.1 | — |
+| Persian (`fa`) | **210.1** | 497.1 | — | 32,732.3 | — |
+| Finnish (`fi`) | **227.8** | 258.8 | — | 12,339.5 | — |
+| French (`fr`) | **234.2** | 503.7 | — | 36,161.9 | — |
+| Hebrew (`he`) | **228.6** | — | — | — | — |
+| Hungarian (`hu`) | **198.2** | 264.7 | — | 13,694.3 | — |
+| Italian (`it`) | **170.8** | 517.0 | — | 34,504.6 | — |
+| Norwegian Bokmål (`nb`) | **187.1** | 239.7 | — | 7,457.6 | — |
+| Dutch (`nl`) | **187.1** | 354.8 | — | 18,148.2 | — |
+| Norwegian Nynorsk (`nn`) | **168.7** | 231.2 | — | 7,489.4 | — |
+| Polish (`pl`) | **194.6** | 214.5 | — | 5,282.9 | — |
+| Portuguese (`pt`) | **166.9** | 293.2 | — | 21,157.2 | — |
+| Russian (`ru`) | **273.4** | 414.4 | — | 15,703.8 | — |
+| Swedish (`sv`) | **189.3** | 212.5 | — | 5,351.4 | — |
+| Ukrainian (`uk`) | **221.5** | — | — | — | — |
+| Yiddish (`yi`) | **227.5** | 624.2 | — | 33,251.6 | — |
+
+Radixor won all **18 / 18** direct PyStemmer comparisons. At `N=100`, its
+geometric-mean speedup was **1.67×**; the largest direct advantage was **3.03×**
+for Italian. Across all 20 Radixor languages, throughput ranged from **3.66 to
+5.99 million words/s**.
+
+### CISTEM comparison for German
+
+The German row also provides a direct comparison with CISTEM:
+
+| Engine | Implementation | N=100 | vs radixor |
+|---|---|---|---|
+| **radixor** | Rust trie | **230.9 ns/word** | — |
+| PyStemmer (de) | Snowball C | 635.5 ns/word | 2.75× slower |
+| **CISTEM** | pure Python (`nltk`) | **3,341.9 ns/word** | **14.47× slower** |
+
+CISTEM has no batch entry point (it is a per-word Python loop), so its per-word
+cost is flat across batch sizes and batching cannot amortize it. It is a compact
+~40-rule German heuristic with no dictionary — a different design point that
+trades coverage for simplicity. Because CISTEM's unavoidable normalization work
+modestly biases the measurement in radixor's favour (point 2 above), the 14.47×
+result is not a perfectly normalization-matched ratio.
+
+The all-language Gradle task does not measure stage-level profiling or cached
+lookup performance. This page therefore does not mix such figures from an older
+workstation into the published run.
+
+## A note on comparability of *quality*
+
+These are **speed** comparisons. Radixor is a **lexicon-trained transformation
+stemmer**: it learns patch commands from UniMorph-grounded word–stem evidence
+and can generalize those transformations beyond exact training entries.
+Snowball, Porter, and CISTEM use hand-written rule systems. They produce
+different stems and are not directly comparable on output; see the shared
+[linguistic quality
+methodology](../benchmarks/reference/linguistic-quality.md) for how stemming
+quality is assessed separately from throughput.
+
+## Reproduce
+
+```bash
+pip install -r python/benchmarks/requirements-bench.txt
+./gradlew pythonBenchmarkAllLanguagesBatch --rerun-tasks
+```
+
+The run prints the machine/Python/engine versions and each engine's backing
+module (provenance), and writes per-point rows (CSV) plus the full report
+including environment (JSON). Methodology and fairness notes live in
+`python/benchmarks/README.md`.
--- a/docs/python/quick-start.md
+++ b/docs/python/quick-start.md
@@ -0,0 +1,130 @@
+# Python Quick Start
+
+Radixor's Python implementation is a native Rust extension with a Python API.
+It uses the same learned patch-command model and version 7 compiled-trie format
+as the Java implementation, without requiring a JVM.
+
+## 1. Install the runtime and standard models
+
+Create an isolated environment and install Radixor:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate       # Windows: .venv\Scripts\activate
+```
+
+=== "PyPI"
+
+    ```bash
+    python -m pip install --only-binary=:all: radixor
+    ```
+
+=== "GitHub Releases"
+
+    ```bash
+    python -m pip install --only-binary=:all: \
+      --index-url https://leogalambos.github.io/Radixor/python/simple/ radixor
+    ```
+
+PyPI publication is pending. The GitHub option becomes live when the first
+Python GitHub Releases populate the Pages-backed package index. See
+[Installation and Builds](installation.md) for availability and source builds.
+
+The `radixor` wheel contains code. Its required
+`radixor-models-standard` dependency contains 20 precompiled models. The
+standard package excludes textual source dictionaries and optional PoliMorf
+data, which keeps startup on the direct compiled-model path.
+
+## 2. Select and reuse a stemmer
+
+Construct a stemmer once and retain it for the lifetime of the application:
+
+```python
+from radixor import Stemmer
+
+english = Stemmer("en")
+polish = Stemmer("pl")
+
+print(english.stem("running"))  # 'run'
+print(polish.stem("koty"))
+```
+
+Short aliases such as `en`, `de`, and `pl` resolve to the documented default
+model IDs. A full ID such as `us-uk-default` selects the same model explicitly.
+The complete mapping is listed under [Built-in Languages](../built-in-languages.md).
+
+## 3. Prefer batch calls for collections
+
+Crossing the Python/native boundary once per collection is substantially more
+efficient than a Python loop of scalar calls:
+
+```python
+words = ["running", "studies", "better", "cars"]
+stems = english.stem_batch(words)
+```
+
+`stem_batch()` preserves input order and returns one item per word. Entries can
+be `None` when the trie has no applicable patch command.
+
+Repeated natural-language tokens use a bounded result cache shared by the
+scalar and batch APIs. Its default capacity is 10,000 entries, matching
+PyStemmer; choose another bound or pass `0` to disable it:
+
+```python
+english = Stemmer("en", cache_size=10_000)
+uncached = Stemmer("en", cache_size=0)
+```
+
+The cache covers `stem()`, `stemWord()`, `stem_batch()`, and `stemWords()`;
+the `stem_all*()` methods are not cached.
+
+## 4. Migrate from PyStemmer
+
+Radixor exposes PyStemmer's familiar scalar and batch method names:
+
+```python
+stemmer = Stemmer("en")
+
+stemmer.stemWord("running")
+stemmer.stemWords(["running", "unknown_word"])
+```
+
+`stemWord()` and `stemWords()` return unmatched input unchanged. This removes
+the `None` fallback checks required by Radixor's original `stem()` and
+`stem_batch()` methods, so most migration work is limited to the package import
+and dependency change.
+
+## 5. Load a custom compiled model
+
+The standard installation covers the maintained default catalog. A custom
+version 7 model can be loaded directly:
+
+```python
+custom = Stemmer(compiled="models/domain-english.rxc")
+```
+
+To compile a maintained textual dictionary during a preparation step:
+
+```python
+from radixor import compile
+
+compile("dictionaries/domain.tsv.gz", "models/domain-english.rxc", language="en")
+```
+
+Deploy the resulting `.rxc` file and load it at application startup. See
+[Dictionary Compilation](model-compilation.md) for format interoperability and
+the production compilation profile.
+
+## 6. Production checklist
+
+- Pin compatible `radixor` and `radixor-models-standard` releases in the
+  application's dependency lock.
+- Construct and reuse stemmers instead of rebuilding them per request.
+- Use batch calls for token collections.
+- Choose `stem*` or `stemWord*` semantics deliberately for unmatched words.
+- Treat custom dictionaries and compiled models as trusted application input.
+- Regression-test representative vocabulary before changing model versions.
+
+Continue with [Installation and Builds](installation.md) for wheel/platform
+details, [Usage and API](usage.md) for the complete call surface, or
+[Performance](performance.md) for benchmark methodology and results.
--- a/docs/python/usage.md
+++ b/docs/python/usage.md
@@ -0,0 +1,173 @@
+# Usage and examples
+
+## Creating a stemmer
+
+```python
+from radixor import Stemmer
+
+s = Stemmer("en")                      # by language code (bundled model)
+s = Stemmer("us-uk-default")           # by full model ID
+s = Stemmer(path="my_dictionary.gz")   # a custom gzipped TSV source dictionary
+s = Stemmer(compiled="en.rxc")         # a pre-compiled binary (instant load)
+```
+
+The traversal direction is derived from the language (right-to-left `fa`/`he`/`yi`
+use FORWARD, all others BACKWARD); override with `backward=True|False` for a
+custom `path=`.
+
+## Stemming a single word
+
+```python
+s.stem("running")     # 'run'
+s.stem("cats")        # 'cat'
+s.stem("zzzzz")       # None  -> not reducible / unknown
+```
+
+`stem()` returns the single **dominant** stem, or `None`.
+
+!!! info "Why a known word may return itself"
+    A surface form that is *also* a canonical headword (e.g. an English word
+    that is both its own lemma and an inflection of another lemma) returns
+    itself, because the dominant sense is “this word is its own stem”. The
+    inflectional reading is still available via `stem_all()`.
+
+## Batch stemming — the fast path
+
+For anything beyond a handful of words, use the batch API. It crosses the
+Python↔native boundary **once** for the whole list, which is the dominant cost
+when stemming from Python.
+
+```python
+words = ["running", "cats", "stemming", "quickly"]
+s.stem_batch(words)          # ['run', 'cat', 'stem', 'quick'] (None for unknowns)
+```
+
+```python
+# Multiple candidate stems per word (ambiguity preserved):
+s.stem_all("running")               # e.g. ['run', 'runn']
+s.stem_all_batch(["running", "cats"])
+```
+
+## PyStemmer-compatible methods
+
+Radixor also exposes PyStemmer's scalar and batch method names. They differ
+from the native Radixor methods only when the trie cannot find a patch command:
+
+| Method | Recognized word | Word without a patch command | Return type |
+| --- | --- | --- | --- |
+| `stem(word)` | dominant stem | `None` | `str | None` |
+| `stem_batch(words)` | dominant stem at the same position | `None` at the same position | `list[str | None]` |
+| `stemWord(word)` | dominant stem | original input word | `str` |
+| `stemWords(words)` | dominant stem at the same position | original input word at the same position | `list[str]` |
+
+Use `stemWord()` and `stemWords()` when migrating code that expects
+PyStemmer's no-`None` contract:
+
+```python
+import radixor as Stemmer
+
+# The rest of this common PyStemmer call pattern remains unchanged.
+s = Stemmer.Stemmer("english")
+
+s.stemWord("running")                  # 'run'
+s.stemWord("unknown_word")             # 'unknown_word'
+s.stemWords(["running", "unknown_word"])
+# ['run', 'unknown_word']
+```
+
+`stemWords()` retains input order and makes one Python-to-Rust call for the
+whole list. PyStemmer's full language names, such as `"english"` and
+`"czech"`, are accepted for bundled Radixor languages alongside two-letter
+codes and full model IDs.
+
+The compatibility contract covers these method names, full language aliases,
+and unmatched-word fallback behavior. Radixor configuration keywords remain
+Radixor-specific: use `cache_size`, not PyStemmer's `maxCacheSize`. Both
+libraries default to a cache capacity of 10,000 entries.
+
+## Bounded result cache
+
+Real text repeats tokens. The default bounded cache returns the already-built
+result object on a recognized-word hit (a reference-count bump — no
+re-stemming, no new result string). Unknown words are cached as misses, so
+`stemWord()` and `stemWords()` still create their required original-word
+result. Its default capacity is **10,000 entries**, matching PyStemmer:
+
+```python
+s = Stemmer("en")                       # cache up to 10,000 distinct input words
+s = Stemmer("en", cache_size=50_000)    # choose a custom capacity
+s = Stemmer("en", cache_size=0)         # explicitly disable caching
+```
+
+One cache is shared by `stem()`, `stemWord()`, `stem_batch()`, and
+`stemWords()`. The `stem_all()` and `stem_all_batch()` methods are not cached.
+Caching never changes results; it only avoids recomputation. Entries are
+inserted until the configured capacity is reached; there is no eviction. For a
+high-cardinality stream without useful token repetition, use `cache_size=0`.
+
+## Skipping lowercasing for pre-normalized input
+
+By default lookups lowercase the input (`LOWERCASE_WITH_LOCALE_ROOT`). If your
+pipeline already lowercases tokens, skip the redundant work:
+
+```python
+s = Stemmer("en", lowercase=False)     # assume already-lowercased input
+s.stem("running")                      # 'run'
+s.stem("Running")                      # None  -> not lowercased, so no match
+```
+
+The model's keys are always lowercase; `lowercase=False` only turns off
+per-lookup normalization. On already-lowercased input the results are identical.
+
+## Compile once, load instantly
+
+Compiling a trie from text costs a few seconds for large languages. Compile it
+once to Radixor's binary format and load it directly afterwards:
+
+```python
+import radixor
+
+radixor.compile("stemmer.gz", "en.rxc", language="en")
+s = radixor.Stemmer(compiled="en.rxc")
+```
+
+See [Compiling Dictionaries in Python](model-compilation.md) for the source
+format, traversal and normalization options, deployment guidance, Java
+interoperability, and the controls that remain Java-only.
+
+## Using a custom dictionary
+
+A source dictionary is a gzipped (or plain) TSV file, one entry per line, the
+first column the canonical stem and the rest its variants; `#` and `//` start
+line remarks:
+
+```
+run	running	runs	ran
+cat	cats
+```
+
+```python
+s = Stemmer(path="custom.gz", backward=True, store_original=True)
+```
+
+`store_original=True` (default) maps each stem to itself (a no-op patch) so the
+stem is recognised. See [Dictionary Format](../dictionary-format.md) for the
+authoritative specification shared with the Java project.
+
+## Thread-safety
+
+A `Stemmer` is safe to share across threads. The bounded cache is guarded
+internally; the compiled trie is immutable after construction.
+
+## API summary
+
+| Call | Returns | Notes |
+|---|---|---|
+| `Stemmer(lang \| path= \| compiled=, *, backward, store_original, lowercase, cache_size=10_000)` | stemmer | auto-detects compiled vs textual for `path=`; `cache_size=0` disables caching |
+| `stem(word)` | `str \| None` | dominant stem |
+| `stem_batch(words)` | `list[str \| None]` | **preferred** for many words |
+| `stemWord(word)` | `str` | PyStemmer-compatible; returns an unmatched word unchanged |
+| `stemWords(words)` | `list[str]` | PyStemmer-compatible batch call; preserves unmatched words and input order |
+| `stem_all(word)` | `list[str]` | all candidate stems, best first |
+| `stem_all_batch(words)` | `list[list[str]]` | |
+| `radixor.compile(source, out, *, language, backward, store_original, lowercase)` | `None` | writes a v7 binary |