Egothor/Radixor
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(python): add native distribution and release infrastructure

Browse Source 
- add the Rust-backed Python API with PyStemmer compatibility
- distribute standard compiled models as a separate Python package
- generate model artifacts during builds instead of storing them in Git
- add GitHub release and Pages-backed package index workflows
- add Python tests, benchmarks, documentation, and Gradle integration
- refresh the documentation site, branding, and language benchmarks
This commit is contained in:
Leo Galambos
2026-08-10 22:34:32 +02:00
parent b45e143c84
commit 5e3d3c7c7d
139 changed files with 11420 additions and 747 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
python/benchmarks/README.md Normal file
View File

@@ -0,0 +1,111 @@
# Runtime stemming benchmarks
These scripts measure **runtime stemming throughput only** — model construction
and dictionary compilation happen once during setup and are **excluded** from
every timing. Anyone can reproduce the numbers on their own machine.
## What is measured, and why batch sizes
Each engine is driven through its batch entry point over a fixed word budget
(default 5 000 tokens), split into batches of **10, 20, 50, 100** words. The
harness fits the descriptive line `per_call(N) ≈ intercept + N · slope` across
those sizes. The fit is unconstrained and timing noise can make its intercept
negative, so it describes observed scaling rather than physically separating
overhead from word work. We report the *best* (minimum) of many repeats — the
microbenchmark convention that suppresses OS/GC noise.
## Data — identical to the Java JMH benchmarks
The workload is the **changed-token corpus** built from the repository's
canonical dictionaries (`models/<model>/src/modelInput/stemmer.gz`),
mirroring `LanguageBenchmarkCorpus` in the Java project: each dictionary field
is paired with its line's root, normalized `trim().lower()`, and only tokens
that **differ** from their root are kept, in dictionary order, padded to ≥ 5 000
tokens. See `corpus.py`.
## Fairness — three things that quietly break stemmer comparisons
Getting a *fair* comparison turned out to matter more than any micro-optimization.
Three asymmetries, if left in, make the numbers meaningless:
1. **Result caching.** PyStemmer caches results by default (`maxCacheSize=10000`).
Because a benchmark stems the same corpus every repeat, that cache turns
measured passes into dict lookups rather than stemming. **The harness
explicitly disables both caches**: PyStemmer uses `maxCacheSize=0` and
radixor uses `cache_size=0`, so both engines do real stemming.
snowballstemmer-pure, nltk-porter, and cistem have no cache.
2. **Lowercasing.** Snowball/PyStemmer do no case handling — they assume the
caller pre-lowercased the input (our corpus is pre-lowercased for everyone).
radixor normally lowercases internally; for a same-work comparison the
harness runs radixor with **`lowercase=False`** (assume-already-lowercased),
so Snowball and radixor do identical normalization work on identical input.
CISTEM always performs its own lowercasing and German umlaut normalization;
that unavoidable extra work modestly biases its comparison in radixor's
favour.
3. **Delegation.** `snowballstemmer` delegates to PyStemmer when it is installed
(they become the same C code). The harness bypasses that and uses
snowballstemmer's genuine pure-Python backend, and records each engine's
backing module + whether it is a compiled extension (`--json`) as proof.
## Engines compared
| Engine | Implementation | Batch API |
|---|---|---|
| `radixor` | Rust patch-command trie (cache disabled) | `stem_batch` — one FFI call per batch |
| `PyStemmer` | Snowball C `libstemmer` (cache disabled) | `stemWords(list)` — one C call per batch |
| `snowballstemmer-pure` | Official **pure-Python** Snowball | `stemWords(list)` — Python loop |
| `nltk-porter` | Porter (English), pure Python | scalar loop |
| `cistem` | CISTEM (German), pure Python (`nltk`) | scalar loop |
## Reproduce
```bash
cd python/
maturin develop --release # build the radixor extension into your env
cd ..
./gradlew pythonBuildStandardModels # generate and package standard models
pip install --no-deps build/python/dist/standard/radixor_models_standard-0.0.0-py3-none-any.whl
cd python/
pip install -r benchmarks/requirements-bench.txt
python benchmarks/run_benchmark.py --language en de fr ru fi \
--sizes 10 20 50 100 --repeats 21 \
--json benchmarks/results.json --csv benchmarks/results.csv
```
From the repository root, the Gradle integration builds an isolated host wheel
and benchmarks Radixor plus every available comparison engine over all
supported languages with the fixed 10/20/50/100 size sweep:
```bash
./gradlew pythonBenchmarkAllLanguagesBatch
```
The generated CSV and JSON reports are placed in
`build/reports/python-benchmarks/`.
Comparison engines are auto-detected in the environment selected by the Gradle
`pythonExecutable` property. Install `requirements-bench.txt` in that environment
to enable the complete comparison set.
The run prints machine/Python/engine versions and each engine's backing module,
and writes per-point rows (CSV) plus the full report incl. environment and
provenance (JSON), so results are self-describing and verifiable.
## Published results
The canonical, current single-machine results and complete environment metadata
are published on the documentation site's [Python performance
page](../../docs/python/performance.md). Keeping the measured table in one place
prevents results from different CPUs or benchmark runs from being mixed.
## Interpretation
- In the published 2026-08-08 run, **radixor is the fastest stemmer measured in
Python** in all 18 languages directly shared with PyStemmer.
- The benchmark intentionally disables caches. Cached-operation performance is
outside this suite and must not be inferred from its results.
- radixor and Snowball remain different *classes* of stemmer: radixor is
dictionary-based (UniMorph gold coverage), Snowball is rule-based. radixor
gives dictionary-quality stems *and* the best measured throughput.

103
python/benchmarks/corpus.py Normal file
View File

@@ -0,0 +1,103 @@
###############################################################################
# Copyright (C) 2026, Leo Galambos
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
###############################################################################
"""Deterministic benchmark corpus, mirroring the Java JMH LanguageBenchmarkCorpus.
The timing workload contains dictionary forms that differ from their canonical
root. Each field is normalized with ``strip().lower()``, retained when it needs
stemming, and repeated in stable dictionary order until the corpus reaches
``MINIMUM_TIMING_TOKEN_COUNT``. This mirrors
``LanguageBenchmarkCorpus.buildChangedTimingCorpus`` and supplies the same
token sequence to every benchmark engine.
"""
from __future__ import annotations
import gzip
from pathlib import Path
MINIMUM_TIMING_TOKEN_COUNT = 5_000
def _normalize(token: str) -> str:
# Java: token.trim().toLowerCase(Locale.ROOT)
return token.strip().lower()
def _contains_whitespace(token: str) -> bool:
return any(ch.isspace() for ch in token)
def read_changed_tokens(dict_gz_path: str | Path) -> list[str]:
"""Return the changed-token list (token != root) in dictionary order.
Not yet padded to the timing minimum; see :func:`build_timing_corpus`.
"""
tokens: list[str] = []
with gzip.open(dict_gz_path, "rt", encoding="utf-8") as fh:
for line in fh:
if not line or line.isspace():
continue
# Match the Java benchmark: only a leading marker starts a comment;
# inline markers remain part of the dictionary field.
if line.startswith("#") or line.startswith("//"):
continue
fields = line.split("\t")
if not fields:
continue
root = _normalize(fields[0])
if not root or _contains_whitespace(root):
continue
for field in fields:
token = _normalize(field)
if not token or _contains_whitespace(token):
continue
if token != root: # changed-token workload
tokens.append(token)
return tokens
def build_timing_corpus(
dict_gz_path: str | Path,
minimum_token_count: int = MINIMUM_TIMING_TOKEN_COUNT,
) -> list[str]:
"""Return the padded changed-token timing corpus (>= minimum_token_count)."""
changed = read_changed_tokens(dict_gz_path)
if not changed:
raise ValueError(f"No changed-token corpus tokens available in {dict_gz_path}")
if len(changed) >= minimum_token_count:
return changed
# Repeat in stable order to reach the minimum, exactly like the Java code.
out: list[str] = []
n = len(changed)
for i in range(minimum_token_count):
out.append(changed[i % n])
return out

360
python/benchmarks/engines.py Normal file
View File

@@ -0,0 +1,360 @@
###############################################################################
# Copyright (C) 2026, Leo Galambos
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
###############################################################################
"""Stemmer engine adapters for the benchmark.
Every engine exposes a uniform interface:
engine.name -> str
engine.kind -> "native-batch" | "c-batch" | "py-batch" | "py-loop"
engine.supports(code) -> bool (ISO-639-1 language code)
engine.make(code) -> callable(list[str]) -> list[str] (the batch fn)
``kind`` records how batching is implemented:
- ``native-batch``: one native batch call
- ``c-batch``: C extension with a list entry point
- ``py-batch``: pure-Python object with a list method
- ``py-loop``: repeated scalar calls in Python
Engines whose package is not installed are simply reported as unavailable, so
the benchmark runs with whatever the user has.
"""
from __future__ import annotations
import inspect
from pathlib import Path
from typing import Callable, Optional
def _module_info(obj) -> dict:
"""Provenance of the module that actually provides ``obj``.
The returned metadata identifies the backing module and distinguishes
native extensions from pure-Python implementations.
Resolves via ``type(obj).__module__`` -> ``sys.modules`` because
``inspect.getmodule`` returns ``None`` for Cython extension instances.
"""
import sys
name = type(obj).__module__
mod = sys.modules.get(name) or inspect.getmodule(obj)
file = getattr(mod, "__file__", None) or ""
lower = file.lower()
compiled = (
lower.endswith((".pyd", ".so", ".dll")) or "cpython" in lower or "abi3" in lower
)
return {
"backing_module": name,
"backing_file": file,
"compiled_extension": compiled,
}
# ISO-639-1 -> Snowball algorithm name (matches the Java SnowballLanguageCase
# mapping; nb/nn both map to the single Snowball "norwegian" algorithm).
_SNOWBALL_NAMES: dict[str, str] = {
"cs": "czech",
"fa": "persian",
"pl": "polish",
"da": "danish",
"nl": "dutch",
"en": "english",
"fi": "finnish",
"fr": "french",
"de": "german",
"hu": "hungarian",
"it": "italian",
"nb": "norwegian",
"nn": "norwegian",
"pt": "portuguese",
"ru": "russian",
"es": "spanish",
"sv": "swedish",
"yi": "yiddish",
}
BatchFn = Callable[[list[str]], list[str]]
class Engine:
name: str = "engine"
kind: str = "py-loop"
def available(self) -> bool:
raise NotImplementedError
def supports(self, code: str) -> bool:
raise NotImplementedError
def make(self, code: str) -> BatchFn:
raise NotImplementedError
def provenance(self, code: str) -> dict:
return {
"backing_module": self.name,
"backing_file": "",
"compiled_extension": False,
"algorithm": None,
}
class RadixorEngine(Engine):
name = "radixor"
kind = "native-batch"
def __init__(self, lowercase: bool = False) -> None:
# The shared corpus is already lowercase. Disable Radixor's redundant
# normalization unless a benchmark explicitly includes that cost.
self._lowercase = lowercase
def available(self) -> bool:
try:
import radixor # noqa: F401
return True
except Exception:
return False
def supports(self, code: str) -> bool:
try:
from radixor import _LANGUAGE_ALIASES
return code in _LANGUAGE_ALIASES
except Exception:
return False
def make(self, code: str) -> BatchFn:
from radixor import Stemmer
s = Stemmer(code, lowercase=self._lowercase, cache_size=0)
return s.stem_batch
def provenance(self, code: str) -> dict:
from radixor import Stemmer
s = Stemmer(code, lowercase=self._lowercase, cache_size=0)
info = _module_info(s._core)
info["algorithm"] = "radixor-trie"
info["lowercase"] = self._lowercase
info["cache_disabled"] = True
return info
class PyStemmerEngine(Engine):
name = "PyStemmer"
kind = "c-batch"
def available(self) -> bool:
try:
import Stemmer # noqa: F401
return True
except Exception:
return False
def _algorithms(self) -> set[str]:
import Stemmer
return {a.lower() for a in Stemmer.algorithms()}
def supports(self, code: str) -> bool:
name = _SNOWBALL_NAMES.get(code)
return bool(name) and name in self._algorithms()
@staticmethod
def _new_stemmer(code: str):
import Stemmer
stemmer = Stemmer.Stemmer(_SNOWBALL_NAMES[code])
# Repeated passes would otherwise measure PyStemmer's default cache
# after the first pass. Both native engines therefore run uncached.
stemmer.maxCacheSize = 0
return stemmer
def make(self, code: str) -> BatchFn:
stemmer = self._new_stemmer(code)
# PyStemmer's native list entry point: one C call for the whole batch.
return stemmer.stemWords
def provenance(self, code: str) -> dict:
stemmer = self._new_stemmer(code)
info = _module_info(stemmer)
info["algorithm"] = _SNOWBALL_NAMES[code]
info["cache_disabled"] = True
# Record whether PyStemmer resolves to an independent native extension.
try:
import snowballstemmer
snowball_dir = str(Path(snowballstemmer.__file__).resolve().parent).lower()
except Exception:
snowball_dir = None
backing = info["backing_file"].lower()
info["independent_of_snowballstemmer"] = bool(
info["compiled_extension"]
and (snowball_dir is None or snowball_dir not in backing)
)
return info
class SnowballStemmerEngine(Engine):
"""Pure-Python Snowball backend.
``snowballstemmer.stemmer()`` delegates to PyStemmer (the C extension) when
PyStemmer is installed, which would make this engine a duplicate of
``PyStemmer``. To retain a distinct implementation, this adapter imports
the language's pure-Python class directly from
``snowballstemmer.<name>_stemmer``.
"""
name = "snowballstemmer-pure"
kind = "py-batch"
def available(self) -> bool:
try:
import snowballstemmer # noqa: F401
return True
except Exception:
return False
def _load_class(self, code: str):
import importlib
name = _SNOWBALL_NAMES.get(code)
if not name:
return None
module = importlib.import_module(f"snowballstemmer.{name}_stemmer")
class_name = name.capitalize() + "Stemmer"
return getattr(module, class_name, None)
def supports(self, code: str) -> bool:
try:
return self._load_class(code) is not None
except Exception:
return False
def make(self, code: str) -> BatchFn:
stemmer = self._load_class(code)()
return stemmer.stemWords # pure-Python loop over the list, internally
def provenance(self, code: str) -> dict:
stemmer = self._load_class(code)()
info = _module_info(stemmer)
info["algorithm"] = _SNOWBALL_NAMES[code]
return info
class NltkPorterEngine(Engine):
name = "nltk-porter"
kind = "py-loop"
def available(self) -> bool:
try:
from nltk.stem import PorterStemmer # noqa: F401
return True
except Exception:
return False
def supports(self, code: str) -> bool:
return code == "en" # Porter is English-only
def make(self, code: str) -> BatchFn:
from nltk.stem import PorterStemmer
ps = PorterStemmer()
stem = ps.stem
def batch(words: list[str]) -> list[str]:
return [stem(w) for w in words]
return batch
def provenance(self, code: str) -> dict:
from nltk.stem import PorterStemmer
info = _module_info(PorterStemmer())
info["algorithm"] = "porter"
return info
class CistemEngine(Engine):
"""CISTEM — a fast lightweight German stemmer (German only)."""
name = "cistem"
kind = "py-loop"
def available(self) -> bool:
try:
from nltk.stem.cistem import Cistem # noqa: F401
return True
except Exception:
return False
def supports(self, code: str) -> bool:
return code == "de"
def make(self, code: str) -> BatchFn:
from nltk.stem.cistem import Cistem
stem = Cistem().stem
def batch(words: list[str]) -> list[str]:
return [stem(w) for w in words]
return batch
def provenance(self, code: str) -> dict:
from nltk.stem.cistem import Cistem
info = _module_info(Cistem())
info["algorithm"] = "cistem"
return info
ALL_ENGINES: list[Engine] = [
RadixorEngine(),
PyStemmerEngine(),
SnowballStemmerEngine(),
NltkPorterEngine(),
CistemEngine(),
]
def available_engines(names: Optional[set[str]] = None) -> list[Engine]:
engines = [e for e in ALL_ENGINES if e.available()]
if names:
engines = [e for e in engines if e.name in names]
return engines

6
python/benchmarks/requirements-bench.txt Normal file
View File

@@ -0,0 +1,6 @@
# Optional comparison engines for the runtime stemming benchmark.
# radixor itself must already be built (maturin develop) in the same env.
# Any subset may be installed; the harness auto-detects what is present.
PyStemmer # Snowball as a C extension (libstemmer); real batch API
snowballstemmer # Official pure-Python Snowball (used via its pure backend)
nltk # Porter (English) pure-Python reference baseline

436
python/benchmarks/run_benchmark.py Normal file
View File

@@ -0,0 +1,436 @@
#!/usr/bin/env python3
###############################################################################
# Copyright (C) 2026, Leo Galambos
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
###############################################################################
"""Runtime stemming benchmark for the radixor Python extension vs alternatives.
Measures ONLY runtime stemming throughput — model construction / dictionary
compilation happens once in setup and is excluded from all timings.
Batch sizes are swept (default 10/20/50/100) and an unconstrained descriptive
line is fitted for each engine: per_call_time(N) = intercept + slope * N. The
fit summarizes scaling across the measured sizes; timing noise can make its
intercept negative, so it must not be read as a physical overhead measurement.
Data is the same as the Java JMH benchmarks: the changed-token corpus derived
from the bundled UniMorph gold-standard dictionaries (see corpus.py).
Examples
--------
python run_benchmark.py --language en
python run_benchmark.py --all-languages --engines radixor
python run_benchmark.py --language en de ru --repeats 15 --csv results.csv
python run_benchmark.py --language en --sizes 10 20 50 100 200 --json out.json
"""
from __future__ import annotations
import argparse
import csv as csvmod
import gc
import json
import platform
import statistics
import sys
import time
from pathlib import Path
from typing import Optional
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE)) # allow running as a plain script
import corpus as corpus_mod # noqa: E402
import engines as engines_mod # noqa: E402
def _chunks(seq: list[str], n: int) -> list[list[str]]:
return [seq[i : i + n] for i in range(0, len(seq), n)]
def _time_sequence_ns(batch_fn, batches: list[list[str]]) -> int:
"""Time one full pass over all batches (nanoseconds)."""
start = time.perf_counter_ns()
for b in batches:
batch_fn(b)
return time.perf_counter_ns() - start
def _linfit(xs: list[float], ys: list[float]) -> tuple[float, float]:
"""Ordinary least squares: returns (intercept, slope)."""
n = len(xs)
mean_x = sum(xs) / n
mean_y = sum(ys) / n
sxx = sum((x - mean_x) ** 2 for x in xs)
sxy = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
slope = sxy / sxx if sxx else 0.0
intercept = mean_y - slope * mean_x
return intercept, slope
_DIST_NAMES = {
"radixor": "radixor",
"PyStemmer": "PyStemmer",
"snowballstemmer-pure": "snowballstemmer",
"nltk-porter": "nltk",
}
def _engine_version(name: str) -> Optional[str]:
import importlib.metadata as md
dist = _DIST_NAMES.get(name)
if not dist:
return None
try:
return md.version(dist)
except Exception:
return "editable" if name == "radixor" else None
def _processor_name() -> str:
"""Return a useful CPU model name without adding a platform dependency."""
name = platform.processor().strip()
if name:
return name
cpuinfo = Path("/proc/cpuinfo")
if cpuinfo.is_file():
for line in cpuinfo.read_text(encoding="utf-8", errors="replace").splitlines():
key, separator, value = line.partition(":")
if separator and key.strip() in {"model name", "Hardware"}:
name = value.strip()
if name:
return name
return "unknown"
def run(args) -> dict:
from radixor import _LANGUAGE_ALIASES
engine_filter = set(args.engines) if args.engines else None
engines = engines_mod.available_engines(engine_filter)
if not engines:
print(
"No stemmer engines available. Install PyStemmer / snowballstemmer / nltk.",
file=sys.stderr,
)
sys.exit(2)
results: list[dict] = []
strict_engine_names = (
(engine_filter or {"radixor"}) if args.all_languages else set()
)
failures: list[str] = []
available_engine_names = {engine.name for engine in engines}
for missing_engine in sorted(strict_engine_names - available_engine_names):
failures.append(f"engine unavailable: {missing_engine}")
for code in args.language:
model_id = _LANGUAGE_ALIASES.get(code, code)
if args.model_path:
dict_path = Path(args.model_path)
else:
# Corpus construction deliberately uses the canonical repository
# source. Runtime distributions contain only compiled model data.
dict_path = (
HERE.parents[1]
/ "models"
/ model_id
/ "src"
/ "modelInput"
/ "stemmer.gz"
)
if not dict_path.is_file():
print(
f"[{code}] canonical benchmark dictionary not found: {dict_path}",
file=sys.stderr,
)
if args.all_languages:
failures.append(f"{code}: dictionary not found: {dict_path}")
continue
full = corpus_mod.build_timing_corpus(dict_path)
budget = min(args.words, len(full)) if args.words > 0 else len(full)
pool = full[:budget]
print(
f"\n=== language={code} model={model_id} "
f"corpus={len(pool)} changed tokens ==="
)
for engine in engines:
if not engine.supports(code):
if engine.name in strict_engine_names:
failures.append(f"{code}: engine does not support {engine.name}")
continue
try:
batch_fn = engine.make(code)
except Exception as exc: # pragma: no cover - engine setup failure
print(f" [{engine.name}] setup failed: {exc}", file=sys.stderr)
if engine.name in strict_engine_names:
failures.append(f"{code}: {engine.name} setup failed: {exc}")
continue
prov = engine.provenance(code)
results.append(
{
"language": code,
"model": model_id,
"engine": engine.name,
"kind": engine.kind,
"batch_size": "PROVENANCE",
**prov,
}
)
print(
f" {engine.name:<16} backing={prov.get('backing_module')} "
f"compiled={prov.get('compiled_extension')} "
f"algo={prov.get('algorithm')}"
)
# sanity: output length must equal input length
probe = batch_fn(pool[: min(8, len(pool))])
if len(probe) != min(8, len(pool)):
print(
f" [{engine.name}] unexpected output shape; skipping",
file=sys.stderr,
)
if engine.name in strict_engine_names:
failures.append(
f"{code}: {engine.name} returned an unexpected output shape"
)
continue
per_call_best: list[float] = []
for size in args.sizes:
batches = _chunks(pool, size)
n_calls = len(batches)
n_words = len(pool)
# warmup
for _ in range(args.warmup):
_time_sequence_ns(batch_fn, batches)
gc_was_enabled = gc.isenabled()
gc.disable()
try:
totals = [
_time_sequence_ns(batch_fn, batches)
for _ in range(args.repeats)
]
finally:
if gc_was_enabled:
gc.enable()
med_total = statistics.median(totals)
min_total = min(totals)
# Per-word/per-call reported from the best (min) pass — the
# microbenchmark convention that suppresses OS/GC scheduling
# noise. The later OLS fit is descriptive and unconstrained.
per_word_ns = min_total / n_words
per_call_ns = min_total / n_calls
per_call_best.append(per_call_ns)
throughput = n_words / (min_total / 1e9)
row = {
"language": code,
"model": model_id,
"engine": engine.name,
"kind": engine.kind,
"batch_size": size,
"calls": n_calls,
"words": n_words,
"repeats": args.repeats,
"median_total_ms": med_total / 1e6,
"min_total_ms": min_total / 1e6,
"per_word_ns": per_word_ns,
"per_call_us": per_call_ns / 1e3,
"throughput_words_per_s": throughput,
}
results.append(row)
print(
f" {engine.name:<16} [{engine.kind:<12}] "
f"N={size:<4} {per_word_ns:8.1f} ns/word "
f"{per_call_ns / 1e3:8.2f} us/call "
f"{throughput / 1e6:6.2f} M words/s"
)
# Unconstrained descriptive OLS fit across batch sizes. Keep the
# historical JSON key for report compatibility.
if len(args.sizes) >= 2:
intercept_ns, slope_ns = _linfit(
[float(s) for s in args.sizes], per_call_best
)
results.append(
{
"language": code,
"model": model_id,
"engine": engine.name,
"kind": engine.kind,
"batch_size": "FIT",
"regie_ns_per_call": intercept_ns,
"real_ns_per_word": slope_ns,
}
)
print(
f" {engine.name:<16} -> estimated intercept/call = "
f"{intercept_ns / 1e3:7.2f} us "
f"estimated slope = {slope_ns:7.1f} ns/word"
)
if args.all_languages:
expected_measurements = {
(code, engine_name, size)
for code in args.language
for engine_name in strict_engine_names
for size in args.sizes
}
actual_measurements = {
(row["language"], row["engine"], row["batch_size"])
for row in results
if isinstance(row.get("batch_size"), int)
}
missing_measurements = sorted(expected_measurements - actual_measurements)
if missing_measurements:
failures.append(f"missing measurement rows: {missing_measurements}")
if failures:
raise RuntimeError(
"Incomplete all-language benchmark: " + "; ".join(failures)
)
return {
"environment": {
"platform": platform.platform(),
"processor": _processor_name(),
"python": sys.version.split()[0],
"python_impl": platform.python_implementation(),
"engine_versions": {e.name: _engine_version(e.name) for e in engines},
},
"parameters": {
"languages": args.language,
"sizes": args.sizes,
"words_budget": args.words,
"repeats": args.repeats,
"warmup": args.warmup,
},
"results": results,
}
def main() -> None:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
languages = p.add_mutually_exclusive_group()
languages.add_argument(
"--language",
"-l",
nargs="+",
default=None,
help="ISO-639-1 language code(s) or model id(s). Default: en",
)
languages.add_argument(
"--all-languages",
action="store_true",
help="Benchmark every language alias bundled by radixor",
)
p.add_argument(
"--sizes",
"-s",
type=int,
nargs="+",
default=[10, 20, 50, 100],
help="Batch sizes to sweep. Default: 10 20 50 100",
)
p.add_argument(
"--words",
"-w",
type=int,
default=5000,
help="Words processed per measurement (<=0 = whole corpus). Default: 5000",
)
p.add_argument(
"--repeats",
"-r",
type=int,
default=15,
help="Timed repeats per point (best/min reported). Default: 15",
)
p.add_argument("--warmup", type=int, default=3, help="Warmup passes. Default: 3")
p.add_argument(
"--engines",
nargs="+",
default=None,
help="Restrict to named engines (radixor PyStemmer snowballstemmer nltk-porter)",
)
p.add_argument(
"--model-path",
default=None,
help="Explicit gzipped dictionary path (single-language runs)",
)
p.add_argument("--csv", default=None, help="Write per-point rows to this CSV file")
p.add_argument(
"--json", default=None, help="Write full results (incl. environment) to JSON"
)
args = p.parse_args()
if args.all_languages:
from radixor import _LANGUAGE_ALIASES
args.language = sorted(_LANGUAGE_ALIASES)
elif args.language is None:
args.language = ["en"]
report = run(args)
if args.json:
Path(args.json).write_text(json.dumps(report, indent=2), encoding="utf-8")
print(f"\nwrote {args.json}")
if args.csv:
rows = [
r
for r in report["results"]
if r.get("batch_size") not in ("FIT", "PROVENANCE")
]
if rows:
with open(args.csv, "w", newline="", encoding="utf-8") as fh:
w = csvmod.DictWriter(fh, fieldnames=list(rows[0].keys()))
w.writeheader()
w.writerows(rows)
print(f"wrote {args.csv}")
print("\nEnvironment:")
for k, v in report["environment"].items():
print(f" {k}: {v}")
if __name__ == "__main__":
main()