Radixor/docs/programmatic-loading-and-building.md

# Loading and Building Stemmers in Java

This document explains how to acquire a compiled Radixor stemmer in Java.

For Python construction and binary preparation, use [Python Usage and API](python/usage.md)
and [Compiling Dictionaries in Python](python/model-compilation.md).

## Load a registered default model

Language-oriented entry points resolve a registered default model and compile its GZip textual dictionary into a `FrequencyTrie<CompiledPatchCommand>`. The corresponding model JAR must be on the runtime classpath; the core contains no dictionary.

```java
import java.io.IOException;

import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class RegisteredLanguageModelExample {

    private RegisteredLanguageModelExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
                StemmerPatchTrieLoader.Language.US_UK,
                true,
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);
    }
}
```

The `storeOriginal` flag controls whether the canonical stem is inserted as a no-op patch entry for the stem itself.

Language-oriented `loadCompiled(...)` entry points build the runtime trie with the same contracted
representation used by the published benchmarks. During compilation, uniform preferred-command
subtrees are collapsed into accepting leaves, so lookup can stop before consuming the entire input
when the remaining characters cannot change the selected patch command.

## Load a textual dictionary

Loading from a dictionary file follows the same trie preparation model as registered model resources, but the source comes from your own file or path and bypasses registry metadata. The input may be plain UTF-8 text or GZip-compressed UTF-8 text; the loader detects GZip data from the stream header. The textual format is tab-separated values, meaning that columns are separated by the tab character. Each non-empty logical line starts with the stem column and may contain zero or more variant columns. Input case normalization is controlled by `CaseProcessingMode` (default: `LOWERCASE_WITH_LOCALE_ROOT`), trailing remarks introduced by `#` or `//` are ignored, and dictionary items containing embedded whitespace are currently ignored with warning-level diagnostics.

For explicit model IDs, multiple variants, and ClassLoader control, see [Model Selection and Loading](model-selection-and-loading.md).

```java
import java.io.IOException;
import java.nio.file.Path;

import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class LoadTextDictionaryExample {

    private LoadTextDictionaryExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
                Path.of("data", "stemmer.tsv"),
                true,
                ReductionSettings.withDefaults(
                        ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS));
    }
}
```

Additional `StemmerPatchTrieLoader.loadCompiled(...)` overloads let callers provide explicit `WordTraversalDirection`, `CaseProcessingMode`, `DiacriticProcessingMode`, or a complete `TrieMetadata` instance. Use those overloads when a custom dictionary must be compiled with forward traversal for right-to-left languages, case-sensitive keys, or diacritic stripping.

When `ReductionSettings` are supplied through these compiled loader APIs, uniform-subtree
contraction is still enabled as an internal pre-reduction step. The public `ReductionMode` remains
the semantic policy for subtree equivalence after that contraction has removed regions whose
preferred command is already uniform.

## Load a compiled binary artifact

Binary loading is typically the preferred runtime path because it avoids reparsing the textual source and skips the preparation step entirely.

```java
import java.io.IOException;
import java.nio.file.Path;

import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class LoadBinaryExample {

    private LoadBinaryExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadBinaryCompiled(
                Path.of("stemmers", "english.radixor.gz"));
    }
}
```

The binary format is the native `FrequencyTrie` serialization wrapped in GZip compression. It includes persisted `TrieMetadata`, so lookup after loading uses the traversal, case-processing, diacritic-processing, and reduction settings captured when the trie was compiled.

## Tune child lookup density when loading binaries

To optimize hot-path latency, you can tune direct child indexing by passing `maxExpandedIndex`
at load time. This does not change persisted metadata, only the materialized in-memory form.

```java
import java.io.IOException;
import java.nio.file.Path;

import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class LoadBinaryWithDenseLookupExample {

    private LoadBinaryWithDenseLookupExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<CompiledPatchCommand> balanced = StemmerPatchTrieLoader.loadBinaryCompiled(
                Path.of("stemmers", "english.radixor.gz"));

        final FrequencyTrie<CompiledPatchCommand> fast = StemmerPatchTrieLoader.loadBinaryCompiled(
                Path.of("stemmers", "english.radixor.gz"),
                1024);

        final FrequencyTrie<CompiledPatchCommand> compact = StemmerPatchTrieLoader.loadBinaryCompiled(
                Path.of("stemmers", "english.radixor.gz"),
                0);
    }
}
```

Negative values still use `FrequencyTrie.DEFAULT_MAX_EXPANDED_INDEX`.

[Lookup Edge Optimization](lookup-edge-optimization.md) describes the trade-off in detail and examples for build-time tuning as well.

## Build directly with a mutable builder

A `FrequencyTrie.Builder<V>` accepts repeated `put(key, value)` calls and compiles the final read-only trie through `build()`. Compilation performs bottom-up reduction and produces the compact immutable runtime representation.

```java
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;

public final class BuilderExample {

    private BuilderExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) {
        final ReductionSettings settings = ReductionSettings.withDefaults(
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);

        final FrequencyTrie.Builder<String> builder =
                new FrequencyTrie.Builder<>(String[]::new, settings);

        final PatchCommandEncoder encoder = PatchCommandEncoder.builder().build();

        builder.put("running", encoder.encode("running", "run"));
        builder.put("runs", encoder.encode("runs", "run"));
        builder.put("ran", encoder.encode("ran", "run"));
        builder.put("runner", encoder.encode("runner", "run"));

        final FrequencyTrie<String> trie = builder.build();
        System.out.println("Canonical node count: " + trie.size());
    }
}
```

## Preparation-time memory characteristics

Compilation is commonly a one-time preparation activity and is generally fast enough not to be the main operational concern. The more important constraint is memory usage while building from textual dictionary data. Before reduction produces the compact immutable structure, the mutable build-time representation keeps the inserted data in memory. This is precisely why very large source dictionaries may require noticeably more memory during preparation than after compilation. The resulting compiled trie, by contrast, is designed as the compact runtime form.

This makes offline preparation especially attractive for large dictionaries.

## Continue with

- [Querying and Ambiguity Handling](programmatic-querying-and-ambiguity.md)
- [Extending and Persisting Compiled Tries](programmatic-extending-and-persistence.md)