# Built-in Languages

Radixor ships with a curated set of bundled stemmer dictionaries that can be loaded directly from the library distribution. These resources are intended to provide an immediately usable baseline for evaluation, prototyping, integration, and general-purpose stemming workloads, while still fitting naturally into workflows where the bundled baseline is later refined, extended, or replaced with custom lexical data.

## Overview

Bundled dictionaries are exposed through:

```java
org.egothor.stemmer.StemmerPatchTrieLoader.Language
```

Each bundled dictionary is packaged with the library as a compressed UTF-8 text resource. When loaded through the runtime API, the resource is parsed by `StemmerDictionaryParser`, transformed into patch-command mappings, and compiled into a read-only `FrequencyTrie<CompiledPatchCommand>` by `StemmerPatchTrieLoader`.

The bundled language definition also carries a language-level right-to-left flag. That flag is used by the loader to derive the `WordTraversalDirection` used for both trie-key construction and patch-command generation. In practice, left-to-right bundled languages use historical backward Egothor traversal, while right-to-left bundled languages use forward traversal over the stored form.

## Supported bundled languages

The following bundled language identifiers are currently available:

| Language | Enum constant | Writing direction | Notes | Benchmark page |
|---|---|---:|---|---|
| Czech | `CS_CZ` | LTR | Bundled general-purpose dictionary | [Czech](benchmarks/languages/czech.md) |
| Danish | `DA_DK` | LTR | Bundled general-purpose dictionary | [Danish](benchmarks/languages/danish.md) |
| German | `DE_DE` | LTR | Bundled general-purpose dictionary | [German](benchmarks/languages/german.md) |
| Spanish | `ES_ES` | LTR | Bundled general-purpose dictionary | [Spanish](benchmarks/languages/spanish.md) |
| Persian | `FA_IR` | RTL | Bundled dictionary uses forward traversal over the stored form | [Persian](benchmarks/languages/persian.md) |
| Finnish | `FI_FI` | LTR | Bundled general-purpose dictionary | [Finnish](benchmarks/languages/finnish.md) |
| French | `FR_FR` | LTR | Bundled general-purpose dictionary | [French](benchmarks/languages/french.md) |
| Hebrew | `HE_IL` | RTL | Bundled dictionary uses forward traversal over the stored form | No same-language external benchmark in this run |
| Hungarian | `HU_HU` | LTR | Bundled general-purpose dictionary | [Hungarian](benchmarks/languages/hungarian.md) |
| Italian | `IT_IT` | LTR | Bundled general-purpose dictionary | [Italian](benchmarks/languages/italian.md) |
| Norwegian Bokmål | `NB_NO` | LTR | Bundled general-purpose dictionary | [Norwegian Bokmal](benchmarks/languages/norwegian-bokmal.md) |
| Dutch | `NL_NL` | LTR | Bundled general-purpose dictionary | [Dutch](benchmarks/languages/dutch.md) |
| Norwegian Nynorsk | `NN_NO` | LTR | Bundled general-purpose dictionary | [Norwegian Nynorsk](benchmarks/languages/norwegian-nynorsk.md) |
| Polish | `PL_PL` | LTR | Bundled general-purpose dictionary | [Polish](benchmarks/languages/polish.md) |
| Portuguese | `PT_PT` | LTR | Bundled general-purpose dictionary | [Portuguese](benchmarks/languages/portuguese.md) |
| Russian | `RU_RU` | LTR | Bundled general-purpose dictionary | [Russian](benchmarks/languages/russian.md) |
| Swedish | `SV_SE` | LTR | Bundled general-purpose dictionary | [Swedish](benchmarks/languages/swedish.md) |
| Ukrainian | `UK_UA` | LTR | Bundled general-purpose dictionary | [Ukrainian](benchmarks/languages/ukrainian.md) |
| English | `US_UK` | LTR | Bundled general-purpose dictionary | [English](benchmarks/languages/english.md) |
| Yiddish | `YI` | RTL | Bundled dictionary uses forward traversal over the stored form | [Yiddish](benchmarks/languages/yiddish.md) |

## Basic usage

Load a bundled dictionary like this:

```java
import java.io.IOException;

import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class BuiltInExample {

    private BuiltInExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
                StemmerPatchTrieLoader.Language.US_UK,
                true,
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);

        System.out.println(trie.traversalDirection());
    }
}
```

This call loads the bundled dictionary resource for the selected language, parses its lexical entries, derives patch-command mappings, and compiles the result into a read-only trie.

## Example: stemming with a bundled dictionary

```java
import java.io.IOException;

import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class EnglishExample {

    private EnglishExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<CompiledPatchCommand> trie = StemmerPatchTrieLoader.loadCompiled(
                StemmerPatchTrieLoader.Language.US_UK,
                true,
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);

        final String word = "running";
        final CompiledPatchCommand patch = trie.get(word);
        final String stem = patch == null ? word : patch.apply(word);

        System.out.println(word + " -> " + stem);
    }
}
```

`CompiledPatchCommand` values are compiled with the traversal direction used when the trie and its patch commands were produced.

## Traversal behavior and right-to-left languages

Bundled dictionaries are not all processed identically.

For traditional left-to-right suffix-oriented resources, Radixor preserves historical Egothor behavior and traverses logical word characters backward. That means trie paths are constructed from the logical end of the stored word toward its beginning, and patch commands are interpreted with the same backward traversal model.

For bundled right-to-left languages such as Persian, Hebrew, and Yiddish, Radixor uses forward traversal over the stored form. In those cases:

- trie keys are traversed from the logical beginning of the stored form,
- patch commands are generated in that same forward direction,
- compiled patch-command application uses `WordTraversalDirection.FORWARD`, which is naturally captured when `loadCompiled(...)` creates `CompiledPatchCommand` values.

This design keeps the traversal policy explicit and consistent across dictionary loading, trie lookup, binary persistence, builder reconstruction, and patch application.

## Reduction behavior

Bundled dictionaries can be compiled using any supported `ReductionMode`. The reduction configuration controls how semantically equivalent subtrees are merged during trie compilation, while preserving the contract of the selected mode.

Typical entry points are:

- `StemmerPatchTrieLoader.loadCompiled(language, storeOriginal, reductionMode)`
- `StemmerPatchTrieLoader.loadCompiled(language, storeOriginal, reductionSettings)`

For most users, `ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS` is the most conservative general-purpose choice because it preserves ranked `getAll(...)` behavior.

Compiled bundled dictionaries also use internal uniform-subtree contraction. If a whole subtree
would return the same preferred patch command, Radixor stores that subtree as an accepting leaf and
removes the deeper branches. This is the contracted trie representation used by the published
benchmark tables and is independent of the public reduction mode selected by the caller.

## Intended role of bundled dictionaries

Bundled dictionaries should be understood as practical default resources.

They are a good fit when:

- a supported language is already available,
- immediate usability matters,
- a reasonable baseline is sufficient,
- the goal is evaluation, prototyping, or straightforward integration.

They are also well suited to staged refinement workflows in which a bundled base is loaded first, then extended with domain-specific vocabulary, and finally persisted as a custom binary artifact.

## Character representation

Bundled dictionaries are ordinary UTF-8 lexical resources. The parser reads them as text, the trie stores standard Java strings, and the patch-command model operates on general character sequences.

This is important for two reasons:

1. the built-in resources are not limited to ASCII-only processing,
2. the traversal model is orthogonal to character encoding and script choice.

In other words, right-to-left handling in the loader is about logical traversal strategy, not about introducing a separate character model.

## When to prefer custom dictionaries

A custom dictionary is usually the better choice when:

- domain-specific vocabulary materially affects stemming quality,
- lexical coverage must be controlled more precisely,
- a stronger lexical resource is available than the bundled baseline,
- operational requirements demand an explicitly curated, versioned artifact.

Typical examples include:

- technical terminology,
- biomedical language,
- legal or financial vocabulary,
- organization-specific product and process names,
- dictionaries maintained with project-specific validation rules.

## Production recommendation

For production systems, the most robust workflow is usually:

1. start from a bundled dictionary when it is suitable,
2. extend it with domain-specific forms if needed,
3. rebuild it into a binary artifact,
4. deploy that compiled binary artifact,
5. load it at runtime through `loadBinaryCompiled(...)`.

This avoids repeated startup parsing and makes the deployed stemming behavior explicit, reproducible, and versionable.

## Example refinement workflow

```java
import java.io.IOException;
import java.nio.file.Path;

import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.FrequencyTrieBuilders;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;
import org.egothor.stemmer.StemmerPatchTrieLoader;

public final class BundledRefinementExample {

    private BundledRefinementExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<String> base = StemmerPatchTrieLoader.load(
                StemmerPatchTrieLoader.Language.US_UK,
                true,
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);

        final FrequencyTrie.Builder<String> builder = FrequencyTrieBuilders.copyOf(
                base,
                String[]::new,
                ReductionSettings.withDefaults(
                        ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS));

        final PatchCommandEncoder encoder = PatchCommandEncoder.builder()
                .traversalDirection(base.traversalDirection())
                .build();

        builder.put("microservices", encoder.encode("microservices", "microservice"));

        final FrequencyTrie<String> compiled = builder.build();

        StemmerPatchTrieBinaryIO.write(compiled, Path.of("english-custom.radixor.gz"));
    }
}
```

The reconstructed builder preserves the traversal direction of the source trie, so refinements remain semantically aligned with the original bundled dictionary.

## Extending language support

The built-in set is intentionally a practical baseline rather than a closed catalog. Additional languages, stronger lexical coverage, and improved dictionaries for currently supported languages are all natural extension paths.

What matters most is not only the number of entries, but the quality, consistency, maintainability, and operational usefulness of the lexical resource being added.

## Related API surface

The following types are typically involved when working with bundled dictionaries:

- `StemmerPatchTrieLoader`
- `StemmerPatchTrieLoader.Language`
- `FrequencyTrie`
- `PatchCommandEncoder`
- `WordTraversalDirection`
- `ReductionMode`
- `ReductionSettings`
- `StemmerPatchTrieBinaryIO`
- `FrequencyTrieBuilders`

## Next steps

- [Quick start](quick-start.md)
- [Dictionary format](dictionary-format.md)
- [CLI compilation](cli-compilation.md)
- [Programmatic usage](programmatic-usage.md)

## Summary

Radixor’s built-in language support provides immediate usability, a professionally defined baseline API, and a practical starting point for custom refinement. The bundled set now includes both left-to-right and right-to-left languages, and the library models that distinction explicitly through `WordTraversalDirection` so that trie construction, lookup, and patch application remain consistent.
