feat: EGOTHOR v4 hot-path additions
This commit is contained in:
27
.project
27
.project
@@ -2,21 +2,22 @@
|
|||||||
<projectDescription>
|
<projectDescription>
|
||||||
<name>Radixor</name>
|
<name>Radixor</name>
|
||||||
<comment></comment>
|
<comment></comment>
|
||||||
<projects/>
|
<projects>
|
||||||
|
</projects>
|
||||||
|
<buildSpec>
|
||||||
|
<buildCommand>
|
||||||
|
<name>org.eclipse.jdt.core.javabuilder</name>
|
||||||
|
<arguments>
|
||||||
|
</arguments>
|
||||||
|
</buildCommand>
|
||||||
|
<buildCommand>
|
||||||
|
<name>org.eclipse.buildship.core.gradleprojectbuilder</name>
|
||||||
|
<arguments>
|
||||||
|
</arguments>
|
||||||
|
</buildCommand>
|
||||||
|
</buildSpec>
|
||||||
<natures>
|
<natures>
|
||||||
<nature>org.eclipse.jdt.core.javanature</nature>
|
<nature>org.eclipse.jdt.core.javanature</nature>
|
||||||
<nature>org.eclipse.buildship.core.gradleprojectnature</nature>
|
<nature>org.eclipse.buildship.core.gradleprojectnature</nature>
|
||||||
</natures>
|
</natures>
|
||||||
<buildSpec>
|
|
||||||
<buildCommand>
|
|
||||||
<name>org.eclipse.jdt.core.javabuilder</name>
|
|
||||||
<arguments/>
|
|
||||||
</buildCommand>
|
|
||||||
<buildCommand>
|
|
||||||
<name>org.eclipse.buildship.core.gradleprojectbuilder</name>
|
|
||||||
<arguments/>
|
|
||||||
</buildCommand>
|
|
||||||
</buildSpec>
|
|
||||||
<linkedResources/>
|
|
||||||
<filteredResources/>
|
|
||||||
</projectDescription>
|
</projectDescription>
|
||||||
|
|||||||
@@ -75,6 +75,14 @@ The distinction between preferred-result lookup and multi-result lookup is part
|
|||||||
|
|
||||||
That model is part of how the public API should be understood.
|
That model is part of how the public API should be understood.
|
||||||
|
|
||||||
|
Visitor lookup methods such as `getAllNormalized(..., EntrySink, maxResults)` are additive hot-path APIs. They expose the same local ordering and count semantics without allocating result containers, but they do not replace `get()`, `getAll()`, or `getEntries()`.
|
||||||
|
|
||||||
|
Compiled `FrequencyTrie` instances are immutable and thread-safe for concurrent reads. Visitor sinks are caller-owned and are not retained by the trie. Stored values passed to sinks are the model-owned trie values; for `FrequencyTrie<String>` patch tries, those patch strings are immutable stored strings rather than fresh per-result strings.
|
||||||
|
|
||||||
|
### Stable patch application behavior
|
||||||
|
|
||||||
|
`PatchCommandEncoder.apply(...)` remains the compatibility API for string-returning patch application. Buffer-oriented `applyTo(...)` overloads are additive APIs for caller-owned output storage. They do not retain output arrays, report insufficient capacity with `APPLY_INSUFFICIENT_CAPACITY`, and preserve the existing malformed-patch compatibility behavior where `apply(...)` preserves the source.
|
||||||
|
|
||||||
### Stable reduction-mode intent
|
### Stable reduction-mode intent
|
||||||
|
|
||||||
Each public `ReductionMode` constant carries a semantic contract that should remain meaningful across versions.
|
Each public `ReductionMode` constant carries a semantic contract that should remain meaningful across versions.
|
||||||
|
|||||||
@@ -33,6 +33,28 @@ import org.egothor.stemmer.ValueCount;
|
|||||||
final List<ValueCount<String>> entries = trie.getEntries("axes");
|
final List<ValueCount<String>> entries = trie.getEntries("axes");
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Visitor lookup for hot paths
|
||||||
|
|
||||||
|
For allocation-sensitive token loops, use the visitor-style lookup methods. They visit the same ordered local values and counts without allocating a result array, list, or `ValueCount` objects.
|
||||||
|
|
||||||
|
```java
|
||||||
|
trie.getAll("axes", (patch, count, rank) -> {
|
||||||
|
// rank is zero-based and follows the same ordering as getAll(String).
|
||||||
|
return true; // return false to stop after this callback
|
||||||
|
}, 8);
|
||||||
|
```
|
||||||
|
|
||||||
|
If the caller has already normalized the input exactly as required by `trie.metadata()`, the normalized methods avoid lookup normalization buffers too:
|
||||||
|
|
||||||
|
```java
|
||||||
|
final char[] token = "axes".toCharArray();
|
||||||
|
trie.getAllNormalized(token, 0, token.length, (patch, count, rank) -> {
|
||||||
|
return true;
|
||||||
|
}, 8);
|
||||||
|
```
|
||||||
|
|
||||||
|
`getAllNormalized(...)` bypasses `caseProcessingMode` and `diacriticProcessingMode`; callers are responsible for supplying canonical input. `maxResults == 0` visits nothing, negative values are rejected, and a sink returning `false` stops iteration after the current callback.
|
||||||
|
|
||||||
## Apply patch commands
|
## Apply patch commands
|
||||||
|
|
||||||
A patch command is not the final stem. It must be applied to the original input token. `PatchCommandEncoder.apply(source, patchCommand)` performs that transformation directly on the serialized command format. If the source is `null`, the method returns `null`. If the patch is `null`, empty, or malformed in compatibility-relevant ways, the original source word is preserved. Equal source and target words are represented by the canonical no-op patch.
|
A patch command is not the final stem. It must be applied to the original input token. `PatchCommandEncoder.apply(source, patchCommand)` performs that transformation directly on the serialized command format. If the source is `null`, the method returns `null`. If the patch is `null`, empty, or malformed in compatibility-relevant ways, the original source word is preserved. Equal source and target words are represented by the canonical no-op patch.
|
||||||
@@ -45,6 +67,25 @@ final String patch = trie.get(word);
|
|||||||
final String stem = PatchCommandEncoder.apply(word, patch);
|
final String stem = PatchCommandEncoder.apply(word, patch);
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Hot paths can apply a patch into caller-owned character storage:
|
||||||
|
|
||||||
|
```java
|
||||||
|
final char[] output = new char[32];
|
||||||
|
final int produced = PatchCommandEncoder.applyTo(
|
||||||
|
word,
|
||||||
|
patch,
|
||||||
|
trie.traversalDirection(),
|
||||||
|
output,
|
||||||
|
0,
|
||||||
|
output.length);
|
||||||
|
|
||||||
|
if (produced != PatchCommandEncoder.APPLY_INSUFFICIENT_CAPACITY) {
|
||||||
|
final String stem = new String(output, 0, produced);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`applyTo(...)` returns the produced character count on success and `APPLY_INSUFFICIENT_CAPACITY` when the output range is too small. Capacity failure does not write partial output. The source and output ranges of the `char[]` overload must not overlap.
|
||||||
|
|
||||||
For multiple candidates:
|
For multiple candidates:
|
||||||
|
|
||||||
```java
|
```java
|
||||||
|
|||||||
@@ -31,11 +31,13 @@
|
|||||||
package org.egothor.stemmer.benchmark;
|
package org.egothor.stemmer.benchmark;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
import java.util.List;
|
||||||
import java.util.concurrent.TimeUnit;
|
import java.util.concurrent.TimeUnit;
|
||||||
import org.egothor.stemmer.FrequencyTrie;
|
import org.egothor.stemmer.FrequencyTrie;
|
||||||
import org.egothor.stemmer.PatchCommandEncoder;
|
import org.egothor.stemmer.PatchCommandEncoder;
|
||||||
import org.egothor.stemmer.ReductionMode;
|
import org.egothor.stemmer.ReductionMode;
|
||||||
import org.egothor.stemmer.ReductionSettings;
|
import org.egothor.stemmer.ReductionSettings;
|
||||||
|
import org.egothor.stemmer.ValueCount;
|
||||||
import org.openjdk.jmh.annotations.Benchmark;
|
import org.openjdk.jmh.annotations.Benchmark;
|
||||||
import org.openjdk.jmh.annotations.BenchmarkMode;
|
import org.openjdk.jmh.annotations.BenchmarkMode;
|
||||||
import org.openjdk.jmh.annotations.Level;
|
import org.openjdk.jmh.annotations.Level;
|
||||||
@@ -97,12 +99,45 @@ public class FrequencyTrieLookupBenchmark {
|
|||||||
*/
|
*/
|
||||||
private String[] lookupKeys;
|
private String[] lookupKeys;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Lookup keys as normalized caller-owned character storage.
|
||||||
|
*/
|
||||||
|
private char[][] lookupKeyCharacters;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Keys that are known to return multiple patch candidates from
|
* Keys that are known to return multiple patch candidates from
|
||||||
* {@code getAll()}.
|
* {@code getAll()}.
|
||||||
*/
|
*/
|
||||||
private String[] ambiguousLookupKeys;
|
private String[] ambiguousLookupKeys;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Ambiguous lookup keys as normalized caller-owned character storage.
|
||||||
|
*/
|
||||||
|
private char[][] ambiguousLookupKeyCharacters;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Preferred patches aligned with {@link #lookupKeys}.
|
||||||
|
*/
|
||||||
|
private String[] preferredPatches;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reusable output buffer for patch application benchmarks.
|
||||||
|
*/
|
||||||
|
private char[] outputBuffer;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Mutable field consumed by visitor sinks.
|
||||||
|
*/
|
||||||
|
private int visitorAccumulator;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sink used by visitor lookup benchmarks without per-invocation allocation.
|
||||||
|
*/
|
||||||
|
private final FrequencyTrie.EntrySink<String> visitorSink = (value, count, rank) -> {
|
||||||
|
this.visitorAccumulator += value.length() + count + rank;
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initializes the benchmark state.
|
* Initializes the benchmark state.
|
||||||
*
|
*
|
||||||
@@ -116,6 +151,23 @@ public class FrequencyTrieLookupBenchmark {
|
|||||||
this.trie = BenchmarkCorpusSupport.compilePatchTrie(corpus.dictionaryText(), settings, true);
|
this.trie = BenchmarkCorpusSupport.compilePatchTrie(corpus.dictionaryText(), settings, true);
|
||||||
this.lookupKeys = corpus.lookupKeys();
|
this.lookupKeys = corpus.lookupKeys();
|
||||||
this.ambiguousLookupKeys = corpus.ambiguousLookupKeys();
|
this.ambiguousLookupKeys = corpus.ambiguousLookupKeys();
|
||||||
|
this.lookupKeyCharacters = toCharArrays(this.lookupKeys);
|
||||||
|
this.ambiguousLookupKeyCharacters = toCharArrays(this.ambiguousLookupKeys);
|
||||||
|
this.preferredPatches = new String[this.lookupKeys.length];
|
||||||
|
int maxKeyLength = 0;
|
||||||
|
for (int index = 0; index < this.lookupKeys.length; index++) {
|
||||||
|
this.preferredPatches[index] = this.trie.get(this.lookupKeys[index]);
|
||||||
|
maxKeyLength = Math.max(maxKeyLength, this.lookupKeys[index].length());
|
||||||
|
}
|
||||||
|
this.outputBuffer = new char[maxKeyLength + 32];
|
||||||
|
}
|
||||||
|
|
||||||
|
private static char[][] toCharArrays(final String[] values) {
|
||||||
|
final char[][] characters = new char[values.length][];
|
||||||
|
for (int index = 0; index < values.length; index++) {
|
||||||
|
characters[index] = values[index].toCharArray();
|
||||||
|
}
|
||||||
|
return characters;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -155,6 +207,61 @@ public class FrequencyTrieLookupBenchmark {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures retrieval of all patch candidates through caller-owned normalized
|
||||||
|
* character storage and a visitor sink.
|
||||||
|
*
|
||||||
|
* @param state prepared lookup state
|
||||||
|
* @param blackhole sink preventing dead-code elimination
|
||||||
|
*/
|
||||||
|
@Benchmark
|
||||||
|
public void lookupAllPatchesWithNormalizedCharVisitor(final LookupState state, final Blackhole blackhole) {
|
||||||
|
final char[][] keys = state.ambiguousLookupKeyCharacters;
|
||||||
|
for (char[] key : keys) {
|
||||||
|
final int count = state.trie.getAllNormalized(key, 0, key.length, state.visitorSink, Integer.MAX_VALUE);
|
||||||
|
if (count < 2) {
|
||||||
|
throw new IllegalStateException("Expected multiple patches for benchmark key.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
blackhole.consume(state.visitorAccumulator);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures counted candidate retrieval through the allocating entry API.
|
||||||
|
*
|
||||||
|
* @param state prepared lookup state
|
||||||
|
* @param blackhole sink preventing dead-code elimination
|
||||||
|
*/
|
||||||
|
@Benchmark
|
||||||
|
public void lookupPatchEntries(final LookupState state, final Blackhole blackhole) {
|
||||||
|
final String[] keys = state.ambiguousLookupKeys;
|
||||||
|
for (String key : keys) {
|
||||||
|
final List<ValueCount<String>> entries = state.trie.getEntries(key);
|
||||||
|
if (entries.size() < 2) {
|
||||||
|
throw new IllegalStateException("Expected multiple entries for key " + key + '.');
|
||||||
|
}
|
||||||
|
blackhole.consume(entries);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures counted candidate retrieval through the visitor API.
|
||||||
|
*
|
||||||
|
* @param state prepared lookup state
|
||||||
|
* @param blackhole sink preventing dead-code elimination
|
||||||
|
*/
|
||||||
|
@Benchmark
|
||||||
|
public void lookupPatchEntriesWithVisitor(final LookupState state, final Blackhole blackhole) {
|
||||||
|
final char[][] keys = state.ambiguousLookupKeyCharacters;
|
||||||
|
for (char[] key : keys) {
|
||||||
|
final int count = state.trie.getAllNormalized(key, 0, key.length, state.visitorSink, Integer.MAX_VALUE);
|
||||||
|
if (count < 2) {
|
||||||
|
throw new IllegalStateException("Expected multiple entries for benchmark key.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
blackhole.consume(state.visitorAccumulator);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Measures end-to-end preferred stemming from lookup plus patch application.
|
* Measures end-to-end preferred stemming from lookup plus patch application.
|
||||||
*
|
*
|
||||||
@@ -170,6 +277,28 @@ public class FrequencyTrieLookupBenchmark {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measures patch application into caller-owned output storage.
|
||||||
|
*
|
||||||
|
* @param state prepared lookup state
|
||||||
|
* @param blackhole sink preventing dead-code elimination
|
||||||
|
*/
|
||||||
|
@Benchmark
|
||||||
|
public void applyPreferredPatchToBuffer(final LookupState state, final Blackhole blackhole) {
|
||||||
|
final String[] keys = state.lookupKeys;
|
||||||
|
final String[] patches = state.preferredPatches;
|
||||||
|
final char[] output = state.outputBuffer;
|
||||||
|
for (int index = 0; index < keys.length; index++) {
|
||||||
|
final int length = PatchCommandEncoder.applyTo(keys[index], patches[index],
|
||||||
|
state.trie.traversalDirection(), output, 0, output.length);
|
||||||
|
if (length == PatchCommandEncoder.APPLY_INSUFFICIENT_CAPACITY) {
|
||||||
|
throw new IllegalStateException("Output buffer too small for key " + keys[index] + '.');
|
||||||
|
}
|
||||||
|
blackhole.consume(length);
|
||||||
|
blackhole.consume(output[0]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Measures end-to-end full candidate stemming from {@code getAll()} plus
|
* Measures end-to-end full candidate stemming from {@code getAll()} plus
|
||||||
* patch application.
|
* patch application.
|
||||||
|
|||||||
@@ -48,8 +48,7 @@ public enum CaseProcessingMode {
|
|||||||
AS_IS,
|
AS_IS,
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Normalizes all dictionary content to lower case using
|
* Normalizes all dictionary content to lower case using {@link Locale#ROOT}.
|
||||||
* {@link Locale#ROOT}.
|
|
||||||
*/
|
*/
|
||||||
LOWERCASE_WITH_LOCALE_ROOT
|
LOWERCASE_WITH_LOCALE_ROOT
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -93,12 +93,12 @@ final class DiacriticStripper {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Removes supported diacritic marks and common Latin ligatures from the supplied
|
* Removes supported diacritic marks and common Latin ligatures from the
|
||||||
* text.
|
* supplied text.
|
||||||
*
|
*
|
||||||
* <p>
|
* <p>
|
||||||
* The method returns the original {@link String} instance when no replacement is
|
* The method returns the original {@link String} instance when no replacement
|
||||||
* required, avoiding an unnecessary allocation on the common ASCII path.
|
* is required, avoiding an unnecessary allocation on the common ASCII path.
|
||||||
* </p>
|
* </p>
|
||||||
*
|
*
|
||||||
* @param input text to normalize
|
* @param input text to normalize
|
||||||
|
|||||||
@@ -119,7 +119,8 @@ public final class FrequencyTrie<V> {
|
|||||||
private final boolean removeDiacritics;
|
private final boolean removeDiacritics;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Shared empty array instance for empty lookup results from {@link #getAll(String)}.
|
* Shared empty array instance for empty lookup results from
|
||||||
|
* {@link #getAll(String)}.
|
||||||
*/
|
*/
|
||||||
private final V[] emptyValues;
|
private final V[] emptyValues;
|
||||||
|
|
||||||
@@ -165,13 +166,18 @@ public final class FrequencyTrie<V> {
|
|||||||
*/
|
*/
|
||||||
private static final int CASE_VERSION = 4;
|
private static final int CASE_VERSION = 4;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Argument name for lookup keys.
|
||||||
|
*/
|
||||||
|
private static final String ARG_KEY = "key";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Default dense child lookup span in code points used when materializing
|
* Default dense child lookup span in code points used when materializing
|
||||||
* compiled nodes without an explicit override.
|
* compiled nodes without an explicit override.
|
||||||
* <p>
|
* <p>
|
||||||
* Increasing this value increases the chance of direct array indexing for
|
* Increasing this value increases the chance of direct array indexing for child
|
||||||
* child lookup at runtime at the cost of per-node dense table memory for
|
* lookup at runtime at the cost of per-node dense table memory for compact
|
||||||
* compact character spans.
|
* character spans.
|
||||||
* </p>
|
* </p>
|
||||||
*/
|
*/
|
||||||
public static final int DEFAULT_MAX_EXPANDED_INDEX = 512;
|
public static final int DEFAULT_MAX_EXPANDED_INDEX = 512;
|
||||||
@@ -191,6 +197,30 @@ public final class FrequencyTrie<V> {
|
|||||||
return STREAM_VERSION;
|
return STREAM_VERSION;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Receives trie values during visitor-style lookup.
|
||||||
|
*
|
||||||
|
* <p>
|
||||||
|
* Implementations are caller-owned and are not retained by the trie. Returning
|
||||||
|
* {@code false} stops iteration after the current callback.
|
||||||
|
* </p>
|
||||||
|
*
|
||||||
|
* @param <V> value type
|
||||||
|
*/
|
||||||
|
@FunctionalInterface
|
||||||
|
public interface EntrySink<V> {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Accepts one ordered local value.
|
||||||
|
*
|
||||||
|
* @param value stored value
|
||||||
|
* @param count stored local occurrence count
|
||||||
|
* @param rank zero-based rank in deterministic local ordering
|
||||||
|
* @return {@code true} to continue iteration, {@code false} to stop
|
||||||
|
*/
|
||||||
|
boolean accept(V value, int count, int rank);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Creates a new compiled trie instance.
|
* Creates a new compiled trie instance.
|
||||||
*
|
*
|
||||||
@@ -229,7 +259,7 @@ public final class FrequencyTrie<V> {
|
|||||||
* @throws NullPointerException if {@code key} is {@code null}
|
* @throws NullPointerException if {@code key} is {@code null}
|
||||||
*/
|
*/
|
||||||
public V get(final String key) {
|
public V get(final String key) {
|
||||||
Objects.requireNonNull(key, "key");
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
final CompiledNode<V> node = findNode(normalizeLookupKey(key));
|
final CompiledNode<V> node = findNode(normalizeLookupKey(key));
|
||||||
if (node == null) {
|
if (node == null) {
|
||||||
return null;
|
return null;
|
||||||
@@ -266,7 +296,7 @@ public final class FrequencyTrie<V> {
|
|||||||
*/
|
*/
|
||||||
@SuppressWarnings("PMD.MethodReturnsInternalArray")
|
@SuppressWarnings("PMD.MethodReturnsInternalArray")
|
||||||
public V[] getAll(final String key) {
|
public V[] getAll(final String key) {
|
||||||
Objects.requireNonNull(key, "key");
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
final CompiledNode<V> node = findNode(normalizeLookupKey(key));
|
final CompiledNode<V> node = findNode(normalizeLookupKey(key));
|
||||||
if (node == null) {
|
if (node == null) {
|
||||||
return this.emptyValues;
|
return this.emptyValues;
|
||||||
@@ -301,7 +331,7 @@ public final class FrequencyTrie<V> {
|
|||||||
* @throws NullPointerException if {@code key} is {@code null}
|
* @throws NullPointerException if {@code key} is {@code null}
|
||||||
*/
|
*/
|
||||||
public List<ValueCount<V>> getEntries(final String key) {
|
public List<ValueCount<V>> getEntries(final String key) {
|
||||||
Objects.requireNonNull(key, "key");
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
final CompiledNode<V> node = findNode(normalizeLookupKey(key));
|
final CompiledNode<V> node = findNode(normalizeLookupKey(key));
|
||||||
if (node == null) {
|
if (node == null) {
|
||||||
return List.of();
|
return List.of();
|
||||||
@@ -325,6 +355,132 @@ public final class FrequencyTrie<V> {
|
|||||||
return Collections.unmodifiableList(entries);
|
return Collections.unmodifiableList(entries);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits all values stored at the node addressed by an already-normalized
|
||||||
|
* {@code char[]} key slice.
|
||||||
|
*
|
||||||
|
* <p>
|
||||||
|
* This method bypasses {@link TrieMetadata#caseProcessingMode()} and
|
||||||
|
* {@link TrieMetadata#diacriticProcessingMode()}. The caller must provide input
|
||||||
|
* normalized exactly as required by this trie's metadata. The trie is immutable
|
||||||
|
* and thread-safe for concurrent reads; the supplied sink is caller-owned and
|
||||||
|
* is not retained.
|
||||||
|
* </p>
|
||||||
|
*
|
||||||
|
* @param key normalized key storage
|
||||||
|
* @param offset first character offset
|
||||||
|
* @param length number of characters to read
|
||||||
|
* @param sink value sink
|
||||||
|
* @param maxResults maximum number of results to visit
|
||||||
|
* @return number of visited values
|
||||||
|
* @throws NullPointerException if {@code key} or {@code sink} is
|
||||||
|
* {@code null}
|
||||||
|
* @throws IndexOutOfBoundsException if the key slice is invalid
|
||||||
|
* @throws IllegalArgumentException if {@code maxResults} is negative
|
||||||
|
*/
|
||||||
|
public int getAllNormalized(final char[] key, final int offset, final int length, final EntrySink<? super V> sink,
|
||||||
|
final int maxResults) {
|
||||||
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
|
Objects.requireNonNull(sink, "sink");
|
||||||
|
Objects.checkFromIndexSize(offset, length, key.length);
|
||||||
|
validateMaxResults(maxResults);
|
||||||
|
if (maxResults == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return visitNode(findNode(key, offset, length), sink, maxResults);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits all values stored at the node addressed by an already-normalized
|
||||||
|
* character sequence.
|
||||||
|
*
|
||||||
|
* @param key normalized key
|
||||||
|
* @param sink value sink
|
||||||
|
* @param maxResults maximum number of results to visit
|
||||||
|
* @return number of visited values
|
||||||
|
* @throws NullPointerException if {@code key} or {@code sink} is
|
||||||
|
* {@code null}
|
||||||
|
* @throws IllegalArgumentException if {@code maxResults} is negative
|
||||||
|
* @see #getAllNormalized(char[], int, int, EntrySink, int)
|
||||||
|
*/
|
||||||
|
public int getAllNormalized(final CharSequence key, final EntrySink<? super V> sink, final int maxResults) {
|
||||||
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
|
Objects.requireNonNull(sink, "sink");
|
||||||
|
validateMaxResults(maxResults);
|
||||||
|
if (maxResults == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
return visitNode(findNode(key), sink, maxResults);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits the first value stored at the node addressed by an already-normalized
|
||||||
|
* {@code char[]} key slice.
|
||||||
|
*
|
||||||
|
* @param key normalized key storage
|
||||||
|
* @param offset first character offset
|
||||||
|
* @param length number of characters to read
|
||||||
|
* @param sink value sink
|
||||||
|
* @return {@code true} when a value was visited, otherwise {@code false}
|
||||||
|
* @see #getAllNormalized(char[], int, int, EntrySink, int)
|
||||||
|
*/
|
||||||
|
public boolean getFirstNormalized(final char[] key, final int offset, final int length,
|
||||||
|
final EntrySink<? super V> sink) {
|
||||||
|
return getAllNormalized(key, offset, length, sink, 1) == 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits the first value stored at the node addressed by an already-normalized
|
||||||
|
* character sequence.
|
||||||
|
*
|
||||||
|
* @param key normalized key
|
||||||
|
* @param sink value sink
|
||||||
|
* @return {@code true} when a value was visited, otherwise {@code false}
|
||||||
|
* @see #getAllNormalized(CharSequence, EntrySink, int)
|
||||||
|
*/
|
||||||
|
public boolean getFirstNormalized(final CharSequence key, final EntrySink<? super V> sink) {
|
||||||
|
return getAllNormalized(key, sink, 1) == 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits all values stored at the node addressed by the supplied key, applying
|
||||||
|
* metadata-driven lookup normalization when required.
|
||||||
|
*
|
||||||
|
* <p>
|
||||||
|
* This method preserves the same lookup normalization semantics as
|
||||||
|
* {@link #getAll(String)}. It may allocate when metadata requires lowercase or
|
||||||
|
* diacritic normalization.
|
||||||
|
* </p>
|
||||||
|
*
|
||||||
|
* @param key key to resolve
|
||||||
|
* @param sink value sink
|
||||||
|
* @param maxResults maximum number of results to visit
|
||||||
|
* @return number of visited values
|
||||||
|
*/
|
||||||
|
public int getAll(final CharSequence key, final EntrySink<? super V> sink, final int maxResults) {
|
||||||
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
|
Objects.requireNonNull(sink, "sink");
|
||||||
|
validateMaxResults(maxResults);
|
||||||
|
if (maxResults == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
final CharSequence normalized = normalizeLookupKey(key);
|
||||||
|
return visitNode(findNode(normalized), sink, maxResults);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits the first value stored at the node addressed by the supplied key,
|
||||||
|
* applying metadata-driven lookup normalization when required.
|
||||||
|
*
|
||||||
|
* @param key key to resolve
|
||||||
|
* @param sink value sink
|
||||||
|
* @return {@code true} when a value was visited, otherwise {@code false}
|
||||||
|
* @see #getAll(CharSequence, EntrySink, int)
|
||||||
|
*/
|
||||||
|
public boolean getFirst(final CharSequence key, final EntrySink<? super V> sink) {
|
||||||
|
return getAll(key, sink, 1) == 1;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Returns the logical key traversal direction used by this trie.
|
* Returns the logical key traversal direction used by this trie.
|
||||||
*
|
*
|
||||||
@@ -431,16 +587,17 @@ public final class FrequencyTrie<V> {
|
|||||||
* dense child-index span configuration.
|
* dense child-index span configuration.
|
||||||
* <p>
|
* <p>
|
||||||
* This setting is applied only while materializing the in-memory compiled
|
* This setting is applied only while materializing the in-memory compiled
|
||||||
* representation during load. It is not serialized in {@link TrieMetadata},
|
* representation during load. It is not serialized in {@link TrieMetadata}, so
|
||||||
* so each load can independently choose its own runtime lookup trade-off.
|
* each load can independently choose its own runtime lookup trade-off.
|
||||||
* </p>
|
* </p>
|
||||||
*
|
*
|
||||||
* @param inputStream source input stream
|
* @param inputStream source input stream
|
||||||
* @param arrayFactory array factory used to create typed arrays
|
* @param arrayFactory array factory used to create typed arrays
|
||||||
* @param valueCodec codec used to read values
|
* @param valueCodec codec used to read values
|
||||||
* @param maxExpandedIndex dense lookup span override; zero disables dense lookup,
|
* @param maxExpandedIndex dense lookup span override; zero disables dense
|
||||||
* negative values use {@link #DEFAULT_MAX_EXPANDED_INDEX}
|
* lookup, negative values use
|
||||||
* @param <V> value type
|
* {@link #DEFAULT_MAX_EXPANDED_INDEX}
|
||||||
|
* @param <V> value type
|
||||||
* @return deserialized compiled trie
|
* @return deserialized compiled trie
|
||||||
* @throws NullPointerException if any argument is {@code null}
|
* @throws NullPointerException if any argument is {@code null}
|
||||||
* @throws IOException if reading fails or the binary format is invalid
|
* @throws IOException if reading fails or the binary format is invalid
|
||||||
@@ -573,7 +730,8 @@ public final class FrequencyTrie<V> {
|
|||||||
|
|
||||||
final TrieMetadata sourceMetadata = readMetadata(dataInput, version);
|
final TrieMetadata sourceMetadata = readMetadata(dataInput, version);
|
||||||
final int effectiveMaxExpandedIndex = maxExpandedIndex >= 0 ? maxExpandedIndex : DEFAULT_MAX_EXPANDED_INDEX;
|
final int effectiveMaxExpandedIndex = maxExpandedIndex >= 0 ? maxExpandedIndex : DEFAULT_MAX_EXPANDED_INDEX;
|
||||||
final CompiledNode<V>[] nodes = readNodes(dataInput, arrayFactory, valueCodec, nodeCount, effectiveMaxExpandedIndex);
|
final CompiledNode<V>[] nodes = readNodes(dataInput, arrayFactory, valueCodec, nodeCount,
|
||||||
|
effectiveMaxExpandedIndex);
|
||||||
final CompiledNode<V> rootNode = nodes[rootNodeId];
|
final CompiledNode<V> rootNode = nodes[rootNodeId];
|
||||||
|
|
||||||
if (LOGGER.isLoggable(Level.FINE)) {
|
if (LOGGER.isLoggable(Level.FINE)) {
|
||||||
@@ -584,12 +742,12 @@ public final class FrequencyTrie<V> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private static DataInputStream wrapInputStream(final InputStream inputStream) {
|
private static DataInputStream wrapInputStream(final InputStream inputStream) {
|
||||||
return inputStream instanceof DataInputStream
|
return inputStream instanceof DataInputStream ? (DataInputStream) inputStream
|
||||||
? (DataInputStream) inputStream
|
|
||||||
: new DataInputStream(inputStream);
|
: new DataInputStream(inputStream);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static TrieMetadata readMetadata(final DataInputStream dataInput, final int version) throws IOException {
|
private static TrieMetadata readMetadata(final DataInputStream dataInput, final int version)
|
||||||
|
throws IOException {
|
||||||
if (version == STREAM_VERSION) {
|
if (version == STREAM_VERSION) {
|
||||||
return readTextMetadata(dataInput);
|
return readTextMetadata(dataInput);
|
||||||
}
|
}
|
||||||
@@ -600,12 +758,12 @@ public final class FrequencyTrie<V> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
final ReductionSettings reductionSettings = readReductionSettings(dataInput);
|
final ReductionSettings reductionSettings = readReductionSettings(dataInput);
|
||||||
final DiacriticProcessingMode diacriticProcessingMode = readEnumByOrdinal(dataInput, DiacriticProcessingMode.values(),
|
final DiacriticProcessingMode diacriticProcessingMode = readEnumByOrdinal(dataInput,
|
||||||
"diacritic processing mode");
|
DiacriticProcessingMode.values(), "diacritic processing mode");
|
||||||
final CaseProcessingMode caseProcessingMode = version >= CASE_VERSION
|
final CaseProcessingMode caseProcessingMode = version >= CASE_VERSION ? readCaseProcessingMode(dataInput)
|
||||||
? readCaseProcessingMode(dataInput)
|
|
||||||
: CaseProcessingMode.LOWERCASE_WITH_LOCALE_ROOT;
|
: CaseProcessingMode.LOWERCASE_WITH_LOCALE_ROOT;
|
||||||
return new TrieMetadata(version, traversalDirection, reductionSettings, diacriticProcessingMode, caseProcessingMode);
|
return new TrieMetadata(version, traversalDirection, reductionSettings, diacriticProcessingMode,
|
||||||
|
caseProcessingMode);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static TrieMetadata readTextMetadata(final DataInputStream dataInput) throws IOException {
|
private static TrieMetadata readTextMetadata(final DataInputStream dataInput) throws IOException {
|
||||||
@@ -644,8 +802,9 @@ public final class FrequencyTrie<V> {
|
|||||||
return values[ordinal];
|
return values[ordinal];
|
||||||
}
|
}
|
||||||
|
|
||||||
private static <V> CompiledNode<V>[] readNodes(final DataInputStream dataInput, final IntFunction<V[]> arrayFactory,
|
private static <V> CompiledNode<V>[] readNodes(final DataInputStream dataInput,
|
||||||
final ValueStreamCodec<V> valueCodec, final int nodeCount, final int maxExpandedIndex) throws IOException {
|
final IntFunction<V[]> arrayFactory, final ValueStreamCodec<V> valueCodec, final int nodeCount,
|
||||||
|
final int maxExpandedIndex) throws IOException {
|
||||||
final char[][] edgeLabelsByNode = new char[nodeCount][];
|
final char[][] edgeLabelsByNode = new char[nodeCount][];
|
||||||
final int[][] childNodeIdsByNode = new int[nodeCount][];
|
final int[][] childNodeIdsByNode = new int[nodeCount][];
|
||||||
@SuppressWarnings("unchecked")
|
@SuppressWarnings("unchecked")
|
||||||
@@ -700,14 +859,16 @@ public final class FrequencyTrie<V> {
|
|||||||
|
|
||||||
private static <V> CompiledNode<V> resolveNode(final int nodeIndex, final char[][] edgeLabelsByNode,
|
private static <V> CompiledNode<V> resolveNode(final int nodeIndex, final char[][] edgeLabelsByNode,
|
||||||
final int[][] childNodeIdsByNode, final V[][] orderedValuesByNode, final int[][] orderedCountsByNode,
|
final int[][] childNodeIdsByNode, final V[][] orderedValuesByNode, final int[][] orderedCountsByNode,
|
||||||
final CompiledNode<V>[] nodes, final boolean[] inProgress, final int maxExpandedIndex) throws IOException {
|
final CompiledNode<V>[] nodes, final boolean[] inProgress, final int maxExpandedIndex)
|
||||||
|
throws IOException {
|
||||||
final CompiledNode<V> cachedNode = nodes[nodeIndex];
|
final CompiledNode<V> cachedNode = nodes[nodeIndex];
|
||||||
if (cachedNode != null) {
|
if (cachedNode != null) {
|
||||||
return cachedNode;
|
return cachedNode;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (inProgress[nodeIndex]) {
|
if (inProgress[nodeIndex]) {
|
||||||
throw new IOException("Invalid serialized node graph: cyclic reference detected at node " + nodeIndex + '.');
|
throw new IOException(
|
||||||
|
"Invalid serialized node graph: cyclic reference detected at node " + nodeIndex + '.');
|
||||||
}
|
}
|
||||||
inProgress[nodeIndex] = true;
|
inProgress[nodeIndex] = true;
|
||||||
try {
|
try {
|
||||||
@@ -720,16 +881,15 @@ public final class FrequencyTrie<V> {
|
|||||||
for (int edgeIndex = 0; edgeIndex < edgeCount; edgeIndex++) {
|
for (int edgeIndex = 0; edgeIndex < edgeCount; edgeIndex++) {
|
||||||
final int childNodeId = childNodeIds[edgeIndex];
|
final int childNodeId = childNodeIds[edgeIndex];
|
||||||
if (childNodeId < 0 || childNodeId >= edgeLabelsByNode.length) {
|
if (childNodeId < 0 || childNodeId >= edgeLabelsByNode.length) {
|
||||||
throw new IOException(
|
throw new IOException("Invalid child node id at node " + nodeIndex + ", edge index " + edgeIndex
|
||||||
"Invalid child node id at node " + nodeIndex + ", edge index " + edgeIndex + ": "
|
+ ": " + childNodeId);
|
||||||
+ childNodeId);
|
|
||||||
}
|
}
|
||||||
children[edgeIndex] = resolveNode(childNodeId, edgeLabelsByNode, childNodeIdsByNode,
|
children[edgeIndex] = resolveNode(childNodeId, edgeLabelsByNode, childNodeIdsByNode,
|
||||||
orderedValuesByNode, orderedCountsByNode, nodes, inProgress, maxExpandedIndex);
|
orderedValuesByNode, orderedCountsByNode, nodes, inProgress, maxExpandedIndex);
|
||||||
}
|
}
|
||||||
|
|
||||||
final CompiledNode<V> node = new CompiledNode<>(edgeLabels, children, orderedValuesByNode[nodeIndex], maxExpandedIndex,
|
final CompiledNode<V> node = new CompiledNode<>(edgeLabels, children, orderedValuesByNode[nodeIndex],
|
||||||
orderedCountsByNode[nodeIndex]);
|
maxExpandedIndex, orderedCountsByNode[nodeIndex]);
|
||||||
nodes[nodeIndex] = node;
|
nodes[nodeIndex] = node;
|
||||||
return node;
|
return node;
|
||||||
} finally {
|
} finally {
|
||||||
@@ -740,8 +900,9 @@ public final class FrequencyTrie<V> {
|
|||||||
private static void validateSerializedEdges(final int nodeIndex, final char... edgeLabels) throws IOException {
|
private static void validateSerializedEdges(final int nodeIndex, final char... edgeLabels) throws IOException {
|
||||||
for (int edgeIndex = 1; edgeIndex < edgeLabels.length; edgeIndex++) {
|
for (int edgeIndex = 1; edgeIndex < edgeLabels.length; edgeIndex++) {
|
||||||
if (edgeLabels[edgeIndex - 1] >= edgeLabels[edgeIndex]) {
|
if (edgeLabels[edgeIndex - 1] >= edgeLabels[edgeIndex]) {
|
||||||
throw new IOException("Edge labels must be strictly ascending at node " + nodeIndex + ", edge index "
|
throw new IOException(
|
||||||
+ edgeIndex + ": '" + edgeLabels[edgeIndex - 1] + "' then '" + edgeLabels[edgeIndex] + "'.");
|
"Edge labels must be strictly ascending at node " + nodeIndex + ", edge index " + edgeIndex
|
||||||
|
+ ": '" + edgeLabels[edgeIndex - 1] + "' then '" + edgeLabels[edgeIndex] + "'.");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -754,6 +915,16 @@ public final class FrequencyTrie<V> {
|
|||||||
* @return compiled node, or {@code null} if the path does not exist
|
* @return compiled node, or {@code null} if the path does not exist
|
||||||
*/
|
*/
|
||||||
private CompiledNode<V> findNode(final String key) {
|
private CompiledNode<V> findNode(final String key) {
|
||||||
|
return findNode((CharSequence) key);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Locates the compiled node for the supplied key.
|
||||||
|
*
|
||||||
|
* @param key already-normalized key to resolve
|
||||||
|
* @return compiled node, or {@code null} if the path does not exist
|
||||||
|
*/
|
||||||
|
private CompiledNode<V> findNode(final CharSequence key) {
|
||||||
CompiledNode<V> current = this.root;
|
CompiledNode<V> current = this.root;
|
||||||
if (this.lookupTraversalDirection == WordTraversalDirection.BACKWARD) {
|
if (this.lookupTraversalDirection == WordTraversalDirection.BACKWARD) {
|
||||||
for (int traversalOffset = key.length() - 1; traversalOffset >= 0; traversalOffset--) {
|
for (int traversalOffset = key.length() - 1; traversalOffset >= 0; traversalOffset--) {
|
||||||
@@ -774,6 +945,77 @@ public final class FrequencyTrie<V> {
|
|||||||
return current;
|
return current;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Locates the compiled node for the supplied key slice.
|
||||||
|
*
|
||||||
|
* @param key already-normalized key storage
|
||||||
|
* @param offset first character offset
|
||||||
|
* @param length number of characters to read
|
||||||
|
* @return compiled node, or {@code null} if the path does not exist
|
||||||
|
*/
|
||||||
|
private CompiledNode<V> findNode(final char[] key, final int offset, final int length) {
|
||||||
|
CompiledNode<V> current = this.root;
|
||||||
|
if (this.lookupTraversalDirection == WordTraversalDirection.BACKWARD) {
|
||||||
|
for (int traversalOffset = offset + length - 1; traversalOffset >= offset; traversalOffset--) {
|
||||||
|
current = current.findChild(key[traversalOffset]);
|
||||||
|
if (current == null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
|
||||||
|
final int endExclusive = offset + length;
|
||||||
|
for (int traversalOffset = offset; traversalOffset < endExclusive; traversalOffset++) {
|
||||||
|
current = current.findChild(key[traversalOffset]);
|
||||||
|
if (current == null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visits node-local values without allocating result containers.
|
||||||
|
*
|
||||||
|
* @param node resolved node, or {@code null}
|
||||||
|
* @param sink value sink
|
||||||
|
* @param maxResults maximum values to visit
|
||||||
|
* @return number of visited values
|
||||||
|
*/
|
||||||
|
private int visitNode(final CompiledNode<V> node, final EntrySink<? super V> sink, final int maxResults) {
|
||||||
|
if (node == null) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
final V[] orderedValues = node.orderedValues();
|
||||||
|
final int valueCount = Math.min(orderedValues.length, maxResults);
|
||||||
|
if (valueCount == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
final int[] orderedCounts = node.orderedCounts();
|
||||||
|
int visited = 0;
|
||||||
|
for (int rank = 0; rank < valueCount; rank++) {
|
||||||
|
visited++;
|
||||||
|
if (!sink.accept(orderedValues[rank], orderedCounts[rank], rank)) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return visited;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validates visitor maximum result count.
|
||||||
|
*
|
||||||
|
* @param maxResults maximum result count
|
||||||
|
*/
|
||||||
|
private static void validateMaxResults(final int maxResults) {
|
||||||
|
if (maxResults < 0) {
|
||||||
|
throw new IllegalArgumentException("maxResults must be non-negative.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Applies lookup-time case normalization according to persisted metadata.
|
* Applies lookup-time case normalization according to persisted metadata.
|
||||||
*
|
*
|
||||||
@@ -781,11 +1023,21 @@ public final class FrequencyTrie<V> {
|
|||||||
* @return normalized key for trie traversal
|
* @return normalized key for trie traversal
|
||||||
*/
|
*/
|
||||||
private String normalizeLookupKey(final String key) {
|
private String normalizeLookupKey(final String key) {
|
||||||
|
return normalizeLookupKey((CharSequence) key).toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies lookup-time normalization according to persisted metadata.
|
||||||
|
*
|
||||||
|
* @param key lookup key
|
||||||
|
* @return normalized key for trie traversal
|
||||||
|
*/
|
||||||
|
private CharSequence normalizeLookupKey(final CharSequence key) {
|
||||||
if (!this.lowercasesLookupKeys && !this.removeDiacritics) {
|
if (!this.lowercasesLookupKeys && !this.removeDiacritics) {
|
||||||
return key;
|
return key;
|
||||||
}
|
}
|
||||||
|
|
||||||
String normalized = key;
|
String normalized = key.toString();
|
||||||
if (this.lowercasesLookupKeys) {
|
if (this.lowercasesLookupKeys) {
|
||||||
normalized = normalized.toLowerCase(Locale.ROOT);
|
normalized = normalized.toLowerCase(Locale.ROOT);
|
||||||
}
|
}
|
||||||
@@ -846,9 +1098,9 @@ public final class FrequencyTrie<V> {
|
|||||||
/**
|
/**
|
||||||
* Dense edge lookup span threshold.
|
* Dense edge lookup span threshold.
|
||||||
* <p>
|
* <p>
|
||||||
* This value controls a speed/memory trade-off during freezing:
|
* This value controls a speed/memory trade-off during freezing: dense child
|
||||||
* dense child lookup tables are allocated only for nodes whose child
|
* lookup tables are allocated only for nodes whose child labels fit in this
|
||||||
* labels fit in this span.
|
* span.
|
||||||
* </p>
|
* </p>
|
||||||
*/
|
*/
|
||||||
private final int maxExpandedIndex;
|
private final int maxExpandedIndex;
|
||||||
@@ -925,8 +1177,8 @@ public final class FrequencyTrie<V> {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Creates a new builder with the provided settings, explicit traversal
|
* Creates a new builder with the provided settings, explicit traversal
|
||||||
* direction, explicit case processing mode, explicit diacritic processing
|
* direction, explicit case processing mode, explicit diacritic processing mode,
|
||||||
* mode, and an explicit dense child lookup threshold.
|
* and an explicit dense child lookup threshold.
|
||||||
*
|
*
|
||||||
* @param arrayFactory array factory
|
* @param arrayFactory array factory
|
||||||
* @param reductionSettings reduction configuration
|
* @param reductionSettings reduction configuration
|
||||||
@@ -934,10 +1186,10 @@ public final class FrequencyTrie<V> {
|
|||||||
* @param caseProcessingMode dictionary case processing mode
|
* @param caseProcessingMode dictionary case processing mode
|
||||||
* @param diacriticProcessingMode dictionary diacritic processing mode
|
* @param diacriticProcessingMode dictionary diacritic processing mode
|
||||||
* @param maxExpandedIndex dense lookup span override; zero disables
|
* @param maxExpandedIndex dense lookup span override; zero disables
|
||||||
* dense lookup. Larger values increase direct
|
* dense lookup. Larger values increase direct
|
||||||
* indexing opportunities while potentially
|
* indexing opportunities while potentially
|
||||||
* increasing materialization memory in nodes
|
* increasing materialization memory in nodes
|
||||||
* whose edge label span is within the limit.
|
* whose edge label span is within the limit.
|
||||||
* @throws NullPointerException if any argument is {@code null}
|
* @throws NullPointerException if any argument is {@code null}
|
||||||
*/
|
*/
|
||||||
public Builder(final IntFunction<V[]> arrayFactory, final ReductionSettings reductionSettings,
|
public Builder(final IntFunction<V[]> arrayFactory, final ReductionSettings reductionSettings,
|
||||||
@@ -1052,7 +1304,7 @@ public final class FrequencyTrie<V> {
|
|||||||
* @throws IllegalArgumentException if {@code count} is less than {@code 1}
|
* @throws IllegalArgumentException if {@code count} is less than {@code 1}
|
||||||
*/
|
*/
|
||||||
public Builder<V> put(final String key, final V value, final int count) {
|
public Builder<V> put(final String key, final V value, final int count) {
|
||||||
Objects.requireNonNull(key, "key");
|
Objects.requireNonNull(key, ARG_KEY);
|
||||||
Objects.requireNonNull(value, "value");
|
Objects.requireNonNull(value, "value");
|
||||||
|
|
||||||
if (count < 1) { // NOPMD
|
if (count < 1) { // NOPMD
|
||||||
|
|||||||
@@ -119,11 +119,11 @@ public final class FrequencyTrieBuilders {
|
|||||||
* Copies one compiled node and all reachable descendants into the target
|
* Copies one compiled node and all reachable descendants into the target
|
||||||
* builder.
|
* builder.
|
||||||
*
|
*
|
||||||
* @param node current compiled node
|
* @param node current compiled node
|
||||||
* @param keyBuilder current key builder
|
* @param keyBuilder current key builder
|
||||||
* @param builder target mutable builder
|
* @param builder target mutable builder
|
||||||
* @param traversalDirection logical key traversal direction used by the source
|
* @param traversalDirection logical key traversal direction used by the source
|
||||||
* @param <V> value type
|
* @param <V> value type
|
||||||
*/
|
*/
|
||||||
private static <V> void copyNode(final CompiledNode<V> node, final StringBuilder keyBuilder,
|
private static <V> void copyNode(final CompiledNode<V> node, final StringBuilder keyBuilder,
|
||||||
final FrequencyTrie.Builder<V> builder, final WordTraversalDirection traversalDirection) {
|
final FrequencyTrie.Builder<V> builder, final WordTraversalDirection traversalDirection) {
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ import java.util.concurrent.locks.ReentrantLock;
|
|||||||
* instance can still be used safely when needed.
|
* instance can still be used safely when needed.
|
||||||
* </p>
|
* </p>
|
||||||
*/
|
*/
|
||||||
@SuppressWarnings("PMD.CyclomaticComplexity")
|
@SuppressWarnings({ "PMD.AvoidLiteralsInIfCondition", "PMD.CyclomaticComplexity", "PMD.ForLoopVariableCount" })
|
||||||
public final class PatchCommandEncoder {
|
public final class PatchCommandEncoder {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -121,6 +121,13 @@ public final class PatchCommandEncoder {
|
|||||||
*/
|
*/
|
||||||
/* default */ static final String NOOP_PATCH = String.valueOf(new char[] { NOOP_OPCODE, NOOP_ARGUMENT });
|
/* default */ static final String NOOP_PATCH = String.valueOf(new char[] { NOOP_OPCODE, NOOP_ARGUMENT });
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return value used by
|
||||||
|
* {@link #applyTo(CharSequence, String, WordTraversalDirection, char[], int, int)}
|
||||||
|
* when the caller-owned output range is too small for the transformed text.
|
||||||
|
*/
|
||||||
|
public static final int APPLY_INSUFFICIENT_CAPACITY = -1;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Prefix used in unsupported NOOP patch argument exceptions.
|
* Prefix used in unsupported NOOP patch argument exceptions.
|
||||||
*/
|
*/
|
||||||
@@ -346,6 +353,78 @@ public final class PatchCommandEncoder {
|
|||||||
return applyStrategyFor(traversalDirection).apply(source, patchCommand);
|
return applyStrategyFor(traversalDirection).apply(source, patchCommand);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a compact patch command into a caller-owned output buffer.
|
||||||
|
*
|
||||||
|
* <p>
|
||||||
|
* The output array is not retained. Capacity failure is reported by
|
||||||
|
* {@link #APPLY_INSUFFICIENT_CAPACITY} and leaves the output range unchanged.
|
||||||
|
* Malformed compatibility cases preserve the source exactly as
|
||||||
|
* {@link #apply(String, String, WordTraversalDirection)} does.
|
||||||
|
* </p>
|
||||||
|
*
|
||||||
|
* @param source original source text
|
||||||
|
* @param patchCommand compact patch command
|
||||||
|
* @param traversalDirection traversal direction used by the patch command
|
||||||
|
* @param output caller-owned output storage
|
||||||
|
* @param outputOffset first writable output offset
|
||||||
|
* @param outputLength writable output capacity
|
||||||
|
* @return produced character count, or {@link #APPLY_INSUFFICIENT_CAPACITY}
|
||||||
|
* when {@code outputLength} is too small
|
||||||
|
*/
|
||||||
|
public static int applyTo(final CharSequence source, final String patchCommand,
|
||||||
|
final WordTraversalDirection traversalDirection, final char[] output, final int outputOffset,
|
||||||
|
final int outputLength) {
|
||||||
|
Objects.requireNonNull(source, "source");
|
||||||
|
Objects.requireNonNull(traversalDirection, "traversalDirection");
|
||||||
|
Objects.requireNonNull(output, "output");
|
||||||
|
Objects.checkFromIndexSize(outputOffset, outputLength, output.length);
|
||||||
|
|
||||||
|
final int sourceLength = source.length();
|
||||||
|
final int producedLength = computeAppliedLength(sourceLength, patchCommand, traversalDirection);
|
||||||
|
if (producedLength > outputLength) {
|
||||||
|
return APPLY_INSUFFICIENT_CAPACITY;
|
||||||
|
}
|
||||||
|
applyToOutput(source, 0, sourceLength, patchCommand, traversalDirection, output, outputOffset, producedLength);
|
||||||
|
return producedLength;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a compact patch command from a caller-owned source slice into a
|
||||||
|
* caller-owned output buffer.
|
||||||
|
*
|
||||||
|
* @param source source storage
|
||||||
|
* @param sourceOffset first source character offset
|
||||||
|
* @param sourceLength number of source characters
|
||||||
|
* @param patchCommand compact patch command
|
||||||
|
* @param traversalDirection traversal direction used by the patch command
|
||||||
|
* @param output caller-owned output storage
|
||||||
|
* @param outputOffset first writable output offset
|
||||||
|
* @param outputLength writable output capacity
|
||||||
|
* @return produced character count, or {@link #APPLY_INSUFFICIENT_CAPACITY}
|
||||||
|
* when {@code outputLength} is too small
|
||||||
|
* @throws IllegalArgumentException when source and output ranges overlap in the
|
||||||
|
* same array
|
||||||
|
*/
|
||||||
|
public static int applyTo(final char[] source, final int sourceOffset, final int sourceLength,
|
||||||
|
final String patchCommand, final WordTraversalDirection traversalDirection, final char[] output,
|
||||||
|
final int outputOffset, final int outputLength) {
|
||||||
|
Objects.requireNonNull(source, "source");
|
||||||
|
Objects.requireNonNull(traversalDirection, "traversalDirection");
|
||||||
|
Objects.requireNonNull(output, "output");
|
||||||
|
Objects.checkFromIndexSize(sourceOffset, sourceLength, source.length);
|
||||||
|
Objects.checkFromIndexSize(outputOffset, outputLength, output.length);
|
||||||
|
validateNonOverlappingRanges(source, sourceOffset, sourceLength, output, outputOffset, outputLength);
|
||||||
|
|
||||||
|
final int producedLength = computeAppliedLength(sourceLength, patchCommand, traversalDirection);
|
||||||
|
if (producedLength > outputLength) {
|
||||||
|
return APPLY_INSUFFICIENT_CAPACITY;
|
||||||
|
}
|
||||||
|
applyToOutput(source, sourceOffset, sourceLength, patchCommand, traversalDirection, output, outputOffset,
|
||||||
|
producedLength);
|
||||||
|
return producedLength;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Encodes a patch command using the historical backward Egothor semantics.
|
* Encodes a patch command using the historical backward Egothor semantics.
|
||||||
*
|
*
|
||||||
@@ -409,7 +488,6 @@ public final class PatchCommandEncoder {
|
|||||||
* @param patchCommand compact patch command
|
* @param patchCommand compact patch command
|
||||||
* @return transformed word, or {@code null} when {@code source} is {@code null}
|
* @return transformed word, or {@code null} when {@code source} is {@code null}
|
||||||
*/
|
*/
|
||||||
@SuppressWarnings({ "PMD.CyclomaticComplexity", "PMD.AvoidLiteralsInIfCondition" })
|
|
||||||
private static String applyBackward(final String source, final String patchCommand) {
|
private static String applyBackward(final String source, final String patchCommand) {
|
||||||
if (source == null) {
|
if (source == null) {
|
||||||
return null;
|
return null;
|
||||||
@@ -435,7 +513,7 @@ public final class PatchCommandEncoder {
|
|||||||
int position = result.length() - 1;
|
int position = result.length() - 1;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) { // NOPMD
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
final char opcode = patchCommand.charAt(patchIndex);
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
final char argument = patchCommand.charAt(patchIndex + 1);
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
@@ -493,7 +571,6 @@ public final class PatchCommandEncoder {
|
|||||||
* @param patchCommand compact patch command
|
* @param patchCommand compact patch command
|
||||||
* @return transformed word, or {@code null} when {@code source} is {@code null}
|
* @return transformed word, or {@code null} when {@code source} is {@code null}
|
||||||
*/
|
*/
|
||||||
@SuppressWarnings({ "PMD.CyclomaticComplexity", "PMD.AvoidLiteralsInIfCondition" })
|
|
||||||
private static String applyForward(final String source, final String patchCommand) {
|
private static String applyForward(final String source, final String patchCommand) {
|
||||||
if (source == null) {
|
if (source == null) {
|
||||||
return null;
|
return null;
|
||||||
@@ -519,7 +596,7 @@ public final class PatchCommandEncoder {
|
|||||||
int position = 0;
|
int position = 0;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) { // NOPMD
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
final char opcode = patchCommand.charAt(patchIndex);
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
final char argument = patchCommand.charAt(patchIndex + 1);
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
@@ -681,7 +758,7 @@ public final class PatchCommandEncoder {
|
|||||||
*/
|
*/
|
||||||
private static String applyBackwardToEmptySource(final StringBuilder result, final String patchCommand) {
|
private static String applyBackwardToEmptySource(final StringBuilder result, final String patchCommand) {
|
||||||
try {
|
try {
|
||||||
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) { // NOPMD
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
final char opcode = patchCommand.charAt(patchIndex);
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
final char argument = patchCommand.charAt(patchIndex + 1);
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
@@ -722,7 +799,7 @@ public final class PatchCommandEncoder {
|
|||||||
*/
|
*/
|
||||||
private static String applyForwardToEmptySource(final StringBuilder result, final String patchCommand) {
|
private static String applyForwardToEmptySource(final StringBuilder result, final String patchCommand) {
|
||||||
try {
|
try {
|
||||||
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) { // NOPMD
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
final char opcode = patchCommand.charAt(patchIndex);
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
final char argument = patchCommand.charAt(patchIndex + 1);
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
@@ -753,6 +830,711 @@ public final class PatchCommandEncoder {
|
|||||||
return result.toString();
|
return result.toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes the transformed length or the preserved source length for malformed
|
||||||
|
* compatibility cases.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param traversalDirection traversal direction
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeAppliedLength(final int sourceLength, final String patchCommand,
|
||||||
|
final WordTraversalDirection traversalDirection) {
|
||||||
|
if (patchCommand == null || patchCommand.isEmpty() || NOOP_PATCH.equals(patchCommand)
|
||||||
|
|| (patchCommand.length() & 1) != 0) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
if (traversalDirection == WordTraversalDirection.BACKWARD) {
|
||||||
|
return computeBackwardAppliedLength(sourceLength, patchCommand);
|
||||||
|
}
|
||||||
|
return computeForwardAppliedLength(sourceLength, patchCommand);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes the backward traversal output length.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeBackwardAppliedLength(final int sourceLength, final String patchCommand) {
|
||||||
|
if (patchCommand.length() == 2) {
|
||||||
|
return computeSingleBackwardAppliedLength(sourceLength, patchCommand.charAt(0), patchCommand.charAt(1));
|
||||||
|
}
|
||||||
|
if (sourceLength == 0) {
|
||||||
|
return computeBackwardEmptyAppliedLength(patchCommand);
|
||||||
|
}
|
||||||
|
|
||||||
|
int currentLength = sourceLength;
|
||||||
|
int position = sourceLength - 1;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
|
switch (opcode) {
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
final int skipCount = decodeEncodedCount(argument);
|
||||||
|
if (skipCount < 1) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
position = position - skipCount + 1;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
if (position < 0 || position >= currentLength) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int deleteCount = decodeEncodedCount(argument);
|
||||||
|
if (deleteCount < 1) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
final int deleteEndExclusive = position + 1;
|
||||||
|
position -= deleteCount - 1;
|
||||||
|
if (position < 0 || deleteEndExclusive > currentLength || position > deleteEndExclusive) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
currentLength -= deleteEndExclusive - position;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
if (position < -1 || position >= currentLength) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
currentLength++;
|
||||||
|
position++;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return sourceLength;
|
||||||
|
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
|
||||||
|
position--;
|
||||||
|
}
|
||||||
|
return currentLength;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes the forward traversal output length.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeForwardAppliedLength(final int sourceLength, final String patchCommand) {
|
||||||
|
if (patchCommand.length() == 2) {
|
||||||
|
return computeSingleForwardAppliedLength(sourceLength, patchCommand.charAt(0), patchCommand.charAt(1));
|
||||||
|
}
|
||||||
|
if (sourceLength == 0) {
|
||||||
|
return computeForwardEmptyAppliedLength(patchCommand);
|
||||||
|
}
|
||||||
|
|
||||||
|
int currentLength = sourceLength;
|
||||||
|
int position = 0;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
|
switch (opcode) {
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
final int skipCount = decodeEncodedCount(argument);
|
||||||
|
if (skipCount < 1) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
position = position + skipCount - 1;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
if (position < 0 || position >= currentLength) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int deleteCount = decodeEncodedCount(argument);
|
||||||
|
if (deleteCount < 1 || position < 0 || position + deleteCount > currentLength) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
currentLength -= deleteCount;
|
||||||
|
position--;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
if (position < 0 || position > currentLength) {
|
||||||
|
return sourceLength;
|
||||||
|
}
|
||||||
|
currentLength++;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return sourceLength;
|
||||||
|
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
|
||||||
|
position++;
|
||||||
|
}
|
||||||
|
return currentLength;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes a single backward instruction output length.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param opcode opcode
|
||||||
|
* @param argument argument
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeSingleBackwardAppliedLength(final int sourceLength, final char opcode,
|
||||||
|
final char argument) {
|
||||||
|
final int encodedValue;
|
||||||
|
switch (opcode) {
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
encodedValue = decodeEncodedCount(argument);
|
||||||
|
return encodedValue < 1 || encodedValue > sourceLength ? sourceLength : sourceLength - encodedValue;
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
return sourceLength + 1;
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
return sourceLength;
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return sourceLength;
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes a single forward instruction output length.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param opcode opcode
|
||||||
|
* @param argument argument
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeSingleForwardAppliedLength(final int sourceLength, final char opcode,
|
||||||
|
final char argument) {
|
||||||
|
return computeSingleBackwardAppliedLength(sourceLength, opcode, argument);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes output length for an empty source in backward traversal.
|
||||||
|
*
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeBackwardEmptyAppliedLength(final String patchCommand) {
|
||||||
|
int currentLength = 0;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
switch (opcode) {
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
currentLength++;
|
||||||
|
break;
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
return 0;
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return currentLength;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computes output length for an empty source in forward traversal.
|
||||||
|
*
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return produced length
|
||||||
|
*/
|
||||||
|
private static int computeForwardEmptyAppliedLength(final String patchCommand) {
|
||||||
|
return computeBackwardEmptyAppliedLength(patchCommand);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies an already-sized patch into caller output.
|
||||||
|
*
|
||||||
|
* @param source source text
|
||||||
|
* @param sourceOffset source offset
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param traversalDirection traversal direction
|
||||||
|
* @param output output storage
|
||||||
|
* @param outputOffset output offset
|
||||||
|
* @param producedLength already-validated produced length
|
||||||
|
*/
|
||||||
|
private static void applyToOutput(final CharSequence source, final int sourceOffset, final int sourceLength,
|
||||||
|
final String patchCommand, final WordTraversalDirection traversalDirection, final char[] output,
|
||||||
|
final int outputOffset, final int producedLength) {
|
||||||
|
if (isPreservedSource(sourceLength, producedLength, patchCommand, traversalDirection)) {
|
||||||
|
copySource(source, sourceOffset, sourceLength, output, outputOffset);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (sourceLength > 0) {
|
||||||
|
copySource(source, sourceOffset, sourceLength, output, outputOffset);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (traversalDirection == WordTraversalDirection.BACKWARD) {
|
||||||
|
applyBackwardToOutput(sourceLength, patchCommand, output, outputOffset);
|
||||||
|
} else {
|
||||||
|
applyForwardToOutput(sourceLength, patchCommand, output, outputOffset);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies an already-sized patch into caller output.
|
||||||
|
*
|
||||||
|
* @param source source storage
|
||||||
|
* @param sourceOffset source offset
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param traversalDirection traversal direction
|
||||||
|
* @param output output storage
|
||||||
|
* @param outputOffset output offset
|
||||||
|
* @param producedLength already-validated produced length
|
||||||
|
*/
|
||||||
|
private static void applyToOutput(final char[] source, final int sourceOffset, final int sourceLength,
|
||||||
|
final String patchCommand, final WordTraversalDirection traversalDirection, final char[] output,
|
||||||
|
final int outputOffset, final int producedLength) {
|
||||||
|
if (isPreservedSource(sourceLength, producedLength, patchCommand, traversalDirection)) {
|
||||||
|
System.arraycopy(source, sourceOffset, output, outputOffset, sourceLength);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (sourceLength > 0) {
|
||||||
|
System.arraycopy(source, sourceOffset, output, outputOffset, sourceLength);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (traversalDirection == WordTraversalDirection.BACKWARD) {
|
||||||
|
applyBackwardToOutput(sourceLength, patchCommand, output, outputOffset);
|
||||||
|
} else {
|
||||||
|
applyForwardToOutput(sourceLength, patchCommand, output, outputOffset);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Determines whether the output is exactly the original source.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param producedLength produced length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param traversalDirection traversal direction
|
||||||
|
* @return {@code true} if copying the source is sufficient
|
||||||
|
*/
|
||||||
|
private static boolean isPreservedSource(final int sourceLength, final int producedLength,
|
||||||
|
final String patchCommand, final WordTraversalDirection traversalDirection) {
|
||||||
|
return producedLength == sourceLength
|
||||||
|
&& isKnownPreserveOnlyPatch(sourceLength, patchCommand, traversalDirection);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Returns whether equal length also means no mutation is needed.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param traversalDirection traversal direction
|
||||||
|
* @return {@code true} when the command preserves source content
|
||||||
|
*/
|
||||||
|
private static boolean isKnownPreserveOnlyPatch(final int sourceLength, final String patchCommand,
|
||||||
|
final WordTraversalDirection traversalDirection) {
|
||||||
|
if (patchCommand == null || patchCommand.isEmpty() || NOOP_PATCH.equals(patchCommand)
|
||||||
|
|| (patchCommand.length() & 1) != 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (patchCommand.length() == 2) {
|
||||||
|
return isSingleInstructionPreserveOnly(sourceLength, patchCommand.charAt(0), patchCommand.charAt(1));
|
||||||
|
}
|
||||||
|
if (sourceLength == 0) {
|
||||||
|
return hasEmptySourcePreserveOnlyPatch(patchCommand);
|
||||||
|
}
|
||||||
|
return traversalDirection == WordTraversalDirection.BACKWARD
|
||||||
|
? hasBackwardPreserveOnlyPatch(sourceLength, patchCommand)
|
||||||
|
: hasForwardPreserveOnlyPatch(sourceLength, patchCommand);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests whether a single instruction preserves the source content.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param opcode opcode
|
||||||
|
* @param argument argument
|
||||||
|
* @return {@code true} when no mutation should be applied
|
||||||
|
*/
|
||||||
|
private static boolean isSingleInstructionPreserveOnly(final int sourceLength, final char opcode,
|
||||||
|
final char argument) {
|
||||||
|
switch (opcode) {
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int encodedValue = decodeEncodedCount(argument);
|
||||||
|
return encodedValue < 1 || encodedValue > sourceLength;
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
return false;
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
return sourceLength == 0;
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
return true;
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests whether an empty-source patch preserves the source.
|
||||||
|
*
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return {@code true} when no mutation should be applied
|
||||||
|
*/
|
||||||
|
private static boolean hasEmptySourcePreserveOnlyPatch(final String patchCommand) {
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
switch (opcode) {
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
break;
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
return true;
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests whether a backward patch preserves the source because it is malformed
|
||||||
|
* or a NOOP.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return {@code true} when no mutation should be applied
|
||||||
|
*/
|
||||||
|
private static boolean hasBackwardPreserveOnlyPatch(final int sourceLength, final String patchCommand) {
|
||||||
|
int currentLength = sourceLength;
|
||||||
|
int position = sourceLength - 1;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
switch (opcode) {
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
final int skipCount = decodeEncodedCount(argument);
|
||||||
|
if (skipCount < 1) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
position = position - skipCount + 1;
|
||||||
|
break;
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
if (position < 0 || position >= currentLength) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int deleteCount = decodeEncodedCount(argument);
|
||||||
|
if (deleteCount < 1) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
final int deleteEndExclusive = position + 1;
|
||||||
|
position -= deleteCount - 1;
|
||||||
|
if (position < 0 || deleteEndExclusive > currentLength || position > deleteEndExclusive) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
currentLength -= deleteEndExclusive - position;
|
||||||
|
break;
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
if (position < -1 || position >= currentLength) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
currentLength++;
|
||||||
|
position++;
|
||||||
|
break;
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
position--;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests whether a forward patch preserves the source because it is malformed or
|
||||||
|
* a NOOP.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @return {@code true} when no mutation should be applied
|
||||||
|
*/
|
||||||
|
private static boolean hasForwardPreserveOnlyPatch(final int sourceLength, final String patchCommand) {
|
||||||
|
int currentLength = sourceLength;
|
||||||
|
int position = 0;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
switch (opcode) {
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
final int skipCount = decodeEncodedCount(argument);
|
||||||
|
if (skipCount < 1) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
position = position + skipCount - 1;
|
||||||
|
break;
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
if (position < 0 || position >= currentLength) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int deleteCount = decodeEncodedCount(argument);
|
||||||
|
if (deleteCount < 1 || position < 0 || position + deleteCount > currentLength) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
currentLength -= deleteCount;
|
||||||
|
position--;
|
||||||
|
break;
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
if (position < 0 || position > currentLength) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
currentLength++;
|
||||||
|
break;
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
if (argument != NOOP_ARGUMENT) {
|
||||||
|
throw new IllegalArgumentException(MSG_NOOP + argument);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
default:
|
||||||
|
throw new IllegalArgumentException(MSG_OPCODE + opcode);
|
||||||
|
}
|
||||||
|
position++;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Copies source characters from a sequence.
|
||||||
|
*
|
||||||
|
* @param source source text
|
||||||
|
* @param sourceOffset source offset
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param output output storage
|
||||||
|
* @param outputOffset output offset
|
||||||
|
*/
|
||||||
|
private static void copySource(final CharSequence source, final int sourceOffset, final int sourceLength,
|
||||||
|
final char[] output, final int outputOffset) {
|
||||||
|
for (int index = 0; index < sourceLength; index++) {
|
||||||
|
output[outputOffset + index] = source.charAt(sourceOffset + index);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a backward patch after validation.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param output output storage initialized with source
|
||||||
|
* @param outputOffset output offset
|
||||||
|
*/
|
||||||
|
private static void applyBackwardToOutput(final int sourceLength, final String patchCommand, final char[] output,
|
||||||
|
final int outputOffset) {
|
||||||
|
if (sourceLength == 0) {
|
||||||
|
applyBackwardEmptyToOutput(patchCommand, output, outputOffset);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
int currentLength = sourceLength;
|
||||||
|
int position = sourceLength - 1;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
|
switch (opcode) {
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
position = position - decodeEncodedCount(argument) + 1;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
output[outputOffset + position] = argument;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int deleteEndExclusive = position + 1;
|
||||||
|
position -= decodeEncodedCount(argument) - 1;
|
||||||
|
System.arraycopy(output, outputOffset + deleteEndExclusive, output, outputOffset + position,
|
||||||
|
currentLength - deleteEndExclusive);
|
||||||
|
currentLength -= deleteEndExclusive - position;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
final int insertIndex = position + 1;
|
||||||
|
System.arraycopy(output, outputOffset + insertIndex, output, outputOffset + insertIndex + 1,
|
||||||
|
currentLength - insertIndex);
|
||||||
|
output[outputOffset + insertIndex] = argument;
|
||||||
|
currentLength++;
|
||||||
|
position++;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
return;
|
||||||
|
|
||||||
|
default:
|
||||||
|
throw new AssertionError("Patch command was not validated.");
|
||||||
|
}
|
||||||
|
|
||||||
|
position--;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a forward patch after validation.
|
||||||
|
*
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param output output storage initialized with source
|
||||||
|
* @param outputOffset output offset
|
||||||
|
*/
|
||||||
|
private static void applyForwardToOutput(final int sourceLength, final String patchCommand, final char[] output,
|
||||||
|
final int outputOffset) {
|
||||||
|
if (sourceLength == 0) {
|
||||||
|
applyForwardEmptyToOutput(patchCommand, output, outputOffset);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
int currentLength = sourceLength;
|
||||||
|
int position = 0;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char opcode = patchCommand.charAt(patchIndex);
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
|
||||||
|
switch (opcode) {
|
||||||
|
case SKIP_OPCODE:
|
||||||
|
position = position + decodeEncodedCount(argument) - 1;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case REPLACE_OPCODE:
|
||||||
|
output[outputOffset + position] = argument;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case DELETE_OPCODE:
|
||||||
|
final int deleteCount = decodeEncodedCount(argument);
|
||||||
|
System.arraycopy(output, outputOffset + position + deleteCount, output, outputOffset + position,
|
||||||
|
currentLength - position - deleteCount);
|
||||||
|
currentLength -= deleteCount;
|
||||||
|
position--;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case INSERT_OPCODE:
|
||||||
|
System.arraycopy(output, outputOffset + position, output, outputOffset + position + 1,
|
||||||
|
currentLength - position);
|
||||||
|
output[outputOffset + position] = argument;
|
||||||
|
currentLength++;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case NOOP_OPCODE:
|
||||||
|
return;
|
||||||
|
|
||||||
|
default:
|
||||||
|
throw new AssertionError("Patch command was not validated.");
|
||||||
|
}
|
||||||
|
|
||||||
|
position++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies an empty-source backward patch after validation.
|
||||||
|
*
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param output output storage
|
||||||
|
* @param outputOffset output offset
|
||||||
|
*/
|
||||||
|
private static void applyBackwardEmptyToOutput(final String patchCommand, final char[] output,
|
||||||
|
final int outputOffset) {
|
||||||
|
int currentLength = 0;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
final char argument = patchCommand.charAt(patchIndex + 1);
|
||||||
|
System.arraycopy(output, outputOffset, output, outputOffset + 1, currentLength);
|
||||||
|
output[outputOffset] = argument;
|
||||||
|
currentLength++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies an empty-source forward patch after validation.
|
||||||
|
*
|
||||||
|
* @param patchCommand patch command
|
||||||
|
* @param output output storage
|
||||||
|
* @param outputOffset output offset
|
||||||
|
*/
|
||||||
|
private static void applyForwardEmptyToOutput(final String patchCommand, final char[] output,
|
||||||
|
final int outputOffset) {
|
||||||
|
int currentLength = 0;
|
||||||
|
for (int patchIndex = 0, patchLength = patchCommand.length(); patchIndex < patchLength; patchIndex += 2) {
|
||||||
|
output[outputOffset + currentLength] = patchCommand.charAt(patchIndex + 1);
|
||||||
|
currentLength++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validates that source and output slices do not overlap when backed by the
|
||||||
|
* same array.
|
||||||
|
*
|
||||||
|
* @param source source storage
|
||||||
|
* @param sourceOffset source offset
|
||||||
|
* @param sourceLength source length
|
||||||
|
* @param output output storage
|
||||||
|
* @param outputOffset output offset
|
||||||
|
* @param outputLength output length
|
||||||
|
*/
|
||||||
|
private static void validateNonOverlappingRanges(final char[] source, final int sourceOffset,
|
||||||
|
final int sourceLength, final char[] output, final int outputOffset, final int outputLength) {
|
||||||
|
if (!source.equals(output) || sourceLength == 0 || outputLength == 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
final int sourceEnd = sourceOffset + sourceLength;
|
||||||
|
final int outputEnd = outputOffset + outputLength;
|
||||||
|
if (sourceOffset < outputEnd && outputOffset < sourceEnd) {
|
||||||
|
throw new IllegalArgumentException("source and output ranges must not overlap.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Returns the direction-specialized apply strategy.
|
* Returns the direction-specialized apply strategy.
|
||||||
*
|
*
|
||||||
@@ -769,7 +1551,6 @@ public final class PatchCommandEncoder {
|
|||||||
* @param argument serialized count argument
|
* @param argument serialized count argument
|
||||||
* @return decoded positive count, or {@code -1} when the argument is malformed
|
* @return decoded positive count, or {@code -1} when the argument is malformed
|
||||||
*/
|
*/
|
||||||
@SuppressWarnings("PMD.AvoidLiteralsInIfCondition")
|
|
||||||
private static int decodeEncodedCount(final char argument) {
|
private static int decodeEncodedCount(final char argument) {
|
||||||
if (argument < 'a') {
|
if (argument < 'a') {
|
||||||
return -1;
|
return -1;
|
||||||
|
|||||||
@@ -95,8 +95,8 @@ public final class StemmerPatchTrieBinaryIO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Reads a GZip-compressed binary patch-command trie from a filesystem path
|
* Reads a GZip-compressed binary patch-command trie from a filesystem path with
|
||||||
* with an optional dense child lookup span override.
|
* an optional dense child lookup span override.
|
||||||
* <p>
|
* <p>
|
||||||
* This is a runtime-only tuning parameter. The dense-span setting is not
|
* This is a runtime-only tuning parameter. The dense-span setting is not
|
||||||
* persisted in the file and does not change the compiled metadata.
|
* persisted in the file and does not change the compiled metadata.
|
||||||
@@ -183,14 +183,15 @@ public final class StemmerPatchTrieBinaryIO {
|
|||||||
* persisted in the file and does not change the compiled metadata.
|
* persisted in the file and does not change the compiled metadata.
|
||||||
* </p>
|
* </p>
|
||||||
*
|
*
|
||||||
* @param inputStream source stream
|
* @param inputStream source stream
|
||||||
* @param maxExpandedIndex dense lookup span override; negative values use
|
* @param maxExpandedIndex dense lookup span override; negative values use
|
||||||
* {@link FrequencyTrie#DEFAULT_MAX_EXPANDED_INDEX}
|
* {@link FrequencyTrie#DEFAULT_MAX_EXPANDED_INDEX}
|
||||||
* @return deserialized trie
|
* @return deserialized trie
|
||||||
* @throws NullPointerException if {@code inputStream} is {@code null}
|
* @throws NullPointerException if {@code inputStream} is {@code null}
|
||||||
* @throws IOException if reading or decompression fails
|
* @throws IOException if reading or decompression fails
|
||||||
*/
|
*/
|
||||||
public static FrequencyTrie<String> read(final InputStream inputStream, final int maxExpandedIndex) throws IOException {
|
public static FrequencyTrie<String> read(final InputStream inputStream, final int maxExpandedIndex)
|
||||||
|
throws IOException {
|
||||||
Objects.requireNonNull(inputStream, "inputStream");
|
Objects.requireNonNull(inputStream, "inputStream");
|
||||||
|
|
||||||
try (GZIPInputStream gzipInputStream = new GZIPInputStream(new BufferedInputStream(inputStream));
|
try (GZIPInputStream gzipInputStream = new GZIPInputStream(new BufferedInputStream(inputStream));
|
||||||
|
|||||||
@@ -461,7 +461,7 @@ public final class StemmerPatchTrieLoader {
|
|||||||
public static FrequencyTrie<String> load(final Path path, final boolean storeOriginal,
|
public static FrequencyTrie<String> load(final Path path, final boolean storeOriginal,
|
||||||
final ReductionSettings reductionSettings, final WordTraversalDirection traversalDirection,
|
final ReductionSettings reductionSettings, final WordTraversalDirection traversalDirection,
|
||||||
final CaseProcessingMode caseProcessingMode, final DiacriticProcessingMode diacriticProcessingMode)
|
final CaseProcessingMode caseProcessingMode, final DiacriticProcessingMode diacriticProcessingMode)
|
||||||
throws IOException {
|
throws IOException {
|
||||||
Objects.requireNonNull(path, PARAMETER_PATH);
|
Objects.requireNonNull(path, PARAMETER_PATH);
|
||||||
final TrieMetadata metadata = metadataForCompilation(traversalDirection, reductionSettings, caseProcessingMode,
|
final TrieMetadata metadata = metadataForCompilation(traversalDirection, reductionSettings, caseProcessingMode,
|
||||||
diacriticProcessingMode);
|
diacriticProcessingMode);
|
||||||
@@ -816,7 +816,8 @@ public final class StemmerPatchTrieLoader {
|
|||||||
* @throws IOException if the file cannot be opened, decompressed, or
|
* @throws IOException if the file cannot be opened, decompressed, or
|
||||||
* read
|
* read
|
||||||
*/
|
*/
|
||||||
public static FrequencyTrie<String> loadBinary(final String fileName, final int maxExpandedIndex) throws IOException {
|
public static FrequencyTrie<String> loadBinary(final String fileName, final int maxExpandedIndex)
|
||||||
|
throws IOException {
|
||||||
Objects.requireNonNull(fileName, FILENAME_REQUIRED);
|
Objects.requireNonNull(fileName, FILENAME_REQUIRED);
|
||||||
return StemmerPatchTrieBinaryIO.read(fileName, maxExpandedIndex);
|
return StemmerPatchTrieBinaryIO.read(fileName, maxExpandedIndex);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -58,17 +58,17 @@
|
|||||||
* {@link org.egothor.stemmer.StemmerPatchTrieLoader}, which reads the
|
* {@link org.egothor.stemmer.StemmerPatchTrieLoader}, which reads the
|
||||||
* traditional line-oriented tab-separated values resource format in which each
|
* traditional line-oriented tab-separated values resource format in which each
|
||||||
* non-empty logical line starts with a canonical stem followed by known surface
|
* non-empty logical line starts with a canonical stem followed by known surface
|
||||||
* variants in subsequent tab-separated columns.
|
* variants in subsequent tab-separated columns. Parsing is delegated to
|
||||||
* Parsing is delegated to {@link org.egothor.stemmer.StemmerDictionaryParser},
|
* {@link org.egothor.stemmer.StemmerDictionaryParser}, which applies
|
||||||
* which applies configurable case processing through
|
* configurable case processing through
|
||||||
* {@link org.egothor.stemmer.CaseProcessingMode} (default:
|
* {@link org.egothor.stemmer.CaseProcessingMode} (default:
|
||||||
* {@link org.egothor.stemmer.CaseProcessingMode#LOWERCASE_WITH_LOCALE_ROOT}),
|
* {@link org.egothor.stemmer.CaseProcessingMode#LOWERCASE_WITH_LOCALE_ROOT}),
|
||||||
* supports whole-line as well as trailing remarks introduced by {@code #} or
|
* supports whole-line as well as trailing remarks introduced by {@code #} or
|
||||||
* {@code //}, and currently ignores dictionary items containing Unicode
|
* {@code //}, and currently ignores dictionary items containing Unicode
|
||||||
* whitespace characters while reporting them through warning-level diagnostics.
|
* whitespace characters while reporting them through warning-level diagnostics.
|
||||||
* During loading, each variant is converted into a patch command
|
* During loading, each variant is converted into a patch command targeting the
|
||||||
* targeting the canonical stem, and the stem itself may optionally be stored
|
* canonical stem, and the stem itself may optionally be stored under the
|
||||||
* under the canonical no-operation patch.
|
* canonical no-operation patch.
|
||||||
* </p>
|
* </p>
|
||||||
*
|
*
|
||||||
* <p>
|
* <p>
|
||||||
|
|||||||
@@ -48,8 +48,8 @@ import java.util.Objects;
|
|||||||
public final class CompiledNode<V> {
|
public final class CompiledNode<V> {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Default dense child lookup span in characters used when an explicit override is
|
* Default dense child lookup span in characters used when an explicit override
|
||||||
* not provided.
|
* is not provided.
|
||||||
*/
|
*/
|
||||||
public static final int DEFAULT_MAX_EXPANDED_INDEX = 512;
|
public static final int DEFAULT_MAX_EXPANDED_INDEX = 512;
|
||||||
|
|
||||||
@@ -71,8 +71,8 @@ public final class CompiledNode<V> {
|
|||||||
/**
|
/**
|
||||||
* Dense child lookup table used when labels fit into a compact char interval.
|
* Dense child lookup table used when labels fit into a compact char interval.
|
||||||
* <p>
|
* <p>
|
||||||
* The table enables direct O(1) indexing for child lookup and is allocated
|
* The table enables direct O(1) indexing for child lookup and is allocated only
|
||||||
* only when the character span of this node's edges is within the configured
|
* when the character span of this node's edges is within the configured
|
||||||
* threshold.
|
* threshold.
|
||||||
* </p>
|
* </p>
|
||||||
*/
|
*/
|
||||||
@@ -111,8 +111,8 @@ public final class CompiledNode<V> {
|
|||||||
*
|
*
|
||||||
* @param maxExpandedIndex upper bound for the dense lookup interval size; zero
|
* @param maxExpandedIndex upper bound for the dense lookup interval size; zero
|
||||||
* disables dense lookup. Larger values improve
|
* disables dense lookup. Larger values improve
|
||||||
* direct-index likelihood while increasing dense
|
* direct-index likelihood while increasing dense table
|
||||||
* table memory in compact-label nodes.
|
* memory in compact-label nodes.
|
||||||
* @throws NullPointerException if any array argument is {@code null}
|
* @throws NullPointerException if any array argument is {@code null}
|
||||||
* @throws IllegalArgumentException if the edge-related arrays or value-related
|
* @throws IllegalArgumentException if the edge-related arrays or value-related
|
||||||
* arrays do not have matching lengths or the
|
* arrays do not have matching lengths or the
|
||||||
@@ -288,7 +288,8 @@ public final class CompiledNode<V> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Returns a small memory-related metric describing this node's dense table size.
|
* Returns a small memory-related metric describing this node's dense table
|
||||||
|
* size.
|
||||||
*
|
*
|
||||||
* @return number of dense table slots, or {@code 0} when dense lookup is not
|
* @return number of dense table slots, or {@code 0} when dense lookup is not
|
||||||
* enabled
|
* enabled
|
||||||
@@ -328,8 +329,9 @@ public final class CompiledNode<V> {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
return Arrays.equals(this.edgeLabels, other.edgeLabels) && Arrays.equals(this.children, other.children)
|
return Arrays.equals(this.edgeLabels, other.edgeLabels) && Arrays.equals(this.children, other.children)
|
||||||
&& Arrays.equals(this.orderedValues, other.orderedValues) && Arrays.equals(this.orderedCounts, other.orderedCounts)
|
&& Arrays.equals(this.orderedValues, other.orderedValues)
|
||||||
&& this.denseEdgeMin == other.denseEdgeMin && Arrays.equals(this.denseChildren, other.denseChildren);
|
&& Arrays.equals(this.orderedCounts, other.orderedCounts) && this.denseEdgeMin == other.denseEdgeMin
|
||||||
|
&& Arrays.equals(this.denseChildren, other.denseChildren);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -339,9 +341,8 @@ public final class CompiledNode<V> {
|
|||||||
*/
|
*/
|
||||||
@Override
|
@Override
|
||||||
public String toString() {
|
public String toString() {
|
||||||
return "CompiledNode{"
|
return "CompiledNode{" + "edgeCount=" + this.edgeLabels.length + ", orderedValueCount="
|
||||||
+ "edgeCount=" + this.edgeLabels.length + ", orderedValueCount=" + this.orderedValues.length
|
+ this.orderedValues.length + ", denseTableLength=" + denseTableLength() + '}';
|
||||||
+ ", denseTableLength=" + denseTableLength() + '}';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -350,8 +351,8 @@ public final class CompiledNode<V> {
|
|||||||
* Lookup order is:
|
* Lookup order is:
|
||||||
* <ol>
|
* <ol>
|
||||||
* <li>dense array index (if the label interval is compact enough),</li>
|
* <li>dense array index (if the label interval is compact enough),</li>
|
||||||
* <li>small-child linear scan when the fallback node has {@value #LINEAR_CHILD_COUNT_THRESHOLD}
|
* <li>small-child linear scan when the fallback node has
|
||||||
* or fewer edges,</li>
|
* {@value #LINEAR_CHILD_COUNT_THRESHOLD} or fewer edges,</li>
|
||||||
* <li>binary search over sorted labels.</li>
|
* <li>binary search over sorted labels.</li>
|
||||||
* </ol>
|
* </ol>
|
||||||
* </p>
|
* </p>
|
||||||
|
|||||||
@@ -45,6 +45,7 @@ import java.io.ByteArrayOutputStream;
|
|||||||
import java.io.DataInputStream;
|
import java.io.DataInputStream;
|
||||||
import java.io.DataOutputStream;
|
import java.io.DataOutputStream;
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import org.junit.jupiter.api.DisplayName;
|
import org.junit.jupiter.api.DisplayName;
|
||||||
@@ -63,6 +64,7 @@ import org.junit.jupiter.api.Test;
|
|||||||
@Tag("unit")
|
@Tag("unit")
|
||||||
@Tag("trie")
|
@Tag("trie")
|
||||||
@Tag("frequency-trie")
|
@Tag("frequency-trie")
|
||||||
|
@Tag("lookup")
|
||||||
@DisplayName("FrequencyTrie")
|
@DisplayName("FrequencyTrie")
|
||||||
class FrequencyTrieTest {
|
class FrequencyTrieTest {
|
||||||
|
|
||||||
@@ -398,6 +400,149 @@ class FrequencyTrieTest {
|
|||||||
() -> assertThrows(UnsupportedOperationException.class, () -> entries.add(new ValueCount<String>("z", 1))));
|
() -> assertThrows(UnsupportedOperationException.class, () -> entries.add(new ValueCount<String>("z", 1))));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies that visitor lookup returns the same deterministic order and counts
|
||||||
|
* as the allocating APIs.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("Visitor lookup matches getAll order and getEntries counts")
|
||||||
|
void visitorLookupMatchesGetAllOrderAndGetEntriesCounts() {
|
||||||
|
final FrequencyTrie.Builder<String> builder = rankedBuilder();
|
||||||
|
builder.put("house", "noun", 3);
|
||||||
|
builder.put("house", "verb", 2);
|
||||||
|
builder.put("house", "adjective", 1);
|
||||||
|
final FrequencyTrie<String> trie = builder.build();
|
||||||
|
final List<String> values = new ArrayList<>();
|
||||||
|
final List<Integer> counts = new ArrayList<>();
|
||||||
|
final List<Integer> ranks = new ArrayList<>();
|
||||||
|
|
||||||
|
final int visited = trie.getAllNormalized("house", (value, count, rank) -> {
|
||||||
|
values.add(value);
|
||||||
|
counts.add(count);
|
||||||
|
ranks.add(rank);
|
||||||
|
return true;
|
||||||
|
}, 10);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(3, visited),
|
||||||
|
() -> assertEquals(List.of("noun", "verb", "adjective"), values),
|
||||||
|
() -> assertEquals(List.of(3, 2, 1), counts),
|
||||||
|
() -> assertEquals(List.of(0, 1, 2), ranks));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies visitor maximum result and early-stop behavior.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("Visitor lookup honors maxResults and sink early stop")
|
||||||
|
void visitorLookupHonorsMaxResultsAndSinkEarlyStop() {
|
||||||
|
final FrequencyTrie.Builder<String> builder = rankedBuilder();
|
||||||
|
builder.put("house", "noun", 3);
|
||||||
|
builder.put("house", "verb", 2);
|
||||||
|
builder.put("house", "adjective", 1);
|
||||||
|
final FrequencyTrie<String> trie = builder.build();
|
||||||
|
final List<String> limited = new ArrayList<>();
|
||||||
|
final List<String> stopped = new ArrayList<>();
|
||||||
|
|
||||||
|
final int limitedCount = trie.getAllNormalized("house", (value, count, rank) -> {
|
||||||
|
limited.add(value);
|
||||||
|
return true;
|
||||||
|
}, 2);
|
||||||
|
final int stoppedCount = trie.getAllNormalized("house", (value, count, rank) -> {
|
||||||
|
stopped.add(value);
|
||||||
|
return false;
|
||||||
|
}, 10);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(2, limitedCount),
|
||||||
|
() -> assertEquals(List.of("noun", "verb"), limited),
|
||||||
|
() -> assertEquals(1, stoppedCount),
|
||||||
|
() -> assertEquals(List.of("noun"), stopped));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies visitor zero, negative, missing, and first-result behavior.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("Visitor lookup handles zero, negative, missing, and first-result cases")
|
||||||
|
void visitorLookupHandlesBoundaryCases() {
|
||||||
|
final FrequencyTrie.Builder<String> builder = rankedBuilder();
|
||||||
|
builder.put("house", "noun");
|
||||||
|
final FrequencyTrie<String> trie = builder.build();
|
||||||
|
final int[] calls = new int[1];
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(0, trie.getAllNormalized("house", (value, count, rank) -> {
|
||||||
|
calls[0]++;
|
||||||
|
return true;
|
||||||
|
}, 0)),
|
||||||
|
() -> assertEquals(0, calls[0]),
|
||||||
|
() -> assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> trie.getAllNormalized("house", (value, count, rank) -> true, -1)),
|
||||||
|
() -> assertEquals(0, trie.getAllNormalized("missing", (value, count, rank) -> true, 10)),
|
||||||
|
() -> assertFalse(trie.getFirstNormalized("missing", (value, count, rank) -> true)),
|
||||||
|
() -> assertTrue(trie.getFirstNormalized("house", (value, count, rank) -> {
|
||||||
|
assertEquals("noun", value);
|
||||||
|
assertEquals(1, count);
|
||||||
|
assertEquals(0, rank);
|
||||||
|
return true;
|
||||||
|
})));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies visitor API argument validation.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("Visitor lookup rejects null and invalid range arguments")
|
||||||
|
void visitorLookupRejectsNullAndInvalidRangeArguments() {
|
||||||
|
final FrequencyTrie<String> trie = rankedBuilder().build();
|
||||||
|
final char[] key = "house".toCharArray();
|
||||||
|
final FrequencyTrie.EntrySink<String> sink = (value, count, rank) -> true;
|
||||||
|
|
||||||
|
assertAll(() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> trie.getAllNormalized((char[]) null, 0, 0, sink, 1)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> trie.getAllNormalized(key, 0, key.length, null, 1)),
|
||||||
|
() -> assertThrows(IndexOutOfBoundsException.class,
|
||||||
|
() -> trie.getAllNormalized(key, -1, key.length, sink, 1)),
|
||||||
|
() -> assertThrows(IndexOutOfBoundsException.class,
|
||||||
|
() -> trie.getAllNormalized(key, 1, key.length, sink, 1)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> trie.getAllNormalized((CharSequence) null, sink, 1)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> trie.getAllNormalized("house", null, 1)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> trie.getAll((CharSequence) null, sink, 1)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> trie.getAll("house", null, 1)),
|
||||||
|
() -> assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> trie.getAll("house", sink, -1)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies normalized char-array slices and metadata-aware CharSequence visitor
|
||||||
|
* lookup.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("Visitor lookup supports normalized char slices and metadata-aware CharSequence keys")
|
||||||
|
void visitorLookupSupportsCharSlicesAndMetadataAwareKeys() {
|
||||||
|
final FrequencyTrie.Builder<String> builder = new FrequencyTrie.Builder<>(String[]::new,
|
||||||
|
ReductionSettings.withDefaults(ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS),
|
||||||
|
WordTraversalDirection.BACKWARD, CaseProcessingMode.LOWERCASE_WITH_LOCALE_ROOT);
|
||||||
|
builder.put("house", "noun");
|
||||||
|
final FrequencyTrie<String> trie = builder.build();
|
||||||
|
final char[] padded = "__house__".toCharArray();
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(1,
|
||||||
|
trie.getAllNormalized(padded, 2, 5, (value, count, rank) -> {
|
||||||
|
assertEquals("noun", value);
|
||||||
|
return true;
|
||||||
|
}, 10)),
|
||||||
|
() -> assertFalse(trie.getFirstNormalized("HOUSE", (value, count, rank) -> true),
|
||||||
|
"Normalized lookup must bypass metadata lowercasing."),
|
||||||
|
() -> assertTrue(trie.getFirst("HOUSE", (value, count, rank) -> {
|
||||||
|
assertEquals("noun", value);
|
||||||
|
return true;
|
||||||
|
})));
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Verifies that equal frequencies prefer the shorter string representation.
|
* Verifies that equal frequencies prefer the shorter string representation.
|
||||||
*/
|
*/
|
||||||
|
|||||||
@@ -31,6 +31,7 @@
|
|||||||
package org.egothor.stemmer;
|
package org.egothor.stemmer;
|
||||||
|
|
||||||
import static org.junit.jupiter.api.Assertions.assertAll;
|
import static org.junit.jupiter.api.Assertions.assertAll;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
|
||||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||||
@@ -68,6 +69,8 @@ import org.junit.jupiter.params.provider.MethodSource;
|
|||||||
@Tag("unit")
|
@Tag("unit")
|
||||||
@Tag("stemmer")
|
@Tag("stemmer")
|
||||||
@Tag("patch")
|
@Tag("patch")
|
||||||
|
@Tag("encoding")
|
||||||
|
@Tag("apply")
|
||||||
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
|
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
|
||||||
class PatchCommandEncoderTest {
|
class PatchCommandEncoderTest {
|
||||||
|
|
||||||
@@ -147,6 +150,63 @@ class PatchCommandEncoderTest {
|
|||||||
Arguments.of(10, "teacher", PatchCommandEncoder.NOOP_PATCH, "teacher"));
|
Arguments.of(10, "teacher", PatchCommandEncoder.NOOP_PATCH, "teacher"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Provides explicit forward-direction single-instruction patch application cases.
|
||||||
|
*
|
||||||
|
* @return test arguments
|
||||||
|
*/
|
||||||
|
private static Stream<Arguments> provideForwardSingleInstructionApplyCases() {
|
||||||
|
return Stream.of(
|
||||||
|
// 1
|
||||||
|
Arguments.of(1, "abcd", "Db", "cd"),
|
||||||
|
// 2
|
||||||
|
Arguments.of(2, "abc", "Ia", "aabc"),
|
||||||
|
// 3
|
||||||
|
Arguments.of(3, "abc", "Ra", "abc"),
|
||||||
|
// 4
|
||||||
|
Arguments.of(4, "abc", "-a", "abc"),
|
||||||
|
// 5
|
||||||
|
Arguments.of(5, "abc", PatchCommandEncoder.NOOP_PATCH, "abc"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Provides forward-direction applyTo cases that exercise preserve-only and
|
||||||
|
* non-preserve branches.
|
||||||
|
*
|
||||||
|
* @return test arguments
|
||||||
|
*/
|
||||||
|
private static Stream<Arguments> provideForwardApplyToCases() {
|
||||||
|
return Stream.of(
|
||||||
|
// 1
|
||||||
|
Arguments.of(1, "book", "-aRa", "baok"),
|
||||||
|
// 2
|
||||||
|
Arguments.of(2, "abc", "-dRa", "abc"),
|
||||||
|
// 3
|
||||||
|
Arguments.of(3, "abc", "DdRa", "abc"),
|
||||||
|
// 4
|
||||||
|
Arguments.of(4, "abc", "-dIa", "abc"),
|
||||||
|
// 5
|
||||||
|
Arguments.of(5, "abc", "Na-a", "abc"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Provides empty-source forward applyTo cases that cover empty-source
|
||||||
|
* instruction handling.
|
||||||
|
*
|
||||||
|
* @return test arguments
|
||||||
|
*/
|
||||||
|
private static Stream<Arguments> provideForwardEmptySourceApplyCases() {
|
||||||
|
return Stream.of(
|
||||||
|
// 1
|
||||||
|
Arguments.of(1, "IaIb", "ab"),
|
||||||
|
// 2
|
||||||
|
Arguments.of(2, "-aRa", ""),
|
||||||
|
// 3
|
||||||
|
Arguments.of(3, "IaRa", ""),
|
||||||
|
// 4
|
||||||
|
Arguments.of(4, "Na-a", ""));
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Provides malformed or index-invalid patch inputs that must preserve the
|
* Provides malformed or index-invalid patch inputs that must preserve the
|
||||||
* original source according to the implementation contract.
|
* original source according to the implementation contract.
|
||||||
@@ -236,12 +296,31 @@ class PatchCommandEncoderTest {
|
|||||||
return new StringBuilder(text).reverse().toString();
|
return new StringBuilder(text).reverse().toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a patch into a right-sized output buffer and returns the produced
|
||||||
|
* string.
|
||||||
|
*
|
||||||
|
* @param source source text
|
||||||
|
* @param patch patch command
|
||||||
|
* @param traversalDirection traversal direction
|
||||||
|
* @return transformed text
|
||||||
|
*/
|
||||||
|
private static String applyToString(final String source, final String patch,
|
||||||
|
final WordTraversalDirection traversalDirection) {
|
||||||
|
final char[] output = new char[Math.max(source.length() + 16, 16)];
|
||||||
|
final int produced = PatchCommandEncoder.applyTo(source, patch, traversalDirection, output, 0, output.length);
|
||||||
|
return new String(output, 0, produced);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Tests constructor validation and basic instantiation behavior.
|
* Tests constructor validation and basic instantiation behavior.
|
||||||
*/
|
*/
|
||||||
@Nested
|
@Nested
|
||||||
@DisplayName("construction")
|
@DisplayName("construction")
|
||||||
@Tag("construction")
|
@Tag("construction")
|
||||||
|
@Tag("unit")
|
||||||
|
@Tag("stemmer")
|
||||||
|
@Tag("patch")
|
||||||
class ConstructionTests {
|
class ConstructionTests {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -327,6 +406,9 @@ class PatchCommandEncoderTest {
|
|||||||
@Nested
|
@Nested
|
||||||
@DisplayName("encode(String, String)")
|
@DisplayName("encode(String, String)")
|
||||||
@Tag("encoding")
|
@Tag("encoding")
|
||||||
|
@Tag("unit")
|
||||||
|
@Tag("stemmer")
|
||||||
|
@Tag("patch")
|
||||||
class EncodeTests {
|
class EncodeTests {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -461,6 +543,9 @@ class PatchCommandEncoderTest {
|
|||||||
@Nested
|
@Nested
|
||||||
@DisplayName("apply(String, String)")
|
@DisplayName("apply(String, String)")
|
||||||
@Tag("apply")
|
@Tag("apply")
|
||||||
|
@Tag("unit")
|
||||||
|
@Tag("stemmer")
|
||||||
|
@Tag("patch")
|
||||||
class ApplyTests {
|
class ApplyTests {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -535,6 +620,101 @@ class PatchCommandEncoderTest {
|
|||||||
assertEquals("city", PatchCommandEncoder.apply("cities", patch, WordTraversalDirection.FORWARD));
|
assertEquals("city", PatchCommandEncoder.apply("cities", patch, WordTraversalDirection.FORWARD));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies explicit single-instruction forward patch application
|
||||||
|
* semantics.
|
||||||
|
*
|
||||||
|
* @param caseId numeric case identifier
|
||||||
|
* @param source source word
|
||||||
|
* @param patch encoded patch command
|
||||||
|
* @param expected expected transformed word
|
||||||
|
*/
|
||||||
|
@ParameterizedTest(name = "[{index}] case {0}: forward single instruction apply({1}, {2}) -> {3}")
|
||||||
|
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideForwardSingleInstructionApplyCases")
|
||||||
|
@DisplayName("applies forward single instruction patches correctly")
|
||||||
|
void shouldApplyForwardSingleInstructionsExplicitly(int caseId, String source, String patch, String expected) {
|
||||||
|
assertEquals(expected, PatchCommandEncoder.apply(source, patch, WordTraversalDirection.FORWARD),
|
||||||
|
() -> "Case " + caseId + " failed for source='" + source + "', patch='" + patch + "'.");
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies forward single-instruction malformed commands fail fast.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("throws for unsupported forward opcode and NOOP argument")
|
||||||
|
void shouldThrowForUnsupportedForwardOpcodeAndNoopArgument() {
|
||||||
|
assertAll(() -> assertEquals("Unsupported patch opcode: X",
|
||||||
|
assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> PatchCommandEncoder.apply("abc", "Xa", WordTraversalDirection.FORWARD))
|
||||||
|
.getMessage()),
|
||||||
|
() -> assertEquals("Unsupported NOOP patch argument: `",
|
||||||
|
assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> PatchCommandEncoder.apply("abc", "N`", WordTraversalDirection.FORWARD))
|
||||||
|
.getMessage()));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies explicit forward-applyTo cases that exercise preserve-only and
|
||||||
|
* non-preserve branches.
|
||||||
|
*
|
||||||
|
* @param caseId numeric case identifier
|
||||||
|
* @param source source word
|
||||||
|
* @param patch encoded patch command
|
||||||
|
* @param expected expected transformed word
|
||||||
|
*/
|
||||||
|
@ParameterizedTest(name = "[{index}] case {0}: applyToForward({1}, {2}) -> {3}")
|
||||||
|
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideForwardApplyToCases")
|
||||||
|
@DisplayName("applyTo handles forward preserve-only and mutation branches")
|
||||||
|
void shouldApplyToForwardPreserveAndMutationBranches(int caseId, String source, String patch, String expected) {
|
||||||
|
final char[] output = new char[Math.max(source.length() + 16, 16)];
|
||||||
|
|
||||||
|
final int produced = PatchCommandEncoder.applyTo(source, patch, WordTraversalDirection.FORWARD, output, 0,
|
||||||
|
output.length);
|
||||||
|
|
||||||
|
assertAll(
|
||||||
|
() -> assertEquals(expected.length(), produced,
|
||||||
|
() -> "Case " + caseId + " produced wrong length."),
|
||||||
|
() -> assertEquals(expected, new String(output, 0, produced),
|
||||||
|
() -> "Case " + caseId + " failed for patch='" + patch + "'."));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies empty-source forward applyTo behavior for insert-only and malformed
|
||||||
|
* instructions.
|
||||||
|
*
|
||||||
|
* @param caseId numeric case identifier
|
||||||
|
* @param patch encoded patch command
|
||||||
|
* @param expected expected transformed word
|
||||||
|
*/
|
||||||
|
@ParameterizedTest(name = "[{index}] case {0}: applyToForward(\"\", {1}) -> \"{2}\"")
|
||||||
|
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideForwardEmptySourceApplyCases")
|
||||||
|
@DisplayName("applies forward empty-source patches correctly")
|
||||||
|
void shouldApplyToForwardEmptySourceCases(int caseId, String patch, String expected) {
|
||||||
|
final char[] output = new char[Math.max(expected.length() + 16, 16)];
|
||||||
|
|
||||||
|
final int produced = PatchCommandEncoder.applyTo("", patch, WordTraversalDirection.FORWARD, output, 0,
|
||||||
|
output.length);
|
||||||
|
|
||||||
|
assertAll(
|
||||||
|
() -> assertEquals(expected.length(), produced,
|
||||||
|
() -> "Case " + caseId + " produced wrong length."),
|
||||||
|
() -> assertEquals(expected, new String(output, 0, produced),
|
||||||
|
() -> "Case " + caseId + " failed for patch='" + patch + "'."));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies malformed empty-source forward patches fail fast and preserve
|
||||||
|
* empty-source semantics.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("throws for unsupported NOOP argument on empty-source forward patch")
|
||||||
|
void shouldThrowForUnsupportedNoopArgumentOnForwardEmptySource() {
|
||||||
|
IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> PatchCommandEncoder.apply("", "N`Ra", WordTraversalDirection.FORWARD));
|
||||||
|
|
||||||
|
assertEquals("Unsupported NOOP patch argument: `", exception.getMessage());
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Verifies explicit patch application cases.
|
* Verifies explicit patch application cases.
|
||||||
*
|
*
|
||||||
@@ -590,6 +770,181 @@ class PatchCommandEncoderTest {
|
|||||||
assertEquals(source, PatchCommandEncoder.apply(source, malformedPatch), () -> "Case " + caseId
|
assertEquals(source, PatchCommandEncoder.apply(source, malformedPatch), () -> "Case " + caseId
|
||||||
+ " failed for source='" + source + "', malformedPatch='" + malformedPatch + "'.");
|
+ " failed for source='" + source + "', malformedPatch='" + malformedPatch + "'.");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies buffer application against string-returning application.
|
||||||
|
*
|
||||||
|
* @param caseId numeric case identifier
|
||||||
|
* @param source source word
|
||||||
|
* @param patch patch command
|
||||||
|
* @param expected expected transformed word
|
||||||
|
*/
|
||||||
|
@ParameterizedTest(name = "[{index}] case {0}: applyTo({1}, {2}) -> {3}")
|
||||||
|
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideApplyCases")
|
||||||
|
@DisplayName("applyTo matches apply for explicit backward patch commands")
|
||||||
|
void shouldApplyToBufferLikeApplyForBackwardCommands(int caseId, String source, String patch, String expected) {
|
||||||
|
final char[] output = "___..............".toCharArray();
|
||||||
|
|
||||||
|
final int produced = PatchCommandEncoder.applyTo(source, patch, WordTraversalDirection.BACKWARD, output, 3,
|
||||||
|
output.length - 3);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(expected.length(), produced, () -> "Case " + caseId + " produced wrong length."),
|
||||||
|
() -> assertEquals(expected, new String(output, 3, produced)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies char-array source slices.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo supports char-array source slices")
|
||||||
|
void shouldApplyToCharArraySourceSlice() {
|
||||||
|
final char[] source = "__teacher__".toCharArray();
|
||||||
|
final char[] output = new char[16];
|
||||||
|
|
||||||
|
final int produced = PatchCommandEncoder.applyTo(source, 2, 7, "Db", WordTraversalDirection.BACKWARD,
|
||||||
|
output, 1, output.length - 1);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(5, produced), () -> assertEquals("teach", new String(output, 1, produced)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies null and range validation for buffer application.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo rejects null and invalid range arguments")
|
||||||
|
void shouldRejectNullAndInvalidRangeArguments() {
|
||||||
|
final char[] source = "teacher".toCharArray();
|
||||||
|
final char[] output = new char[16];
|
||||||
|
|
||||||
|
assertAll(() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo((CharSequence) null, "Db", WordTraversalDirection.BACKWARD,
|
||||||
|
output, 0, output.length)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo("teacher", "Db", null, output, 0, output.length)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo("teacher", "Db", WordTraversalDirection.BACKWARD, null,
|
||||||
|
0, output.length)),
|
||||||
|
() -> assertThrows(IndexOutOfBoundsException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo("teacher", "Db", WordTraversalDirection.BACKWARD,
|
||||||
|
output, -1, output.length)),
|
||||||
|
() -> assertThrows(NullPointerException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo((char[]) null, 0, 7, "Db",
|
||||||
|
WordTraversalDirection.BACKWARD, output, 0, output.length)),
|
||||||
|
() -> assertThrows(IndexOutOfBoundsException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo(source, 1, source.length, "Db",
|
||||||
|
WordTraversalDirection.BACKWARD, output, 0, output.length)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies null, empty, and canonical NOOP patch preservation.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo preserves source for null, empty, and canonical NOOP patches")
|
||||||
|
void shouldApplyToPreserveSourceForEmptyCompatibilityPatches() {
|
||||||
|
final char[] nullPatchOutput = new char[8];
|
||||||
|
final char[] emptyPatchOutput = new char[8];
|
||||||
|
final char[] noopOutput = new char[8];
|
||||||
|
|
||||||
|
final int nullPatchLength = PatchCommandEncoder.applyTo("teacher", null, WordTraversalDirection.BACKWARD,
|
||||||
|
nullPatchOutput, 0, nullPatchOutput.length);
|
||||||
|
final int emptyPatchLength = PatchCommandEncoder.applyTo("teacher", "", WordTraversalDirection.BACKWARD,
|
||||||
|
emptyPatchOutput, 0, emptyPatchOutput.length);
|
||||||
|
final int noopLength = PatchCommandEncoder.applyTo("teacher", PatchCommandEncoder.NOOP_PATCH,
|
||||||
|
WordTraversalDirection.BACKWARD, noopOutput, 0, noopOutput.length);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(7, nullPatchLength),
|
||||||
|
() -> assertEquals("teacher", new String(nullPatchOutput, 0, nullPatchLength)),
|
||||||
|
() -> assertEquals(7, emptyPatchLength),
|
||||||
|
() -> assertEquals("teacher", new String(emptyPatchOutput, 0, emptyPatchLength)),
|
||||||
|
() -> assertEquals(7, noopLength),
|
||||||
|
() -> assertEquals("teacher", new String(noopOutput, 0, noopLength)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies insufficient capacity behavior.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo reports insufficient capacity without writing output")
|
||||||
|
void shouldReportInsufficientCapacityWithoutWritingOutput() {
|
||||||
|
final char[] output = "xxxx".toCharArray();
|
||||||
|
|
||||||
|
final int produced = PatchCommandEncoder.applyTo("abc", "Ic", WordTraversalDirection.BACKWARD, output, 0,
|
||||||
|
output.length - 1);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(PatchCommandEncoder.APPLY_INSUFFICIENT_CAPACITY, produced),
|
||||||
|
() -> assertArrayEquals("xxxx".toCharArray(), output));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies exception parity with string-returning application.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo throws for unsupported opcode and NOOP argument")
|
||||||
|
void shouldApplyToThrowForUnsupportedOpcodeAndNoopArgument() {
|
||||||
|
final char[] output = new char[8];
|
||||||
|
|
||||||
|
assertAll(() -> {
|
||||||
|
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo("abc", "Xa", WordTraversalDirection.BACKWARD, output, 0,
|
||||||
|
output.length));
|
||||||
|
assertEquals("Unsupported patch opcode: X", exception.getMessage());
|
||||||
|
}, () -> {
|
||||||
|
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo("abc", "Nb", WordTraversalDirection.BACKWARD, output, 0,
|
||||||
|
output.length));
|
||||||
|
assertEquals("Unsupported NOOP patch argument: b", exception.getMessage());
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies malformed compatibility behavior for buffer application.
|
||||||
|
*
|
||||||
|
* @param caseId numeric case identifier
|
||||||
|
* @param source original source
|
||||||
|
* @param malformedPatch malformed patch
|
||||||
|
*/
|
||||||
|
@ParameterizedTest(name = "[{index}] case {0}: malformed applyTo patch {2} preserves {1}")
|
||||||
|
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideMalformedPatchCases")
|
||||||
|
@DisplayName("applyTo preserves source for malformed or index-invalid patch commands")
|
||||||
|
void shouldApplyToPreserveSourceForMalformedOrIndexInvalidPatchCommands(int caseId, String source,
|
||||||
|
String malformedPatch) {
|
||||||
|
final char[] output = new char[Math.max(source.length(), 1)];
|
||||||
|
|
||||||
|
final int produced = PatchCommandEncoder.applyTo(source, malformedPatch, WordTraversalDirection.BACKWARD,
|
||||||
|
output, 0, output.length);
|
||||||
|
|
||||||
|
assertAll(() -> assertEquals(source.length(), produced, () -> "Case " + caseId + " produced wrong length."),
|
||||||
|
() -> assertEquals(source, new String(output, 0, produced)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies explicit traversal direction for buffer application.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo follows explicit forward traversal direction")
|
||||||
|
void shouldApplyToWithForwardTraversalDirection() {
|
||||||
|
final PatchCommandEncoder encoder = PatchCommandEncoder.builder()
|
||||||
|
.traversalDirection(WordTraversalDirection.FORWARD)
|
||||||
|
.build();
|
||||||
|
final String patch = encoder.encode("cities", "city");
|
||||||
|
|
||||||
|
assertEquals(PatchCommandEncoder.apply("cities", patch, WordTraversalDirection.FORWARD),
|
||||||
|
applyToString("cities", patch, WordTraversalDirection.FORWARD));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Verifies overlapping source/output slices are rejected.
|
||||||
|
*/
|
||||||
|
@Test
|
||||||
|
@DisplayName("applyTo rejects overlapping char-array source and output ranges")
|
||||||
|
void shouldRejectOverlappingSourceAndOutputRanges() {
|
||||||
|
final char[] buffer = "teacher....".toCharArray();
|
||||||
|
|
||||||
|
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
||||||
|
() -> PatchCommandEncoder.applyTo(buffer, 0, 7, "Db", WordTraversalDirection.BACKWARD, buffer, 2, 5));
|
||||||
|
|
||||||
|
assertEquals("source and output ranges must not overlap.", exception.getMessage());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -598,6 +953,9 @@ class PatchCommandEncoderTest {
|
|||||||
@Nested
|
@Nested
|
||||||
@DisplayName("stemming-oriented scenarios")
|
@DisplayName("stemming-oriented scenarios")
|
||||||
@Tag("regression")
|
@Tag("regression")
|
||||||
|
@Tag("unit")
|
||||||
|
@Tag("stemmer")
|
||||||
|
@Tag("patch")
|
||||||
class StemmingScenarioTests {
|
class StemmingScenarioTests {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -659,6 +1017,9 @@ class PatchCommandEncoderTest {
|
|||||||
@Nested
|
@Nested
|
||||||
@DisplayName("reversed-word processing")
|
@DisplayName("reversed-word processing")
|
||||||
@Tag("normalization")
|
@Tag("normalization")
|
||||||
|
@Tag("unit")
|
||||||
|
@Tag("stemmer")
|
||||||
|
@Tag("patch")
|
||||||
class ReversedWordProcessingTests {
|
class ReversedWordProcessingTests {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -743,6 +1104,7 @@ class PatchCommandEncoderTest {
|
|||||||
*/
|
*/
|
||||||
@ParameterizedTest(name = "[{index}] case {0}: mirrored consistency for {1} -> {2}")
|
@ParameterizedTest(name = "[{index}] case {0}: mirrored consistency for {1} -> {2}")
|
||||||
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideReversedRoundTripPairs")
|
@MethodSource("org.egothor.stemmer.PatchCommandEncoderTest#provideReversedRoundTripPairs")
|
||||||
|
@Tag("normalization")
|
||||||
@DisplayName("preserves correctness under mirrored input orientation")
|
@DisplayName("preserves correctness under mirrored input orientation")
|
||||||
void shouldPreserveCorrectnessUnderMirroredInputOrientation(int caseId, String source, String target) {
|
void shouldPreserveCorrectnessUnderMirroredInputOrientation(int caseId, String source, String target) {
|
||||||
PatchCommandEncoder encoder = PatchCommandEncoder.builder().build();
|
PatchCommandEncoder encoder = PatchCommandEncoder.builder().build();
|
||||||
|
|||||||
@@ -46,9 +46,39 @@ import org.junit.jupiter.api.Test;
|
|||||||
*/
|
*/
|
||||||
@Tag("unit")
|
@Tag("unit")
|
||||||
@Tag("trie")
|
@Tag("trie")
|
||||||
|
@Tag("lookup")
|
||||||
@DisplayName("CompiledNode and NodeData")
|
@DisplayName("CompiledNode and NodeData")
|
||||||
class CompiledNodeAndNodeDataTest {
|
class CompiledNodeAndNodeDataTest {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Creates a typed child array for compiled-node tests.
|
||||||
|
*
|
||||||
|
* @param length requested array length
|
||||||
|
* @return typed child array
|
||||||
|
*/
|
||||||
|
@SuppressWarnings("unchecked")
|
||||||
|
private static CompiledNode<String>[] children(final int length) {
|
||||||
|
return new CompiledNode[length];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Creates an empty child array for leaf compiled-node tests.
|
||||||
|
*
|
||||||
|
* @return empty typed child array
|
||||||
|
*/
|
||||||
|
private static CompiledNode<String>[] noChildren() {
|
||||||
|
return children(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Creates a leaf node used as a child in lookup tests.
|
||||||
|
*
|
||||||
|
* @return leaf node
|
||||||
|
*/
|
||||||
|
private static CompiledNode<String> leaf() {
|
||||||
|
return new CompiledNode<>(new char[0], noChildren(), new String[0], new int[0]);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Verifies that {@link NodeData} rejects mismatched edge-related array lengths.
|
* Verifies that {@link NodeData} rejects mismatched edge-related array lengths.
|
||||||
*/
|
*/
|
||||||
@@ -99,8 +129,7 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode rejects mismatched edge and child arrays")
|
@DisplayName("CompiledNode rejects mismatched edge and child arrays")
|
||||||
void compiledNodeShouldRejectMismatchedEdgeAndChildArrays() {
|
void compiledNodeShouldRejectMismatchedEdgeAndChildArrays() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] children = noChildren();
|
||||||
final CompiledNode<String>[] children = new CompiledNode[0];
|
|
||||||
|
|
||||||
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
||||||
() -> new CompiledNode<String>(new char[] { 'a' }, children, new String[0], new int[0]));
|
() -> new CompiledNode<String>(new char[] { 'a' }, children, new String[0], new int[0]));
|
||||||
@@ -114,8 +143,7 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode rejects mismatched value arrays")
|
@DisplayName("CompiledNode rejects mismatched value arrays")
|
||||||
void compiledNodeShouldRejectMismatchedValueArrays() {
|
void compiledNodeShouldRejectMismatchedValueArrays() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] children = noChildren();
|
||||||
final CompiledNode<String>[] children = new CompiledNode[0];
|
|
||||||
|
|
||||||
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
final IllegalArgumentException exception = assertThrows(IllegalArgumentException.class,
|
||||||
() -> new CompiledNode<String>(new char[0], children, new String[] { "stem" }, new int[0]));
|
() -> new CompiledNode<String>(new char[0], children, new String[] { "stem" }, new int[0]));
|
||||||
@@ -131,8 +159,7 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@DisplayName("CompiledNode accessors expose documented backing arrays")
|
@DisplayName("CompiledNode accessors expose documented backing arrays")
|
||||||
void compiledNodeAccessorsShouldExposeDocumentedBackingArrays() {
|
void compiledNodeAccessorsShouldExposeDocumentedBackingArrays() {
|
||||||
final char[] edgeLabels = new char[] { 'a' };
|
final char[] edgeLabels = new char[] { 'a' };
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] children = children(1);
|
||||||
final CompiledNode<String>[] children = new CompiledNode[1];
|
|
||||||
final String[] orderedValues = new String[] { "stem" };
|
final String[] orderedValues = new String[] { "stem" };
|
||||||
final int[] orderedCounts = new int[] { 5 };
|
final int[] orderedCounts = new int[] { 5 };
|
||||||
final CompiledNode<String> node = new CompiledNode<>(edgeLabels, children, orderedValues, orderedCounts);
|
final CompiledNode<String> node = new CompiledNode<>(edgeLabels, children, orderedValues, orderedCounts);
|
||||||
@@ -149,12 +176,11 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode can resolve child via dense lookup table")
|
@DisplayName("CompiledNode can resolve child via dense lookup table")
|
||||||
void compiledNodeUsesDenseLookupForCompactIntervals() {
|
void compiledNodeUsesDenseLookupForCompactIntervals() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] children = children(4);
|
||||||
final CompiledNode<String>[] children = new CompiledNode[4];
|
children[0] = leaf();
|
||||||
children[0] = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
children[1] = leaf();
|
||||||
children[1] = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
children[2] = leaf();
|
||||||
children[2] = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
children[3] = leaf();
|
||||||
children[3] = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
|
||||||
|
|
||||||
final CompiledNode<String> node = new CompiledNode<>(new char[] { 'a', 'b', 'c', 'd' }, children,
|
final CompiledNode<String> node = new CompiledNode<>(new char[] { 'a', 'b', 'c', 'd' }, children,
|
||||||
new String[] { "1", "2", "3", "4" }, new int[] { 1, 1, 1, 1 });
|
new String[] { "1", "2", "3", "4" }, new int[] { 1, 1, 1, 1 });
|
||||||
@@ -172,12 +198,11 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode resolves child by linear scan for small degree")
|
@DisplayName("CompiledNode resolves child by linear scan for small degree")
|
||||||
void compiledNodeUsesLinearScanForSmallDegree() {
|
void compiledNodeUsesLinearScanForSmallDegree() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] children = children(4);
|
||||||
final CompiledNode<String>[] children = new CompiledNode[4];
|
final CompiledNode<String> childA = leaf();
|
||||||
final CompiledNode<String> childA = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childB = leaf();
|
||||||
final CompiledNode<String> childB = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childC = leaf();
|
||||||
final CompiledNode<String> childC = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childD = leaf();
|
||||||
final CompiledNode<String> childD = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
|
||||||
children[0] = childA;
|
children[0] = childA;
|
||||||
children[1] = childB;
|
children[1] = childB;
|
||||||
children[2] = childC;
|
children[2] = childC;
|
||||||
@@ -200,13 +225,12 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode resolves child by binary search for large degree")
|
@DisplayName("CompiledNode resolves child by binary search for large degree")
|
||||||
void compiledNodeUsesBinarySearchForLargeDegree() {
|
void compiledNodeUsesBinarySearchForLargeDegree() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] children = children(5);
|
||||||
final CompiledNode<String>[] children = new CompiledNode[5];
|
final CompiledNode<String> childA = leaf();
|
||||||
final CompiledNode<String> childA = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childB = leaf();
|
||||||
final CompiledNode<String> childB = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childC = leaf();
|
||||||
final CompiledNode<String> childC = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childD = leaf();
|
||||||
final CompiledNode<String> childD = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
final CompiledNode<String> childE = leaf();
|
||||||
final CompiledNode<String> childE = new CompiledNode<>(new char[0], new CompiledNode[0], new String[0], new int[0]);
|
|
||||||
children[0] = childA;
|
children[0] = childA;
|
||||||
children[1] = childB;
|
children[1] = childB;
|
||||||
children[2] = childC;
|
children[2] = childC;
|
||||||
@@ -230,8 +254,7 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode reports leaf, value and edge presence state")
|
@DisplayName("CompiledNode reports leaf, value and edge presence state")
|
||||||
void compiledNodeReportsNodeStateHelpers() {
|
void compiledNodeReportsNodeStateHelpers() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] childless = noChildren();
|
||||||
final CompiledNode<String>[] childless = new CompiledNode[0];
|
|
||||||
final CompiledNode<String> leaf = new CompiledNode<>(new char[0], childless, new String[0], new int[0]);
|
final CompiledNode<String> leaf = new CompiledNode<>(new char[0], childless, new String[0], new int[0]);
|
||||||
|
|
||||||
assertTrue(leaf.isLeaf());
|
assertTrue(leaf.isLeaf());
|
||||||
@@ -239,11 +262,10 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
assertFalse(leaf.hasValues());
|
assertFalse(leaf.hasValues());
|
||||||
assertFalse(leaf.hasEdge('a'));
|
assertFalse(leaf.hasEdge('a'));
|
||||||
|
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] child = children(1);
|
||||||
final CompiledNode<String>[] child = new CompiledNode[1];
|
|
||||||
final String[] orderedValues = new String[] { "leaf" };
|
final String[] orderedValues = new String[] { "leaf" };
|
||||||
final int[] orderedCounts = new int[] { 1 };
|
final int[] orderedCounts = new int[] { 1 };
|
||||||
child[0] = new CompiledNode<>(new char[0], new CompiledNode[0], orderedValues, orderedCounts);
|
child[0] = new CompiledNode<>(new char[0], noChildren(), orderedValues, orderedCounts);
|
||||||
final CompiledNode<String> node = new CompiledNode<>(new char[] { 'a' }, child, orderedValues, orderedCounts);
|
final CompiledNode<String> node = new CompiledNode<>(new char[] { 'a' }, child, orderedValues, orderedCounts);
|
||||||
|
|
||||||
assertFalse(node.isLeaf());
|
assertFalse(node.isLeaf());
|
||||||
@@ -260,9 +282,8 @@ class CompiledNodeAndNodeDataTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("CompiledNode equals and hashCode align for identical structure")
|
@DisplayName("CompiledNode equals and hashCode align for identical structure")
|
||||||
void compiledNodeEqualsAndHashCodeAlignForIdenticalStructure() {
|
void compiledNodeEqualsAndHashCodeAlignForIdenticalStructure() {
|
||||||
@SuppressWarnings("unchecked")
|
final CompiledNode<String>[] child = children(1);
|
||||||
final CompiledNode<String>[] child = new CompiledNode[1];
|
final CompiledNode<String> leaf = new CompiledNode<>(new char[0], noChildren(), new String[] { "v" },
|
||||||
final CompiledNode<String> leaf = new CompiledNode<>(new char[0], new CompiledNode[0], new String[] { "v" },
|
|
||||||
new int[] { 1 });
|
new int[] { 1 });
|
||||||
child[0] = leaf;
|
child[0] = leaf;
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user