Package com.google.genai.proto

Interface SentencepieceModel.NormalizerSpecOrBuilder

  • All Implemented Interfaces:
    com.google.protobuf.GeneratedMessageV3.ExtendableMessageOrBuilder , com.google.protobuf.MessageLiteOrBuilder , com.google.protobuf.MessageOrBuilder 
    
public interface SentencepieceModel.NormalizerSpecOrBuilder
 implements GeneratedMessageV3.ExtendableMessageOrBuilder<MessageT>

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description

    • Field Summary

      Fields 
      Modifier and Type Field Description

    • Constructor Summary

      Constructors 
      Constructor Description

    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description

    • Method Summary

      Modifier and Type Method Description
      abstract boolean hasName() 
      name of normalization rule.
      optional string name = 1;
      abstract String getName() 
      name of normalization rule.
      optional string name = 1;
      abstract ByteString getNameBytes() 
      name of normalization rule.
      optional string name = 1;
      abstract boolean hasPrecompiledCharsmap() 
      Pre-compiled normalization rule created by
Builder::GetPrecompiledCharsMap() or Builder::CompileCharsMap() method.
Usually this field is set by Builder::GetNormalizerSpec() method.
      optional bytes precompiled_charsmap = 2;
      abstract ByteString getPrecompiledCharsmap() 
      Pre-compiled normalization rule created by
Builder::GetPrecompiledCharsMap() or Builder::CompileCharsMap() method.
Usually this field is set by Builder::GetNormalizerSpec() method.
      optional bytes precompiled_charsmap = 2;
      abstract boolean hasAddDummyPrefix() 
      Adds dummy whitespace at the beginning of text in order to
treat "world" in "world" and "hello world" in the same way.
      optional bool add_dummy_prefix = 3 [default = true];
      abstract boolean getAddDummyPrefix() 
      Adds dummy whitespace at the beginning of text in order to
treat "world" in "world" and "hello world" in the same way.
      optional bool add_dummy_prefix = 3 [default = true];
      abstract boolean hasRemoveExtraWhitespaces() 
      Removes leading, trailing, and duplicate internal whitespace.
      optional bool remove_extra_whitespaces = 4 [default = true];
      abstract boolean getRemoveExtraWhitespaces() 
      Removes leading, trailing, and duplicate internal whitespace.
      optional bool remove_extra_whitespaces = 4 [default = true];
      abstract boolean hasEscapeWhitespaces() 
      Replaces whitespace with meta symbol.
This field must be true to train sentence piece model.
      optional bool escape_whitespaces = 5 [default = true];
      abstract boolean getEscapeWhitespaces() 
      Replaces whitespace with meta symbol.
This field must be true to train sentence piece model.
      optional bool escape_whitespaces = 5 [default = true];
      abstract boolean hasNormalizationRuleTsv() 
      Custom normalization rule file in TSV format.
https://github.com/google/sentencepiece/blob/master/doc/normalization.md
This field is only used in SentencePieceTrainer::Train() method, which
compiles the rule into the binary rule stored in `precompiled_charsmap`.
      optional string normalization_rule_tsv = 6;
      abstract String getNormalizationRuleTsv() 
      Custom normalization rule file in TSV format.
https://github.com/google/sentencepiece/blob/master/doc/normalization.md
This field is only used in SentencePieceTrainer::Train() method, which
compiles the rule into the binary rule stored in `precompiled_charsmap`.
      optional string normalization_rule_tsv = 6;
      abstract ByteString getNormalizationRuleTsvBytes() 
      Custom normalization rule file in TSV format.
https://github.com/google/sentencepiece/blob/master/doc/normalization.md
This field is only used in SentencePieceTrainer::Train() method, which
compiles the rule into the binary rule stored in `precompiled_charsmap`.
      optional string normalization_rule_tsv = 6;

      • Methods inherited from class com.google.protobuf.GeneratedMessageV3.ExtendableMessageOrBuilder

        getDefaultInstanceForType, getExtension, getExtensionCount, hasExtension

      • Methods inherited from class com.google.protobuf.MessageOrBuilder

        findInitializationErrors, getAllFields, getDescriptorForType, getField, getInitializationErrorString, getOneofFieldDescriptor, getRepeatedField, getRepeatedFieldCount, getUnknownFields, hasField, hasOneof

      • Methods inherited from class com.google.protobuf.MessageLiteOrBuilder

        getDefaultInstanceForType, isInitialized

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

    • Method Detail

      • hasName

         abstract boolean hasName()
        name of normalization rule.
        optional string name = 1;
        Returns:

        Whether the name field is set.

      • getName

         abstract String getName()
        name of normalization rule.
        optional string name = 1;
        Returns:

        The name.

      • getNameBytes

         abstract ByteString getNameBytes()
        name of normalization rule.
        optional string name = 1;
        Returns:

        The bytes for name.

      • hasPrecompiledCharsmap

         abstract boolean hasPrecompiledCharsmap()
        Pre-compiled normalization rule created by
Builder::GetPrecompiledCharsMap() or Builder::CompileCharsMap() method.
Usually this field is set by Builder::GetNormalizerSpec() method.
        optional bytes precompiled_charsmap = 2;
        Returns:

        Whether the precompiledCharsmap field is set.

      • getPrecompiledCharsmap

         abstract ByteString getPrecompiledCharsmap()
        Pre-compiled normalization rule created by
Builder::GetPrecompiledCharsMap() or Builder::CompileCharsMap() method.
Usually this field is set by Builder::GetNormalizerSpec() method.
        optional bytes precompiled_charsmap = 2;
        Returns:

        The precompiledCharsmap.

      • hasAddDummyPrefix

         abstract boolean hasAddDummyPrefix()
        Adds dummy whitespace at the beginning of text in order to
treat "world" in "world" and "hello world" in the same way.
        optional bool add_dummy_prefix = 3 [default = true];
        Returns:

        Whether the addDummyPrefix field is set.

      • getAddDummyPrefix

         abstract boolean getAddDummyPrefix()
        Adds dummy whitespace at the beginning of text in order to
treat "world" in "world" and "hello world" in the same way.
        optional bool add_dummy_prefix = 3 [default = true];
        Returns:

        The addDummyPrefix.

      • hasRemoveExtraWhitespaces

         abstract boolean hasRemoveExtraWhitespaces()
        Removes leading, trailing, and duplicate internal whitespace.
        optional bool remove_extra_whitespaces = 4 [default = true];
        Returns:

        Whether the removeExtraWhitespaces field is set.

      • getRemoveExtraWhitespaces

         abstract boolean getRemoveExtraWhitespaces()
        Removes leading, trailing, and duplicate internal whitespace.
        optional bool remove_extra_whitespaces = 4 [default = true];
        Returns:

        The removeExtraWhitespaces.

      • hasEscapeWhitespaces

         abstract boolean hasEscapeWhitespaces()
        Replaces whitespace with meta symbol.
This field must be true to train sentence piece model.
        optional bool escape_whitespaces = 5 [default = true];
        Returns:

        Whether the escapeWhitespaces field is set.

      • getEscapeWhitespaces

         abstract boolean getEscapeWhitespaces()
        Replaces whitespace with meta symbol.
This field must be true to train sentence piece model.
        optional bool escape_whitespaces = 5 [default = true];
        Returns:

        The escapeWhitespaces.

      • hasNormalizationRuleTsv

         abstract boolean hasNormalizationRuleTsv()
        Custom normalization rule file in TSV format.
https://github.com/google/sentencepiece/blob/master/doc/normalization.md
This field is only used in SentencePieceTrainer::Train() method, which
compiles the rule into the binary rule stored in `precompiled_charsmap`.
        optional string normalization_rule_tsv = 6;
        Returns:

        Whether the normalizationRuleTsv field is set.

      • getNormalizationRuleTsv

         abstract String getNormalizationRuleTsv()
        Custom normalization rule file in TSV format.
https://github.com/google/sentencepiece/blob/master/doc/normalization.md
This field is only used in SentencePieceTrainer::Train() method, which
compiles the rule into the binary rule stored in `precompiled_charsmap`.
        optional string normalization_rule_tsv = 6;
        Returns:

        The normalizationRuleTsv.

      • getNormalizationRuleTsvBytes

         abstract ByteString getNormalizationRuleTsvBytes()
        Custom normalization rule file in TSV format.
https://github.com/google/sentencepiece/blob/master/doc/normalization.md
This field is only used in SentencePieceTrainer::Train() method, which
compiles the rule into the binary rule stored in `precompiled_charsmap`.
        optional string normalization_rule_tsv = 6;
        Returns:

        The bytes for normalizationRuleTsv.