Constructor Detail

Method Detail

• getInputList

   abstract List<String> getInputList()

  /////////////////////////////////////////////////////////////////
  General parameters
  
  Input corpus files.
   Trainer accepts the following two formats:
   A) Monolingual: plain text, one sentence per line.
   B) Bilingual:   TSV, source sentence <tab> target sentence
   When bilingual data is passed, shared vocabulary model is built.
   Note that the input file must be raw corpus, not a preprocessed corpus.
   Trainer only loads the first `input_sentence_size` sentences specified
   with this parameter.
  

  repeated string input = 1;

  Returns:

  A list containing the input.

• getInputCount

   abstract int getInputCount()

  /////////////////////////////////////////////////////////////////
  General parameters
  
  Input corpus files.
   Trainer accepts the following two formats:
   A) Monolingual: plain text, one sentence per line.
   B) Bilingual:   TSV, source sentence <tab> target sentence
   When bilingual data is passed, shared vocabulary model is built.
   Note that the input file must be raw corpus, not a preprocessed corpus.
   Trainer only loads the first `input_sentence_size` sentences specified
   with this parameter.
  

  repeated string input = 1;

  Returns:

  The count of input.

• getInput

   abstract String getInput(int index)

  /////////////////////////////////////////////////////////////////
  General parameters
  
  Input corpus files.
   Trainer accepts the following two formats:
   A) Monolingual: plain text, one sentence per line.
   B) Bilingual:   TSV, source sentence <tab> target sentence
   When bilingual data is passed, shared vocabulary model is built.
   Note that the input file must be raw corpus, not a preprocessed corpus.
   Trainer only loads the first `input_sentence_size` sentences specified
   with this parameter.
  

  repeated string input = 1;

  Parameters:

  index - The index of the element to return.

  Returns:

  The input at the given index.

• getInputBytes

   abstract ByteString getInputBytes(int index)

  /////////////////////////////////////////////////////////////////
  General parameters
  
  Input corpus files.
   Trainer accepts the following two formats:
   A) Monolingual: plain text, one sentence per line.
   B) Bilingual:   TSV, source sentence <tab> target sentence
   When bilingual data is passed, shared vocabulary model is built.
   Note that the input file must be raw corpus, not a preprocessed corpus.
   Trainer only loads the first `input_sentence_size` sentences specified
   with this parameter.
  

  repeated string input = 1;

  Parameters:

  index - The index of the value to return.

  Returns:

  The bytes of the input at the given index.

• hasInputFormat

   abstract boolean hasInputFormat()

  Input corpus format:
  "text": one-sentence-per-line text format (default)
  "tsv":  sentence <tab> freq
  

  optional string input_format = 7;

  Returns:

  Whether the inputFormat field is set.

• getInputFormat

   abstract String getInputFormat()

  Input corpus format:
  "text": one-sentence-per-line text format (default)
  "tsv":  sentence <tab> freq
  

  optional string input_format = 7;

  Returns:

  The inputFormat.

• getInputFormatBytes

   abstract ByteString getInputFormatBytes()

  Input corpus format:
  "text": one-sentence-per-line text format (default)
  "tsv":  sentence <tab> freq
  

  optional string input_format = 7;

  Returns:

  The bytes for inputFormat.

• hasModelPrefix

   abstract boolean hasModelPrefix()

  Output model file prefix.
  <model_prefix>.model and <model_prefix>.vocab are generated.
  

  optional string model_prefix = 2;

  Returns:

  Whether the modelPrefix field is set.

• getModelPrefix

   abstract String getModelPrefix()

  Output model file prefix.
  <model_prefix>.model and <model_prefix>.vocab are generated.
  

  optional string model_prefix = 2;

  Returns:

  The modelPrefix.

• getModelPrefixBytes

   abstract ByteString getModelPrefixBytes()

  Output model file prefix.
  <model_prefix>.model and <model_prefix>.vocab are generated.
  

  optional string model_prefix = 2;

  Returns:

  The bytes for modelPrefix.

• hasModelType

   abstract boolean hasModelType()

  optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];

  Returns:

  Whether the modelType field is set.

• getModelType

   abstract SentencepieceModel.TrainerSpec.ModelType getModelType()

  optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];

  Returns:

  The modelType.

• hasVocabSize

   abstract boolean hasVocabSize()

  Vocabulary size. 8k is the default size.
  

  optional int32 vocab_size = 4 [default = 8000];

  Returns:

  Whether the vocabSize field is set.

• getVocabSize

   abstract int getVocabSize()

  Vocabulary size. 8k is the default size.
  

  optional int32 vocab_size = 4 [default = 8000];

  Returns:

  The vocabSize.

• getAcceptLanguageList

   abstract List<String> getAcceptLanguageList()

  List of the languages this model can accept.
  Since the model is language-agnostic, this field is used as a reference.
  

  repeated string accept_language = 5;

  Returns:

  A list containing the acceptLanguage.

• getAcceptLanguageCount

   abstract int getAcceptLanguageCount()

  List of the languages this model can accept.
  Since the model is language-agnostic, this field is used as a reference.
  

  repeated string accept_language = 5;

  Returns:

  The count of acceptLanguage.

• getAcceptLanguage

   abstract String getAcceptLanguage(int index)

  List of the languages this model can accept.
  Since the model is language-agnostic, this field is used as a reference.
  

  repeated string accept_language = 5;

  Parameters:

  index - The index of the element to return.

  Returns:

  The acceptLanguage at the given index.

• getAcceptLanguageBytes

   abstract ByteString getAcceptLanguageBytes(int index)

  List of the languages this model can accept.
  Since the model is language-agnostic, this field is used as a reference.
  

  repeated string accept_language = 5;

  Parameters:

  index - The index of the value to return.

  Returns:

  The bytes of the acceptLanguage at the given index.

• hasSelfTestSampleSize

   abstract boolean hasSelfTestSampleSize()

  Size of self-test samples, which are encoded in the model file.
  

  optional int32 self_test_sample_size = 6 [default = 0];

  Returns:

  Whether the selfTestSampleSize field is set.

• getSelfTestSampleSize

   abstract int getSelfTestSampleSize()

  Size of self-test samples, which are encoded in the model file.
  

  optional int32 self_test_sample_size = 6 [default = 0];

  Returns:

  The selfTestSampleSize.

• hasEnableDifferentialPrivacy

   abstract boolean hasEnableDifferentialPrivacy()

  Whether to use DP version of sentencepiece. Use it with TSV input format
  (requires precomputed word tab counts to work).
  

  optional bool enable_differential_privacy = 50 [default = false];

  Returns:

  Whether the enableDifferentialPrivacy field is set.

• getEnableDifferentialPrivacy

   abstract boolean getEnableDifferentialPrivacy()

  Whether to use DP version of sentencepiece. Use it with TSV input format
  (requires precomputed word tab counts to work).
  

  optional bool enable_differential_privacy = 50 [default = false];

  Returns:

  The enableDifferentialPrivacy.

• hasDifferentialPrivacyNoiseLevel

   abstract boolean hasDifferentialPrivacyNoiseLevel()

  Set these parameters if you need DP version of sentencepiece.
  std of noise to add.
  

  optional float differential_privacy_noise_level = 51 [default = 0];

  Returns:

  Whether the differentialPrivacyNoiseLevel field is set.

• getDifferentialPrivacyNoiseLevel

   abstract float getDifferentialPrivacyNoiseLevel()

  Set these parameters if you need DP version of sentencepiece.
  std of noise to add.
  

  optional float differential_privacy_noise_level = 51 [default = 0];

  Returns:

  The differentialPrivacyNoiseLevel.

• hasDifferentialPrivacyClippingThreshold

   abstract boolean hasDifferentialPrivacyClippingThreshold()

  Clipping threshold to apply after adding noise. All the words with
  frequency less than this value are dropped.
  

  optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];

  Returns:

  Whether the differentialPrivacyClippingThreshold field is set.

• getDifferentialPrivacyClippingThreshold

   abstract long getDifferentialPrivacyClippingThreshold()

  Clipping threshold to apply after adding noise. All the words with
  frequency less than this value are dropped.
  

  optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];

  Returns:

  The differentialPrivacyClippingThreshold.

• hasCharacterCoverage

   abstract boolean hasCharacterCoverage()

  /////////////////////////////////////////////////////////////////
  Training parameters.
  
  Uses characters which cover the corpus with the ratio of `chars_coverage`.
  This parameter determines the set of basic Alphabet of sentence piece.
  1.0 - `chars_coverage` characters are treated as UNK.
  See also required_chars field.
  

  optional float character_coverage = 10 [default = 0.9995];

  Returns:

  Whether the characterCoverage field is set.

• getCharacterCoverage

   abstract float getCharacterCoverage()

  /////////////////////////////////////////////////////////////////
  Training parameters.
  
  Uses characters which cover the corpus with the ratio of `chars_coverage`.
  This parameter determines the set of basic Alphabet of sentence piece.
  1.0 - `chars_coverage` characters are treated as UNK.
  See also required_chars field.
  

  optional float character_coverage = 10 [default = 0.9995];

  Returns:

  The characterCoverage.

• hasInputSentenceSize

   abstract boolean hasInputSentenceSize()

  Maximum size of sentences the trainer loads from `input` parameter.
  Trainer simply loads the `input` files in sequence.
  It is better to shuffle the input corpus randomly.
  

  optional uint64 input_sentence_size = 11 [default = 0];

  Returns:

  Whether the inputSentenceSize field is set.

• getInputSentenceSize

   abstract long getInputSentenceSize()

  Maximum size of sentences the trainer loads from `input` parameter.
  Trainer simply loads the `input` files in sequence.
  It is better to shuffle the input corpus randomly.
  

  optional uint64 input_sentence_size = 11 [default = 0];

  Returns:

  The inputSentenceSize.

• hasShuffleInputSentence

   abstract boolean hasShuffleInputSentence()

  optional bool shuffle_input_sentence = 19 [default = true];

  Returns:

  Whether the shuffleInputSentence field is set.

• getShuffleInputSentence

   abstract boolean getShuffleInputSentence()

  optional bool shuffle_input_sentence = 19 [default = true];

  Returns:

  The shuffleInputSentence.

• hasMiningSentenceSize

  @Deprecated() abstract boolean hasMiningSentenceSize()

  Maximum size of sentences to make seed sentence pieces.
  Extended suffix array is constructed to extract frequent
  sub-strings from the corpus. This uses 20N working space,
  where N is the size of corpus.
  

  optional int32 mining_sentence_size = 12 [deprecated = true];

  Returns:

  Whether the miningSentenceSize field is set.

• getMiningSentenceSize

  @Deprecated() abstract int getMiningSentenceSize()

  Maximum size of sentences to make seed sentence pieces.
  Extended suffix array is constructed to extract frequent
  sub-strings from the corpus. This uses 20N working space,
  where N is the size of corpus.
  

  optional int32 mining_sentence_size = 12 [deprecated = true];

  Returns:

  The miningSentenceSize.

• hasTrainingSentenceSize

  @Deprecated() abstract boolean hasTrainingSentenceSize()

  Maximum size of sentences to train sentence pieces.
  

  optional int32 training_sentence_size = 13 [deprecated = true];

  Returns:

  Whether the trainingSentenceSize field is set.

• getTrainingSentenceSize

  @Deprecated() abstract int getTrainingSentenceSize()

  Maximum size of sentences to train sentence pieces.
  

  optional int32 training_sentence_size = 13 [deprecated = true];

  Returns:

  The trainingSentenceSize.

• hasSeedSentencepieceSize

   abstract boolean hasSeedSentencepieceSize()

  The size of seed sentencepieces.
  `seed_sentencepiece_size` must be larger than `vocab_size`.
  

  optional int32 seed_sentencepiece_size = 14 [default = 1000000];

  Returns:

  Whether the seedSentencepieceSize field is set.

• getSeedSentencepieceSize

   abstract int getSeedSentencepieceSize()

  The size of seed sentencepieces.
  `seed_sentencepiece_size` must be larger than `vocab_size`.
  

  optional int32 seed_sentencepiece_size = 14 [default = 1000000];

  Returns:

  The seedSentencepieceSize.

• hasShrinkingFactor

   abstract boolean hasShrinkingFactor()

  In every EM sub-iterations, keeps top
  `shrinking_factor` * `current sentencepieces size` with respect to
  the loss of the sentence piece. This value should be smaller than 1.0.
  

  optional float shrinking_factor = 15 [default = 0.75];

  Returns:

  Whether the shrinkingFactor field is set.

• getShrinkingFactor

   abstract float getShrinkingFactor()

  In every EM sub-iterations, keeps top
  `shrinking_factor` * `current sentencepieces size` with respect to
  the loss of the sentence piece. This value should be smaller than 1.0.
  

  optional float shrinking_factor = 15 [default = 0.75];

  Returns:

  The shrinkingFactor.

• hasMaxSentenceLength

   abstract boolean hasMaxSentenceLength()

  The maximum sentence length in byte. The sentences with the length
  larger than `max_sentence_length` is simply ignored.
  Longer input tends to bring the following risks:
   * Overflow during EM training (unigram language model only)
   * Performance drop because of O(n log n) cost in BPE.
  

  optional int32 max_sentence_length = 18 [default = 4192];

  Returns:

  Whether the maxSentenceLength field is set.

• getMaxSentenceLength

   abstract int getMaxSentenceLength()

  The maximum sentence length in byte. The sentences with the length
  larger than `max_sentence_length` is simply ignored.
  Longer input tends to bring the following risks:
   * Overflow during EM training (unigram language model only)
   * Performance drop because of O(n log n) cost in BPE.
  

  optional int32 max_sentence_length = 18 [default = 4192];

  Returns:

  The maxSentenceLength.

• hasNumThreads

   abstract boolean hasNumThreads()

  Number of threads in the training.
  

  optional int32 num_threads = 16 [default = 16];

  Returns:

  Whether the numThreads field is set.

• getNumThreads

   abstract int getNumThreads()

  Number of threads in the training.
  

  optional int32 num_threads = 16 [default = 16];

  Returns:

  The numThreads.

• hasNumSubIterations

   abstract boolean hasNumSubIterations()

  Number of EM sub iterations.
  

  optional int32 num_sub_iterations = 17 [default = 2];

  Returns:

  Whether the numSubIterations field is set.

• getNumSubIterations

   abstract int getNumSubIterations()

  Number of EM sub iterations.
  

  optional int32 num_sub_iterations = 17 [default = 2];

  Returns:

  The numSubIterations.

• hasMaxSentencepieceLength

   abstract boolean hasMaxSentencepieceLength()

  /////////////////////////////////////////////////////////////////
  SentencePiece parameters which control the shapes of sentence piece.
  
  Maximum length of sentencepiece.
  

  optional int32 max_sentencepiece_length = 20 [default = 16];

  Returns:

  Whether the maxSentencepieceLength field is set.

• getMaxSentencepieceLength

   abstract int getMaxSentencepieceLength()

  /////////////////////////////////////////////////////////////////
  SentencePiece parameters which control the shapes of sentence piece.
  
  Maximum length of sentencepiece.
  

  optional int32 max_sentencepiece_length = 20 [default = 16];

  Returns:

  The maxSentencepieceLength.

• hasSplitByUnicodeScript

   abstract boolean hasSplitByUnicodeScript()

  Uses Unicode script to split sentence pieces.
  When `split_by_unicode_script` is true, we do not allow sentence piece to
  include multiple Unicode scripts, e.g. "F1" is not a valid piece.
  Exception: CJ characters (Hiragana/Katakana/Han) are all handled
  as one script type, since Japanese word can consist of multiple scripts.
  This exception is always applied regardless of the accept-language
  parameter.
  

  optional bool split_by_unicode_script = 21 [default = true];

  Returns:

  Whether the splitByUnicodeScript field is set.

• getSplitByUnicodeScript

   abstract boolean getSplitByUnicodeScript()

  Uses Unicode script to split sentence pieces.
  When `split_by_unicode_script` is true, we do not allow sentence piece to
  include multiple Unicode scripts, e.g. "F1" is not a valid piece.
  Exception: CJ characters (Hiragana/Katakana/Han) are all handled
  as one script type, since Japanese word can consist of multiple scripts.
  This exception is always applied regardless of the accept-language
  parameter.
  

  optional bool split_by_unicode_script = 21 [default = true];

  Returns:

  The splitByUnicodeScript.

• hasSplitByNumber

   abstract boolean hasSplitByNumber()

  When `split_by_number` is true, put a boundary between number and
  non-number transition. If we want to treat "F1" is one token, set this flag
  to be false.
  

  optional bool split_by_number = 23 [default = true];

  Returns:

  Whether the splitByNumber field is set.

• getSplitByNumber

   abstract boolean getSplitByNumber()

  When `split_by_number` is true, put a boundary between number and
  non-number transition. If we want to treat "F1" is one token, set this flag
  to be false.
  

  optional bool split_by_number = 23 [default = true];

  Returns:

  The splitByNumber.

• hasSplitByWhitespace

   abstract boolean hasSplitByWhitespace()

  Use a white space to split sentence pieces.
  When `split_by_whitespace` is false, we may have the piece containing
  a white space in the middle. e.g., "in_the".
  

  optional bool split_by_whitespace = 22 [default = true];

  Returns:

  Whether the splitByWhitespace field is set.

• getSplitByWhitespace

   abstract boolean getSplitByWhitespace()

  Use a white space to split sentence pieces.
  When `split_by_whitespace` is false, we may have the piece containing
  a white space in the middle. e.g., "in_the".
  

  optional bool split_by_whitespace = 22 [default = true];

  Returns:

  The splitByWhitespace.

• hasTreatWhitespaceAsSuffix

   abstract boolean hasTreatWhitespaceAsSuffix()

  Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
  hello_. When `treat_whitespace_as_suffix` is true,
  NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
  of sentence.
  

  optional bool treat_whitespace_as_suffix = 24 [default = false];

  Returns:

  Whether the treatWhitespaceAsSuffix field is set.

• getTreatWhitespaceAsSuffix

   abstract boolean getTreatWhitespaceAsSuffix()

  Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
  hello_. When `treat_whitespace_as_suffix` is true,
  NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
  of sentence.
  

  optional bool treat_whitespace_as_suffix = 24 [default = false];

  Returns:

  The treatWhitespaceAsSuffix.

• hasAllowWhitespaceOnlyPieces

   abstract boolean hasAllowWhitespaceOnlyPieces()

  Allows pieces that only contain whitespaces instead of appearing only as
  prefix or suffix of other pieces.
  

  optional bool allow_whitespace_only_pieces = 26 [default = false];

  Returns:

  Whether the allowWhitespaceOnlyPieces field is set.

• getAllowWhitespaceOnlyPieces

   abstract boolean getAllowWhitespaceOnlyPieces()

  Allows pieces that only contain whitespaces instead of appearing only as
  prefix or suffix of other pieces.
  

  optional bool allow_whitespace_only_pieces = 26 [default = false];

  Returns:

  The allowWhitespaceOnlyPieces.

• hasSplitDigits

   abstract boolean hasSplitDigits()

  Split all digits (0-9) into separate pieces.
  

  optional bool split_digits = 25 [default = false];

  Returns:

  Whether the splitDigits field is set.

• getSplitDigits

   abstract boolean getSplitDigits()

  Split all digits (0-9) into separate pieces.
  

  optional bool split_digits = 25 [default = false];

  Returns:

  The splitDigits.

• hasPretokenizationDelimiter

   abstract boolean hasPretokenizationDelimiter()

  Defines the pre-tokenization delimiter.
  When specified, no pieces crossing this delimiter is not included
  in the vocab. Then the delimiter string is virtually ignored
  during the training. This field can allows constraints on the vocabulary
  selection. Note that this field is available on unigram mode.
  

  optional string pretokenization_delimiter = 53 [default = ""];

  Returns:

  Whether the pretokenizationDelimiter field is set.

• getPretokenizationDelimiter

   abstract String getPretokenizationDelimiter()

  Defines the pre-tokenization delimiter.
  When specified, no pieces crossing this delimiter is not included
  in the vocab. Then the delimiter string is virtually ignored
  during the training. This field can allows constraints on the vocabulary
  selection. Note that this field is available on unigram mode.
  

  optional string pretokenization_delimiter = 53 [default = ""];

  Returns:

  The pretokenizationDelimiter.

• getPretokenizationDelimiterBytes

   abstract ByteString getPretokenizationDelimiterBytes()

  Defines the pre-tokenization delimiter.
  When specified, no pieces crossing this delimiter is not included
  in the vocab. Then the delimiter string is virtually ignored
  during the training. This field can allows constraints on the vocabulary
  selection. Note that this field is available on unigram mode.
  

  optional string pretokenization_delimiter = 53 [default = ""];

  Returns:

  The bytes for pretokenizationDelimiter.

• getControlSymbolsList

   abstract List<String> getControlSymbolsList()

  /////////////////////////////////////////////////////////////////
  Vocabulary management
  
  Defines control symbols used as an indicator to
  change the behavior of the decoder. <s> and </s> are pre-defined.
  We can use this field to encode various meta information,
  including language indicator in multilingual model.
  These symbols are not visible to users, but visible to
  the decoder. Note that when the input sentence contains control symbols,
  they are not treated as one token, but segmented into normal pieces.
  Control symbols must be inserted independently from the segmentation.
  

  repeated string control_symbols = 30;

  Returns:

  A list containing the controlSymbols.

• getControlSymbolsCount

   abstract int getControlSymbolsCount()

  /////////////////////////////////////////////////////////////////
  Vocabulary management
  
  Defines control symbols used as an indicator to
  change the behavior of the decoder. <s> and </s> are pre-defined.
  We can use this field to encode various meta information,
  including language indicator in multilingual model.
  These symbols are not visible to users, but visible to
  the decoder. Note that when the input sentence contains control symbols,
  they are not treated as one token, but segmented into normal pieces.
  Control symbols must be inserted independently from the segmentation.
  

  repeated string control_symbols = 30;

  Returns:

  The count of controlSymbols.

• getControlSymbols

   abstract String getControlSymbols(int index)

  /////////////////////////////////////////////////////////////////
  Vocabulary management
  
  Defines control symbols used as an indicator to
  change the behavior of the decoder. <s> and </s> are pre-defined.
  We can use this field to encode various meta information,
  including language indicator in multilingual model.
  These symbols are not visible to users, but visible to
  the decoder. Note that when the input sentence contains control symbols,
  they are not treated as one token, but segmented into normal pieces.
  Control symbols must be inserted independently from the segmentation.
  

  repeated string control_symbols = 30;

  Parameters:

  index - The index of the element to return.

  Returns:

  The controlSymbols at the given index.

• getControlSymbolsBytes

   abstract ByteString getControlSymbolsBytes(int index)

  /////////////////////////////////////////////////////////////////
  Vocabulary management
  
  Defines control symbols used as an indicator to
  change the behavior of the decoder. <s> and </s> are pre-defined.
  We can use this field to encode various meta information,
  including language indicator in multilingual model.
  These symbols are not visible to users, but visible to
  the decoder. Note that when the input sentence contains control symbols,
  they are not treated as one token, but segmented into normal pieces.
  Control symbols must be inserted independently from the segmentation.
  

  repeated string control_symbols = 30;

  Parameters:

  index - The index of the value to return.

  Returns:

  The bytes of the controlSymbols at the given index.

• getUserDefinedSymbolsList

   abstract List<String> getUserDefinedSymbolsList()

  Defines user defined symbols.
  These symbols are added with extremely high score
  so they are always treated as one unique symbol in any context.
  Typical usage of user_defined_symbols is placeholder for named entities.
  

  repeated string user_defined_symbols = 31;

  Returns:

  A list containing the userDefinedSymbols.

• getUserDefinedSymbolsCount

   abstract int getUserDefinedSymbolsCount()

  Defines user defined symbols.
  These symbols are added with extremely high score
  so they are always treated as one unique symbol in any context.
  Typical usage of user_defined_symbols is placeholder for named entities.
  

  repeated string user_defined_symbols = 31;

  Returns:

  The count of userDefinedSymbols.

• getUserDefinedSymbols

   abstract String getUserDefinedSymbols(int index)

  Defines user defined symbols.
  These symbols are added with extremely high score
  so they are always treated as one unique symbol in any context.
  Typical usage of user_defined_symbols is placeholder for named entities.
  

  repeated string user_defined_symbols = 31;

  Parameters:

  index - The index of the element to return.

  Returns:

  The userDefinedSymbols at the given index.

• getUserDefinedSymbolsBytes

   abstract ByteString getUserDefinedSymbolsBytes(int index)

  Defines user defined symbols.
  These symbols are added with extremely high score
  so they are always treated as one unique symbol in any context.
  Typical usage of user_defined_symbols is placeholder for named entities.
  

  repeated string user_defined_symbols = 31;

  Parameters:

  index - The index of the value to return.

  Returns:

  The bytes of the userDefinedSymbols at the given index.

• hasRequiredChars

   abstract boolean hasRequiredChars()

  Defines required characters. Each UTF8 character in this string is included
  in the character set regardless of character_coverage value. Unlike
  user_defined_symbols, these characters have scores based on the frequency
  on input sentences, and the model can form subwords using characters
  in this field.
  

  optional string required_chars = 36;

  Returns:

  Whether the requiredChars field is set.

• getRequiredChars

   abstract String getRequiredChars()

  Defines required characters. Each UTF8 character in this string is included
  in the character set regardless of character_coverage value. Unlike
  user_defined_symbols, these characters have scores based on the frequency
  on input sentences, and the model can form subwords using characters
  in this field.
  

  optional string required_chars = 36;

  Returns:

  The requiredChars.

• getRequiredCharsBytes

   abstract ByteString getRequiredCharsBytes()

  Defines required characters. Each UTF8 character in this string is included
  in the character set regardless of character_coverage value. Unlike
  user_defined_symbols, these characters have scores based on the frequency
  on input sentences, and the model can form subwords using characters
  in this field.
  

  optional string required_chars = 36;

  Returns:

  The bytes for requiredChars.

• hasByteFallback

   abstract boolean hasByteFallback()

  Decomposes unknown pieces into UTF-8 bytes.
  

  optional bool byte_fallback = 35 [default = false];

  Returns:

  Whether the byteFallback field is set.

• getByteFallback

   abstract boolean getByteFallback()

  Decomposes unknown pieces into UTF-8 bytes.
  

  optional bool byte_fallback = 35 [default = false];

  Returns:

  The byteFallback.

• hasVocabularyOutputPieceScore

   abstract boolean hasVocabularyOutputPieceScore()

  When creating the vocabulary file, defines whether or not to additionally
  output the score for each piece.
  

  optional bool vocabulary_output_piece_score = 32 [default = true];

  Returns:

  Whether the vocabularyOutputPieceScore field is set.

• getVocabularyOutputPieceScore

   abstract boolean getVocabularyOutputPieceScore()

  When creating the vocabulary file, defines whether or not to additionally
  output the score for each piece.
  

  optional bool vocabulary_output_piece_score = 32 [default = true];

  Returns:

  The vocabularyOutputPieceScore.

• hasHardVocabLimit

   abstract boolean hasHardVocabLimit()

  `vocab_size` is treated as hard limit. Crash if
  the model can not produce the vocab of size `vocab_size`,
  When `hard_vocab_limit` is false, vocab_size is treated
  as soft limit. Note that when model_type=char,
  always assumes hard_vocab_limit = false.
  

  optional bool hard_vocab_limit = 33 [default = true];

  Returns:

  Whether the hardVocabLimit field is set.

• getHardVocabLimit

   abstract boolean getHardVocabLimit()

  `vocab_size` is treated as hard limit. Crash if
  the model can not produce the vocab of size `vocab_size`,
  When `hard_vocab_limit` is false, vocab_size is treated
  as soft limit. Note that when model_type=char,
  always assumes hard_vocab_limit = false.
  

  optional bool hard_vocab_limit = 33 [default = true];

  Returns:

  The hardVocabLimit.

• hasUseAllVocab

   abstract boolean hasUseAllVocab()

  use all symbols for vocab extraction. This flag is valid
  if model type is either CHAR or WORD
  

  optional bool use_all_vocab = 34 [default = false];

  Returns:

  Whether the useAllVocab field is set.

• getUseAllVocab

   abstract boolean getUseAllVocab()

  use all symbols for vocab extraction. This flag is valid
  if model type is either CHAR or WORD
  

  optional bool use_all_vocab = 34 [default = false];

  Returns:

  The useAllVocab.

• hasUnkId

   abstract boolean hasUnkId()

  /////////////////////////////////////////////////////////////////
  Reserved special meta tokens.
  * -1 is not used.
  * unk_id must not be -1.
  Id must starts with 0 and be contiguous.
  

  optional int32 unk_id = 40 [default = 0];

  Returns:

  Whether the unkId field is set.

• getUnkId

   abstract int getUnkId()

  /////////////////////////////////////////////////////////////////
  Reserved special meta tokens.
  * -1 is not used.
  * unk_id must not be -1.
  Id must starts with 0 and be contiguous.
  

  optional int32 unk_id = 40 [default = 0];

  Returns:

  The unkId.

• hasBosId

   abstract boolean hasBosId()

  <s>
  

  optional int32 bos_id = 41 [default = 1];

  Returns:

  Whether the bosId field is set.

• getBosId

   abstract int getBosId()

  <s>
  

  optional int32 bos_id = 41 [default = 1];

  Returns:

  The bosId.

• hasEosId

   abstract boolean hasEosId()

  </s>
  

  optional int32 eos_id = 42 [default = 2];

  Returns:

  Whether the eosId field is set.

• getEosId

   abstract int getEosId()

  </s>
  

  optional int32 eos_id = 42 [default = 2];

  Returns:

  The eosId.

• hasPadId

   abstract boolean hasPadId()

  <pad> (padding)
  

  optional int32 pad_id = 43 [default = -1];

  Returns:

  Whether the padId field is set.

• getPadId

   abstract int getPadId()

  <pad> (padding)
  

  optional int32 pad_id = 43 [default = -1];

  Returns:

  The padId.

• hasUnkPiece

   abstract boolean hasUnkPiece()

  optional string unk_piece = 45 [default = "<unk>"];

  Returns:

  Whether the unkPiece field is set.

• getUnkPiece

   abstract String getUnkPiece()

  optional string unk_piece = 45 [default = "<unk>"];

  Returns:

  The unkPiece.

• getUnkPieceBytes

   abstract ByteString getUnkPieceBytes()

  optional string unk_piece = 45 [default = "<unk>"];

  Returns:

  The bytes for unkPiece.

• hasBosPiece

   abstract boolean hasBosPiece()

  optional string bos_piece = 46 [default = "<s>"];

  Returns:

  Whether the bosPiece field is set.

• getBosPiece

   abstract String getBosPiece()

  optional string bos_piece = 46 [default = "<s>"];

  Returns:

  The bosPiece.

• getBosPieceBytes

   abstract ByteString getBosPieceBytes()

  optional string bos_piece = 46 [default = "<s>"];

  Returns:

  The bytes for bosPiece.

• hasEosPiece

   abstract boolean hasEosPiece()

  optional string eos_piece = 47 [default = "</s>"];

  Returns:

  Whether the eosPiece field is set.

• getEosPiece

   abstract String getEosPiece()

  optional string eos_piece = 47 [default = "</s>"];

  Returns:

  The eosPiece.

• getEosPieceBytes

   abstract ByteString getEosPieceBytes()

  optional string eos_piece = 47 [default = "</s>"];

  Returns:

  The bytes for eosPiece.

• hasPadPiece

   abstract boolean hasPadPiece()

  optional string pad_piece = 48 [default = "<pad>"];

  Returns:

  Whether the padPiece field is set.

• getPadPiece

   abstract String getPadPiece()

  optional string pad_piece = 48 [default = "<pad>"];

  Returns:

  The padPiece.

• getPadPieceBytes

   abstract ByteString getPadPieceBytes()

  optional string pad_piece = 48 [default = "<pad>"];

  Returns:

  The bytes for padPiece.

• hasUnkSurface

   abstract boolean hasUnkSurface()

  Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
  since this character can be useful both for user and
  developer. We can easily figure out that <unk> is emitted.
  

  optional string unk_surface = 44 [default = " \342\201\207 "];

  Returns:

  Whether the unkSurface field is set.

• getUnkSurface

   abstract String getUnkSurface()

  Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
  since this character can be useful both for user and
  developer. We can easily figure out that <unk> is emitted.
  

  optional string unk_surface = 44 [default = " \342\201\207 "];

  Returns:

  The unkSurface.

• getUnkSurfaceBytes

   abstract ByteString getUnkSurfaceBytes()

  Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
  since this character can be useful both for user and
  developer. We can easily figure out that <unk> is emitted.
  

  optional string unk_surface = 44 [default = " \342\201\207 "];

  Returns:

  The bytes for unkSurface.

• hasTrainExtremelyLargeCorpus

   abstract boolean hasTrainExtremelyLargeCorpus()

  Increase bit depth to allow unigram model training on large
  (>10M sentences) corpora. A Side-effect of enabling this flag
  is increased memory usage.
  

  optional bool train_extremely_large_corpus = 49 [default = false];

  Returns:

  Whether the trainExtremelyLargeCorpus field is set.

• getTrainExtremelyLargeCorpus

   abstract boolean getTrainExtremelyLargeCorpus()

  Increase bit depth to allow unigram model training on large
  (>10M sentences) corpora. A Side-effect of enabling this flag
  is increased memory usage.
  

  optional bool train_extremely_large_corpus = 49 [default = false];

  Returns:

  The trainExtremelyLargeCorpus.

• hasSeedSentencepiecesFile

   abstract boolean hasSeedSentencepiecesFile()

  Path to a seed sentencepieces file, with one tab-separated
  seed sentencepiece <tab> frequency per line.
  

  optional string seed_sentencepieces_file = 54 [default = ""];

  Returns:

  Whether the seedSentencepiecesFile field is set.

• getSeedSentencepiecesFile

   abstract String getSeedSentencepiecesFile()

  Path to a seed sentencepieces file, with one tab-separated
  seed sentencepiece <tab> frequency per line.
  

  optional string seed_sentencepieces_file = 54 [default = ""];

  Returns:

  The seedSentencepiecesFile.

• getSeedSentencepiecesFileBytes

   abstract ByteString getSeedSentencepiecesFileBytes()

  Path to a seed sentencepieces file, with one tab-separated
  seed sentencepiece <tab> frequency per line.
  

  optional string seed_sentencepieces_file = 54 [default = ""];

  Returns:

  The bytes for seedSentencepiecesFile.