Class SentencepieceModel.TrainerSpec.Builder

  • All Implemented Interfaces:
    com.google.genai.proto.SentencepieceModel.TrainerSpecOrBuilder , com.google.protobuf.GeneratedMessageV3.ExtendableMessageOrBuilder , com.google.protobuf.Message.Builder , com.google.protobuf.MessageLite.Builder , com.google.protobuf.MessageLiteOrBuilder , com.google.protobuf.MessageOrBuilder , java.lang.Cloneable

    
    public final class SentencepieceModel.TrainerSpec.Builder
    extends GeneratedMessageV3.ExtendableBuilder<MessageT, BuilderT> implements SentencepieceModel.TrainerSpecOrBuilder
                        
    TrainerSpec encodes a various parameters for SentencePiece training.
    Next id: 55
    
    Protobuf type com.google.genai.proto.TrainerSpec
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
    • Field Summary

      Fields 
      Modifier and Type Field Description
    • Constructor Summary

      Constructors 
      Constructor Description
    • Enum Constant Summary

      Enum Constants 
      Enum Constant Description
    • Method Summary

      Modifier and Type Method Description
      final static Descriptors.Descriptor getDescriptor()
      SentencepieceModel.TrainerSpec.Builder clear()
      Descriptors.Descriptor getDescriptorForType()
      SentencepieceModel.TrainerSpec getDefaultInstanceForType()
      SentencepieceModel.TrainerSpec build()
      SentencepieceModel.TrainerSpec buildPartial()
      SentencepieceModel.TrainerSpec.Builder clone()
      SentencepieceModel.TrainerSpec.Builder setField(Descriptors.FieldDescriptor field, Object value)
      SentencepieceModel.TrainerSpec.Builder clearField(Descriptors.FieldDescriptor field)
      SentencepieceModel.TrainerSpec.Builder clearOneof(Descriptors.OneofDescriptor oneof)
      SentencepieceModel.TrainerSpec.Builder setRepeatedField(Descriptors.FieldDescriptor field, int index, Object value)
      SentencepieceModel.TrainerSpec.Builder addRepeatedField(Descriptors.FieldDescriptor field, Object value)
      <Type> SentencepieceModel.TrainerSpec.Builder setExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, Type> extension, Type value)
      <Type> SentencepieceModel.TrainerSpec.Builder setExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, List<Type>> extension, int index, Type value)
      <Type> SentencepieceModel.TrainerSpec.Builder addExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, List<Type>> extension, Type value)
      <T> SentencepieceModel.TrainerSpec.Builder clearExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, T> extension)
      SentencepieceModel.TrainerSpec.Builder mergeFrom(Message other)
      SentencepieceModel.TrainerSpec.Builder mergeFrom(SentencepieceModel.TrainerSpec other)
      final boolean isInitialized()
      SentencepieceModel.TrainerSpec.Builder mergeFrom(CodedInputStream input, ExtensionRegistryLite extensionRegistry)
      ProtocolStringList getInputList()
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      int getInputCount()
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      String getInput(int index)
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      ByteString getInputBytes(int index)
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      SentencepieceModel.TrainerSpec.Builder setInput(int index, String value)
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      SentencepieceModel.TrainerSpec.Builder addInput(String value)
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      SentencepieceModel.TrainerSpec.Builder addAllInput(Iterable<String> values)
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      SentencepieceModel.TrainerSpec.Builder clearInput()
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      SentencepieceModel.TrainerSpec.Builder addInputBytes(ByteString value)
      /////////////////////////////////////////////////////////////////
      General parameters
      
      Input corpus files.
       Trainer accepts the following two formats:
       A) Monolingual: plain text, one sentence per line.
       B) Bilingual:   TSV, source sentence <tab> target sentence
       When bilingual data is passed, shared vocabulary model is built.
       Note that the input file must be raw corpus, not a preprocessed corpus.
       Trainer only loads the first `input_sentence_size` sentences specified
       with this parameter.
      
      repeated string input = 1;
      boolean hasInputFormat()
      Input corpus format:
      "text": one-sentence-per-line text format (default)
      "tsv":  sentence <tab> freq
      
      optional string input_format = 7;
      String getInputFormat()
      Input corpus format:
      "text": one-sentence-per-line text format (default)
      "tsv":  sentence <tab> freq
      
      optional string input_format = 7;
      ByteString getInputFormatBytes()
      Input corpus format:
      "text": one-sentence-per-line text format (default)
      "tsv":  sentence <tab> freq
      
      optional string input_format = 7;
      SentencepieceModel.TrainerSpec.Builder setInputFormat(String value)
      Input corpus format:
      "text": one-sentence-per-line text format (default)
      "tsv":  sentence <tab> freq
      
      optional string input_format = 7;
      SentencepieceModel.TrainerSpec.Builder clearInputFormat()
      Input corpus format:
      "text": one-sentence-per-line text format (default)
      "tsv":  sentence <tab> freq
      
      optional string input_format = 7;
      SentencepieceModel.TrainerSpec.Builder setInputFormatBytes(ByteString value)
      Input corpus format:
      "text": one-sentence-per-line text format (default)
      "tsv":  sentence <tab> freq
      
      optional string input_format = 7;
      boolean hasModelPrefix()
      Output model file prefix.
      <model_prefix>.model and <model_prefix>.vocab are generated.
      
      optional string model_prefix = 2;
      String getModelPrefix()
      Output model file prefix.
      <model_prefix>.model and <model_prefix>.vocab are generated.
      
      optional string model_prefix = 2;
      ByteString getModelPrefixBytes()
      Output model file prefix.
      <model_prefix>.model and <model_prefix>.vocab are generated.
      
      optional string model_prefix = 2;
      SentencepieceModel.TrainerSpec.Builder setModelPrefix(String value)
      Output model file prefix.
      <model_prefix>.model and <model_prefix>.vocab are generated.
      
      optional string model_prefix = 2;
      SentencepieceModel.TrainerSpec.Builder clearModelPrefix()
      Output model file prefix.
      <model_prefix>.model and <model_prefix>.vocab are generated.
      
      optional string model_prefix = 2;
      SentencepieceModel.TrainerSpec.Builder setModelPrefixBytes(ByteString value)
      Output model file prefix.
      <model_prefix>.model and <model_prefix>.vocab are generated.
      
      optional string model_prefix = 2;
      boolean hasModelType() optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];
      SentencepieceModel.TrainerSpec.ModelType getModelType() optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];
      SentencepieceModel.TrainerSpec.Builder setModelType(SentencepieceModel.TrainerSpec.ModelType value) optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];
      SentencepieceModel.TrainerSpec.Builder clearModelType() optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];
      boolean hasVocabSize()
      Vocabulary size. 8k is the default size.
      
      optional int32 vocab_size = 4 [default = 8000];
      int getVocabSize()
      Vocabulary size. 8k is the default size.
      
      optional int32 vocab_size = 4 [default = 8000];
      SentencepieceModel.TrainerSpec.Builder setVocabSize(int value)
      Vocabulary size. 8k is the default size.
      
      optional int32 vocab_size = 4 [default = 8000];
      SentencepieceModel.TrainerSpec.Builder clearVocabSize()
      Vocabulary size. 8k is the default size.
      
      optional int32 vocab_size = 4 [default = 8000];
      ProtocolStringList getAcceptLanguageList()
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      int getAcceptLanguageCount()
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      String getAcceptLanguage(int index)
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      ByteString getAcceptLanguageBytes(int index)
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      SentencepieceModel.TrainerSpec.Builder setAcceptLanguage(int index, String value)
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      SentencepieceModel.TrainerSpec.Builder addAcceptLanguage(String value)
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      SentencepieceModel.TrainerSpec.Builder addAllAcceptLanguage(Iterable<String> values)
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      SentencepieceModel.TrainerSpec.Builder clearAcceptLanguage()
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      SentencepieceModel.TrainerSpec.Builder addAcceptLanguageBytes(ByteString value)
      List of the languages this model can accept.
      Since the model is language-agnostic, this field is used as a reference.
      
      repeated string accept_language = 5;
      boolean hasSelfTestSampleSize()
      Size of self-test samples, which are encoded in the model file.
      
      optional int32 self_test_sample_size = 6 [default = 0];
      int getSelfTestSampleSize()
      Size of self-test samples, which are encoded in the model file.
      
      optional int32 self_test_sample_size = 6 [default = 0];
      SentencepieceModel.TrainerSpec.Builder setSelfTestSampleSize(int value)
      Size of self-test samples, which are encoded in the model file.
      
      optional int32 self_test_sample_size = 6 [default = 0];
      SentencepieceModel.TrainerSpec.Builder clearSelfTestSampleSize()
      Size of self-test samples, which are encoded in the model file.
      
      optional int32 self_test_sample_size = 6 [default = 0];
      boolean hasEnableDifferentialPrivacy()
      Whether to use DP version of sentencepiece. Use it with TSV input format
      (requires precomputed word tab counts to work).
      
      optional bool enable_differential_privacy = 50 [default = false];
      boolean getEnableDifferentialPrivacy()
      Whether to use DP version of sentencepiece. Use it with TSV input format
      (requires precomputed word tab counts to work).
      
      optional bool enable_differential_privacy = 50 [default = false];
      SentencepieceModel.TrainerSpec.Builder setEnableDifferentialPrivacy(boolean value)
      Whether to use DP version of sentencepiece. Use it with TSV input format
      (requires precomputed word tab counts to work).
      
      optional bool enable_differential_privacy = 50 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearEnableDifferentialPrivacy()
      Whether to use DP version of sentencepiece. Use it with TSV input format
      (requires precomputed word tab counts to work).
      
      optional bool enable_differential_privacy = 50 [default = false];
      boolean hasDifferentialPrivacyNoiseLevel()
      Set these parameters if you need DP version of sentencepiece.
      std of noise to add.
      
      optional float differential_privacy_noise_level = 51 [default = 0];
      float getDifferentialPrivacyNoiseLevel()
      Set these parameters if you need DP version of sentencepiece.
      std of noise to add.
      
      optional float differential_privacy_noise_level = 51 [default = 0];
      SentencepieceModel.TrainerSpec.Builder setDifferentialPrivacyNoiseLevel(float value)
      Set these parameters if you need DP version of sentencepiece.
      std of noise to add.
      
      optional float differential_privacy_noise_level = 51 [default = 0];
      SentencepieceModel.TrainerSpec.Builder clearDifferentialPrivacyNoiseLevel()
      Set these parameters if you need DP version of sentencepiece.
      std of noise to add.
      
      optional float differential_privacy_noise_level = 51 [default = 0];
      boolean hasDifferentialPrivacyClippingThreshold()
      Clipping threshold to apply after adding noise. All the words with
      frequency less than this value are dropped.
      
      optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
      long getDifferentialPrivacyClippingThreshold()
      Clipping threshold to apply after adding noise. All the words with
      frequency less than this value are dropped.
      
      optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
      SentencepieceModel.TrainerSpec.Builder setDifferentialPrivacyClippingThreshold(long value)
      Clipping threshold to apply after adding noise. All the words with
      frequency less than this value are dropped.
      
      optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
      SentencepieceModel.TrainerSpec.Builder clearDifferentialPrivacyClippingThreshold()
      Clipping threshold to apply after adding noise. All the words with
      frequency less than this value are dropped.
      
      optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
      boolean hasCharacterCoverage()
      /////////////////////////////////////////////////////////////////
      Training parameters.
      
      Uses characters which cover the corpus with the ratio of `chars_coverage`.
      This parameter determines the set of basic Alphabet of sentence piece.
      1.0 - `chars_coverage` characters are treated as UNK.
      See also required_chars field.
      
      optional float character_coverage = 10 [default = 0.9995];
      float getCharacterCoverage()
      /////////////////////////////////////////////////////////////////
      Training parameters.
      
      Uses characters which cover the corpus with the ratio of `chars_coverage`.
      This parameter determines the set of basic Alphabet of sentence piece.
      1.0 - `chars_coverage` characters are treated as UNK.
      See also required_chars field.
      
      optional float character_coverage = 10 [default = 0.9995];
      SentencepieceModel.TrainerSpec.Builder setCharacterCoverage(float value)
      /////////////////////////////////////////////////////////////////
      Training parameters.
      
      Uses characters which cover the corpus with the ratio of `chars_coverage`.
      This parameter determines the set of basic Alphabet of sentence piece.
      1.0 - `chars_coverage` characters are treated as UNK.
      See also required_chars field.
      
      optional float character_coverage = 10 [default = 0.9995];
      SentencepieceModel.TrainerSpec.Builder clearCharacterCoverage()
      /////////////////////////////////////////////////////////////////
      Training parameters.
      
      Uses characters which cover the corpus with the ratio of `chars_coverage`.
      This parameter determines the set of basic Alphabet of sentence piece.
      1.0 - `chars_coverage` characters are treated as UNK.
      See also required_chars field.
      
      optional float character_coverage = 10 [default = 0.9995];
      boolean hasInputSentenceSize()
      Maximum size of sentences the trainer loads from `input` parameter.
      Trainer simply loads the `input` files in sequence.
      It is better to shuffle the input corpus randomly.
      
      optional uint64 input_sentence_size = 11 [default = 0];
      long getInputSentenceSize()
      Maximum size of sentences the trainer loads from `input` parameter.
      Trainer simply loads the `input` files in sequence.
      It is better to shuffle the input corpus randomly.
      
      optional uint64 input_sentence_size = 11 [default = 0];
      SentencepieceModel.TrainerSpec.Builder setInputSentenceSize(long value)
      Maximum size of sentences the trainer loads from `input` parameter.
      Trainer simply loads the `input` files in sequence.
      It is better to shuffle the input corpus randomly.
      
      optional uint64 input_sentence_size = 11 [default = 0];
      SentencepieceModel.TrainerSpec.Builder clearInputSentenceSize()
      Maximum size of sentences the trainer loads from `input` parameter.
      Trainer simply loads the `input` files in sequence.
      It is better to shuffle the input corpus randomly.
      
      optional uint64 input_sentence_size = 11 [default = 0];
      boolean hasShuffleInputSentence() optional bool shuffle_input_sentence = 19 [default = true];
      boolean getShuffleInputSentence() optional bool shuffle_input_sentence = 19 [default = true];
      SentencepieceModel.TrainerSpec.Builder setShuffleInputSentence(boolean value) optional bool shuffle_input_sentence = 19 [default = true];
      SentencepieceModel.TrainerSpec.Builder clearShuffleInputSentence() optional bool shuffle_input_sentence = 19 [default = true];
      boolean hasMiningSentenceSize()
      Maximum size of sentences to make seed sentence pieces.
      Extended suffix array is constructed to extract frequent
      sub-strings from the corpus. This uses 20N working space,
      where N is the size of corpus.
      
      optional int32 mining_sentence_size = 12 [deprecated = true];
      int getMiningSentenceSize()
      Maximum size of sentences to make seed sentence pieces.
      Extended suffix array is constructed to extract frequent
      sub-strings from the corpus. This uses 20N working space,
      where N is the size of corpus.
      
      optional int32 mining_sentence_size = 12 [deprecated = true];
      SentencepieceModel.TrainerSpec.Builder setMiningSentenceSize(int value)
      Maximum size of sentences to make seed sentence pieces.
      Extended suffix array is constructed to extract frequent
      sub-strings from the corpus. This uses 20N working space,
      where N is the size of corpus.
      
      optional int32 mining_sentence_size = 12 [deprecated = true];
      SentencepieceModel.TrainerSpec.Builder clearMiningSentenceSize()
      Maximum size of sentences to make seed sentence pieces.
      Extended suffix array is constructed to extract frequent
      sub-strings from the corpus. This uses 20N working space,
      where N is the size of corpus.
      
      optional int32 mining_sentence_size = 12 [deprecated = true];
      boolean hasTrainingSentenceSize()
      Maximum size of sentences to train sentence pieces.
      
      optional int32 training_sentence_size = 13 [deprecated = true];
      int getTrainingSentenceSize()
      Maximum size of sentences to train sentence pieces.
      
      optional int32 training_sentence_size = 13 [deprecated = true];
      SentencepieceModel.TrainerSpec.Builder setTrainingSentenceSize(int value)
      Maximum size of sentences to train sentence pieces.
      
      optional int32 training_sentence_size = 13 [deprecated = true];
      SentencepieceModel.TrainerSpec.Builder clearTrainingSentenceSize()
      Maximum size of sentences to train sentence pieces.
      
      optional int32 training_sentence_size = 13 [deprecated = true];
      boolean hasSeedSentencepieceSize()
      The size of seed sentencepieces.
      `seed_sentencepiece_size` must be larger than `vocab_size`.
      
      optional int32 seed_sentencepiece_size = 14 [default = 1000000];
      int getSeedSentencepieceSize()
      The size of seed sentencepieces.
      `seed_sentencepiece_size` must be larger than `vocab_size`.
      
      optional int32 seed_sentencepiece_size = 14 [default = 1000000];
      SentencepieceModel.TrainerSpec.Builder setSeedSentencepieceSize(int value)
      The size of seed sentencepieces.
      `seed_sentencepiece_size` must be larger than `vocab_size`.
      
      optional int32 seed_sentencepiece_size = 14 [default = 1000000];
      SentencepieceModel.TrainerSpec.Builder clearSeedSentencepieceSize()
      The size of seed sentencepieces.
      `seed_sentencepiece_size` must be larger than `vocab_size`.
      
      optional int32 seed_sentencepiece_size = 14 [default = 1000000];
      boolean hasShrinkingFactor()
      In every EM sub-iterations, keeps top
      `shrinking_factor` * `current sentencepieces size` with respect to
      the loss of the sentence piece. This value should be smaller than 1.0.
      
      optional float shrinking_factor = 15 [default = 0.75];
      float getShrinkingFactor()
      In every EM sub-iterations, keeps top
      `shrinking_factor` * `current sentencepieces size` with respect to
      the loss of the sentence piece. This value should be smaller than 1.0.
      
      optional float shrinking_factor = 15 [default = 0.75];
      SentencepieceModel.TrainerSpec.Builder setShrinkingFactor(float value)
      In every EM sub-iterations, keeps top
      `shrinking_factor` * `current sentencepieces size` with respect to
      the loss of the sentence piece. This value should be smaller than 1.0.
      
      optional float shrinking_factor = 15 [default = 0.75];
      SentencepieceModel.TrainerSpec.Builder clearShrinkingFactor()
      In every EM sub-iterations, keeps top
      `shrinking_factor` * `current sentencepieces size` with respect to
      the loss of the sentence piece. This value should be smaller than 1.0.
      
      optional float shrinking_factor = 15 [default = 0.75];
      boolean hasMaxSentenceLength()
      The maximum sentence length in byte. The sentences with the length
      larger than `max_sentence_length` is simply ignored.
      Longer input tends to bring the following risks:
       * Overflow during EM training (unigram language model only)
       * Performance drop because of O(n log n) cost in BPE.
      
      optional int32 max_sentence_length = 18 [default = 4192];
      int getMaxSentenceLength()
      The maximum sentence length in byte. The sentences with the length
      larger than `max_sentence_length` is simply ignored.
      Longer input tends to bring the following risks:
       * Overflow during EM training (unigram language model only)
       * Performance drop because of O(n log n) cost in BPE.
      
      optional int32 max_sentence_length = 18 [default = 4192];
      SentencepieceModel.TrainerSpec.Builder setMaxSentenceLength(int value)
      The maximum sentence length in byte. The sentences with the length
      larger than `max_sentence_length` is simply ignored.
      Longer input tends to bring the following risks:
       * Overflow during EM training (unigram language model only)
       * Performance drop because of O(n log n) cost in BPE.
      
      optional int32 max_sentence_length = 18 [default = 4192];
      SentencepieceModel.TrainerSpec.Builder clearMaxSentenceLength()
      The maximum sentence length in byte. The sentences with the length
      larger than `max_sentence_length` is simply ignored.
      Longer input tends to bring the following risks:
       * Overflow during EM training (unigram language model only)
       * Performance drop because of O(n log n) cost in BPE.
      
      optional int32 max_sentence_length = 18 [default = 4192];
      boolean hasNumThreads()
      Number of threads in the training.
      
      optional int32 num_threads = 16 [default = 16];
      int getNumThreads()
      Number of threads in the training.
      
      optional int32 num_threads = 16 [default = 16];
      SentencepieceModel.TrainerSpec.Builder setNumThreads(int value)
      Number of threads in the training.
      
      optional int32 num_threads = 16 [default = 16];
      SentencepieceModel.TrainerSpec.Builder clearNumThreads()
      Number of threads in the training.
      
      optional int32 num_threads = 16 [default = 16];
      boolean hasNumSubIterations()
      Number of EM sub iterations.
      
      optional int32 num_sub_iterations = 17 [default = 2];
      int getNumSubIterations()
      Number of EM sub iterations.
      
      optional int32 num_sub_iterations = 17 [default = 2];
      SentencepieceModel.TrainerSpec.Builder setNumSubIterations(int value)
      Number of EM sub iterations.
      
      optional int32 num_sub_iterations = 17 [default = 2];
      SentencepieceModel.TrainerSpec.Builder clearNumSubIterations()
      Number of EM sub iterations.
      
      optional int32 num_sub_iterations = 17 [default = 2];
      boolean hasMaxSentencepieceLength()
      /////////////////////////////////////////////////////////////////
      SentencePiece parameters which control the shapes of sentence piece.
      
      Maximum length of sentencepiece.
      
      optional int32 max_sentencepiece_length = 20 [default = 16];
      int getMaxSentencepieceLength()
      /////////////////////////////////////////////////////////////////
      SentencePiece parameters which control the shapes of sentence piece.
      
      Maximum length of sentencepiece.
      
      optional int32 max_sentencepiece_length = 20 [default = 16];
      SentencepieceModel.TrainerSpec.Builder setMaxSentencepieceLength(int value)
      /////////////////////////////////////////////////////////////////
      SentencePiece parameters which control the shapes of sentence piece.
      
      Maximum length of sentencepiece.
      
      optional int32 max_sentencepiece_length = 20 [default = 16];
      SentencepieceModel.TrainerSpec.Builder clearMaxSentencepieceLength()
      /////////////////////////////////////////////////////////////////
      SentencePiece parameters which control the shapes of sentence piece.
      
      Maximum length of sentencepiece.
      
      optional int32 max_sentencepiece_length = 20 [default = 16];
      boolean hasSplitByUnicodeScript()
      Uses Unicode script to split sentence pieces.
      When `split_by_unicode_script` is true, we do not allow sentence piece to
      include multiple Unicode scripts, e.g. "F1" is not a valid piece.
      Exception: CJ characters (Hiragana/Katakana/Han) are all handled
      as one script type, since Japanese word can consist of multiple scripts.
      This exception is always applied regardless of the accept-language
      parameter.
      
      optional bool split_by_unicode_script = 21 [default = true];
      boolean getSplitByUnicodeScript()
      Uses Unicode script to split sentence pieces.
      When `split_by_unicode_script` is true, we do not allow sentence piece to
      include multiple Unicode scripts, e.g. "F1" is not a valid piece.
      Exception: CJ characters (Hiragana/Katakana/Han) are all handled
      as one script type, since Japanese word can consist of multiple scripts.
      This exception is always applied regardless of the accept-language
      parameter.
      
      optional bool split_by_unicode_script = 21 [default = true];
      SentencepieceModel.TrainerSpec.Builder setSplitByUnicodeScript(boolean value)
      Uses Unicode script to split sentence pieces.
      When `split_by_unicode_script` is true, we do not allow sentence piece to
      include multiple Unicode scripts, e.g. "F1" is not a valid piece.
      Exception: CJ characters (Hiragana/Katakana/Han) are all handled
      as one script type, since Japanese word can consist of multiple scripts.
      This exception is always applied regardless of the accept-language
      parameter.
      
      optional bool split_by_unicode_script = 21 [default = true];
      SentencepieceModel.TrainerSpec.Builder clearSplitByUnicodeScript()
      Uses Unicode script to split sentence pieces.
      When `split_by_unicode_script` is true, we do not allow sentence piece to
      include multiple Unicode scripts, e.g. "F1" is not a valid piece.
      Exception: CJ characters (Hiragana/Katakana/Han) are all handled
      as one script type, since Japanese word can consist of multiple scripts.
      This exception is always applied regardless of the accept-language
      parameter.
      
      optional bool split_by_unicode_script = 21 [default = true];
      boolean hasSplitByNumber()
      When `split_by_number` is true, put a boundary between number and
      non-number transition. If we want to treat "F1" is one token, set this flag
      to be false.
      
      optional bool split_by_number = 23 [default = true];
      boolean getSplitByNumber()
      When `split_by_number` is true, put a boundary between number and
      non-number transition. If we want to treat "F1" is one token, set this flag
      to be false.
      
      optional bool split_by_number = 23 [default = true];
      SentencepieceModel.TrainerSpec.Builder setSplitByNumber(boolean value)
      When `split_by_number` is true, put a boundary between number and
      non-number transition. If we want to treat "F1" is one token, set this flag
      to be false.
      
      optional bool split_by_number = 23 [default = true];
      SentencepieceModel.TrainerSpec.Builder clearSplitByNumber()
      When `split_by_number` is true, put a boundary between number and
      non-number transition. If we want to treat "F1" is one token, set this flag
      to be false.
      
      optional bool split_by_number = 23 [default = true];
      boolean hasSplitByWhitespace()
      Use a white space to split sentence pieces.
      When `split_by_whitespace` is false, we may have the piece containing
      a white space in the middle. e.g., "in_the".
      
      optional bool split_by_whitespace = 22 [default = true];
      boolean getSplitByWhitespace()
      Use a white space to split sentence pieces.
      When `split_by_whitespace` is false, we may have the piece containing
      a white space in the middle. e.g., "in_the".
      
      optional bool split_by_whitespace = 22 [default = true];
      SentencepieceModel.TrainerSpec.Builder setSplitByWhitespace(boolean value)
      Use a white space to split sentence pieces.
      When `split_by_whitespace` is false, we may have the piece containing
      a white space in the middle. e.g., "in_the".
      
      optional bool split_by_whitespace = 22 [default = true];
      SentencepieceModel.TrainerSpec.Builder clearSplitByWhitespace()
      Use a white space to split sentence pieces.
      When `split_by_whitespace` is false, we may have the piece containing
      a white space in the middle. e.g., "in_the".
      
      optional bool split_by_whitespace = 22 [default = true];
      boolean hasTreatWhitespaceAsSuffix()
      Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
      hello_. When `treat_whitespace_as_suffix` is true,
      NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
      of sentence.
      
      optional bool treat_whitespace_as_suffix = 24 [default = false];
      boolean getTreatWhitespaceAsSuffix()
      Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
      hello_. When `treat_whitespace_as_suffix` is true,
      NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
      of sentence.
      
      optional bool treat_whitespace_as_suffix = 24 [default = false];
      SentencepieceModel.TrainerSpec.Builder setTreatWhitespaceAsSuffix(boolean value)
      Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
      hello_. When `treat_whitespace_as_suffix` is true,
      NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
      of sentence.
      
      optional bool treat_whitespace_as_suffix = 24 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearTreatWhitespaceAsSuffix()
      Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
      hello_. When `treat_whitespace_as_suffix` is true,
      NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
      of sentence.
      
      optional bool treat_whitespace_as_suffix = 24 [default = false];
      boolean hasAllowWhitespaceOnlyPieces()
      Allows pieces that only contain whitespaces instead of appearing only as
      prefix or suffix of other pieces.
      
      optional bool allow_whitespace_only_pieces = 26 [default = false];
      boolean getAllowWhitespaceOnlyPieces()
      Allows pieces that only contain whitespaces instead of appearing only as
      prefix or suffix of other pieces.
      
      optional bool allow_whitespace_only_pieces = 26 [default = false];
      SentencepieceModel.TrainerSpec.Builder setAllowWhitespaceOnlyPieces(boolean value)
      Allows pieces that only contain whitespaces instead of appearing only as
      prefix or suffix of other pieces.
      
      optional bool allow_whitespace_only_pieces = 26 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearAllowWhitespaceOnlyPieces()
      Allows pieces that only contain whitespaces instead of appearing only as
      prefix or suffix of other pieces.
      
      optional bool allow_whitespace_only_pieces = 26 [default = false];
      boolean hasSplitDigits()
      Split all digits (0-9) into separate pieces.
      
      optional bool split_digits = 25 [default = false];
      boolean getSplitDigits()
      Split all digits (0-9) into separate pieces.
      
      optional bool split_digits = 25 [default = false];
      SentencepieceModel.TrainerSpec.Builder setSplitDigits(boolean value)
      Split all digits (0-9) into separate pieces.
      
      optional bool split_digits = 25 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearSplitDigits()
      Split all digits (0-9) into separate pieces.
      
      optional bool split_digits = 25 [default = false];
      boolean hasPretokenizationDelimiter()
      Defines the pre-tokenization delimiter.
      When specified, no pieces crossing this delimiter is not included
      in the vocab. Then the delimiter string is virtually ignored
      during the training. This field can allows constraints on the vocabulary
      selection. Note that this field is available on unigram mode.
      
      optional string pretokenization_delimiter = 53 [default = ""];
      String getPretokenizationDelimiter()
      Defines the pre-tokenization delimiter.
      When specified, no pieces crossing this delimiter is not included
      in the vocab. Then the delimiter string is virtually ignored
      during the training. This field can allows constraints on the vocabulary
      selection. Note that this field is available on unigram mode.
      
      optional string pretokenization_delimiter = 53 [default = ""];
      ByteString getPretokenizationDelimiterBytes()
      Defines the pre-tokenization delimiter.
      When specified, no pieces crossing this delimiter is not included
      in the vocab. Then the delimiter string is virtually ignored
      during the training. This field can allows constraints on the vocabulary
      selection. Note that this field is available on unigram mode.
      
      optional string pretokenization_delimiter = 53 [default = ""];
      SentencepieceModel.TrainerSpec.Builder setPretokenizationDelimiter(String value)
      Defines the pre-tokenization delimiter.
      When specified, no pieces crossing this delimiter is not included
      in the vocab. Then the delimiter string is virtually ignored
      during the training. This field can allows constraints on the vocabulary
      selection. Note that this field is available on unigram mode.
      
      optional string pretokenization_delimiter = 53 [default = ""];
      SentencepieceModel.TrainerSpec.Builder clearPretokenizationDelimiter()
      Defines the pre-tokenization delimiter.
      When specified, no pieces crossing this delimiter is not included
      in the vocab. Then the delimiter string is virtually ignored
      during the training. This field can allows constraints on the vocabulary
      selection. Note that this field is available on unigram mode.
      
      optional string pretokenization_delimiter = 53 [default = ""];
      SentencepieceModel.TrainerSpec.Builder setPretokenizationDelimiterBytes(ByteString value)
      Defines the pre-tokenization delimiter.
      When specified, no pieces crossing this delimiter is not included
      in the vocab. Then the delimiter string is virtually ignored
      during the training. This field can allows constraints on the vocabulary
      selection. Note that this field is available on unigram mode.
      
      optional string pretokenization_delimiter = 53 [default = ""];
      ProtocolStringList getControlSymbolsList()
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      int getControlSymbolsCount()
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      String getControlSymbols(int index)
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      ByteString getControlSymbolsBytes(int index)
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      SentencepieceModel.TrainerSpec.Builder setControlSymbols(int index, String value)
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      SentencepieceModel.TrainerSpec.Builder addControlSymbols(String value)
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      SentencepieceModel.TrainerSpec.Builder addAllControlSymbols(Iterable<String> values)
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      SentencepieceModel.TrainerSpec.Builder clearControlSymbols()
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      SentencepieceModel.TrainerSpec.Builder addControlSymbolsBytes(ByteString value)
      /////////////////////////////////////////////////////////////////
      Vocabulary management
      
      Defines control symbols used as an indicator to
      change the behavior of the decoder. <s> and </s> are pre-defined.
      We can use this field to encode various meta information,
      including language indicator in multilingual model.
      These symbols are not visible to users, but visible to
      the decoder. Note that when the input sentence contains control symbols,
      they are not treated as one token, but segmented into normal pieces.
      Control symbols must be inserted independently from the segmentation.
      
      repeated string control_symbols = 30;
      ProtocolStringList getUserDefinedSymbolsList()
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      int getUserDefinedSymbolsCount()
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      String getUserDefinedSymbols(int index)
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      ByteString getUserDefinedSymbolsBytes(int index)
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      SentencepieceModel.TrainerSpec.Builder setUserDefinedSymbols(int index, String value)
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      SentencepieceModel.TrainerSpec.Builder addUserDefinedSymbols(String value)
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      SentencepieceModel.TrainerSpec.Builder addAllUserDefinedSymbols(Iterable<String> values)
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      SentencepieceModel.TrainerSpec.Builder clearUserDefinedSymbols()
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      SentencepieceModel.TrainerSpec.Builder addUserDefinedSymbolsBytes(ByteString value)
      Defines user defined symbols.
      These symbols are added with extremely high score
      so they are always treated as one unique symbol in any context.
      Typical usage of user_defined_symbols is placeholder for named entities.
      
      repeated string user_defined_symbols = 31;
      boolean hasRequiredChars()
      Defines required characters. Each UTF8 character in this string is included
      in the character set regardless of character_coverage value. Unlike
      user_defined_symbols, these characters have scores based on the frequency
      on input sentences, and the model can form subwords using characters
      in this field.
      
      optional string required_chars = 36;
      String getRequiredChars()
      Defines required characters. Each UTF8 character in this string is included
      in the character set regardless of character_coverage value. Unlike
      user_defined_symbols, these characters have scores based on the frequency
      on input sentences, and the model can form subwords using characters
      in this field.
      
      optional string required_chars = 36;
      ByteString getRequiredCharsBytes()
      Defines required characters. Each UTF8 character in this string is included
      in the character set regardless of character_coverage value. Unlike
      user_defined_symbols, these characters have scores based on the frequency
      on input sentences, and the model can form subwords using characters
      in this field.
      
      optional string required_chars = 36;
      SentencepieceModel.TrainerSpec.Builder setRequiredChars(String value)
      Defines required characters. Each UTF8 character in this string is included
      in the character set regardless of character_coverage value. Unlike
      user_defined_symbols, these characters have scores based on the frequency
      on input sentences, and the model can form subwords using characters
      in this field.
      
      optional string required_chars = 36;
      SentencepieceModel.TrainerSpec.Builder clearRequiredChars()
      Defines required characters. Each UTF8 character in this string is included
      in the character set regardless of character_coverage value. Unlike
      user_defined_symbols, these characters have scores based on the frequency
      on input sentences, and the model can form subwords using characters
      in this field.
      
      optional string required_chars = 36;
      SentencepieceModel.TrainerSpec.Builder setRequiredCharsBytes(ByteString value)
      Defines required characters. Each UTF8 character in this string is included
      in the character set regardless of character_coverage value. Unlike
      user_defined_symbols, these characters have scores based on the frequency
      on input sentences, and the model can form subwords using characters
      in this field.
      
      optional string required_chars = 36;
      boolean hasByteFallback()
      Decomposes unknown pieces into UTF-8 bytes.
      
      optional bool byte_fallback = 35 [default = false];
      boolean getByteFallback()
      Decomposes unknown pieces into UTF-8 bytes.
      
      optional bool byte_fallback = 35 [default = false];
      SentencepieceModel.TrainerSpec.Builder setByteFallback(boolean value)
      Decomposes unknown pieces into UTF-8 bytes.
      
      optional bool byte_fallback = 35 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearByteFallback()
      Decomposes unknown pieces into UTF-8 bytes.
      
      optional bool byte_fallback = 35 [default = false];
      boolean hasVocabularyOutputPieceScore()
      When creating the vocabulary file, defines whether or not to additionally
      output the score for each piece.
      
      optional bool vocabulary_output_piece_score = 32 [default = true];
      boolean getVocabularyOutputPieceScore()
      When creating the vocabulary file, defines whether or not to additionally
      output the score for each piece.
      
      optional bool vocabulary_output_piece_score = 32 [default = true];
      SentencepieceModel.TrainerSpec.Builder setVocabularyOutputPieceScore(boolean value)
      When creating the vocabulary file, defines whether or not to additionally
      output the score for each piece.
      
      optional bool vocabulary_output_piece_score = 32 [default = true];
      SentencepieceModel.TrainerSpec.Builder clearVocabularyOutputPieceScore()
      When creating the vocabulary file, defines whether or not to additionally
      output the score for each piece.
      
      optional bool vocabulary_output_piece_score = 32 [default = true];
      boolean hasHardVocabLimit()
      `vocab_size` is treated as hard limit. Crash if
      the model can not produce the vocab of size `vocab_size`,
      When `hard_vocab_limit` is false, vocab_size is treated
      as soft limit. Note that when model_type=char,
      always assumes hard_vocab_limit = false.
      
      optional bool hard_vocab_limit = 33 [default = true];
      boolean getHardVocabLimit()
      `vocab_size` is treated as hard limit. Crash if
      the model can not produce the vocab of size `vocab_size`,
      When `hard_vocab_limit` is false, vocab_size is treated
      as soft limit. Note that when model_type=char,
      always assumes hard_vocab_limit = false.
      
      optional bool hard_vocab_limit = 33 [default = true];
      SentencepieceModel.TrainerSpec.Builder setHardVocabLimit(boolean value)
      `vocab_size` is treated as hard limit. Crash if
      the model can not produce the vocab of size `vocab_size`,
      When `hard_vocab_limit` is false, vocab_size is treated
      as soft limit. Note that when model_type=char,
      always assumes hard_vocab_limit = false.
      
      optional bool hard_vocab_limit = 33 [default = true];
      SentencepieceModel.TrainerSpec.Builder clearHardVocabLimit()
      `vocab_size` is treated as hard limit. Crash if
      the model can not produce the vocab of size `vocab_size`,
      When `hard_vocab_limit` is false, vocab_size is treated
      as soft limit. Note that when model_type=char,
      always assumes hard_vocab_limit = false.
      
      optional bool hard_vocab_limit = 33 [default = true];
      boolean hasUseAllVocab()
      use all symbols for vocab extraction. This flag is valid
      if model type is either CHAR or WORD
      
      optional bool use_all_vocab = 34 [default = false];
      boolean getUseAllVocab()
      use all symbols for vocab extraction. This flag is valid
      if model type is either CHAR or WORD
      
      optional bool use_all_vocab = 34 [default = false];
      SentencepieceModel.TrainerSpec.Builder setUseAllVocab(boolean value)
      use all symbols for vocab extraction. This flag is valid
      if model type is either CHAR or WORD
      
      optional bool use_all_vocab = 34 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearUseAllVocab()
      use all symbols for vocab extraction. This flag is valid
      if model type is either CHAR or WORD
      
      optional bool use_all_vocab = 34 [default = false];
      boolean hasUnkId()
      /////////////////////////////////////////////////////////////////
      Reserved special meta tokens.
      * -1 is not used.
      * unk_id must not be -1.
      Id must starts with 0 and be contiguous.
      
      optional int32 unk_id = 40 [default = 0];
      int getUnkId()
      /////////////////////////////////////////////////////////////////
      Reserved special meta tokens.
      * -1 is not used.
      * unk_id must not be -1.
      Id must starts with 0 and be contiguous.
      
      optional int32 unk_id = 40 [default = 0];
      SentencepieceModel.TrainerSpec.Builder setUnkId(int value)
      /////////////////////////////////////////////////////////////////
      Reserved special meta tokens.
      * -1 is not used.
      * unk_id must not be -1.
      Id must starts with 0 and be contiguous.
      
      optional int32 unk_id = 40 [default = 0];
      SentencepieceModel.TrainerSpec.Builder clearUnkId()
      /////////////////////////////////////////////////////////////////
      Reserved special meta tokens.
      * -1 is not used.
      * unk_id must not be -1.
      Id must starts with 0 and be contiguous.
      
      optional int32 unk_id = 40 [default = 0];
      boolean hasBosId()
      <s>
      
      optional int32 bos_id = 41 [default = 1];
      int getBosId()
      <s>
      
      optional int32 bos_id = 41 [default = 1];
      SentencepieceModel.TrainerSpec.Builder setBosId(int value)
      <s>
      
      optional int32 bos_id = 41 [default = 1];
      SentencepieceModel.TrainerSpec.Builder clearBosId()
      <s>
      
      optional int32 bos_id = 41 [default = 1];
      boolean hasEosId()
      </s>
      
      optional int32 eos_id = 42 [default = 2];
      int getEosId()
      </s>
      
      optional int32 eos_id = 42 [default = 2];
      SentencepieceModel.TrainerSpec.Builder setEosId(int value)
      </s>
      
      optional int32 eos_id = 42 [default = 2];
      SentencepieceModel.TrainerSpec.Builder clearEosId()
      </s>
      
      optional int32 eos_id = 42 [default = 2];
      boolean hasPadId()
      <pad> (padding)
      
      optional int32 pad_id = 43 [default = -1];
      int getPadId()
      <pad> (padding)
      
      optional int32 pad_id = 43 [default = -1];
      SentencepieceModel.TrainerSpec.Builder setPadId(int value)
      <pad> (padding)
      
      optional int32 pad_id = 43 [default = -1];
      SentencepieceModel.TrainerSpec.Builder clearPadId()
      <pad> (padding)
      
      optional int32 pad_id = 43 [default = -1];
      boolean hasUnkPiece() optional string unk_piece = 45 [default = "<unk>"];
      String getUnkPiece() optional string unk_piece = 45 [default = "<unk>"];
      ByteString getUnkPieceBytes() optional string unk_piece = 45 [default = "<unk>"];
      SentencepieceModel.TrainerSpec.Builder setUnkPiece(String value) optional string unk_piece = 45 [default = "<unk>"];
      SentencepieceModel.TrainerSpec.Builder clearUnkPiece() optional string unk_piece = 45 [default = "<unk>"];
      SentencepieceModel.TrainerSpec.Builder setUnkPieceBytes(ByteString value) optional string unk_piece = 45 [default = "<unk>"];
      boolean hasBosPiece() optional string bos_piece = 46 [default = "<s>"];
      String getBosPiece() optional string bos_piece = 46 [default = "<s>"];
      ByteString getBosPieceBytes() optional string bos_piece = 46 [default = "<s>"];
      SentencepieceModel.TrainerSpec.Builder setBosPiece(String value) optional string bos_piece = 46 [default = "<s>"];
      SentencepieceModel.TrainerSpec.Builder clearBosPiece() optional string bos_piece = 46 [default = "<s>"];
      SentencepieceModel.TrainerSpec.Builder setBosPieceBytes(ByteString value) optional string bos_piece = 46 [default = "<s>"];
      boolean hasEosPiece() optional string eos_piece = 47 [default = "</s>"];
      String getEosPiece() optional string eos_piece = 47 [default = "</s>"];
      ByteString getEosPieceBytes() optional string eos_piece = 47 [default = "</s>"];
      SentencepieceModel.TrainerSpec.Builder setEosPiece(String value) optional string eos_piece = 47 [default = "</s>"];
      SentencepieceModel.TrainerSpec.Builder clearEosPiece() optional string eos_piece = 47 [default = "</s>"];
      SentencepieceModel.TrainerSpec.Builder setEosPieceBytes(ByteString value) optional string eos_piece = 47 [default = "</s>"];
      boolean hasPadPiece() optional string pad_piece = 48 [default = "<pad>"];
      String getPadPiece() optional string pad_piece = 48 [default = "<pad>"];
      ByteString getPadPieceBytes() optional string pad_piece = 48 [default = "<pad>"];
      SentencepieceModel.TrainerSpec.Builder setPadPiece(String value) optional string pad_piece = 48 [default = "<pad>"];
      SentencepieceModel.TrainerSpec.Builder clearPadPiece() optional string pad_piece = 48 [default = "<pad>"];
      SentencepieceModel.TrainerSpec.Builder setPadPieceBytes(ByteString value) optional string pad_piece = 48 [default = "<pad>"];
      boolean hasUnkSurface()
      Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
      since this character can be useful both for user and
      developer. We can easily figure out that <unk> is emitted.
      
      optional string unk_surface = 44 [default = " \342\201\207 "];
      String getUnkSurface()
      Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
      since this character can be useful both for user and
      developer. We can easily figure out that <unk> is emitted.
      
      optional string unk_surface = 44 [default = " \342\201\207 "];
      ByteString getUnkSurfaceBytes()
      Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
      since this character can be useful both for user and
      developer. We can easily figure out that <unk> is emitted.
      
      optional string unk_surface = 44 [default = " \342\201\207 "];
      SentencepieceModel.TrainerSpec.Builder setUnkSurface(String value)
      Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
      since this character can be useful both for user and
      developer. We can easily figure out that <unk> is emitted.
      
      optional string unk_surface = 44 [default = " \342\201\207 "];
      SentencepieceModel.TrainerSpec.Builder clearUnkSurface()
      Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
      since this character can be useful both for user and
      developer. We can easily figure out that <unk> is emitted.
      
      optional string unk_surface = 44 [default = " \342\201\207 "];
      SentencepieceModel.TrainerSpec.Builder setUnkSurfaceBytes(ByteString value)
      Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
      since this character can be useful both for user and
      developer. We can easily figure out that <unk> is emitted.
      
      optional string unk_surface = 44 [default = " \342\201\207 "];
      boolean hasTrainExtremelyLargeCorpus()
      Increase bit depth to allow unigram model training on large
      (>10M sentences) corpora. A Side-effect of enabling this flag
      is increased memory usage.
      
      optional bool train_extremely_large_corpus = 49 [default = false];
      boolean getTrainExtremelyLargeCorpus()
      Increase bit depth to allow unigram model training on large
      (>10M sentences) corpora. A Side-effect of enabling this flag
      is increased memory usage.
      
      optional bool train_extremely_large_corpus = 49 [default = false];
      SentencepieceModel.TrainerSpec.Builder setTrainExtremelyLargeCorpus(boolean value)
      Increase bit depth to allow unigram model training on large
      (>10M sentences) corpora. A Side-effect of enabling this flag
      is increased memory usage.
      
      optional bool train_extremely_large_corpus = 49 [default = false];
      SentencepieceModel.TrainerSpec.Builder clearTrainExtremelyLargeCorpus()
      Increase bit depth to allow unigram model training on large
      (>10M sentences) corpora. A Side-effect of enabling this flag
      is increased memory usage.
      
      optional bool train_extremely_large_corpus = 49 [default = false];
      boolean hasSeedSentencepiecesFile()
      Path to a seed sentencepieces file, with one tab-separated
      seed sentencepiece <tab> frequency per line.
      
      optional string seed_sentencepieces_file = 54 [default = ""];
      String getSeedSentencepiecesFile()
      Path to a seed sentencepieces file, with one tab-separated
      seed sentencepiece <tab> frequency per line.
      
      optional string seed_sentencepieces_file = 54 [default = ""];
      ByteString getSeedSentencepiecesFileBytes()
      Path to a seed sentencepieces file, with one tab-separated
      seed sentencepiece <tab> frequency per line.
      
      optional string seed_sentencepieces_file = 54 [default = ""];
      SentencepieceModel.TrainerSpec.Builder setSeedSentencepiecesFile(String value)
      Path to a seed sentencepieces file, with one tab-separated
      seed sentencepiece <tab> frequency per line.
      
      optional string seed_sentencepieces_file = 54 [default = ""];
      SentencepieceModel.TrainerSpec.Builder clearSeedSentencepiecesFile()
      Path to a seed sentencepieces file, with one tab-separated
      seed sentencepiece <tab> frequency per line.
      
      optional string seed_sentencepieces_file = 54 [default = ""];
      SentencepieceModel.TrainerSpec.Builder setSeedSentencepiecesFileBytes(ByteString value)
      Path to a seed sentencepieces file, with one tab-separated
      seed sentencepiece <tab> frequency per line.
      
      optional string seed_sentencepieces_file = 54 [default = ""];
      final SentencepieceModel.TrainerSpec.Builder setUnknownFields(UnknownFieldSet unknownFields)
      final SentencepieceModel.TrainerSpec.Builder mergeUnknownFields(UnknownFieldSet unknownFields)
      • Methods inherited from class com.google.protobuf.GeneratedMessageV3.ExtendableBuilder

        addExtension, clearExtension, getAllFields, getExtension, getExtensionCount, getField, getFieldBuilder, getRepeatedField, getRepeatedFieldBuilder, getRepeatedFieldCount, hasExtension, hasField, newBuilderForField, setExtension
      • Methods inherited from class com.google.protobuf.GeneratedMessageV3.Builder

        getOneofFieldDescriptor, getUnknownFields, hasOneof
      • Methods inherited from class com.google.protobuf.AbstractMessage.Builder

        clear, clearOneof, clone, findInitializationErrors, getInitializationErrorString, mergeUnknownFields, toString
      • Methods inherited from class com.google.protobuf.AbstractMessageLite.Builder

        mergeDelimitedFrom
      • Methods inherited from class com.google.protobuf.Message.Builder

        addRepeatedField, clear, clearField, clearOneof, clone, mergeFrom, mergeUnknownFields, setField, setRepeatedField, setUnknownFields
      • Methods inherited from class com.google.protobuf.MessageLite.Builder

        build, buildPartial
      • Methods inherited from class com.google.protobuf.MessageLiteOrBuilder

        getDefaultInstanceForType
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

    • Method Detail

      • getDescriptor

         final static Descriptors.Descriptor getDescriptor()
      • getInputList

         ProtocolStringList getInputList()
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Returns:

        A list containing the input.

      • getInputCount

         int getInputCount()
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Returns:

        The count of input.

      • getInput

         String getInput(int index)
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Parameters:
        index - The index of the element to return.
        Returns:

        The input at the given index.

      • getInputBytes

         ByteString getInputBytes(int index)
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Parameters:
        index - The index of the value to return.
        Returns:

        The bytes of the input at the given index.

      • setInput

         SentencepieceModel.TrainerSpec.Builder setInput(int index, String value)
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Parameters:
        index - The index to set the value at.
        value - The input to set.
        Returns:

        This builder for chaining.

      • addInput

         SentencepieceModel.TrainerSpec.Builder addInput(String value)
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Parameters:
        value - The input to add.
        Returns:

        This builder for chaining.

      • addAllInput

         SentencepieceModel.TrainerSpec.Builder addAllInput(Iterable<String> values)
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Parameters:
        values - The input to add.
        Returns:

        This builder for chaining.

      • clearInput

         SentencepieceModel.TrainerSpec.Builder clearInput()
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Returns:

        This builder for chaining.

      • addInputBytes

         SentencepieceModel.TrainerSpec.Builder addInputBytes(ByteString value)
        /////////////////////////////////////////////////////////////////
        General parameters
        
        Input corpus files.
         Trainer accepts the following two formats:
         A) Monolingual: plain text, one sentence per line.
         B) Bilingual:   TSV, source sentence <tab> target sentence
         When bilingual data is passed, shared vocabulary model is built.
         Note that the input file must be raw corpus, not a preprocessed corpus.
         Trainer only loads the first `input_sentence_size` sentences specified
         with this parameter.
        
        repeated string input = 1;
        Parameters:
        value - The bytes of the input to add.
        Returns:

        This builder for chaining.

      • hasInputFormat

         boolean hasInputFormat()
        Input corpus format:
        "text": one-sentence-per-line text format (default)
        "tsv":  sentence <tab> freq
        
        optional string input_format = 7;
        Returns:

        Whether the inputFormat field is set.

      • getInputFormat

         String getInputFormat()
        Input corpus format:
        "text": one-sentence-per-line text format (default)
        "tsv":  sentence <tab> freq
        
        optional string input_format = 7;
        Returns:

        The inputFormat.

      • getInputFormatBytes

         ByteString getInputFormatBytes()
        Input corpus format:
        "text": one-sentence-per-line text format (default)
        "tsv":  sentence <tab> freq
        
        optional string input_format = 7;
        Returns:

        The bytes for inputFormat.

      • setInputFormat

         SentencepieceModel.TrainerSpec.Builder setInputFormat(String value)
        Input corpus format:
        "text": one-sentence-per-line text format (default)
        "tsv":  sentence <tab> freq
        
        optional string input_format = 7;
        Parameters:
        value - The inputFormat to set.
        Returns:

        This builder for chaining.

      • setInputFormatBytes

         SentencepieceModel.TrainerSpec.Builder setInputFormatBytes(ByteString value)
        Input corpus format:
        "text": one-sentence-per-line text format (default)
        "tsv":  sentence <tab> freq
        
        optional string input_format = 7;
        Parameters:
        value - The bytes for inputFormat to set.
        Returns:

        This builder for chaining.

      • hasModelPrefix

         boolean hasModelPrefix()
        Output model file prefix.
        <model_prefix>.model and <model_prefix>.vocab are generated.
        
        optional string model_prefix = 2;
        Returns:

        Whether the modelPrefix field is set.

      • getModelPrefix

         String getModelPrefix()
        Output model file prefix.
        <model_prefix>.model and <model_prefix>.vocab are generated.
        
        optional string model_prefix = 2;
        Returns:

        The modelPrefix.

      • getModelPrefixBytes

         ByteString getModelPrefixBytes()
        Output model file prefix.
        <model_prefix>.model and <model_prefix>.vocab are generated.
        
        optional string model_prefix = 2;
        Returns:

        The bytes for modelPrefix.

      • setModelPrefixBytes

         SentencepieceModel.TrainerSpec.Builder setModelPrefixBytes(ByteString value)
        Output model file prefix.
        <model_prefix>.model and <model_prefix>.vocab are generated.
        
        optional string model_prefix = 2;
        Parameters:
        value - The bytes for modelPrefix to set.
        Returns:

        This builder for chaining.

      • hasModelType

         boolean hasModelType()

        optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];

        Returns:

        Whether the modelType field is set.

      • hasVocabSize

         boolean hasVocabSize()
        Vocabulary size. 8k is the default size.
        
        optional int32 vocab_size = 4 [default = 8000];
        Returns:

        Whether the vocabSize field is set.

      • getVocabSize

         int getVocabSize()
        Vocabulary size. 8k is the default size.
        
        optional int32 vocab_size = 4 [default = 8000];
        Returns:

        The vocabSize.

      • getAcceptLanguageList

         ProtocolStringList getAcceptLanguageList()
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Returns:

        A list containing the acceptLanguage.

      • getAcceptLanguageCount

         int getAcceptLanguageCount()
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Returns:

        The count of acceptLanguage.

      • getAcceptLanguage

         String getAcceptLanguage(int index)
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Parameters:
        index - The index of the element to return.
        Returns:

        The acceptLanguage at the given index.

      • getAcceptLanguageBytes

         ByteString getAcceptLanguageBytes(int index)
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Parameters:
        index - The index of the value to return.
        Returns:

        The bytes of the acceptLanguage at the given index.

      • setAcceptLanguage

         SentencepieceModel.TrainerSpec.Builder setAcceptLanguage(int index, String value)
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Parameters:
        index - The index to set the value at.
        value - The acceptLanguage to set.
        Returns:

        This builder for chaining.

      • addAcceptLanguage

         SentencepieceModel.TrainerSpec.Builder addAcceptLanguage(String value)
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Parameters:
        value - The acceptLanguage to add.
        Returns:

        This builder for chaining.

      • addAcceptLanguageBytes

         SentencepieceModel.TrainerSpec.Builder addAcceptLanguageBytes(ByteString value)
        List of the languages this model can accept.
        Since the model is language-agnostic, this field is used as a reference.
        
        repeated string accept_language = 5;
        Parameters:
        value - The bytes of the acceptLanguage to add.
        Returns:

        This builder for chaining.

      • hasSelfTestSampleSize

         boolean hasSelfTestSampleSize()
        Size of self-test samples, which are encoded in the model file.
        
        optional int32 self_test_sample_size = 6 [default = 0];
        Returns:

        Whether the selfTestSampleSize field is set.

      • getSelfTestSampleSize

         int getSelfTestSampleSize()
        Size of self-test samples, which are encoded in the model file.
        
        optional int32 self_test_sample_size = 6 [default = 0];
        Returns:

        The selfTestSampleSize.

      • setSelfTestSampleSize

         SentencepieceModel.TrainerSpec.Builder setSelfTestSampleSize(int value)
        Size of self-test samples, which are encoded in the model file.
        
        optional int32 self_test_sample_size = 6 [default = 0];
        Parameters:
        value - The selfTestSampleSize to set.
        Returns:

        This builder for chaining.

      • hasEnableDifferentialPrivacy

         boolean hasEnableDifferentialPrivacy()
        Whether to use DP version of sentencepiece. Use it with TSV input format
        (requires precomputed word tab counts to work).
        
        optional bool enable_differential_privacy = 50 [default = false];
        Returns:

        Whether the enableDifferentialPrivacy field is set.

      • getEnableDifferentialPrivacy

         boolean getEnableDifferentialPrivacy()
        Whether to use DP version of sentencepiece. Use it with TSV input format
        (requires precomputed word tab counts to work).
        
        optional bool enable_differential_privacy = 50 [default = false];
        Returns:

        The enableDifferentialPrivacy.

      • setEnableDifferentialPrivacy

         SentencepieceModel.TrainerSpec.Builder setEnableDifferentialPrivacy(boolean value)
        Whether to use DP version of sentencepiece. Use it with TSV input format
        (requires precomputed word tab counts to work).
        
        optional bool enable_differential_privacy = 50 [default = false];
        Parameters:
        value - The enableDifferentialPrivacy to set.
        Returns:

        This builder for chaining.

      • hasDifferentialPrivacyNoiseLevel

         boolean hasDifferentialPrivacyNoiseLevel()
        Set these parameters if you need DP version of sentencepiece.
        std of noise to add.
        
        optional float differential_privacy_noise_level = 51 [default = 0];
        Returns:

        Whether the differentialPrivacyNoiseLevel field is set.

      • getDifferentialPrivacyNoiseLevel

         float getDifferentialPrivacyNoiseLevel()
        Set these parameters if you need DP version of sentencepiece.
        std of noise to add.
        
        optional float differential_privacy_noise_level = 51 [default = 0];
        Returns:

        The differentialPrivacyNoiseLevel.

      • setDifferentialPrivacyNoiseLevel

         SentencepieceModel.TrainerSpec.Builder setDifferentialPrivacyNoiseLevel(float value)
        Set these parameters if you need DP version of sentencepiece.
        std of noise to add.
        
        optional float differential_privacy_noise_level = 51 [default = 0];
        Parameters:
        value - The differentialPrivacyNoiseLevel to set.
        Returns:

        This builder for chaining.

      • hasDifferentialPrivacyClippingThreshold

         boolean hasDifferentialPrivacyClippingThreshold()
        Clipping threshold to apply after adding noise. All the words with
        frequency less than this value are dropped.
        
        optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
        Returns:

        Whether the differentialPrivacyClippingThreshold field is set.

      • getDifferentialPrivacyClippingThreshold

         long getDifferentialPrivacyClippingThreshold()
        Clipping threshold to apply after adding noise. All the words with
        frequency less than this value are dropped.
        
        optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
        Returns:

        The differentialPrivacyClippingThreshold.

      • setDifferentialPrivacyClippingThreshold

         SentencepieceModel.TrainerSpec.Builder setDifferentialPrivacyClippingThreshold(long value)
        Clipping threshold to apply after adding noise. All the words with
        frequency less than this value are dropped.
        
        optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];
        Parameters:
        value - The differentialPrivacyClippingThreshold to set.
        Returns:

        This builder for chaining.

      • hasCharacterCoverage

         boolean hasCharacterCoverage()
        /////////////////////////////////////////////////////////////////
        Training parameters.
        
        Uses characters which cover the corpus with the ratio of `chars_coverage`.
        This parameter determines the set of basic Alphabet of sentence piece.
        1.0 - `chars_coverage` characters are treated as UNK.
        See also required_chars field.
        
        optional float character_coverage = 10 [default = 0.9995];
        Returns:

        Whether the characterCoverage field is set.

      • getCharacterCoverage

         float getCharacterCoverage()
        /////////////////////////////////////////////////////////////////
        Training parameters.
        
        Uses characters which cover the corpus with the ratio of `chars_coverage`.
        This parameter determines the set of basic Alphabet of sentence piece.
        1.0 - `chars_coverage` characters are treated as UNK.
        See also required_chars field.
        
        optional float character_coverage = 10 [default = 0.9995];
        Returns:

        The characterCoverage.

      • setCharacterCoverage

         SentencepieceModel.TrainerSpec.Builder setCharacterCoverage(float value)
        /////////////////////////////////////////////////////////////////
        Training parameters.
        
        Uses characters which cover the corpus with the ratio of `chars_coverage`.
        This parameter determines the set of basic Alphabet of sentence piece.
        1.0 - `chars_coverage` characters are treated as UNK.
        See also required_chars field.
        
        optional float character_coverage = 10 [default = 0.9995];
        Parameters:
        value - The characterCoverage to set.
        Returns:

        This builder for chaining.

      • clearCharacterCoverage

         SentencepieceModel.TrainerSpec.Builder clearCharacterCoverage()
        /////////////////////////////////////////////////////////////////
        Training parameters.
        
        Uses characters which cover the corpus with the ratio of `chars_coverage`.
        This parameter determines the set of basic Alphabet of sentence piece.
        1.0 - `chars_coverage` characters are treated as UNK.
        See also required_chars field.
        
        optional float character_coverage = 10 [default = 0.9995];
        Returns:

        This builder for chaining.

      • hasInputSentenceSize

         boolean hasInputSentenceSize()
        Maximum size of sentences the trainer loads from `input` parameter.
        Trainer simply loads the `input` files in sequence.
        It is better to shuffle the input corpus randomly.
        
        optional uint64 input_sentence_size = 11 [default = 0];
        Returns:

        Whether the inputSentenceSize field is set.

      • getInputSentenceSize

         long getInputSentenceSize()
        Maximum size of sentences the trainer loads from `input` parameter.
        Trainer simply loads the `input` files in sequence.
        It is better to shuffle the input corpus randomly.
        
        optional uint64 input_sentence_size = 11 [default = 0];
        Returns:

        The inputSentenceSize.

      • setInputSentenceSize

         SentencepieceModel.TrainerSpec.Builder setInputSentenceSize(long value)
        Maximum size of sentences the trainer loads from `input` parameter.
        Trainer simply loads the `input` files in sequence.
        It is better to shuffle the input corpus randomly.
        
        optional uint64 input_sentence_size = 11 [default = 0];
        Parameters:
        value - The inputSentenceSize to set.
        Returns:

        This builder for chaining.

      • clearInputSentenceSize

         SentencepieceModel.TrainerSpec.Builder clearInputSentenceSize()
        Maximum size of sentences the trainer loads from `input` parameter.
        Trainer simply loads the `input` files in sequence.
        It is better to shuffle the input corpus randomly.
        
        optional uint64 input_sentence_size = 11 [default = 0];
        Returns:

        This builder for chaining.

      • hasShuffleInputSentence

         boolean hasShuffleInputSentence()

        optional bool shuffle_input_sentence = 19 [default = true];

        Returns:

        Whether the shuffleInputSentence field is set.

      • getShuffleInputSentence

         boolean getShuffleInputSentence()

        optional bool shuffle_input_sentence = 19 [default = true];

        Returns:

        The shuffleInputSentence.

      • hasMiningSentenceSize

        @Deprecated() boolean hasMiningSentenceSize()
        Maximum size of sentences to make seed sentence pieces.
        Extended suffix array is constructed to extract frequent
        sub-strings from the corpus. This uses 20N working space,
        where N is the size of corpus.
        
        optional int32 mining_sentence_size = 12 [deprecated = true];
        Returns:

        Whether the miningSentenceSize field is set.

      • getMiningSentenceSize

        @Deprecated() int getMiningSentenceSize()
        Maximum size of sentences to make seed sentence pieces.
        Extended suffix array is constructed to extract frequent
        sub-strings from the corpus. This uses 20N working space,
        where N is the size of corpus.
        
        optional int32 mining_sentence_size = 12 [deprecated = true];
        Returns:

        The miningSentenceSize.

      • setMiningSentenceSize

        @Deprecated() SentencepieceModel.TrainerSpec.Builder setMiningSentenceSize(int value)
        Maximum size of sentences to make seed sentence pieces.
        Extended suffix array is constructed to extract frequent
        sub-strings from the corpus. This uses 20N working space,
        where N is the size of corpus.
        
        optional int32 mining_sentence_size = 12 [deprecated = true];
        Parameters:
        value - The miningSentenceSize to set.
        Returns:

        This builder for chaining.

      • clearMiningSentenceSize

        @Deprecated() SentencepieceModel.TrainerSpec.Builder clearMiningSentenceSize()
        Maximum size of sentences to make seed sentence pieces.
        Extended suffix array is constructed to extract frequent
        sub-strings from the corpus. This uses 20N working space,
        where N is the size of corpus.
        
        optional int32 mining_sentence_size = 12 [deprecated = true];
        Returns:

        This builder for chaining.

      • hasTrainingSentenceSize

        @Deprecated() boolean hasTrainingSentenceSize()
        Maximum size of sentences to train sentence pieces.
        
        optional int32 training_sentence_size = 13 [deprecated = true];
        Returns:

        Whether the trainingSentenceSize field is set.

      • getTrainingSentenceSize

        @Deprecated() int getTrainingSentenceSize()
        Maximum size of sentences to train sentence pieces.
        
        optional int32 training_sentence_size = 13 [deprecated = true];
        Returns:

        The trainingSentenceSize.

      • hasSeedSentencepieceSize

         boolean hasSeedSentencepieceSize()
        The size of seed sentencepieces.
        `seed_sentencepiece_size` must be larger than `vocab_size`.
        
        optional int32 seed_sentencepiece_size = 14 [default = 1000000];
        Returns:

        Whether the seedSentencepieceSize field is set.

      • getSeedSentencepieceSize

         int getSeedSentencepieceSize()
        The size of seed sentencepieces.
        `seed_sentencepiece_size` must be larger than `vocab_size`.
        
        optional int32 seed_sentencepiece_size = 14 [default = 1000000];
        Returns:

        The seedSentencepieceSize.

      • setSeedSentencepieceSize

         SentencepieceModel.TrainerSpec.Builder setSeedSentencepieceSize(int value)
        The size of seed sentencepieces.
        `seed_sentencepiece_size` must be larger than `vocab_size`.
        
        optional int32 seed_sentencepiece_size = 14 [default = 1000000];
        Parameters:
        value - The seedSentencepieceSize to set.
        Returns:

        This builder for chaining.

      • hasShrinkingFactor

         boolean hasShrinkingFactor()
        In every EM sub-iterations, keeps top
        `shrinking_factor` * `current sentencepieces size` with respect to
        the loss of the sentence piece. This value should be smaller than 1.0.
        
        optional float shrinking_factor = 15 [default = 0.75];
        Returns:

        Whether the shrinkingFactor field is set.

      • getShrinkingFactor

         float getShrinkingFactor()
        In every EM sub-iterations, keeps top
        `shrinking_factor` * `current sentencepieces size` with respect to
        the loss of the sentence piece. This value should be smaller than 1.0.
        
        optional float shrinking_factor = 15 [default = 0.75];
        Returns:

        The shrinkingFactor.

      • setShrinkingFactor

         SentencepieceModel.TrainerSpec.Builder setShrinkingFactor(float value)
        In every EM sub-iterations, keeps top
        `shrinking_factor` * `current sentencepieces size` with respect to
        the loss of the sentence piece. This value should be smaller than 1.0.
        
        optional float shrinking_factor = 15 [default = 0.75];
        Parameters:
        value - The shrinkingFactor to set.
        Returns:

        This builder for chaining.

      • clearShrinkingFactor

         SentencepieceModel.TrainerSpec.Builder clearShrinkingFactor()
        In every EM sub-iterations, keeps top
        `shrinking_factor` * `current sentencepieces size` with respect to
        the loss of the sentence piece. This value should be smaller than 1.0.
        
        optional float shrinking_factor = 15 [default = 0.75];
        Returns:

        This builder for chaining.

      • hasMaxSentenceLength

         boolean hasMaxSentenceLength()
        The maximum sentence length in byte. The sentences with the length
        larger than `max_sentence_length` is simply ignored.
        Longer input tends to bring the following risks:
         * Overflow during EM training (unigram language model only)
         * Performance drop because of O(n log n) cost in BPE.
        
        optional int32 max_sentence_length = 18 [default = 4192];
        Returns:

        Whether the maxSentenceLength field is set.

      • getMaxSentenceLength

         int getMaxSentenceLength()
        The maximum sentence length in byte. The sentences with the length
        larger than `max_sentence_length` is simply ignored.
        Longer input tends to bring the following risks:
         * Overflow during EM training (unigram language model only)
         * Performance drop because of O(n log n) cost in BPE.
        
        optional int32 max_sentence_length = 18 [default = 4192];
        Returns:

        The maxSentenceLength.

      • setMaxSentenceLength

         SentencepieceModel.TrainerSpec.Builder setMaxSentenceLength(int value)
        The maximum sentence length in byte. The sentences with the length
        larger than `max_sentence_length` is simply ignored.
        Longer input tends to bring the following risks:
         * Overflow during EM training (unigram language model only)
         * Performance drop because of O(n log n) cost in BPE.
        
        optional int32 max_sentence_length = 18 [default = 4192];
        Parameters:
        value - The maxSentenceLength to set.
        Returns:

        This builder for chaining.

      • clearMaxSentenceLength

         SentencepieceModel.TrainerSpec.Builder clearMaxSentenceLength()
        The maximum sentence length in byte. The sentences with the length
        larger than `max_sentence_length` is simply ignored.
        Longer input tends to bring the following risks:
         * Overflow during EM training (unigram language model only)
         * Performance drop because of O(n log n) cost in BPE.
        
        optional int32 max_sentence_length = 18 [default = 4192];
        Returns:

        This builder for chaining.

      • hasNumThreads

         boolean hasNumThreads()
        Number of threads in the training.
        
        optional int32 num_threads = 16 [default = 16];
        Returns:

        Whether the numThreads field is set.

      • getNumThreads

         int getNumThreads()
        Number of threads in the training.
        
        optional int32 num_threads = 16 [default = 16];
        Returns:

        The numThreads.

      • hasNumSubIterations

         boolean hasNumSubIterations()
        Number of EM sub iterations.
        
        optional int32 num_sub_iterations = 17 [default = 2];
        Returns:

        Whether the numSubIterations field is set.

      • getNumSubIterations

         int getNumSubIterations()
        Number of EM sub iterations.
        
        optional int32 num_sub_iterations = 17 [default = 2];
        Returns:

        The numSubIterations.

      • hasMaxSentencepieceLength

         boolean hasMaxSentencepieceLength()
        /////////////////////////////////////////////////////////////////
        SentencePiece parameters which control the shapes of sentence piece.
        
        Maximum length of sentencepiece.
        
        optional int32 max_sentencepiece_length = 20 [default = 16];
        Returns:

        Whether the maxSentencepieceLength field is set.

      • getMaxSentencepieceLength

         int getMaxSentencepieceLength()
        /////////////////////////////////////////////////////////////////
        SentencePiece parameters which control the shapes of sentence piece.
        
        Maximum length of sentencepiece.
        
        optional int32 max_sentencepiece_length = 20 [default = 16];
        Returns:

        The maxSentencepieceLength.

      • setMaxSentencepieceLength

         SentencepieceModel.TrainerSpec.Builder setMaxSentencepieceLength(int value)
        /////////////////////////////////////////////////////////////////
        SentencePiece parameters which control the shapes of sentence piece.
        
        Maximum length of sentencepiece.
        
        optional int32 max_sentencepiece_length = 20 [default = 16];
        Parameters:
        value - The maxSentencepieceLength to set.
        Returns:

        This builder for chaining.

      • clearMaxSentencepieceLength

         SentencepieceModel.TrainerSpec.Builder clearMaxSentencepieceLength()
        /////////////////////////////////////////////////////////////////
        SentencePiece parameters which control the shapes of sentence piece.
        
        Maximum length of sentencepiece.
        
        optional int32 max_sentencepiece_length = 20 [default = 16];
        Returns:

        This builder for chaining.

      • hasSplitByUnicodeScript

         boolean hasSplitByUnicodeScript()
        Uses Unicode script to split sentence pieces.
        When `split_by_unicode_script` is true, we do not allow sentence piece to
        include multiple Unicode scripts, e.g. "F1" is not a valid piece.
        Exception: CJ characters (Hiragana/Katakana/Han) are all handled
        as one script type, since Japanese word can consist of multiple scripts.
        This exception is always applied regardless of the accept-language
        parameter.
        
        optional bool split_by_unicode_script = 21 [default = true];
        Returns:

        Whether the splitByUnicodeScript field is set.

      • getSplitByUnicodeScript

         boolean getSplitByUnicodeScript()
        Uses Unicode script to split sentence pieces.
        When `split_by_unicode_script` is true, we do not allow sentence piece to
        include multiple Unicode scripts, e.g. "F1" is not a valid piece.
        Exception: CJ characters (Hiragana/Katakana/Han) are all handled
        as one script type, since Japanese word can consist of multiple scripts.
        This exception is always applied regardless of the accept-language
        parameter.
        
        optional bool split_by_unicode_script = 21 [default = true];
        Returns:

        The splitByUnicodeScript.

      • setSplitByUnicodeScript

         SentencepieceModel.TrainerSpec.Builder setSplitByUnicodeScript(boolean value)
        Uses Unicode script to split sentence pieces.
        When `split_by_unicode_script` is true, we do not allow sentence piece to
        include multiple Unicode scripts, e.g. "F1" is not a valid piece.
        Exception: CJ characters (Hiragana/Katakana/Han) are all handled
        as one script type, since Japanese word can consist of multiple scripts.
        This exception is always applied regardless of the accept-language
        parameter.
        
        optional bool split_by_unicode_script = 21 [default = true];
        Parameters:
        value - The splitByUnicodeScript to set.
        Returns:

        This builder for chaining.

      • clearSplitByUnicodeScript

         SentencepieceModel.TrainerSpec.Builder clearSplitByUnicodeScript()
        Uses Unicode script to split sentence pieces.
        When `split_by_unicode_script` is true, we do not allow sentence piece to
        include multiple Unicode scripts, e.g. "F1" is not a valid piece.
        Exception: CJ characters (Hiragana/Katakana/Han) are all handled
        as one script type, since Japanese word can consist of multiple scripts.
        This exception is always applied regardless of the accept-language
        parameter.
        
        optional bool split_by_unicode_script = 21 [default = true];
        Returns:

        This builder for chaining.

      • hasSplitByNumber

         boolean hasSplitByNumber()
        When `split_by_number` is true, put a boundary between number and
        non-number transition. If we want to treat "F1" is one token, set this flag
        to be false.
        
        optional bool split_by_number = 23 [default = true];
        Returns:

        Whether the splitByNumber field is set.

      • getSplitByNumber

         boolean getSplitByNumber()
        When `split_by_number` is true, put a boundary between number and
        non-number transition. If we want to treat "F1" is one token, set this flag
        to be false.
        
        optional bool split_by_number = 23 [default = true];
        Returns:

        The splitByNumber.

      • setSplitByNumber

         SentencepieceModel.TrainerSpec.Builder setSplitByNumber(boolean value)
        When `split_by_number` is true, put a boundary between number and
        non-number transition. If we want to treat "F1" is one token, set this flag
        to be false.
        
        optional bool split_by_number = 23 [default = true];
        Parameters:
        value - The splitByNumber to set.
        Returns:

        This builder for chaining.

      • clearSplitByNumber

         SentencepieceModel.TrainerSpec.Builder clearSplitByNumber()
        When `split_by_number` is true, put a boundary between number and
        non-number transition. If we want to treat "F1" is one token, set this flag
        to be false.
        
        optional bool split_by_number = 23 [default = true];
        Returns:

        This builder for chaining.

      • hasSplitByWhitespace

         boolean hasSplitByWhitespace()
        Use a white space to split sentence pieces.
        When `split_by_whitespace` is false, we may have the piece containing
        a white space in the middle. e.g., "in_the".
        
        optional bool split_by_whitespace = 22 [default = true];
        Returns:

        Whether the splitByWhitespace field is set.

      • getSplitByWhitespace

         boolean getSplitByWhitespace()
        Use a white space to split sentence pieces.
        When `split_by_whitespace` is false, we may have the piece containing
        a white space in the middle. e.g., "in_the".
        
        optional bool split_by_whitespace = 22 [default = true];
        Returns:

        The splitByWhitespace.

      • setSplitByWhitespace

         SentencepieceModel.TrainerSpec.Builder setSplitByWhitespace(boolean value)
        Use a white space to split sentence pieces.
        When `split_by_whitespace` is false, we may have the piece containing
        a white space in the middle. e.g., "in_the".
        
        optional bool split_by_whitespace = 22 [default = true];
        Parameters:
        value - The splitByWhitespace to set.
        Returns:

        This builder for chaining.

      • clearSplitByWhitespace

         SentencepieceModel.TrainerSpec.Builder clearSplitByWhitespace()
        Use a white space to split sentence pieces.
        When `split_by_whitespace` is false, we may have the piece containing
        a white space in the middle. e.g., "in_the".
        
        optional bool split_by_whitespace = 22 [default = true];
        Returns:

        This builder for chaining.

      • hasTreatWhitespaceAsSuffix

         boolean hasTreatWhitespaceAsSuffix()
        Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
        hello_. When `treat_whitespace_as_suffix` is true,
        NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
        of sentence.
        
        optional bool treat_whitespace_as_suffix = 24 [default = false];
        Returns:

        Whether the treatWhitespaceAsSuffix field is set.

      • getTreatWhitespaceAsSuffix

         boolean getTreatWhitespaceAsSuffix()
        Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
        hello_. When `treat_whitespace_as_suffix` is true,
        NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
        of sentence.
        
        optional bool treat_whitespace_as_suffix = 24 [default = false];
        Returns:

        The treatWhitespaceAsSuffix.

      • setTreatWhitespaceAsSuffix

         SentencepieceModel.TrainerSpec.Builder setTreatWhitespaceAsSuffix(boolean value)
        Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
        hello_. When `treat_whitespace_as_suffix` is true,
        NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
        of sentence.
        
        optional bool treat_whitespace_as_suffix = 24 [default = false];
        Parameters:
        value - The treatWhitespaceAsSuffix to set.
        Returns:

        This builder for chaining.

      • clearTreatWhitespaceAsSuffix

         SentencepieceModel.TrainerSpec.Builder clearTreatWhitespaceAsSuffix()
        Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello =>
        hello_. When `treat_whitespace_as_suffix` is true,
        NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end
        of sentence.
        
        optional bool treat_whitespace_as_suffix = 24 [default = false];
        Returns:

        This builder for chaining.

      • hasAllowWhitespaceOnlyPieces

         boolean hasAllowWhitespaceOnlyPieces()
        Allows pieces that only contain whitespaces instead of appearing only as
        prefix or suffix of other pieces.
        
        optional bool allow_whitespace_only_pieces = 26 [default = false];
        Returns:

        Whether the allowWhitespaceOnlyPieces field is set.

      • getAllowWhitespaceOnlyPieces

         boolean getAllowWhitespaceOnlyPieces()
        Allows pieces that only contain whitespaces instead of appearing only as
        prefix or suffix of other pieces.
        
        optional bool allow_whitespace_only_pieces = 26 [default = false];
        Returns:

        The allowWhitespaceOnlyPieces.

      • setAllowWhitespaceOnlyPieces

         SentencepieceModel.TrainerSpec.Builder setAllowWhitespaceOnlyPieces(boolean value)
        Allows pieces that only contain whitespaces instead of appearing only as
        prefix or suffix of other pieces.
        
        optional bool allow_whitespace_only_pieces = 26 [default = false];
        Parameters:
        value - The allowWhitespaceOnlyPieces to set.
        Returns:

        This builder for chaining.

      • hasSplitDigits

         boolean hasSplitDigits()
        Split all digits (0-9) into separate pieces.
        
        optional bool split_digits = 25 [default = false];
        Returns:

        Whether the splitDigits field is set.

      • getSplitDigits

         boolean getSplitDigits()
        Split all digits (0-9) into separate pieces.
        
        optional bool split_digits = 25 [default = false];
        Returns:

        The splitDigits.

      • setSplitDigits

         SentencepieceModel.TrainerSpec.Builder setSplitDigits(boolean value)
        Split all digits (0-9) into separate pieces.
        
        optional bool split_digits = 25 [default = false];
        Parameters:
        value - The splitDigits to set.
        Returns:

        This builder for chaining.

      • hasPretokenizationDelimiter

         boolean hasPretokenizationDelimiter()
        Defines the pre-tokenization delimiter.
        When specified, no pieces crossing this delimiter is not included
        in the vocab. Then the delimiter string is virtually ignored
        during the training. This field can allows constraints on the vocabulary
        selection. Note that this field is available on unigram mode.
        
        optional string pretokenization_delimiter = 53 [default = ""];
        Returns:

        Whether the pretokenizationDelimiter field is set.

      • getPretokenizationDelimiter

         String getPretokenizationDelimiter()
        Defines the pre-tokenization delimiter.
        When specified, no pieces crossing this delimiter is not included
        in the vocab. Then the delimiter string is virtually ignored
        during the training. This field can allows constraints on the vocabulary
        selection. Note that this field is available on unigram mode.
        
        optional string pretokenization_delimiter = 53 [default = ""];
        Returns:

        The pretokenizationDelimiter.

      • getPretokenizationDelimiterBytes

         ByteString getPretokenizationDelimiterBytes()
        Defines the pre-tokenization delimiter.
        When specified, no pieces crossing this delimiter is not included
        in the vocab. Then the delimiter string is virtually ignored
        during the training. This field can allows constraints on the vocabulary
        selection. Note that this field is available on unigram mode.
        
        optional string pretokenization_delimiter = 53 [default = ""];
        Returns:

        The bytes for pretokenizationDelimiter.

      • setPretokenizationDelimiter

         SentencepieceModel.TrainerSpec.Builder setPretokenizationDelimiter(String value)
        Defines the pre-tokenization delimiter.
        When specified, no pieces crossing this delimiter is not included
        in the vocab. Then the delimiter string is virtually ignored
        during the training. This field can allows constraints on the vocabulary
        selection. Note that this field is available on unigram mode.
        
        optional string pretokenization_delimiter = 53 [default = ""];
        Parameters:
        value - The pretokenizationDelimiter to set.
        Returns:

        This builder for chaining.

      • clearPretokenizationDelimiter

         SentencepieceModel.TrainerSpec.Builder clearPretokenizationDelimiter()
        Defines the pre-tokenization delimiter.
        When specified, no pieces crossing this delimiter is not included
        in the vocab. Then the delimiter string is virtually ignored
        during the training. This field can allows constraints on the vocabulary
        selection. Note that this field is available on unigram mode.
        
        optional string pretokenization_delimiter = 53 [default = ""];
        Returns:

        This builder for chaining.

      • setPretokenizationDelimiterBytes

         SentencepieceModel.TrainerSpec.Builder setPretokenizationDelimiterBytes(ByteString value)
        Defines the pre-tokenization delimiter.
        When specified, no pieces crossing this delimiter is not included
        in the vocab. Then the delimiter string is virtually ignored
        during the training. This field can allows constraints on the vocabulary
        selection. Note that this field is available on unigram mode.
        
        optional string pretokenization_delimiter = 53 [default = ""];
        Parameters:
        value - The bytes for pretokenizationDelimiter to set.
        Returns:

        This builder for chaining.

      • getControlSymbolsList

         ProtocolStringList getControlSymbolsList()
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Returns:

        A list containing the controlSymbols.

      • getControlSymbolsCount

         int getControlSymbolsCount()
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Returns:

        The count of controlSymbols.

      • getControlSymbols

         String getControlSymbols(int index)
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Parameters:
        index - The index of the element to return.
        Returns:

        The controlSymbols at the given index.

      • getControlSymbolsBytes

         ByteString getControlSymbolsBytes(int index)
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Parameters:
        index - The index of the value to return.
        Returns:

        The bytes of the controlSymbols at the given index.

      • setControlSymbols

         SentencepieceModel.TrainerSpec.Builder setControlSymbols(int index, String value)
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Parameters:
        index - The index to set the value at.
        value - The controlSymbols to set.
        Returns:

        This builder for chaining.

      • addControlSymbols

         SentencepieceModel.TrainerSpec.Builder addControlSymbols(String value)
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Parameters:
        value - The controlSymbols to add.
        Returns:

        This builder for chaining.

      • addAllControlSymbols

         SentencepieceModel.TrainerSpec.Builder addAllControlSymbols(Iterable<String> values)
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Parameters:
        values - The controlSymbols to add.
        Returns:

        This builder for chaining.

      • clearControlSymbols

         SentencepieceModel.TrainerSpec.Builder clearControlSymbols()
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Returns:

        This builder for chaining.

      • addControlSymbolsBytes

         SentencepieceModel.TrainerSpec.Builder addControlSymbolsBytes(ByteString value)
        /////////////////////////////////////////////////////////////////
        Vocabulary management
        
        Defines control symbols used as an indicator to
        change the behavior of the decoder. <s> and </s> are pre-defined.
        We can use this field to encode various meta information,
        including language indicator in multilingual model.
        These symbols are not visible to users, but visible to
        the decoder. Note that when the input sentence contains control symbols,
        they are not treated as one token, but segmented into normal pieces.
        Control symbols must be inserted independently from the segmentation.
        
        repeated string control_symbols = 30;
        Parameters:
        value - The bytes of the controlSymbols to add.
        Returns:

        This builder for chaining.

      • getUserDefinedSymbolsList

         ProtocolStringList getUserDefinedSymbolsList()
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Returns:

        A list containing the userDefinedSymbols.

      • getUserDefinedSymbolsCount

         int getUserDefinedSymbolsCount()
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Returns:

        The count of userDefinedSymbols.

      • getUserDefinedSymbols

         String getUserDefinedSymbols(int index)
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Parameters:
        index - The index of the element to return.
        Returns:

        The userDefinedSymbols at the given index.

      • getUserDefinedSymbolsBytes

         ByteString getUserDefinedSymbolsBytes(int index)
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Parameters:
        index - The index of the value to return.
        Returns:

        The bytes of the userDefinedSymbols at the given index.

      • setUserDefinedSymbols

         SentencepieceModel.TrainerSpec.Builder setUserDefinedSymbols(int index, String value)
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Parameters:
        index - The index to set the value at.
        value - The userDefinedSymbols to set.
        Returns:

        This builder for chaining.

      • addUserDefinedSymbols

         SentencepieceModel.TrainerSpec.Builder addUserDefinedSymbols(String value)
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Parameters:
        value - The userDefinedSymbols to add.
        Returns:

        This builder for chaining.

      • addAllUserDefinedSymbols

         SentencepieceModel.TrainerSpec.Builder addAllUserDefinedSymbols(Iterable<String> values)
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Parameters:
        values - The userDefinedSymbols to add.
        Returns:

        This builder for chaining.

      • clearUserDefinedSymbols

         SentencepieceModel.TrainerSpec.Builder clearUserDefinedSymbols()
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Returns:

        This builder for chaining.

      • addUserDefinedSymbolsBytes

         SentencepieceModel.TrainerSpec.Builder addUserDefinedSymbolsBytes(ByteString value)
        Defines user defined symbols.
        These symbols are added with extremely high score
        so they are always treated as one unique symbol in any context.
        Typical usage of user_defined_symbols is placeholder for named entities.
        
        repeated string user_defined_symbols = 31;
        Parameters:
        value - The bytes of the userDefinedSymbols to add.
        Returns:

        This builder for chaining.

      • hasRequiredChars

         boolean hasRequiredChars()
        Defines required characters. Each UTF8 character in this string is included
        in the character set regardless of character_coverage value. Unlike
        user_defined_symbols, these characters have scores based on the frequency
        on input sentences, and the model can form subwords using characters
        in this field.
        
        optional string required_chars = 36;
        Returns:

        Whether the requiredChars field is set.

      • getRequiredChars

         String getRequiredChars()
        Defines required characters. Each UTF8 character in this string is included
        in the character set regardless of character_coverage value. Unlike
        user_defined_symbols, these characters have scores based on the frequency
        on input sentences, and the model can form subwords using characters
        in this field.
        
        optional string required_chars = 36;
        Returns:

        The requiredChars.

      • getRequiredCharsBytes

         ByteString getRequiredCharsBytes()
        Defines required characters. Each UTF8 character in this string is included
        in the character set regardless of character_coverage value. Unlike
        user_defined_symbols, these characters have scores based on the frequency
        on input sentences, and the model can form subwords using characters
        in this field.
        
        optional string required_chars = 36;
        Returns:

        The bytes for requiredChars.

      • setRequiredChars

         SentencepieceModel.TrainerSpec.Builder setRequiredChars(String value)
        Defines required characters. Each UTF8 character in this string is included
        in the character set regardless of character_coverage value. Unlike
        user_defined_symbols, these characters have scores based on the frequency
        on input sentences, and the model can form subwords using characters
        in this field.
        
        optional string required_chars = 36;
        Parameters:
        value - The requiredChars to set.
        Returns:

        This builder for chaining.

      • clearRequiredChars

         SentencepieceModel.TrainerSpec.Builder clearRequiredChars()
        Defines required characters. Each UTF8 character in this string is included
        in the character set regardless of character_coverage value. Unlike
        user_defined_symbols, these characters have scores based on the frequency
        on input sentences, and the model can form subwords using characters
        in this field.
        
        optional string required_chars = 36;
        Returns:

        This builder for chaining.

      • setRequiredCharsBytes

         SentencepieceModel.TrainerSpec.Builder setRequiredCharsBytes(ByteString value)
        Defines required characters. Each UTF8 character in this string is included
        in the character set regardless of character_coverage value. Unlike
        user_defined_symbols, these characters have scores based on the frequency
        on input sentences, and the model can form subwords using characters
        in this field.
        
        optional string required_chars = 36;
        Parameters:
        value - The bytes for requiredChars to set.
        Returns:

        This builder for chaining.

      • hasByteFallback

         boolean hasByteFallback()
        Decomposes unknown pieces into UTF-8 bytes.
        
        optional bool byte_fallback = 35 [default = false];
        Returns:

        Whether the byteFallback field is set.

      • getByteFallback

         boolean getByteFallback()
        Decomposes unknown pieces into UTF-8 bytes.
        
        optional bool byte_fallback = 35 [default = false];
        Returns:

        The byteFallback.

      • setByteFallback

         SentencepieceModel.TrainerSpec.Builder setByteFallback(boolean value)
        Decomposes unknown pieces into UTF-8 bytes.
        
        optional bool byte_fallback = 35 [default = false];
        Parameters:
        value - The byteFallback to set.
        Returns:

        This builder for chaining.

      • hasVocabularyOutputPieceScore

         boolean hasVocabularyOutputPieceScore()
        When creating the vocabulary file, defines whether or not to additionally
        output the score for each piece.
        
        optional bool vocabulary_output_piece_score = 32 [default = true];
        Returns:

        Whether the vocabularyOutputPieceScore field is set.

      • getVocabularyOutputPieceScore

         boolean getVocabularyOutputPieceScore()
        When creating the vocabulary file, defines whether or not to additionally
        output the score for each piece.
        
        optional bool vocabulary_output_piece_score = 32 [default = true];
        Returns:

        The vocabularyOutputPieceScore.

      • setVocabularyOutputPieceScore

         SentencepieceModel.TrainerSpec.Builder setVocabularyOutputPieceScore(boolean value)
        When creating the vocabulary file, defines whether or not to additionally
        output the score for each piece.
        
        optional bool vocabulary_output_piece_score = 32 [default = true];
        Parameters:
        value - The vocabularyOutputPieceScore to set.
        Returns:

        This builder for chaining.

      • hasHardVocabLimit

         boolean hasHardVocabLimit()
        `vocab_size` is treated as hard limit. Crash if
        the model can not produce the vocab of size `vocab_size`,
        When `hard_vocab_limit` is false, vocab_size is treated
        as soft limit. Note that when model_type=char,
        always assumes hard_vocab_limit = false.
        
        optional bool hard_vocab_limit = 33 [default = true];
        Returns:

        Whether the hardVocabLimit field is set.

      • getHardVocabLimit

         boolean getHardVocabLimit()
        `vocab_size` is treated as hard limit. Crash if
        the model can not produce the vocab of size `vocab_size`,
        When `hard_vocab_limit` is false, vocab_size is treated
        as soft limit. Note that when model_type=char,
        always assumes hard_vocab_limit = false.
        
        optional bool hard_vocab_limit = 33 [default = true];
        Returns:

        The hardVocabLimit.

      • setHardVocabLimit

         SentencepieceModel.TrainerSpec.Builder setHardVocabLimit(boolean value)
        `vocab_size` is treated as hard limit. Crash if
        the model can not produce the vocab of size `vocab_size`,
        When `hard_vocab_limit` is false, vocab_size is treated
        as soft limit. Note that when model_type=char,
        always assumes hard_vocab_limit = false.
        
        optional bool hard_vocab_limit = 33 [default = true];
        Parameters:
        value - The hardVocabLimit to set.
        Returns:

        This builder for chaining.

      • clearHardVocabLimit

         SentencepieceModel.TrainerSpec.Builder clearHardVocabLimit()
        `vocab_size` is treated as hard limit. Crash if
        the model can not produce the vocab of size `vocab_size`,
        When `hard_vocab_limit` is false, vocab_size is treated
        as soft limit. Note that when model_type=char,
        always assumes hard_vocab_limit = false.
        
        optional bool hard_vocab_limit = 33 [default = true];
        Returns:

        This builder for chaining.

      • hasUseAllVocab

         boolean hasUseAllVocab()
        use all symbols for vocab extraction. This flag is valid
        if model type is either CHAR or WORD
        
        optional bool use_all_vocab = 34 [default = false];
        Returns:

        Whether the useAllVocab field is set.

      • getUseAllVocab

         boolean getUseAllVocab()
        use all symbols for vocab extraction. This flag is valid
        if model type is either CHAR or WORD
        
        optional bool use_all_vocab = 34 [default = false];
        Returns:

        The useAllVocab.

      • setUseAllVocab

         SentencepieceModel.TrainerSpec.Builder setUseAllVocab(boolean value)
        use all symbols for vocab extraction. This flag is valid
        if model type is either CHAR or WORD
        
        optional bool use_all_vocab = 34 [default = false];
        Parameters:
        value - The useAllVocab to set.
        Returns:

        This builder for chaining.

      • hasUnkId

         boolean hasUnkId()
        /////////////////////////////////////////////////////////////////
        Reserved special meta tokens.
        * -1 is not used.
        * unk_id must not be -1.
        Id must starts with 0 and be contiguous.
        
        optional int32 unk_id = 40 [default = 0];
        Returns:

        Whether the unkId field is set.

      • getUnkId

         int getUnkId()
        /////////////////////////////////////////////////////////////////
        Reserved special meta tokens.
        * -1 is not used.
        * unk_id must not be -1.
        Id must starts with 0 and be contiguous.
        
        optional int32 unk_id = 40 [default = 0];
        Returns:

        The unkId.

      • setUnkId

         SentencepieceModel.TrainerSpec.Builder setUnkId(int value)
        /////////////////////////////////////////////////////////////////
        Reserved special meta tokens.
        * -1 is not used.
        * unk_id must not be -1.
        Id must starts with 0 and be contiguous.
        
        optional int32 unk_id = 40 [default = 0];
        Parameters:
        value - The unkId to set.
        Returns:

        This builder for chaining.

      • clearUnkId

         SentencepieceModel.TrainerSpec.Builder clearUnkId()
        /////////////////////////////////////////////////////////////////
        Reserved special meta tokens.
        * -1 is not used.
        * unk_id must not be -1.
        Id must starts with 0 and be contiguous.
        
        optional int32 unk_id = 40 [default = 0];
        Returns:

        This builder for chaining.

      • hasBosId

         boolean hasBosId()
        <s>
        
        optional int32 bos_id = 41 [default = 1];
        Returns:

        Whether the bosId field is set.

      • getBosId

         int getBosId()
        <s>
        
        optional int32 bos_id = 41 [default = 1];
        Returns:

        The bosId.

      • hasEosId

         boolean hasEosId()
        </s>
        
        optional int32 eos_id = 42 [default = 2];
        Returns:

        Whether the eosId field is set.

      • getEosId

         int getEosId()
        </s>
        
        optional int32 eos_id = 42 [default = 2];
        Returns:

        The eosId.

      • hasPadId

         boolean hasPadId()
        <pad> (padding)
        
        optional int32 pad_id = 43 [default = -1];
        Returns:

        Whether the padId field is set.

      • getPadId

         int getPadId()
        <pad> (padding)
        
        optional int32 pad_id = 43 [default = -1];
        Returns:

        The padId.

      • hasUnkPiece

         boolean hasUnkPiece()

        optional string unk_piece = 45 [default = "<unk>"];

        Returns:

        Whether the unkPiece field is set.

      • getUnkPiece

         String getUnkPiece()

        optional string unk_piece = 45 [default = "<unk>"];

        Returns:

        The unkPiece.

      • getUnkPieceBytes

         ByteString getUnkPieceBytes()

        optional string unk_piece = 45 [default = "<unk>"];

        Returns:

        The bytes for unkPiece.

      • hasBosPiece

         boolean hasBosPiece()

        optional string bos_piece = 46 [default = "<s>"];

        Returns:

        Whether the bosPiece field is set.

      • getBosPiece

         String getBosPiece()

        optional string bos_piece = 46 [default = "<s>"];

        Returns:

        The bosPiece.

      • getBosPieceBytes

         ByteString getBosPieceBytes()

        optional string bos_piece = 46 [default = "<s>"];

        Returns:

        The bytes for bosPiece.

      • hasEosPiece

         boolean hasEosPiece()

        optional string eos_piece = 47 [default = "</s>"];

        Returns:

        Whether the eosPiece field is set.

      • getEosPiece

         String getEosPiece()

        optional string eos_piece = 47 [default = "</s>"];

        Returns:

        The eosPiece.

      • getEosPieceBytes

         ByteString getEosPieceBytes()

        optional string eos_piece = 47 [default = "</s>"];

        Returns:

        The bytes for eosPiece.

      • hasPadPiece

         boolean hasPadPiece()

        optional string pad_piece = 48 [default = "<pad>"];

        Returns:

        Whether the padPiece field is set.

      • getPadPiece

         String getPadPiece()

        optional string pad_piece = 48 [default = "<pad>"];

        Returns:

        The padPiece.

      • getPadPieceBytes

         ByteString getPadPieceBytes()

        optional string pad_piece = 48 [default = "<pad>"];

        Returns:

        The bytes for padPiece.

      • hasUnkSurface

         boolean hasUnkSurface()
        Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
        since this character can be useful both for user and
        developer. We can easily figure out that <unk> is emitted.
        
        optional string unk_surface = 44 [default = " \342\201\207 "];
        Returns:

        Whether the unkSurface field is set.

      • getUnkSurface

         String getUnkSurface()
        Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
        since this character can be useful both for user and
        developer. We can easily figure out that <unk> is emitted.
        
        optional string unk_surface = 44 [default = " \342\201\207 "];
        Returns:

        The unkSurface.

      • getUnkSurfaceBytes

         ByteString getUnkSurfaceBytes()
        Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
        since this character can be useful both for user and
        developer. We can easily figure out that <unk> is emitted.
        
        optional string unk_surface = 44 [default = " \342\201\207 "];
        Returns:

        The bytes for unkSurface.

      • setUnkSurface

         SentencepieceModel.TrainerSpec.Builder setUnkSurface(String value)
        Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
        since this character can be useful both for user and
        developer. We can easily figure out that <unk> is emitted.
        
        optional string unk_surface = 44 [default = " \342\201\207 "];
        Parameters:
        value - The unkSurface to set.
        Returns:

        This builder for chaining.

      • clearUnkSurface

         SentencepieceModel.TrainerSpec.Builder clearUnkSurface()
        Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
        since this character can be useful both for user and
        developer. We can easily figure out that <unk> is emitted.
        
        optional string unk_surface = 44 [default = " \342\201\207 "];
        Returns:

        This builder for chaining.

      • setUnkSurfaceBytes

         SentencepieceModel.TrainerSpec.Builder setUnkSurfaceBytes(ByteString value)
        Encodes <unk> into U+2047 (DOUBLE QUESTION MARK),
        since this character can be useful both for user and
        developer. We can easily figure out that <unk> is emitted.
        
        optional string unk_surface = 44 [default = " \342\201\207 "];
        Parameters:
        value - The bytes for unkSurface to set.
        Returns:

        This builder for chaining.

      • hasTrainExtremelyLargeCorpus

         boolean hasTrainExtremelyLargeCorpus()
        Increase bit depth to allow unigram model training on large
        (>10M sentences) corpora. A Side-effect of enabling this flag
        is increased memory usage.
        
        optional bool train_extremely_large_corpus = 49 [default = false];
        Returns:

        Whether the trainExtremelyLargeCorpus field is set.

      • getTrainExtremelyLargeCorpus

         boolean getTrainExtremelyLargeCorpus()
        Increase bit depth to allow unigram model training on large
        (>10M sentences) corpora. A Side-effect of enabling this flag
        is increased memory usage.
        
        optional bool train_extremely_large_corpus = 49 [default = false];
        Returns:

        The trainExtremelyLargeCorpus.

      • setTrainExtremelyLargeCorpus

         SentencepieceModel.TrainerSpec.Builder setTrainExtremelyLargeCorpus(boolean value)
        Increase bit depth to allow unigram model training on large
        (>10M sentences) corpora. A Side-effect of enabling this flag
        is increased memory usage.
        
        optional bool train_extremely_large_corpus = 49 [default = false];
        Parameters:
        value - The trainExtremelyLargeCorpus to set.
        Returns:

        This builder for chaining.

      • clearTrainExtremelyLargeCorpus

         SentencepieceModel.TrainerSpec.Builder clearTrainExtremelyLargeCorpus()
        Increase bit depth to allow unigram model training on large
        (>10M sentences) corpora. A Side-effect of enabling this flag
        is increased memory usage.
        
        optional bool train_extremely_large_corpus = 49 [default = false];
        Returns:

        This builder for chaining.

      • hasSeedSentencepiecesFile

         boolean hasSeedSentencepiecesFile()
        Path to a seed sentencepieces file, with one tab-separated
        seed sentencepiece <tab> frequency per line.
        
        optional string seed_sentencepieces_file = 54 [default = ""];
        Returns:

        Whether the seedSentencepiecesFile field is set.

      • getSeedSentencepiecesFile

         String getSeedSentencepiecesFile()
        Path to a seed sentencepieces file, with one tab-separated
        seed sentencepiece <tab> frequency per line.
        
        optional string seed_sentencepieces_file = 54 [default = ""];
        Returns:

        The seedSentencepiecesFile.

      • getSeedSentencepiecesFileBytes

         ByteString getSeedSentencepiecesFileBytes()
        Path to a seed sentencepieces file, with one tab-separated
        seed sentencepiece <tab> frequency per line.
        
        optional string seed_sentencepieces_file = 54 [default = ""];
        Returns:

        The bytes for seedSentencepiecesFile.

      • setSeedSentencepiecesFile

         SentencepieceModel.TrainerSpec.Builder setSeedSentencepiecesFile(String value)
        Path to a seed sentencepieces file, with one tab-separated
        seed sentencepiece <tab> frequency per line.
        
        optional string seed_sentencepieces_file = 54 [default = ""];
        Parameters:
        value - The seedSentencepiecesFile to set.
        Returns:

        This builder for chaining.

      • setSeedSentencepiecesFileBytes

         SentencepieceModel.TrainerSpec.Builder setSeedSentencepiecesFileBytes(ByteString value)
        Path to a seed sentencepieces file, with one tab-separated
        seed sentencepiece <tab> frequency per line.
        
        optional string seed_sentencepieces_file = 54 [default = ""];
        Parameters:
        value - The bytes for seedSentencepiecesFile to set.
        Returns:

        This builder for chaining.