Class SentencepieceModel.TrainerSpec.Builder
-
- All Implemented Interfaces:
-
com.google.genai.proto.SentencepieceModel.TrainerSpecOrBuilder,com.google.protobuf.GeneratedMessageV3.ExtendableMessageOrBuilder,com.google.protobuf.Message.Builder,com.google.protobuf.MessageLite.Builder,com.google.protobuf.MessageLiteOrBuilder,com.google.protobuf.MessageOrBuilder,java.lang.Cloneable
public final class SentencepieceModel.TrainerSpec.Builder extends GeneratedMessageV3.ExtendableBuilder<MessageT, BuilderT> implements SentencepieceModel.TrainerSpecOrBuilder
Protobuf typeTrainerSpec encodes a various parameters for SentencePiece training. Next id: 55com.google.genai.proto.TrainerSpec
-
-
Method Summary
Modifier and Type Method Description final static Descriptors.DescriptorgetDescriptor()SentencepieceModel.TrainerSpec.Builderclear()Descriptors.DescriptorgetDescriptorForType()SentencepieceModel.TrainerSpecgetDefaultInstanceForType()SentencepieceModel.TrainerSpecbuild()SentencepieceModel.TrainerSpecbuildPartial()SentencepieceModel.TrainerSpec.Builderclone()SentencepieceModel.TrainerSpec.BuildersetField(Descriptors.FieldDescriptor field, Object value)SentencepieceModel.TrainerSpec.BuilderclearField(Descriptors.FieldDescriptor field)SentencepieceModel.TrainerSpec.BuilderclearOneof(Descriptors.OneofDescriptor oneof)SentencepieceModel.TrainerSpec.BuildersetRepeatedField(Descriptors.FieldDescriptor field, int index, Object value)SentencepieceModel.TrainerSpec.BuilderaddRepeatedField(Descriptors.FieldDescriptor field, Object value)<Type> SentencepieceModel.TrainerSpec.BuildersetExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, Type> extension, Type value)<Type> SentencepieceModel.TrainerSpec.BuildersetExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, List<Type>> extension, int index, Type value)<Type> SentencepieceModel.TrainerSpec.BuilderaddExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, List<Type>> extension, Type value)<T> SentencepieceModel.TrainerSpec.BuilderclearExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, T> extension)SentencepieceModel.TrainerSpec.BuildermergeFrom(Message other)SentencepieceModel.TrainerSpec.BuildermergeFrom(SentencepieceModel.TrainerSpec other)final booleanisInitialized()SentencepieceModel.TrainerSpec.BuildermergeFrom(CodedInputStream input, ExtensionRegistryLite extensionRegistry)ProtocolStringListgetInputList()///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;intgetInputCount()///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;StringgetInput(int index)///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;ByteStringgetInputBytes(int index)///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;SentencepieceModel.TrainerSpec.BuildersetInput(int index, String value)///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;SentencepieceModel.TrainerSpec.BuilderaddInput(String value)///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;SentencepieceModel.TrainerSpec.BuilderaddAllInput(Iterable<String> values)///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;SentencepieceModel.TrainerSpec.BuilderclearInput()///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;SentencepieceModel.TrainerSpec.BuilderaddInputBytes(ByteString value)///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;booleanhasInputFormat()Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;StringgetInputFormat()Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;ByteStringgetInputFormatBytes()Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;SentencepieceModel.TrainerSpec.BuildersetInputFormat(String value)Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;SentencepieceModel.TrainerSpec.BuilderclearInputFormat()Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;SentencepieceModel.TrainerSpec.BuildersetInputFormatBytes(ByteString value)Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;booleanhasModelPrefix()Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;StringgetModelPrefix()Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;ByteStringgetModelPrefixBytes()Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;SentencepieceModel.TrainerSpec.BuildersetModelPrefix(String value)Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;SentencepieceModel.TrainerSpec.BuilderclearModelPrefix()Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;SentencepieceModel.TrainerSpec.BuildersetModelPrefixBytes(ByteString value)Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;booleanhasModelType()optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];SentencepieceModel.TrainerSpec.ModelTypegetModelType()optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];SentencepieceModel.TrainerSpec.BuildersetModelType(SentencepieceModel.TrainerSpec.ModelType value)optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];SentencepieceModel.TrainerSpec.BuilderclearModelType()optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];booleanhasVocabSize()Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];intgetVocabSize()Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];SentencepieceModel.TrainerSpec.BuildersetVocabSize(int value)Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];SentencepieceModel.TrainerSpec.BuilderclearVocabSize()Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];ProtocolStringListgetAcceptLanguageList()List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;intgetAcceptLanguageCount()List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;StringgetAcceptLanguage(int index)List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;ByteStringgetAcceptLanguageBytes(int index)List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;SentencepieceModel.TrainerSpec.BuildersetAcceptLanguage(int index, String value)List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;SentencepieceModel.TrainerSpec.BuilderaddAcceptLanguage(String value)List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;SentencepieceModel.TrainerSpec.BuilderaddAllAcceptLanguage(Iterable<String> values)List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;SentencepieceModel.TrainerSpec.BuilderclearAcceptLanguage()List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;SentencepieceModel.TrainerSpec.BuilderaddAcceptLanguageBytes(ByteString value)List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;booleanhasSelfTestSampleSize()Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];intgetSelfTestSampleSize()Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];SentencepieceModel.TrainerSpec.BuildersetSelfTestSampleSize(int value)Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];SentencepieceModel.TrainerSpec.BuilderclearSelfTestSampleSize()Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];booleanhasEnableDifferentialPrivacy()Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];booleangetEnableDifferentialPrivacy()Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];SentencepieceModel.TrainerSpec.BuildersetEnableDifferentialPrivacy(boolean value)Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];SentencepieceModel.TrainerSpec.BuilderclearEnableDifferentialPrivacy()Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];booleanhasDifferentialPrivacyNoiseLevel()Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];floatgetDifferentialPrivacyNoiseLevel()Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];SentencepieceModel.TrainerSpec.BuildersetDifferentialPrivacyNoiseLevel(float value)Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];SentencepieceModel.TrainerSpec.BuilderclearDifferentialPrivacyNoiseLevel()Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];booleanhasDifferentialPrivacyClippingThreshold()Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];longgetDifferentialPrivacyClippingThreshold()Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];SentencepieceModel.TrainerSpec.BuildersetDifferentialPrivacyClippingThreshold(long value)Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];SentencepieceModel.TrainerSpec.BuilderclearDifferentialPrivacyClippingThreshold()Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];booleanhasCharacterCoverage()///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];floatgetCharacterCoverage()///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];SentencepieceModel.TrainerSpec.BuildersetCharacterCoverage(float value)///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];SentencepieceModel.TrainerSpec.BuilderclearCharacterCoverage()///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];booleanhasInputSentenceSize()Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];longgetInputSentenceSize()Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];SentencepieceModel.TrainerSpec.BuildersetInputSentenceSize(long value)Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];SentencepieceModel.TrainerSpec.BuilderclearInputSentenceSize()Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];booleanhasShuffleInputSentence()optional bool shuffle_input_sentence = 19 [default = true];booleangetShuffleInputSentence()optional bool shuffle_input_sentence = 19 [default = true];SentencepieceModel.TrainerSpec.BuildersetShuffleInputSentence(boolean value)optional bool shuffle_input_sentence = 19 [default = true];SentencepieceModel.TrainerSpec.BuilderclearShuffleInputSentence()optional bool shuffle_input_sentence = 19 [default = true];booleanhasMiningSentenceSize()Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];intgetMiningSentenceSize()Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];SentencepieceModel.TrainerSpec.BuildersetMiningSentenceSize(int value)Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];SentencepieceModel.TrainerSpec.BuilderclearMiningSentenceSize()Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];booleanhasTrainingSentenceSize()Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];intgetTrainingSentenceSize()Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];SentencepieceModel.TrainerSpec.BuildersetTrainingSentenceSize(int value)Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];SentencepieceModel.TrainerSpec.BuilderclearTrainingSentenceSize()Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];booleanhasSeedSentencepieceSize()The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];intgetSeedSentencepieceSize()The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];SentencepieceModel.TrainerSpec.BuildersetSeedSentencepieceSize(int value)The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];SentencepieceModel.TrainerSpec.BuilderclearSeedSentencepieceSize()The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];booleanhasShrinkingFactor()In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];floatgetShrinkingFactor()In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];SentencepieceModel.TrainerSpec.BuildersetShrinkingFactor(float value)In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];SentencepieceModel.TrainerSpec.BuilderclearShrinkingFactor()In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];booleanhasMaxSentenceLength()The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];intgetMaxSentenceLength()The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];SentencepieceModel.TrainerSpec.BuildersetMaxSentenceLength(int value)The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];SentencepieceModel.TrainerSpec.BuilderclearMaxSentenceLength()The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];booleanhasNumThreads()Number of threads in the training.optional int32 num_threads = 16 [default = 16];intgetNumThreads()Number of threads in the training.optional int32 num_threads = 16 [default = 16];SentencepieceModel.TrainerSpec.BuildersetNumThreads(int value)Number of threads in the training.optional int32 num_threads = 16 [default = 16];SentencepieceModel.TrainerSpec.BuilderclearNumThreads()Number of threads in the training.optional int32 num_threads = 16 [default = 16];booleanhasNumSubIterations()Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];intgetNumSubIterations()Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];SentencepieceModel.TrainerSpec.BuildersetNumSubIterations(int value)Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];SentencepieceModel.TrainerSpec.BuilderclearNumSubIterations()Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];booleanhasMaxSentencepieceLength()///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];intgetMaxSentencepieceLength()///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];SentencepieceModel.TrainerSpec.BuildersetMaxSentencepieceLength(int value)///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];SentencepieceModel.TrainerSpec.BuilderclearMaxSentencepieceLength()///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];booleanhasSplitByUnicodeScript()Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];booleangetSplitByUnicodeScript()Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];SentencepieceModel.TrainerSpec.BuildersetSplitByUnicodeScript(boolean value)Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];SentencepieceModel.TrainerSpec.BuilderclearSplitByUnicodeScript()Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];booleanhasSplitByNumber()When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];booleangetSplitByNumber()When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];SentencepieceModel.TrainerSpec.BuildersetSplitByNumber(boolean value)When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];SentencepieceModel.TrainerSpec.BuilderclearSplitByNumber()When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];booleanhasSplitByWhitespace()Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];booleangetSplitByWhitespace()Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];SentencepieceModel.TrainerSpec.BuildersetSplitByWhitespace(boolean value)Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];SentencepieceModel.TrainerSpec.BuilderclearSplitByWhitespace()Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];booleanhasTreatWhitespaceAsSuffix()Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];booleangetTreatWhitespaceAsSuffix()Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];SentencepieceModel.TrainerSpec.BuildersetTreatWhitespaceAsSuffix(boolean value)Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];SentencepieceModel.TrainerSpec.BuilderclearTreatWhitespaceAsSuffix()Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];booleanhasAllowWhitespaceOnlyPieces()Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];booleangetAllowWhitespaceOnlyPieces()Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];SentencepieceModel.TrainerSpec.BuildersetAllowWhitespaceOnlyPieces(boolean value)Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];SentencepieceModel.TrainerSpec.BuilderclearAllowWhitespaceOnlyPieces()Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];booleanhasSplitDigits()Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];booleangetSplitDigits()Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];SentencepieceModel.TrainerSpec.BuildersetSplitDigits(boolean value)Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];SentencepieceModel.TrainerSpec.BuilderclearSplitDigits()Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];booleanhasPretokenizationDelimiter()Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];StringgetPretokenizationDelimiter()Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];ByteStringgetPretokenizationDelimiterBytes()Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];SentencepieceModel.TrainerSpec.BuildersetPretokenizationDelimiter(String value)Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];SentencepieceModel.TrainerSpec.BuilderclearPretokenizationDelimiter()Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];SentencepieceModel.TrainerSpec.BuildersetPretokenizationDelimiterBytes(ByteString value)Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];ProtocolStringListgetControlSymbolsList()///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;intgetControlSymbolsCount()///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;StringgetControlSymbols(int index)///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;ByteStringgetControlSymbolsBytes(int index)///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;SentencepieceModel.TrainerSpec.BuildersetControlSymbols(int index, String value)///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;SentencepieceModel.TrainerSpec.BuilderaddControlSymbols(String value)///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;SentencepieceModel.TrainerSpec.BuilderaddAllControlSymbols(Iterable<String> values)///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;SentencepieceModel.TrainerSpec.BuilderclearControlSymbols()///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;SentencepieceModel.TrainerSpec.BuilderaddControlSymbolsBytes(ByteString value)///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;ProtocolStringListgetUserDefinedSymbolsList()Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;intgetUserDefinedSymbolsCount()Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;StringgetUserDefinedSymbols(int index)Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;ByteStringgetUserDefinedSymbolsBytes(int index)Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;SentencepieceModel.TrainerSpec.BuildersetUserDefinedSymbols(int index, String value)Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;SentencepieceModel.TrainerSpec.BuilderaddUserDefinedSymbols(String value)Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;SentencepieceModel.TrainerSpec.BuilderaddAllUserDefinedSymbols(Iterable<String> values)Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;SentencepieceModel.TrainerSpec.BuilderclearUserDefinedSymbols()Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;SentencepieceModel.TrainerSpec.BuilderaddUserDefinedSymbolsBytes(ByteString value)Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;booleanhasRequiredChars()Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;StringgetRequiredChars()Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;ByteStringgetRequiredCharsBytes()Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;SentencepieceModel.TrainerSpec.BuildersetRequiredChars(String value)Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;SentencepieceModel.TrainerSpec.BuilderclearRequiredChars()Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;SentencepieceModel.TrainerSpec.BuildersetRequiredCharsBytes(ByteString value)Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;booleanhasByteFallback()Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];booleangetByteFallback()Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];SentencepieceModel.TrainerSpec.BuildersetByteFallback(boolean value)Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];SentencepieceModel.TrainerSpec.BuilderclearByteFallback()Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];booleanhasVocabularyOutputPieceScore()When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];booleangetVocabularyOutputPieceScore()When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];SentencepieceModel.TrainerSpec.BuildersetVocabularyOutputPieceScore(boolean value)When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];SentencepieceModel.TrainerSpec.BuilderclearVocabularyOutputPieceScore()When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];booleanhasHardVocabLimit()`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];booleangetHardVocabLimit()`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];SentencepieceModel.TrainerSpec.BuildersetHardVocabLimit(boolean value)`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];SentencepieceModel.TrainerSpec.BuilderclearHardVocabLimit()`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];booleanhasUseAllVocab()use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];booleangetUseAllVocab()use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];SentencepieceModel.TrainerSpec.BuildersetUseAllVocab(boolean value)use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];SentencepieceModel.TrainerSpec.BuilderclearUseAllVocab()use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];booleanhasUnkId()///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];intgetUnkId()///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];SentencepieceModel.TrainerSpec.BuildersetUnkId(int value)///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];SentencepieceModel.TrainerSpec.BuilderclearUnkId()///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];booleanhasBosId()<s>optional int32 bos_id = 41 [default = 1];intgetBosId()<s>optional int32 bos_id = 41 [default = 1];SentencepieceModel.TrainerSpec.BuildersetBosId(int value)<s>optional int32 bos_id = 41 [default = 1];SentencepieceModel.TrainerSpec.BuilderclearBosId()<s>optional int32 bos_id = 41 [default = 1];booleanhasEosId()</s>optional int32 eos_id = 42 [default = 2];intgetEosId()</s>optional int32 eos_id = 42 [default = 2];SentencepieceModel.TrainerSpec.BuildersetEosId(int value)</s>optional int32 eos_id = 42 [default = 2];SentencepieceModel.TrainerSpec.BuilderclearEosId()</s>optional int32 eos_id = 42 [default = 2];booleanhasPadId()<pad> (padding)optional int32 pad_id = 43 [default = -1];intgetPadId()<pad> (padding)optional int32 pad_id = 43 [default = -1];SentencepieceModel.TrainerSpec.BuildersetPadId(int value)<pad> (padding)optional int32 pad_id = 43 [default = -1];SentencepieceModel.TrainerSpec.BuilderclearPadId()<pad> (padding)optional int32 pad_id = 43 [default = -1];booleanhasUnkPiece()optional string unk_piece = 45 [default = "<unk>"];StringgetUnkPiece()optional string unk_piece = 45 [default = "<unk>"];ByteStringgetUnkPieceBytes()optional string unk_piece = 45 [default = "<unk>"];SentencepieceModel.TrainerSpec.BuildersetUnkPiece(String value)optional string unk_piece = 45 [default = "<unk>"];SentencepieceModel.TrainerSpec.BuilderclearUnkPiece()optional string unk_piece = 45 [default = "<unk>"];SentencepieceModel.TrainerSpec.BuildersetUnkPieceBytes(ByteString value)optional string unk_piece = 45 [default = "<unk>"];booleanhasBosPiece()optional string bos_piece = 46 [default = "<s>"];StringgetBosPiece()optional string bos_piece = 46 [default = "<s>"];ByteStringgetBosPieceBytes()optional string bos_piece = 46 [default = "<s>"];SentencepieceModel.TrainerSpec.BuildersetBosPiece(String value)optional string bos_piece = 46 [default = "<s>"];SentencepieceModel.TrainerSpec.BuilderclearBosPiece()optional string bos_piece = 46 [default = "<s>"];SentencepieceModel.TrainerSpec.BuildersetBosPieceBytes(ByteString value)optional string bos_piece = 46 [default = "<s>"];booleanhasEosPiece()optional string eos_piece = 47 [default = "</s>"];StringgetEosPiece()optional string eos_piece = 47 [default = "</s>"];ByteStringgetEosPieceBytes()optional string eos_piece = 47 [default = "</s>"];SentencepieceModel.TrainerSpec.BuildersetEosPiece(String value)optional string eos_piece = 47 [default = "</s>"];SentencepieceModel.TrainerSpec.BuilderclearEosPiece()optional string eos_piece = 47 [default = "</s>"];SentencepieceModel.TrainerSpec.BuildersetEosPieceBytes(ByteString value)optional string eos_piece = 47 [default = "</s>"];booleanhasPadPiece()optional string pad_piece = 48 [default = "<pad>"];StringgetPadPiece()optional string pad_piece = 48 [default = "<pad>"];ByteStringgetPadPieceBytes()optional string pad_piece = 48 [default = "<pad>"];SentencepieceModel.TrainerSpec.BuildersetPadPiece(String value)optional string pad_piece = 48 [default = "<pad>"];SentencepieceModel.TrainerSpec.BuilderclearPadPiece()optional string pad_piece = 48 [default = "<pad>"];SentencepieceModel.TrainerSpec.BuildersetPadPieceBytes(ByteString value)optional string pad_piece = 48 [default = "<pad>"];booleanhasUnkSurface()Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];StringgetUnkSurface()Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];ByteStringgetUnkSurfaceBytes()Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];SentencepieceModel.TrainerSpec.BuildersetUnkSurface(String value)Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];SentencepieceModel.TrainerSpec.BuilderclearUnkSurface()Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];SentencepieceModel.TrainerSpec.BuildersetUnkSurfaceBytes(ByteString value)Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];booleanhasTrainExtremelyLargeCorpus()Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];booleangetTrainExtremelyLargeCorpus()Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];SentencepieceModel.TrainerSpec.BuildersetTrainExtremelyLargeCorpus(boolean value)Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];SentencepieceModel.TrainerSpec.BuilderclearTrainExtremelyLargeCorpus()Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];booleanhasSeedSentencepiecesFile()Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];StringgetSeedSentencepiecesFile()Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];ByteStringgetSeedSentencepiecesFileBytes()Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];SentencepieceModel.TrainerSpec.BuildersetSeedSentencepiecesFile(String value)Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];SentencepieceModel.TrainerSpec.BuilderclearSeedSentencepiecesFile()Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];SentencepieceModel.TrainerSpec.BuildersetSeedSentencepiecesFileBytes(ByteString value)Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];final SentencepieceModel.TrainerSpec.BuildersetUnknownFields(UnknownFieldSet unknownFields)final SentencepieceModel.TrainerSpec.BuildermergeUnknownFields(UnknownFieldSet unknownFields)-
Methods inherited from class com.google.protobuf.GeneratedMessageV3.ExtendableBuilder
addExtension, clearExtension, getAllFields, getExtension, getExtensionCount, getField, getFieldBuilder, getRepeatedField, getRepeatedFieldBuilder, getRepeatedFieldCount, hasExtension, hasField, newBuilderForField, setExtension -
Methods inherited from class com.google.protobuf.GeneratedMessageV3.Builder
getOneofFieldDescriptor, getUnknownFields, hasOneof -
Methods inherited from class com.google.protobuf.AbstractMessage.Builder
clear, clearOneof, clone, findInitializationErrors, getInitializationErrorString, mergeUnknownFields, toString -
Methods inherited from class com.google.protobuf.AbstractMessageLite.Builder
mergeDelimitedFrom -
Methods inherited from class com.google.protobuf.Message.Builder
addRepeatedField, clear, clearField, clearOneof, clone, mergeFrom, mergeUnknownFields, setField, setRepeatedField, setUnknownFields -
Methods inherited from class com.google.protobuf.MessageLite.Builder
build, buildPartial -
Methods inherited from class com.google.protobuf.MessageLiteOrBuilder
getDefaultInstanceForType -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Method Detail
-
getDescriptor
final static Descriptors.Descriptor getDescriptor()
-
getDescriptorForType
Descriptors.Descriptor getDescriptorForType()
-
getDefaultInstanceForType
SentencepieceModel.TrainerSpec getDefaultInstanceForType()
-
build
SentencepieceModel.TrainerSpec build()
-
buildPartial
SentencepieceModel.TrainerSpec buildPartial()
-
setField
SentencepieceModel.TrainerSpec.Builder setField(Descriptors.FieldDescriptor field, Object value)
-
clearField
SentencepieceModel.TrainerSpec.Builder clearField(Descriptors.FieldDescriptor field)
-
clearOneof
SentencepieceModel.TrainerSpec.Builder clearOneof(Descriptors.OneofDescriptor oneof)
-
setRepeatedField
SentencepieceModel.TrainerSpec.Builder setRepeatedField(Descriptors.FieldDescriptor field, int index, Object value)
-
addRepeatedField
SentencepieceModel.TrainerSpec.Builder addRepeatedField(Descriptors.FieldDescriptor field, Object value)
-
setExtension
<Type> SentencepieceModel.TrainerSpec.Builder setExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, Type> extension, Type value)
-
setExtension
<Type> SentencepieceModel.TrainerSpec.Builder setExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, List<Type>> extension, int index, Type value)
-
addExtension
<Type> SentencepieceModel.TrainerSpec.Builder addExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, List<Type>> extension, Type value)
-
clearExtension
<T> SentencepieceModel.TrainerSpec.Builder clearExtension(GeneratedMessage.GeneratedExtension<SentencepieceModel.TrainerSpec, T> extension)
-
mergeFrom
SentencepieceModel.TrainerSpec.Builder mergeFrom(Message other)
-
isInitialized
final boolean isInitialized()
-
mergeFrom
SentencepieceModel.TrainerSpec.Builder mergeFrom(CodedInputStream input, ExtensionRegistryLite extensionRegistry)
-
getInputList
ProtocolStringList getInputList()
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Returns:
A list containing the input.
-
getInputCount
int getInputCount()
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Returns:
The count of input.
-
getInput
String getInput(int index)
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Parameters:
index- The index of the element to return.- Returns:
The input at the given index.
-
getInputBytes
ByteString getInputBytes(int index)
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Parameters:
index- The index of the value to return.- Returns:
The bytes of the input at the given index.
-
setInput
SentencepieceModel.TrainerSpec.Builder setInput(int index, String value)
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Parameters:
index- The index to set the value at.value- The input to set.- Returns:
This builder for chaining.
-
addInput
SentencepieceModel.TrainerSpec.Builder addInput(String value)
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Parameters:
value- The input to add.- Returns:
This builder for chaining.
-
addAllInput
SentencepieceModel.TrainerSpec.Builder addAllInput(Iterable<String> values)
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Parameters:
values- The input to add.- Returns:
This builder for chaining.
-
clearInput
SentencepieceModel.TrainerSpec.Builder clearInput()
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Returns:
This builder for chaining.
-
addInputBytes
SentencepieceModel.TrainerSpec.Builder addInputBytes(ByteString value)
///////////////////////////////////////////////////////////////// General parameters Input corpus files. Trainer accepts the following two formats: A) Monolingual: plain text, one sentence per line. B) Bilingual: TSV, source sentence <tab> target sentence When bilingual data is passed, shared vocabulary model is built. Note that the input file must be raw corpus, not a preprocessed corpus. Trainer only loads the first `input_sentence_size` sentences specified with this parameter.repeated string input = 1;- Parameters:
value- The bytes of the input to add.- Returns:
This builder for chaining.
-
hasInputFormat
boolean hasInputFormat()
Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;- Returns:
Whether the inputFormat field is set.
-
getInputFormat
String getInputFormat()
Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;- Returns:
The inputFormat.
-
getInputFormatBytes
ByteString getInputFormatBytes()
Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;- Returns:
The bytes for inputFormat.
-
setInputFormat
SentencepieceModel.TrainerSpec.Builder setInputFormat(String value)
Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;- Parameters:
value- The inputFormat to set.- Returns:
This builder for chaining.
-
clearInputFormat
SentencepieceModel.TrainerSpec.Builder clearInputFormat()
Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;- Returns:
This builder for chaining.
-
setInputFormatBytes
SentencepieceModel.TrainerSpec.Builder setInputFormatBytes(ByteString value)
Input corpus format: "text": one-sentence-per-line text format (default) "tsv": sentence <tab> freqoptional string input_format = 7;- Parameters:
value- The bytes for inputFormat to set.- Returns:
This builder for chaining.
-
hasModelPrefix
boolean hasModelPrefix()
Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;- Returns:
Whether the modelPrefix field is set.
-
getModelPrefix
String getModelPrefix()
Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;- Returns:
The modelPrefix.
-
getModelPrefixBytes
ByteString getModelPrefixBytes()
Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;- Returns:
The bytes for modelPrefix.
-
setModelPrefix
SentencepieceModel.TrainerSpec.Builder setModelPrefix(String value)
Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;- Parameters:
value- The modelPrefix to set.- Returns:
This builder for chaining.
-
clearModelPrefix
SentencepieceModel.TrainerSpec.Builder clearModelPrefix()
Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;- Returns:
This builder for chaining.
-
setModelPrefixBytes
SentencepieceModel.TrainerSpec.Builder setModelPrefixBytes(ByteString value)
Output model file prefix. <model_prefix>.model and <model_prefix>.vocab are generated.optional string model_prefix = 2;- Parameters:
value- The bytes for modelPrefix to set.- Returns:
This builder for chaining.
-
hasModelType
boolean hasModelType()
optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];- Returns:
Whether the modelType field is set.
-
getModelType
SentencepieceModel.TrainerSpec.ModelType getModelType()
optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];- Returns:
The modelType.
-
setModelType
SentencepieceModel.TrainerSpec.Builder setModelType(SentencepieceModel.TrainerSpec.ModelType value)
optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];- Parameters:
value- The modelType to set.- Returns:
This builder for chaining.
-
clearModelType
SentencepieceModel.TrainerSpec.Builder clearModelType()
optional .com.google.genai.proto.TrainerSpec.ModelType model_type = 3 [default = UNIGRAM];- Returns:
This builder for chaining.
-
hasVocabSize
boolean hasVocabSize()
Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];- Returns:
Whether the vocabSize field is set.
-
getVocabSize
int getVocabSize()
Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];- Returns:
The vocabSize.
-
setVocabSize
SentencepieceModel.TrainerSpec.Builder setVocabSize(int value)
Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];- Parameters:
value- The vocabSize to set.- Returns:
This builder for chaining.
-
clearVocabSize
SentencepieceModel.TrainerSpec.Builder clearVocabSize()
Vocabulary size. 8k is the default size.optional int32 vocab_size = 4 [default = 8000];- Returns:
This builder for chaining.
-
getAcceptLanguageList
ProtocolStringList getAcceptLanguageList()
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Returns:
A list containing the acceptLanguage.
-
getAcceptLanguageCount
int getAcceptLanguageCount()
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Returns:
The count of acceptLanguage.
-
getAcceptLanguage
String getAcceptLanguage(int index)
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Parameters:
index- The index of the element to return.- Returns:
The acceptLanguage at the given index.
-
getAcceptLanguageBytes
ByteString getAcceptLanguageBytes(int index)
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Parameters:
index- The index of the value to return.- Returns:
The bytes of the acceptLanguage at the given index.
-
setAcceptLanguage
SentencepieceModel.TrainerSpec.Builder setAcceptLanguage(int index, String value)
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Parameters:
index- The index to set the value at.value- The acceptLanguage to set.- Returns:
This builder for chaining.
-
addAcceptLanguage
SentencepieceModel.TrainerSpec.Builder addAcceptLanguage(String value)
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Parameters:
value- The acceptLanguage to add.- Returns:
This builder for chaining.
-
addAllAcceptLanguage
SentencepieceModel.TrainerSpec.Builder addAllAcceptLanguage(Iterable<String> values)
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Parameters:
values- The acceptLanguage to add.- Returns:
This builder for chaining.
-
clearAcceptLanguage
SentencepieceModel.TrainerSpec.Builder clearAcceptLanguage()
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Returns:
This builder for chaining.
-
addAcceptLanguageBytes
SentencepieceModel.TrainerSpec.Builder addAcceptLanguageBytes(ByteString value)
List of the languages this model can accept. Since the model is language-agnostic, this field is used as a reference.repeated string accept_language = 5;- Parameters:
value- The bytes of the acceptLanguage to add.- Returns:
This builder for chaining.
-
hasSelfTestSampleSize
boolean hasSelfTestSampleSize()
Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];- Returns:
Whether the selfTestSampleSize field is set.
-
getSelfTestSampleSize
int getSelfTestSampleSize()
Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];- Returns:
The selfTestSampleSize.
-
setSelfTestSampleSize
SentencepieceModel.TrainerSpec.Builder setSelfTestSampleSize(int value)
Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];- Parameters:
value- The selfTestSampleSize to set.- Returns:
This builder for chaining.
-
clearSelfTestSampleSize
SentencepieceModel.TrainerSpec.Builder clearSelfTestSampleSize()
Size of self-test samples, which are encoded in the model file.optional int32 self_test_sample_size = 6 [default = 0];- Returns:
This builder for chaining.
-
hasEnableDifferentialPrivacy
boolean hasEnableDifferentialPrivacy()
Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];- Returns:
Whether the enableDifferentialPrivacy field is set.
-
getEnableDifferentialPrivacy
boolean getEnableDifferentialPrivacy()
Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];- Returns:
The enableDifferentialPrivacy.
-
setEnableDifferentialPrivacy
SentencepieceModel.TrainerSpec.Builder setEnableDifferentialPrivacy(boolean value)
Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];- Parameters:
value- The enableDifferentialPrivacy to set.- Returns:
This builder for chaining.
-
clearEnableDifferentialPrivacy
SentencepieceModel.TrainerSpec.Builder clearEnableDifferentialPrivacy()
Whether to use DP version of sentencepiece. Use it with TSV input format (requires precomputed word tab counts to work).optional bool enable_differential_privacy = 50 [default = false];- Returns:
This builder for chaining.
-
hasDifferentialPrivacyNoiseLevel
boolean hasDifferentialPrivacyNoiseLevel()
Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];- Returns:
Whether the differentialPrivacyNoiseLevel field is set.
-
getDifferentialPrivacyNoiseLevel
float getDifferentialPrivacyNoiseLevel()
Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];- Returns:
The differentialPrivacyNoiseLevel.
-
setDifferentialPrivacyNoiseLevel
SentencepieceModel.TrainerSpec.Builder setDifferentialPrivacyNoiseLevel(float value)
Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];- Parameters:
value- The differentialPrivacyNoiseLevel to set.- Returns:
This builder for chaining.
-
clearDifferentialPrivacyNoiseLevel
SentencepieceModel.TrainerSpec.Builder clearDifferentialPrivacyNoiseLevel()
Set these parameters if you need DP version of sentencepiece. std of noise to add.optional float differential_privacy_noise_level = 51 [default = 0];- Returns:
This builder for chaining.
-
hasDifferentialPrivacyClippingThreshold
boolean hasDifferentialPrivacyClippingThreshold()
Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];- Returns:
Whether the differentialPrivacyClippingThreshold field is set.
-
getDifferentialPrivacyClippingThreshold
long getDifferentialPrivacyClippingThreshold()
Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];- Returns:
The differentialPrivacyClippingThreshold.
-
setDifferentialPrivacyClippingThreshold
SentencepieceModel.TrainerSpec.Builder setDifferentialPrivacyClippingThreshold(long value)
Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];- Parameters:
value- The differentialPrivacyClippingThreshold to set.- Returns:
This builder for chaining.
-
clearDifferentialPrivacyClippingThreshold
SentencepieceModel.TrainerSpec.Builder clearDifferentialPrivacyClippingThreshold()
Clipping threshold to apply after adding noise. All the words with frequency less than this value are dropped.optional uint64 differential_privacy_clipping_threshold = 52 [default = 0];- Returns:
This builder for chaining.
-
hasCharacterCoverage
boolean hasCharacterCoverage()
///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];- Returns:
Whether the characterCoverage field is set.
-
getCharacterCoverage
float getCharacterCoverage()
///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];- Returns:
The characterCoverage.
-
setCharacterCoverage
SentencepieceModel.TrainerSpec.Builder setCharacterCoverage(float value)
///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];- Parameters:
value- The characterCoverage to set.- Returns:
This builder for chaining.
-
clearCharacterCoverage
SentencepieceModel.TrainerSpec.Builder clearCharacterCoverage()
///////////////////////////////////////////////////////////////// Training parameters. Uses characters which cover the corpus with the ratio of `chars_coverage`. This parameter determines the set of basic Alphabet of sentence piece. 1.0 - `chars_coverage` characters are treated as UNK. See also required_chars field.optional float character_coverage = 10 [default = 0.9995];- Returns:
This builder for chaining.
-
hasInputSentenceSize
boolean hasInputSentenceSize()
Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];- Returns:
Whether the inputSentenceSize field is set.
-
getInputSentenceSize
long getInputSentenceSize()
Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];- Returns:
The inputSentenceSize.
-
setInputSentenceSize
SentencepieceModel.TrainerSpec.Builder setInputSentenceSize(long value)
Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];- Parameters:
value- The inputSentenceSize to set.- Returns:
This builder for chaining.
-
clearInputSentenceSize
SentencepieceModel.TrainerSpec.Builder clearInputSentenceSize()
Maximum size of sentences the trainer loads from `input` parameter. Trainer simply loads the `input` files in sequence. It is better to shuffle the input corpus randomly.optional uint64 input_sentence_size = 11 [default = 0];- Returns:
This builder for chaining.
-
hasShuffleInputSentence
boolean hasShuffleInputSentence()
optional bool shuffle_input_sentence = 19 [default = true];- Returns:
Whether the shuffleInputSentence field is set.
-
getShuffleInputSentence
boolean getShuffleInputSentence()
optional bool shuffle_input_sentence = 19 [default = true];- Returns:
The shuffleInputSentence.
-
setShuffleInputSentence
SentencepieceModel.TrainerSpec.Builder setShuffleInputSentence(boolean value)
optional bool shuffle_input_sentence = 19 [default = true];- Parameters:
value- The shuffleInputSentence to set.- Returns:
This builder for chaining.
-
clearShuffleInputSentence
SentencepieceModel.TrainerSpec.Builder clearShuffleInputSentence()
optional bool shuffle_input_sentence = 19 [default = true];- Returns:
This builder for chaining.
-
hasMiningSentenceSize
@Deprecated() boolean hasMiningSentenceSize()
Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];- Returns:
Whether the miningSentenceSize field is set.
-
getMiningSentenceSize
@Deprecated() int getMiningSentenceSize()
Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];- Returns:
The miningSentenceSize.
-
setMiningSentenceSize
@Deprecated() SentencepieceModel.TrainerSpec.Builder setMiningSentenceSize(int value)
Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];- Parameters:
value- The miningSentenceSize to set.- Returns:
This builder for chaining.
-
clearMiningSentenceSize
@Deprecated() SentencepieceModel.TrainerSpec.Builder clearMiningSentenceSize()
Maximum size of sentences to make seed sentence pieces. Extended suffix array is constructed to extract frequent sub-strings from the corpus. This uses 20N working space, where N is the size of corpus.optional int32 mining_sentence_size = 12 [deprecated = true];- Returns:
This builder for chaining.
-
hasTrainingSentenceSize
@Deprecated() boolean hasTrainingSentenceSize()
Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];- Returns:
Whether the trainingSentenceSize field is set.
-
getTrainingSentenceSize
@Deprecated() int getTrainingSentenceSize()
Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];- Returns:
The trainingSentenceSize.
-
setTrainingSentenceSize
@Deprecated() SentencepieceModel.TrainerSpec.Builder setTrainingSentenceSize(int value)
Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];- Parameters:
value- The trainingSentenceSize to set.- Returns:
This builder for chaining.
-
clearTrainingSentenceSize
@Deprecated() SentencepieceModel.TrainerSpec.Builder clearTrainingSentenceSize()
Maximum size of sentences to train sentence pieces.optional int32 training_sentence_size = 13 [deprecated = true];- Returns:
This builder for chaining.
-
hasSeedSentencepieceSize
boolean hasSeedSentencepieceSize()
The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];- Returns:
Whether the seedSentencepieceSize field is set.
-
getSeedSentencepieceSize
int getSeedSentencepieceSize()
The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];- Returns:
The seedSentencepieceSize.
-
setSeedSentencepieceSize
SentencepieceModel.TrainerSpec.Builder setSeedSentencepieceSize(int value)
The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];- Parameters:
value- The seedSentencepieceSize to set.- Returns:
This builder for chaining.
-
clearSeedSentencepieceSize
SentencepieceModel.TrainerSpec.Builder clearSeedSentencepieceSize()
The size of seed sentencepieces. `seed_sentencepiece_size` must be larger than `vocab_size`.optional int32 seed_sentencepiece_size = 14 [default = 1000000];- Returns:
This builder for chaining.
-
hasShrinkingFactor
boolean hasShrinkingFactor()
In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];- Returns:
Whether the shrinkingFactor field is set.
-
getShrinkingFactor
float getShrinkingFactor()
In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];- Returns:
The shrinkingFactor.
-
setShrinkingFactor
SentencepieceModel.TrainerSpec.Builder setShrinkingFactor(float value)
In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];- Parameters:
value- The shrinkingFactor to set.- Returns:
This builder for chaining.
-
clearShrinkingFactor
SentencepieceModel.TrainerSpec.Builder clearShrinkingFactor()
In every EM sub-iterations, keeps top `shrinking_factor` * `current sentencepieces size` with respect to the loss of the sentence piece. This value should be smaller than 1.0.optional float shrinking_factor = 15 [default = 0.75];- Returns:
This builder for chaining.
-
hasMaxSentenceLength
boolean hasMaxSentenceLength()
The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];- Returns:
Whether the maxSentenceLength field is set.
-
getMaxSentenceLength
int getMaxSentenceLength()
The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];- Returns:
The maxSentenceLength.
-
setMaxSentenceLength
SentencepieceModel.TrainerSpec.Builder setMaxSentenceLength(int value)
The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];- Parameters:
value- The maxSentenceLength to set.- Returns:
This builder for chaining.
-
clearMaxSentenceLength
SentencepieceModel.TrainerSpec.Builder clearMaxSentenceLength()
The maximum sentence length in byte. The sentences with the length larger than `max_sentence_length` is simply ignored. Longer input tends to bring the following risks: * Overflow during EM training (unigram language model only) * Performance drop because of O(n log n) cost in BPE.optional int32 max_sentence_length = 18 [default = 4192];- Returns:
This builder for chaining.
-
hasNumThreads
boolean hasNumThreads()
Number of threads in the training.optional int32 num_threads = 16 [default = 16];- Returns:
Whether the numThreads field is set.
-
getNumThreads
int getNumThreads()
Number of threads in the training.optional int32 num_threads = 16 [default = 16];- Returns:
The numThreads.
-
setNumThreads
SentencepieceModel.TrainerSpec.Builder setNumThreads(int value)
Number of threads in the training.optional int32 num_threads = 16 [default = 16];- Parameters:
value- The numThreads to set.- Returns:
This builder for chaining.
-
clearNumThreads
SentencepieceModel.TrainerSpec.Builder clearNumThreads()
Number of threads in the training.optional int32 num_threads = 16 [default = 16];- Returns:
This builder for chaining.
-
hasNumSubIterations
boolean hasNumSubIterations()
Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];- Returns:
Whether the numSubIterations field is set.
-
getNumSubIterations
int getNumSubIterations()
Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];- Returns:
The numSubIterations.
-
setNumSubIterations
SentencepieceModel.TrainerSpec.Builder setNumSubIterations(int value)
Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];- Parameters:
value- The numSubIterations to set.- Returns:
This builder for chaining.
-
clearNumSubIterations
SentencepieceModel.TrainerSpec.Builder clearNumSubIterations()
Number of EM sub iterations.optional int32 num_sub_iterations = 17 [default = 2];- Returns:
This builder for chaining.
-
hasMaxSentencepieceLength
boolean hasMaxSentencepieceLength()
///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];- Returns:
Whether the maxSentencepieceLength field is set.
-
getMaxSentencepieceLength
int getMaxSentencepieceLength()
///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];- Returns:
The maxSentencepieceLength.
-
setMaxSentencepieceLength
SentencepieceModel.TrainerSpec.Builder setMaxSentencepieceLength(int value)
///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];- Parameters:
value- The maxSentencepieceLength to set.- Returns:
This builder for chaining.
-
clearMaxSentencepieceLength
SentencepieceModel.TrainerSpec.Builder clearMaxSentencepieceLength()
///////////////////////////////////////////////////////////////// SentencePiece parameters which control the shapes of sentence piece. Maximum length of sentencepiece.optional int32 max_sentencepiece_length = 20 [default = 16];- Returns:
This builder for chaining.
-
hasSplitByUnicodeScript
boolean hasSplitByUnicodeScript()
Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];- Returns:
Whether the splitByUnicodeScript field is set.
-
getSplitByUnicodeScript
boolean getSplitByUnicodeScript()
Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];- Returns:
The splitByUnicodeScript.
-
setSplitByUnicodeScript
SentencepieceModel.TrainerSpec.Builder setSplitByUnicodeScript(boolean value)
Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];- Parameters:
value- The splitByUnicodeScript to set.- Returns:
This builder for chaining.
-
clearSplitByUnicodeScript
SentencepieceModel.TrainerSpec.Builder clearSplitByUnicodeScript()
Uses Unicode script to split sentence pieces. When `split_by_unicode_script` is true, we do not allow sentence piece to include multiple Unicode scripts, e.g. "F1" is not a valid piece. Exception: CJ characters (Hiragana/Katakana/Han) are all handled as one script type, since Japanese word can consist of multiple scripts. This exception is always applied regardless of the accept-language parameter.optional bool split_by_unicode_script = 21 [default = true];- Returns:
This builder for chaining.
-
hasSplitByNumber
boolean hasSplitByNumber()
When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];- Returns:
Whether the splitByNumber field is set.
-
getSplitByNumber
boolean getSplitByNumber()
When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];- Returns:
The splitByNumber.
-
setSplitByNumber
SentencepieceModel.TrainerSpec.Builder setSplitByNumber(boolean value)
When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];- Parameters:
value- The splitByNumber to set.- Returns:
This builder for chaining.
-
clearSplitByNumber
SentencepieceModel.TrainerSpec.Builder clearSplitByNumber()
When `split_by_number` is true, put a boundary between number and non-number transition. If we want to treat "F1" is one token, set this flag to be false.optional bool split_by_number = 23 [default = true];- Returns:
This builder for chaining.
-
hasSplitByWhitespace
boolean hasSplitByWhitespace()
Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];- Returns:
Whether the splitByWhitespace field is set.
-
getSplitByWhitespace
boolean getSplitByWhitespace()
Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];- Returns:
The splitByWhitespace.
-
setSplitByWhitespace
SentencepieceModel.TrainerSpec.Builder setSplitByWhitespace(boolean value)
Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];- Parameters:
value- The splitByWhitespace to set.- Returns:
This builder for chaining.
-
clearSplitByWhitespace
SentencepieceModel.TrainerSpec.Builder clearSplitByWhitespace()
Use a white space to split sentence pieces. When `split_by_whitespace` is false, we may have the piece containing a white space in the middle. e.g., "in_the".optional bool split_by_whitespace = 22 [default = true];- Returns:
This builder for chaining.
-
hasTreatWhitespaceAsSuffix
boolean hasTreatWhitespaceAsSuffix()
Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];- Returns:
Whether the treatWhitespaceAsSuffix field is set.
-
getTreatWhitespaceAsSuffix
boolean getTreatWhitespaceAsSuffix()
Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];- Returns:
The treatWhitespaceAsSuffix.
-
setTreatWhitespaceAsSuffix
SentencepieceModel.TrainerSpec.Builder setTreatWhitespaceAsSuffix(boolean value)
Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];- Parameters:
value- The treatWhitespaceAsSuffix to set.- Returns:
This builder for chaining.
-
clearTreatWhitespaceAsSuffix
SentencepieceModel.TrainerSpec.Builder clearTreatWhitespaceAsSuffix()
Adds whitespace symbol (_) as a suffix instead of prefix. e.g., _hello => hello_. When `treat_whitespace_as_suffix` is true, NormalizerSpec::add_dummy_prefix will add the dummy whitespace to the end of sentence.optional bool treat_whitespace_as_suffix = 24 [default = false];- Returns:
This builder for chaining.
-
hasAllowWhitespaceOnlyPieces
boolean hasAllowWhitespaceOnlyPieces()
Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];- Returns:
Whether the allowWhitespaceOnlyPieces field is set.
-
getAllowWhitespaceOnlyPieces
boolean getAllowWhitespaceOnlyPieces()
Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];- Returns:
The allowWhitespaceOnlyPieces.
-
setAllowWhitespaceOnlyPieces
SentencepieceModel.TrainerSpec.Builder setAllowWhitespaceOnlyPieces(boolean value)
Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];- Parameters:
value- The allowWhitespaceOnlyPieces to set.- Returns:
This builder for chaining.
-
clearAllowWhitespaceOnlyPieces
SentencepieceModel.TrainerSpec.Builder clearAllowWhitespaceOnlyPieces()
Allows pieces that only contain whitespaces instead of appearing only as prefix or suffix of other pieces.optional bool allow_whitespace_only_pieces = 26 [default = false];- Returns:
This builder for chaining.
-
hasSplitDigits
boolean hasSplitDigits()
Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];- Returns:
Whether the splitDigits field is set.
-
getSplitDigits
boolean getSplitDigits()
Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];- Returns:
The splitDigits.
-
setSplitDigits
SentencepieceModel.TrainerSpec.Builder setSplitDigits(boolean value)
Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];- Parameters:
value- The splitDigits to set.- Returns:
This builder for chaining.
-
clearSplitDigits
SentencepieceModel.TrainerSpec.Builder clearSplitDigits()
Split all digits (0-9) into separate pieces.optional bool split_digits = 25 [default = false];- Returns:
This builder for chaining.
-
hasPretokenizationDelimiter
boolean hasPretokenizationDelimiter()
Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];- Returns:
Whether the pretokenizationDelimiter field is set.
-
getPretokenizationDelimiter
String getPretokenizationDelimiter()
Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];- Returns:
The pretokenizationDelimiter.
-
getPretokenizationDelimiterBytes
ByteString getPretokenizationDelimiterBytes()
Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];- Returns:
The bytes for pretokenizationDelimiter.
-
setPretokenizationDelimiter
SentencepieceModel.TrainerSpec.Builder setPretokenizationDelimiter(String value)
Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];- Parameters:
value- The pretokenizationDelimiter to set.- Returns:
This builder for chaining.
-
clearPretokenizationDelimiter
SentencepieceModel.TrainerSpec.Builder clearPretokenizationDelimiter()
Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];- Returns:
This builder for chaining.
-
setPretokenizationDelimiterBytes
SentencepieceModel.TrainerSpec.Builder setPretokenizationDelimiterBytes(ByteString value)
Defines the pre-tokenization delimiter. When specified, no pieces crossing this delimiter is not included in the vocab. Then the delimiter string is virtually ignored during the training. This field can allows constraints on the vocabulary selection. Note that this field is available on unigram mode.optional string pretokenization_delimiter = 53 [default = ""];- Parameters:
value- The bytes for pretokenizationDelimiter to set.- Returns:
This builder for chaining.
-
getControlSymbolsList
ProtocolStringList getControlSymbolsList()
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Returns:
A list containing the controlSymbols.
-
getControlSymbolsCount
int getControlSymbolsCount()
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Returns:
The count of controlSymbols.
-
getControlSymbols
String getControlSymbols(int index)
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Parameters:
index- The index of the element to return.- Returns:
The controlSymbols at the given index.
-
getControlSymbolsBytes
ByteString getControlSymbolsBytes(int index)
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Parameters:
index- The index of the value to return.- Returns:
The bytes of the controlSymbols at the given index.
-
setControlSymbols
SentencepieceModel.TrainerSpec.Builder setControlSymbols(int index, String value)
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Parameters:
index- The index to set the value at.value- The controlSymbols to set.- Returns:
This builder for chaining.
-
addControlSymbols
SentencepieceModel.TrainerSpec.Builder addControlSymbols(String value)
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Parameters:
value- The controlSymbols to add.- Returns:
This builder for chaining.
-
addAllControlSymbols
SentencepieceModel.TrainerSpec.Builder addAllControlSymbols(Iterable<String> values)
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Parameters:
values- The controlSymbols to add.- Returns:
This builder for chaining.
-
clearControlSymbols
SentencepieceModel.TrainerSpec.Builder clearControlSymbols()
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Returns:
This builder for chaining.
-
addControlSymbolsBytes
SentencepieceModel.TrainerSpec.Builder addControlSymbolsBytes(ByteString value)
///////////////////////////////////////////////////////////////// Vocabulary management Defines control symbols used as an indicator to change the behavior of the decoder. <s> and </s> are pre-defined. We can use this field to encode various meta information, including language indicator in multilingual model. These symbols are not visible to users, but visible to the decoder. Note that when the input sentence contains control symbols, they are not treated as one token, but segmented into normal pieces. Control symbols must be inserted independently from the segmentation.repeated string control_symbols = 30;- Parameters:
value- The bytes of the controlSymbols to add.- Returns:
This builder for chaining.
-
getUserDefinedSymbolsList
ProtocolStringList getUserDefinedSymbolsList()
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Returns:
A list containing the userDefinedSymbols.
-
getUserDefinedSymbolsCount
int getUserDefinedSymbolsCount()
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Returns:
The count of userDefinedSymbols.
-
getUserDefinedSymbols
String getUserDefinedSymbols(int index)
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Parameters:
index- The index of the element to return.- Returns:
The userDefinedSymbols at the given index.
-
getUserDefinedSymbolsBytes
ByteString getUserDefinedSymbolsBytes(int index)
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Parameters:
index- The index of the value to return.- Returns:
The bytes of the userDefinedSymbols at the given index.
-
setUserDefinedSymbols
SentencepieceModel.TrainerSpec.Builder setUserDefinedSymbols(int index, String value)
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Parameters:
index- The index to set the value at.value- The userDefinedSymbols to set.- Returns:
This builder for chaining.
-
addUserDefinedSymbols
SentencepieceModel.TrainerSpec.Builder addUserDefinedSymbols(String value)
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Parameters:
value- The userDefinedSymbols to add.- Returns:
This builder for chaining.
-
addAllUserDefinedSymbols
SentencepieceModel.TrainerSpec.Builder addAllUserDefinedSymbols(Iterable<String> values)
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Parameters:
values- The userDefinedSymbols to add.- Returns:
This builder for chaining.
-
clearUserDefinedSymbols
SentencepieceModel.TrainerSpec.Builder clearUserDefinedSymbols()
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Returns:
This builder for chaining.
-
addUserDefinedSymbolsBytes
SentencepieceModel.TrainerSpec.Builder addUserDefinedSymbolsBytes(ByteString value)
Defines user defined symbols. These symbols are added with extremely high score so they are always treated as one unique symbol in any context. Typical usage of user_defined_symbols is placeholder for named entities.repeated string user_defined_symbols = 31;- Parameters:
value- The bytes of the userDefinedSymbols to add.- Returns:
This builder for chaining.
-
hasRequiredChars
boolean hasRequiredChars()
Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;- Returns:
Whether the requiredChars field is set.
-
getRequiredChars
String getRequiredChars()
Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;- Returns:
The requiredChars.
-
getRequiredCharsBytes
ByteString getRequiredCharsBytes()
Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;- Returns:
The bytes for requiredChars.
-
setRequiredChars
SentencepieceModel.TrainerSpec.Builder setRequiredChars(String value)
Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;- Parameters:
value- The requiredChars to set.- Returns:
This builder for chaining.
-
clearRequiredChars
SentencepieceModel.TrainerSpec.Builder clearRequiredChars()
Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;- Returns:
This builder for chaining.
-
setRequiredCharsBytes
SentencepieceModel.TrainerSpec.Builder setRequiredCharsBytes(ByteString value)
Defines required characters. Each UTF8 character in this string is included in the character set regardless of character_coverage value. Unlike user_defined_symbols, these characters have scores based on the frequency on input sentences, and the model can form subwords using characters in this field.optional string required_chars = 36;- Parameters:
value- The bytes for requiredChars to set.- Returns:
This builder for chaining.
-
hasByteFallback
boolean hasByteFallback()
Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];- Returns:
Whether the byteFallback field is set.
-
getByteFallback
boolean getByteFallback()
Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];- Returns:
The byteFallback.
-
setByteFallback
SentencepieceModel.TrainerSpec.Builder setByteFallback(boolean value)
Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];- Parameters:
value- The byteFallback to set.- Returns:
This builder for chaining.
-
clearByteFallback
SentencepieceModel.TrainerSpec.Builder clearByteFallback()
Decomposes unknown pieces into UTF-8 bytes.optional bool byte_fallback = 35 [default = false];- Returns:
This builder for chaining.
-
hasVocabularyOutputPieceScore
boolean hasVocabularyOutputPieceScore()
When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];- Returns:
Whether the vocabularyOutputPieceScore field is set.
-
getVocabularyOutputPieceScore
boolean getVocabularyOutputPieceScore()
When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];- Returns:
The vocabularyOutputPieceScore.
-
setVocabularyOutputPieceScore
SentencepieceModel.TrainerSpec.Builder setVocabularyOutputPieceScore(boolean value)
When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];- Parameters:
value- The vocabularyOutputPieceScore to set.- Returns:
This builder for chaining.
-
clearVocabularyOutputPieceScore
SentencepieceModel.TrainerSpec.Builder clearVocabularyOutputPieceScore()
When creating the vocabulary file, defines whether or not to additionally output the score for each piece.optional bool vocabulary_output_piece_score = 32 [default = true];- Returns:
This builder for chaining.
-
hasHardVocabLimit
boolean hasHardVocabLimit()
`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];- Returns:
Whether the hardVocabLimit field is set.
-
getHardVocabLimit
boolean getHardVocabLimit()
`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];- Returns:
The hardVocabLimit.
-
setHardVocabLimit
SentencepieceModel.TrainerSpec.Builder setHardVocabLimit(boolean value)
`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];- Parameters:
value- The hardVocabLimit to set.- Returns:
This builder for chaining.
-
clearHardVocabLimit
SentencepieceModel.TrainerSpec.Builder clearHardVocabLimit()
`vocab_size` is treated as hard limit. Crash if the model can not produce the vocab of size `vocab_size`, When `hard_vocab_limit` is false, vocab_size is treated as soft limit. Note that when model_type=char, always assumes hard_vocab_limit = false.optional bool hard_vocab_limit = 33 [default = true];- Returns:
This builder for chaining.
-
hasUseAllVocab
boolean hasUseAllVocab()
use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];- Returns:
Whether the useAllVocab field is set.
-
getUseAllVocab
boolean getUseAllVocab()
use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];- Returns:
The useAllVocab.
-
setUseAllVocab
SentencepieceModel.TrainerSpec.Builder setUseAllVocab(boolean value)
use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];- Parameters:
value- The useAllVocab to set.- Returns:
This builder for chaining.
-
clearUseAllVocab
SentencepieceModel.TrainerSpec.Builder clearUseAllVocab()
use all symbols for vocab extraction. This flag is valid if model type is either CHAR or WORDoptional bool use_all_vocab = 34 [default = false];- Returns:
This builder for chaining.
-
hasUnkId
boolean hasUnkId()
///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];- Returns:
Whether the unkId field is set.
-
getUnkId
int getUnkId()
///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];- Returns:
The unkId.
-
setUnkId
SentencepieceModel.TrainerSpec.Builder setUnkId(int value)
///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];- Parameters:
value- The unkId to set.- Returns:
This builder for chaining.
-
clearUnkId
SentencepieceModel.TrainerSpec.Builder clearUnkId()
///////////////////////////////////////////////////////////////// Reserved special meta tokens. * -1 is not used. * unk_id must not be -1. Id must starts with 0 and be contiguous.optional int32 unk_id = 40 [default = 0];- Returns:
This builder for chaining.
-
hasBosId
boolean hasBosId()
<s>optional int32 bos_id = 41 [default = 1];- Returns:
Whether the bosId field is set.
-
getBosId
int getBosId()
<s>optional int32 bos_id = 41 [default = 1];- Returns:
The bosId.
-
setBosId
SentencepieceModel.TrainerSpec.Builder setBosId(int value)
<s>optional int32 bos_id = 41 [default = 1];- Parameters:
value- The bosId to set.- Returns:
This builder for chaining.
-
clearBosId
SentencepieceModel.TrainerSpec.Builder clearBosId()
<s>optional int32 bos_id = 41 [default = 1];- Returns:
This builder for chaining.
-
hasEosId
boolean hasEosId()
</s>optional int32 eos_id = 42 [default = 2];- Returns:
Whether the eosId field is set.
-
getEosId
int getEosId()
</s>optional int32 eos_id = 42 [default = 2];- Returns:
The eosId.
-
setEosId
SentencepieceModel.TrainerSpec.Builder setEosId(int value)
</s>optional int32 eos_id = 42 [default = 2];- Parameters:
value- The eosId to set.- Returns:
This builder for chaining.
-
clearEosId
SentencepieceModel.TrainerSpec.Builder clearEosId()
</s>optional int32 eos_id = 42 [default = 2];- Returns:
This builder for chaining.
-
hasPadId
boolean hasPadId()
<pad> (padding)optional int32 pad_id = 43 [default = -1];- Returns:
Whether the padId field is set.
-
getPadId
int getPadId()
<pad> (padding)optional int32 pad_id = 43 [default = -1];- Returns:
The padId.
-
setPadId
SentencepieceModel.TrainerSpec.Builder setPadId(int value)
<pad> (padding)optional int32 pad_id = 43 [default = -1];- Parameters:
value- The padId to set.- Returns:
This builder for chaining.
-
clearPadId
SentencepieceModel.TrainerSpec.Builder clearPadId()
<pad> (padding)optional int32 pad_id = 43 [default = -1];- Returns:
This builder for chaining.
-
hasUnkPiece
boolean hasUnkPiece()
optional string unk_piece = 45 [default = "<unk>"];- Returns:
Whether the unkPiece field is set.
-
getUnkPiece
String getUnkPiece()
optional string unk_piece = 45 [default = "<unk>"];- Returns:
The unkPiece.
-
getUnkPieceBytes
ByteString getUnkPieceBytes()
optional string unk_piece = 45 [default = "<unk>"];- Returns:
The bytes for unkPiece.
-
setUnkPiece
SentencepieceModel.TrainerSpec.Builder setUnkPiece(String value)
optional string unk_piece = 45 [default = "<unk>"];- Parameters:
value- The unkPiece to set.- Returns:
This builder for chaining.
-
clearUnkPiece
SentencepieceModel.TrainerSpec.Builder clearUnkPiece()
optional string unk_piece = 45 [default = "<unk>"];- Returns:
This builder for chaining.
-
setUnkPieceBytes
SentencepieceModel.TrainerSpec.Builder setUnkPieceBytes(ByteString value)
optional string unk_piece = 45 [default = "<unk>"];- Parameters:
value- The bytes for unkPiece to set.- Returns:
This builder for chaining.
-
hasBosPiece
boolean hasBosPiece()
optional string bos_piece = 46 [default = "<s>"];- Returns:
Whether the bosPiece field is set.
-
getBosPiece
String getBosPiece()
optional string bos_piece = 46 [default = "<s>"];- Returns:
The bosPiece.
-
getBosPieceBytes
ByteString getBosPieceBytes()
optional string bos_piece = 46 [default = "<s>"];- Returns:
The bytes for bosPiece.
-
setBosPiece
SentencepieceModel.TrainerSpec.Builder setBosPiece(String value)
optional string bos_piece = 46 [default = "<s>"];- Parameters:
value- The bosPiece to set.- Returns:
This builder for chaining.
-
clearBosPiece
SentencepieceModel.TrainerSpec.Builder clearBosPiece()
optional string bos_piece = 46 [default = "<s>"];- Returns:
This builder for chaining.
-
setBosPieceBytes
SentencepieceModel.TrainerSpec.Builder setBosPieceBytes(ByteString value)
optional string bos_piece = 46 [default = "<s>"];- Parameters:
value- The bytes for bosPiece to set.- Returns:
This builder for chaining.
-
hasEosPiece
boolean hasEosPiece()
optional string eos_piece = 47 [default = "</s>"];- Returns:
Whether the eosPiece field is set.
-
getEosPiece
String getEosPiece()
optional string eos_piece = 47 [default = "</s>"];- Returns:
The eosPiece.
-
getEosPieceBytes
ByteString getEosPieceBytes()
optional string eos_piece = 47 [default = "</s>"];- Returns:
The bytes for eosPiece.
-
setEosPiece
SentencepieceModel.TrainerSpec.Builder setEosPiece(String value)
optional string eos_piece = 47 [default = "</s>"];- Parameters:
value- The eosPiece to set.- Returns:
This builder for chaining.
-
clearEosPiece
SentencepieceModel.TrainerSpec.Builder clearEosPiece()
optional string eos_piece = 47 [default = "</s>"];- Returns:
This builder for chaining.
-
setEosPieceBytes
SentencepieceModel.TrainerSpec.Builder setEosPieceBytes(ByteString value)
optional string eos_piece = 47 [default = "</s>"];- Parameters:
value- The bytes for eosPiece to set.- Returns:
This builder for chaining.
-
hasPadPiece
boolean hasPadPiece()
optional string pad_piece = 48 [default = "<pad>"];- Returns:
Whether the padPiece field is set.
-
getPadPiece
String getPadPiece()
optional string pad_piece = 48 [default = "<pad>"];- Returns:
The padPiece.
-
getPadPieceBytes
ByteString getPadPieceBytes()
optional string pad_piece = 48 [default = "<pad>"];- Returns:
The bytes for padPiece.
-
setPadPiece
SentencepieceModel.TrainerSpec.Builder setPadPiece(String value)
optional string pad_piece = 48 [default = "<pad>"];- Parameters:
value- The padPiece to set.- Returns:
This builder for chaining.
-
clearPadPiece
SentencepieceModel.TrainerSpec.Builder clearPadPiece()
optional string pad_piece = 48 [default = "<pad>"];- Returns:
This builder for chaining.
-
setPadPieceBytes
SentencepieceModel.TrainerSpec.Builder setPadPieceBytes(ByteString value)
optional string pad_piece = 48 [default = "<pad>"];- Parameters:
value- The bytes for padPiece to set.- Returns:
This builder for chaining.
-
hasUnkSurface
boolean hasUnkSurface()
Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];- Returns:
Whether the unkSurface field is set.
-
getUnkSurface
String getUnkSurface()
Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];- Returns:
The unkSurface.
-
getUnkSurfaceBytes
ByteString getUnkSurfaceBytes()
Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];- Returns:
The bytes for unkSurface.
-
setUnkSurface
SentencepieceModel.TrainerSpec.Builder setUnkSurface(String value)
Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];- Parameters:
value- The unkSurface to set.- Returns:
This builder for chaining.
-
clearUnkSurface
SentencepieceModel.TrainerSpec.Builder clearUnkSurface()
Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];- Returns:
This builder for chaining.
-
setUnkSurfaceBytes
SentencepieceModel.TrainerSpec.Builder setUnkSurfaceBytes(ByteString value)
Encodes <unk> into U+2047 (DOUBLE QUESTION MARK), since this character can be useful both for user and developer. We can easily figure out that <unk> is emitted.optional string unk_surface = 44 [default = " \342\201\207 "];- Parameters:
value- The bytes for unkSurface to set.- Returns:
This builder for chaining.
-
hasTrainExtremelyLargeCorpus
boolean hasTrainExtremelyLargeCorpus()
Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];- Returns:
Whether the trainExtremelyLargeCorpus field is set.
-
getTrainExtremelyLargeCorpus
boolean getTrainExtremelyLargeCorpus()
Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];- Returns:
The trainExtremelyLargeCorpus.
-
setTrainExtremelyLargeCorpus
SentencepieceModel.TrainerSpec.Builder setTrainExtremelyLargeCorpus(boolean value)
Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];- Parameters:
value- The trainExtremelyLargeCorpus to set.- Returns:
This builder for chaining.
-
clearTrainExtremelyLargeCorpus
SentencepieceModel.TrainerSpec.Builder clearTrainExtremelyLargeCorpus()
Increase bit depth to allow unigram model training on large (>10M sentences) corpora. A Side-effect of enabling this flag is increased memory usage.optional bool train_extremely_large_corpus = 49 [default = false];- Returns:
This builder for chaining.
-
hasSeedSentencepiecesFile
boolean hasSeedSentencepiecesFile()
Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];- Returns:
Whether the seedSentencepiecesFile field is set.
-
getSeedSentencepiecesFile
String getSeedSentencepiecesFile()
Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];- Returns:
The seedSentencepiecesFile.
-
getSeedSentencepiecesFileBytes
ByteString getSeedSentencepiecesFileBytes()
Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];- Returns:
The bytes for seedSentencepiecesFile.
-
setSeedSentencepiecesFile
SentencepieceModel.TrainerSpec.Builder setSeedSentencepiecesFile(String value)
Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];- Parameters:
value- The seedSentencepiecesFile to set.- Returns:
This builder for chaining.
-
clearSeedSentencepiecesFile
SentencepieceModel.TrainerSpec.Builder clearSeedSentencepiecesFile()
Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];- Returns:
This builder for chaining.
-
setSeedSentencepiecesFileBytes
SentencepieceModel.TrainerSpec.Builder setSeedSentencepiecesFileBytes(ByteString value)
Path to a seed sentencepieces file, with one tab-separated seed sentencepiece <tab> frequency per line.optional string seed_sentencepieces_file = 54 [default = ""];- Parameters:
value- The bytes for seedSentencepiecesFile to set.- Returns:
This builder for chaining.
-
setUnknownFields
final SentencepieceModel.TrainerSpec.Builder setUnknownFields(UnknownFieldSet unknownFields)
-
mergeUnknownFields
final SentencepieceModel.TrainerSpec.Builder mergeUnknownFields(UnknownFieldSet unknownFields)
-
-
-
-