org.apache.nifinifi-gcp-nar2.5.0org.apache.nifinifi-gcp-services-api-nar2.5.02.2.0rel/nifi-2.5.0main1org.apache.nifi.processors.gcp.bigquery.PutBigQueryPROCESSORWrites the contents of a FlowFile to a Google BigQuery table. The processor is record based so the schema that is used is driven by the RecordReader. Attributes that are not matched to the target schema are skipped. Exactly once delivery semantics are achieved via stream offsets.googlegoogle cloudbqbigqueryGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDtruefalsetrueENVIRONMENTfalsefalsebigquery-api-endpointBigQuery API EndpointCan be used to override the default BigQuery endpoint. Default is bigquerystorage.googleapis.com:443. Format must be hostname:port.bigquerystorage.googleapis.com:443truefalsetrueENVIRONMENTfalsefalsebq.datasetDatasetBigQuery dataset name (Note - The dataset must exist in GCP)${bq.dataset}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsebq.table.nameTable NameBigQuery table name${bq.table.name}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsebq.record.readerRecord ReaderSpecifies the Controller Service to use for parsing incoming data.org.apache.nifi.serialization.RecordReaderFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalsebq.transfer.typeTransfer TypeDefines the preferred transfer type streaming or batchingSTREAMSTREAMSTREAMUse streaming record handling strategyBATCHBATCHUse batching record handling strategytruefalsefalseNONEfalsefalsebq.append.record.countAppend Record CountThe number of records to be appended to the write stream at once. Applicable for both batch and stream types20truefalsefalseNONEfalsefalsegcp-retry-countNumber of retriesHow many retry attempts should be made before routing to the failure relationship.6truefalsefalseNONEfalsefalsebq.skip.invalid.rowsSkip Invalid RowsSets whether to insert all valid rows of a request, even if invalid rows exist. If not set the entire insert request will fail if it contains an invalid row.falsetruefalsetrueFLOWFILE_ATTRIBUTESfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalsefailureFlowFiles are routed to this relationship if the Google BigQuery operation fails.falsesuccessFlowFiles are routed to this relationship after a successful Google BigQuery operation.falsebq.records.countNumber of records successfully insertedtrueINPUT_REQUIREDorg.apache.nifi.processors.gcp.drive.FetchGoogleDrivePROCESSORFetches files from a Google Drive Folder. Designed to be used in tandem with ListGoogleDrive. Please see Additional Details to set up access to Google Drive.googledrivestoragefetchgcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsedrive-file-idFile IDThe Drive ID of the File to fetch. Please see Additional Details for information on how to obtain the Drive ID.${drive.id}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalseGoogle Doc Export TypeGoogle Doc Export TypeGoogle Documents cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Document, this property specifies the MIME Type to export the document to.application/pdfPDFapplication/pdfMicrosoft Wordapplication/vnd.openxmlformats-officedocument.wordprocessingml.documentOpenDocument Textapplication/vnd.oasis.opendocument.textPlain Texttext/plainRich Textapplication/rtfWeb Page (HTML)application/zipEPUBapplication/epub+ziptruefalsefalseNONEfalsefalseGoogle Spreadsheet Export TypeGoogle Spreadsheet Export TypeGoogle Spreadsheets cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Spreadsheet, this property specifies the MIME Type to export the spreadsheet to.application/pdfPDFapplication/pdfMicrosoft Excelapplication/vnd.openxmlformats-officedocument.spreadsheetml.sheetOpenDocument Spreadsheetapplication/x-vnd.oasis.opendocument.spreadsheetCSV (first sheet only)text/csvComma-separated values. Only the first sheet will be exported.TSV (first sheet only)text/tab-separated-valuesTab-separate values. Only the first sheet will be exported.Web Page (HTML)application/ziptruefalsefalseNONEfalsefalseGoogle Presentation Export TypeGoogle Presentation Export TypeGoogle Presentations cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Presentation, this property specifies the MIME Type to export the presentation to.application/pdfPDFapplication/pdfMicrosoft PowerPointapplication/vnd.openxmlformats-officedocument.presentationml.presentationOpenDocument Presentationapplication/vnd.oasis.opendocument.presentationPlain Texttext/plaintruefalsefalseNONEfalsefalseGoogle Drawing Export TypeGoogle Drawing Export TypeGoogle Drawings cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Drawing, this property specifies the MIME Type to export the drawing to.application/pdfPDFapplication/pdfPNGimage/pngJPEGimage/jpegSVGimage/svg+xmltruefalsefalseNONEfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseconnect-timeoutConnect TimeoutMaximum wait time for connection to Google Drive service.20 sectruefalsetrueENVIRONMENTfalsefalseread-timeoutRead TimeoutMaximum wait time for response from Google Drive service.60 sectruefalsetrueENVIRONMENTfalsefalsefailureA FlowFile will be routed here for each File for which fetch was attempted but failed.falsesuccessA FlowFile will be routed here for each successfully fetched File.falsedrive.idThe id of the filedrive.idThe id of the filefilenameThe name of the filemime.typeThe MIME type of the filedrive.sizeThe size of the file. Set to 0 when the file size is not available (e.g. externally stored files).drive.size.availableIndicates if the file size is known / availabledrive.timestampThe last modified time or created time (whichever is greater) of the file. The reason for this is that the original modified date of a file is preserved when uploaded to Google Drive. 'Created time' takes the time when the upload occurs. However uploaded files can still be modified later.drive.created.timeThe file's creation timedrive.modified.timeThe file's last modification timedrive.ownerThe owner of the filedrive.last.modifying.userThe last modifying user of the filedrive.web.view.linkWeb view link to the filedrive.web.content.linkWeb content link to the filedrive.parent.folder.idThe id of the file's parent folderdrive.parent.folder.nameThe name of the file's parent folderdrive.shared.drive.idThe id of the shared drive (if the file is located on a shared drive)drive.shared.drive.nameThe name of the shared drive (if the file is located on a shared drive)error.codeThe error code returned by Google Driveerror.messageThe error message returned by Google DriveINPUT_REQUIREDRetrieve all files in a Google Drive foldergoogledrivegoogle cloudstateretrievefetchallstreamorg.apache.nifi.processors.gcp.drive.ListGoogleDriveThe "Folder ID" property should be set to the ID of the Google Drive folder that files reside in. See processor documentation / additional details for more information on how to determine a Google Drive folder's ID. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like `#{GOOGLE_DRIVE_FOLDER_ID}`. The "GCP Credentials Provider Service" property should specify an instance of the GCPCredentialsService in order to provide credentials for accessing the folder. The 'success' Relationship of this Processor is then connected to FetchGoogleDrive. org.apache.nifi.processors.gcp.drive.FetchGoogleDrive"File ID" = "${drive.id}" The "GCP Credentials Provider Service" property should specify an instance of the GCPCredentialsService in order to provide credentials for accessing the bucket. org.apache.nifi.processors.gcp.drive.ListGoogleDriveorg.apache.nifi.processors.gcp.drive.PutGoogleDriveorg.apache.nifi.processors.gcp.drive.ListGoogleDrivePROCESSORPerforms a listing of concrete files (shortcuts are ignored) in a Google Drive folder. If the 'Record Writer' property is set, a single Output FlowFile is created, and each file in the listing is written as a single record to the output file. Otherwise, for each file in the listing, an individual FlowFile is created, the metadata being written as FlowFile attributes. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data. Please see Additional Details to set up access to Google Drive.googledrivestoragegcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsefolder-idFolder IDThe ID of the folder from which to pull list of files. Please see Additional Details to set up access to Google Drive and obtain Folder ID. WARNING: Unauthorized access to the folder is treated as if the folder was empty. This results in the processor not creating outgoing FlowFiles. No additional error message is provided.truefalsetrueENVIRONMENTfalsefalserecursive-searchSearch RecursivelyWhen 'true', will include list of files from concrete sub-folders (ignores shortcuts). Otherwise, will return only files that have the defined 'Folder ID' as their parent directly. WARNING: The listing may fail if there are too many sub-folders (500+).truetruetruefalsefalsetruefalsefalseNONEfalsefalsemin-ageMinimum File AgeThe minimum age a file must be in order to be considered; any files younger than this will be ignored.0 sectruefalsefalseNONEfalsefalselisting-strategyListing StrategySpecify how to determine new/updated entities. See each strategy descriptions for detail.timestampsTracking TimestampstimestampsThis strategy tracks the latest timestamp of listed entity to determine new/updated entities. Since it only tracks few timestamps, it can manage listing state efficiently. However, any newly added, or updated entity having timestamp older than the tracked latest timestamp can not be picked by this strategy. For example, such situation can happen in a file system if a file with old timestamp is copied or moved into the target directory without its last modified timestamp being updated. Also may miss files when multiple subdirectories are being written at the same time while listing is running.Tracking EntitiesentitiesThis strategy tracks information of all the listed entities within the latest 'Entity Tracking Time Window' to determine new/updated entities. This strategy can pick entities having old timestamp that can be missed with 'Tracking Timestamps'. Works even when multiple subdirectories are being written at the same time while listing is running. However additional DistributedMapCache controller service is required and more JVM heap memory is used. See the description of 'Entity Tracking Time Window' property for further details on how it works.Time Windowtime-windowThis strategy uses a sliding time window. The window starts where the previous window ended and ends with the 'current time'. One cycle will list files with modification time falling within the time window. Works even when multiple subdirectories are being written at the same time while listing is running. IMPORTANT: This strategy works properly only if the time on both the system hosting NiFi and the one hosting the files are accurate.No TrackingnoneThis strategy lists an entity without any tracking. The same entity will be listed each time on executing this processor. It is recommended to change the default run schedule value. Any property that related to the persisting state will be disregarded.truefalsefalseNONEfalsefalseet-state-cacheEntity Tracking State CacheListed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities' strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::{processorId}(::{nodeId})'. If it tracks per node listed entities, then the optional '::{nodeId}' part is added to manage state separately. E.g. cluster wide cache key = 'ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key = 'ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities' strategy.org.apache.nifi.distributed.cache.client.DistributedMapCacheClientorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalselisting-strategyListing Strategyentitieset-time-windowEntity Tracking Time WindowSpecify how long this processor should track already-listed entities. 'Tracking Entities' strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by 'Tracking Entities' strategy.3 hoursfalsefalsetrueENVIRONMENTfalsefalselisting-strategyListing Strategyentitieset-initial-listing-targetEntity Tracking Initial Listing TargetSpecify how initial listing should be handled. Used by 'Tracking Entities' strategy.allTracking Time WindowwindowIgnore entities having timestamp older than the specified 'Tracking Time Window' at the initial listing activity.All AvailableallRegardless of entities timestamp, all existing entities will be listed at the initial listing activity.falsefalsefalseNONEfalsefalselisting-strategyListing Strategyentitiesrecord-writerRecord WriterSpecifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles.org.apache.nifi.serialization.RecordSetWriterFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseconnect-timeoutConnect TimeoutMaximum wait time for connection to Google Drive service.20 sectruefalsetrueENVIRONMENTfalsefalseread-timeoutRead TimeoutMaximum wait time for response from Google Drive service.60 sectruefalsetrueENVIRONMENTfalsefalsesuccessAll FlowFiles that are received are routed to successfalsedrive.idThe id of the filefilenameThe name of the filemime.typeThe MIME type of the filedrive.sizeThe size of the file. Set to 0 when the file size is not available (e.g. externally stored files).drive.size.availableIndicates if the file size is known / availabledrive.timestampThe last modified time or created time (whichever is greater) of the file. The reason for this is that the original modified date of a file is preserved when uploaded to Google Drive. 'Created time' takes the time when the upload occurs. However uploaded files can still be modified later.drive.created.timeThe file's creation timedrive.modified.timeThe file's last modification timedrive.pathThe path of the file's directory from the base directory. The path contains the folder names in URL encoded form because Google Drive allows special characters in file names, including '/' (slash) and '\' (backslash). The URL encoded folder names are separated by '/' in the path.drive.ownerThe owner of the filedrive.last.modifying.userThe last modifying user of the filedrive.web.view.linkWeb view link to the filedrive.web.content.linkWeb content link to the filedrive.parent.folder.idThe id of the file's parent folderdrive.parent.folder.nameThe name of the file's parent folderdrive.listed.folder.idThe id of the base folder that was listeddrive.listed.folder.nameThe name of the base folder that was listeddrive.shared.drive.idThe id of the shared drive (if the file is located on a shared drive)drive.shared.drive.nameThe name of the shared drive (if the file is located on a shared drive)truetrueThe processor stores necessary data to be able to keep track what files have been listed already. What exactly needs to be stored depends on the 'Listing Strategy'. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.CLUSTERINPUT_FORBIDDENorg.apache.nifi.processors.gcp.drive.FetchGoogleDriveorg.apache.nifi.processors.gcp.drive.PutGoogleDriveTIMER_DRIVEN1 min1org.apache.nifi.processors.gcp.drive.PutGoogleDrivePROCESSORWrites the contents of a FlowFile as a file in Google Drive.googledrivestorageputgcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsefolder-idFolder IDThe ID of the shared folder. Please see Additional Details to set up access to Google Drive and obtain Folder ID.truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsefile-nameFilenameThe name of the file to upload to the specified Google Drive folder.${filename}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalseconflict-resolution-strategyConflict Resolution StrategyIndicates what should happen when a file with the same name already exists in the specified Google Drive folder.failfailfailHandle file conflict as failure.ignoreignoreIgnore conflict, do not change the original file.replacereplaceReplace existing file in case of conflict.truefalsefalseNONEfalsefalsechunked-upload-thresholdChunked Upload ThresholdThe maximum size of the content which is uploaded at once. FlowFiles larger than this threshold are uploaded in chunks.100 MBfalsefalsefalseNONEfalsefalsechunked-upload-sizeChunked Upload SizeDefines the size of a chunk. Used when a FlowFile's size exceeds 'Chunked Upload Threshold' and content is uploaded in smaller chunks. Minimum allowed chunk size is 256 KB, maximum allowed chunk size is 1 GB.10 MBfalsefalsefalseNONEfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseconnect-timeoutConnect TimeoutMaximum wait time for connection to Google Drive service.20 sectruefalsetrueENVIRONMENTfalsefalseread-timeoutRead TimeoutMaximum wait time for response from Google Drive service.60 sectruefalsetrueENVIRONMENTfalsefalsefailureFiles that could not be written to Google Drive for some reason are transferred to this relationship.falsesuccessFiles that have been successfully written to Google Drive are transferred to this relationship.falsefilenameUses the FlowFile's filename as the filename for the Google Drive object.drive.idThe id of the filefilenameThe name of the filemime.typeThe MIME type of the filedrive.sizeThe size of the file. Set to 0 when the file size is not available (e.g. externally stored files).drive.size.availableIndicates if the file size is known / availabledrive.timestampThe last modified time or created time (whichever is greater) of the file. The reason for this is that the original modified date of a file is preserved when uploaded to Google Drive. 'Created time' takes the time when the upload occurs. However uploaded files can still be modified later.drive.created.timeThe file's creation timedrive.modified.timeThe file's last modification timeerror.codeThe error code returned by Google Driveerror.messageThe error message returned by Google DriveINPUT_REQUIREDorg.apache.nifi.processors.gcp.drive.ListGoogleDriveorg.apache.nifi.processors.gcp.drive.FetchGoogleDriveorg.apache.nifi.processors.gcp.pubsub.ConsumeGCPubSubPROCESSOR Consumes messages from the configured Google Cloud PubSub subscription. The 'Batch Size' property specified the maximum number of messages that will be pulled from the subscription in a single request. The 'Processing Strategy' property specifies if each message should be its own FlowFile or if messages should be grouped into a single FlowFile. Using the Demarcator strategy will provide best throughput when the format allows it. Using Record allows to convert data format as well as doing schema enforcement. Using the FlowFile strategy will generate one FlowFile per message and will have the message's attributes as FlowFile attributes. googlegoogle-cloudgcpmessagepubsubconsumeGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDfalsefalsetrueENVIRONMENTfalsefalsegcp-pubsub-subscriptionSubscriptionName of the Google Cloud Pub/Sub SubscriptiontruefalsetrueENVIRONMENTfalsefalsegcp-pubsub-publish-batch-sizeBatch Size ThresholdIndicates the number of messages the cloud service should bundle together in a batch. If not set and left empty, only one message will be used in a batch15truefalsefalseNONEfalsefalseProcessing StrategyProcessing StrategyStrategy for processing PubSub Records and writing serialized output to FlowFilesFLOW_FILEFLOW_FILEFLOW_FILEWrite one FlowFile for each PubSub message consumedDEMARCATORDEMARCATORWrite one FlowFile for each batch of PubSub messages consumedRECORDRECORDWrite one FlowFile containing multiple PubSub messages consumed and processed with Record Reader and Record WritertruefalsefalseNONEfalsefalseRecord ReaderRecord ReaderThe Record Reader to use for incoming messagesorg.apache.nifi.serialization.RecordReaderFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalseProcessing StrategyProcessing StrategyRECORDRecord WriterRecord WriterThe Record Writer to use in order to serialize the outgoing FlowFilesorg.apache.nifi.serialization.RecordSetWriterFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalseProcessing StrategyProcessing StrategyRECORDOutput StrategyOutput StrategyThe format used to output the Kafka Record into a FlowFile Record.USE_VALUEUse Content as ValueUSE_VALUEWrite only the message payload to the FlowFile record.Use WrapperUSE_WRAPPERWrite the message's attributes into the FlowFile record.truefalsefalseNONEfalsefalseProcessing StrategyProcessing StrategyRECORDMessage DemarcatorMessage DemarcatorSince the PubSub client receives messages in batches, this Processor has an option to output FlowFiles which contains all the messages in a single batch. This property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple messages. To enter special character such as 'new line' use CTRL+Enter or Shift+Enter depending on the OS. truefalsefalseNONEfalsefalseProcessing StrategyProcessing StrategyDEMARCATORapi-endpointAPI EndpointOverride the gRPC endpoint in the form of [host:port]pubsub.googleapis.com:443truefalsetrueENVIRONMENTfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalsesuccessFlowFiles are routed to this relationship after a successful Google Cloud Pub/Sub operation.falsegcp.pubsub.ackIdAcknowledgement Id of the consumed Google Cloud PubSub messagegcp.pubsub.messageSizeSerialized size of the consumed Google Cloud PubSub messagegcp.pubsub.attributesCountNumber of attributes the consumed PubSub message has, if anygcp.pubsub.publishTimeTimestamp value when the message was publishedgcp.pubsub.subscriptionName of the PubSub subscriptionDynamic AttributesOther than the listed attributes, this processor may write zero or more attributes, if the original Google Cloud Publisher client added any attributes to the message while sendingINPUT_FORBIDDENorg.apache.nifi.processors.gcp.pubsub.PublishGCPubSuborg.apache.nifi.processors.gcp.pubsub.PublishGCPubSubPROCESSORPublishes the content of the incoming flowfile to the configured Google Cloud PubSub topic. The processor supports dynamic properties. If any dynamic properties are present, they will be sent along with the message in the form of 'attributes'.googlegoogle-cloudgcpmessagepubsubpublishGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDfalsefalsetrueENVIRONMENTfalsefalsegcp-pubsub-topicTopic NameName of the Google Cloud PubSub TopictruefalsetrueENVIRONMENTfalsefalseMessage Derivation StrategyMessage Derivation StrategyThe strategy used to publish the incoming FlowFile to the Google Cloud PubSub endpoint.FLOWFILE_ORIENTEDFlowFile OrientedFLOWFILE_ORIENTEDEach incoming FlowFile is sent as a Google Cloud PubSub messageRecord OrientedRECORD_ORIENTEDEach incoming FlowFile is parsed into NiFi records, which are each sent as a Google Cloud PubSub messagetruefalsefalseNONEfalsefalseRecord ReaderRecord ReaderThe Record Reader to use for incoming FlowFilesorg.apache.nifi.serialization.RecordReaderFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalseMessage Derivation StrategyMessage Derivation StrategyRECORD_ORIENTEDRecord WriterRecord WriterThe Record Writer to use in order to serialize the data before sending to GCPubSub endpointorg.apache.nifi.serialization.RecordSetWriterFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalseMessage Derivation StrategyMessage Derivation StrategyRECORD_ORIENTEDInput Batch SizeInput Batch SizeMaximum number of FlowFiles processed for each Processor invocation100truefalsefalseNONEfalsefalseMaximum Message SizeMaximum Message SizeThe maximum size of a Google PubSub message in bytes. Defaults to 1 MB (1048576 bytes)1 MBtruefalsefalseNONEfalsefalseMessage Derivation StrategyMessage Derivation StrategyFLOWFILE_ORIENTEDgcp-pubsub-publish-batch-sizeBatch Size ThresholdIndicates the number of messages the cloud service should bundle together in a batch. If not set and left empty, only one message will be used in a batch15truefalsefalseNONEfalsefalsegcp-batch-bytesBatch Bytes ThresholdPublish request gets triggered based on this Batch Bytes Threshold property and the Batch Size Threshold property, whichever condition is met first.3 MBtruefalsetrueENVIRONMENTfalsefalsegcp-pubsub-publish-batch-delayBatch Delay ThresholdIndicates the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. This value should not be set too high, usually on the order of milliseconds. Otherwise, calls might appear to never complete.100 millistruefalsefalseNONEfalsefalseapi-endpointAPI EndpointOverride the gRPC endpoint in the form of [host:port]pubsub.googleapis.com:443truefalsetrueENVIRONMENTfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseAttribute nameValue to be set to the attributeAttributes to be set for the outgoing Google Cloud PubSub messageFLOWFILE_ATTRIBUTESfailureFlowFiles are routed to this relationship if the Google Cloud Pub/Sub operation fails.falseretryFlowFiles are routed to this relationship if the Google Cloud Pub/Sub operation fails but attempting the operation again may succeed.falsesuccessFlowFiles are routed to this relationship after a successful Google Cloud Pub/Sub operation.falsegcp.pubsub.messageIdID of the pubsub message published to the configured Google Cloud PubSub topicgcp.pubsub.count.recordsCount of pubsub messages published to the configured Google Cloud PubSub topicgcp.pubsub.topicName of the Google Cloud PubSub topic the message was published toINPUT_REQUIREDMEMORYThe entirety of the FlowFile's content will be read into memory to be sent as a PubSub message.org.apache.nifi.processors.gcp.pubsub.ConsumeGCPubSuborg.apache.nifi.processors.gcp.storage.DeleteGCSObjectPROCESSORDeletes objects from a Google Cloud Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.google cloudgcsgooglestoragedeleteGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDfalsefalsetrueENVIRONMENTfalsefalsegcs-bucketBucketBucket of the object.${gcs.bucket}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-keyKeyName of the object.${filename}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-generationGenerationThe generation of the object to be deleted. If null, will use latest version of the object.falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcp-retry-countNumber of retriesHow many retry attempts should be made before routing to the failure relationship.6truefalsefalseNONEfalsefalsestorage-api-urlStorage API URLOverrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections.falsefalsetrueENVIRONMENTfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalsefailureFlowFiles are routed to this relationship if the Google Cloud Storage operation fails.falsesuccessFlowFiles are routed to this relationship after a successful Google Cloud Storage operation.falsetrueINPUT_REQUIREDorg.apache.nifi.processors.gcp.storage.PutGCSObjectorg.apache.nifi.processors.gcp.storage.FetchGCSObjectorg.apache.nifi.processors.gcp.storage.ListGCSBucketorg.apache.nifi.processors.gcp.storage.FetchGCSObjectPROCESSORFetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.google cloudgooglestoragegcsfetchGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDfalsefalsetrueENVIRONMENTfalsefalsegcs-bucketBucketBucket of the object.${gcs.bucket}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-keyKeyName of the object.${filename}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-generationObject GenerationThe generation of the Object to download. If not set, the latest generation will be downloaded.falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-server-side-encryption-keyServer Side Encryption KeyAn AES256 Key (encoded in base64) which the object has been encrypted in.falsetruetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-object-range-startRange StartThe byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object.falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-object-range-lengthRange LengthThe number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object.falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcp-retry-countNumber of retriesHow many retry attempts should be made before routing to the failure relationship.6truefalsefalseNONEfalsefalsestorage-api-urlStorage API URLOverrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections.falsefalsetrueENVIRONMENTfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalsefailureFlowFiles are routed to this relationship if the Google Cloud Storage operation fails.falsesuccessFlowFiles are routed to this relationship after a successful Google Cloud Storage operation.falsefilenameThe name of the file, parsed if possible from the Content-Disposition response headergcs.bucketBucket of the object.gcs.keyName of the object.gcs.sizeSize of the object.gcs.cache.controlData cache control of the object.gcs.component.countThe number of components which make up the object.gcs.content.dispositionThe data content disposition of the object.gcs.content.encodingThe content encoding of the object.gcs.content.languageThe content language of the object.mime.typeThe MIME/Content-Type of the objectgcs.crc32cThe CRC32C checksum of object's data, encoded in base64 in big-endian order.gcs.create.timeThe creation time of the object (milliseconds)gcs.update.timeThe last modification time of the object (milliseconds)gcs.encryption.algorithmThe algorithm used to encrypt the object.gcs.encryption.sha256The SHA256 hash of the key used to encrypt the objectgcs.etagThe HTTP 1.1 Entity tag for the object.gcs.generated.idThe service-generated for the objectgcs.generationThe data generation of the object.gcs.md5The MD5 hash of the object's data encoded in base64.gcs.media.linkThe media download link to the object.gcs.metagenerationThe metageneration of the object.gcs.ownerThe owner (uploader) of the object.gcs.owner.typeThe ACL entity type of the uploader of the object.gcs.acl.ownerA comma-delimited list of ACL entities that have owner access to the object. Entities will be either email addresses, domains, or project IDs.gcs.acl.writerA comma-delimited list of ACL entities that have write access to the object. Entities will be either email addresses, domains, or project IDs.gcs.acl.readerA comma-delimited list of ACL entities that have read access to the object. Entities will be either email addresses, domains, or project IDs.gcs.uriThe URI of the object as a string.INPUT_REQUIREDRetrieve all files in a Google Compute Storage (GCS) bucketgcpgcsgoogle cloudgoogle compute storagestateretrievefetchallstreamorg.apache.nifi.processors.gcp.storage.ListGCSBucketThe "Bucket" property should be set to the name of the GCS bucket that files reside in. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like `#{GCS_SOURCE_BUCKET}`. Configure the "Project ID" property to reflect the ID of your Google Compute Cloud Project. The "GCP Credentials Provider Service" property should specify an instance of the GCPCredentialsService in order to provide credentials for accessing the bucket. The 'success' Relationship of this Processor is then connected to FetchGCSObject. org.apache.nifi.processors.gcp.storage.FetchGCSObject"Bucket" = "${gcs.bucket}" "Name" = "${filename}" The "GCP Credentials Provider Service" property should specify an instance of the GCPCredentialsService in order to provide credentials for accessing the bucket. org.apache.nifi.processors.gcp.storage.ListGCSBucketorg.apache.nifi.processors.gcp.storage.PutGCSObjectorg.apache.nifi.processors.gcp.storage.DeleteGCSObjectorg.apache.nifi.processors.gcp.storage.ListGCSBucketPROCESSORRetrieves a listing of objects from a GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.google cloudgooglestoragegcslistGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDfalsefalsetrueENVIRONMENTfalsefalsegcs-bucketBucketBucket of the object.truefalsetrueENVIRONMENTfalsefalsegcs-prefixPrefixThe prefix used to filter the object list. In most cases, it should end with a forward slash ('/').falsefalsetrueENVIRONMENTfalsefalselisting-strategyListing StrategySpecify how to determine new/updated entities. See each strategy descriptions for detail.timestampsTracking TimestampstimestampsThis strategy tracks the latest timestamp of listed entity to determine new/updated entities. Since it only tracks few timestamps, it can manage listing state efficiently. This strategy will not pick up any newly added or modified entity if their timestamps are older than the tracked latest timestamp. Also may miss files when multiple subdirectories are being written at the same time while listing is running.Tracking EntitiesentitiesThis strategy tracks information of all the listed entities within the latest 'Entity Tracking Time Window' to determine new/updated entities. This strategy can pick entities having old timestamp that can be missed with 'Tracing Timestamps'. Works even when multiple subdirectories are being written at the same time while listing is running. However an additional DistributedMapCache controller service is required and more JVM heap memory is used. For more information on how the 'Entity Tracking Time Window' property works, see the description.No TrackingnoneThis strategy lists all entities without any tracking. The same entities will be listed each time this processor is scheduled. It is recommended to change the default run schedule value. Any property that relates to the persisting state will be ignored.truefalsefalseNONEfalsefalseet-state-cacheEntity Tracking State CacheListed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities' strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::{processorId}(::{nodeId})'. If it tracks per node listed entities, then the optional '::{nodeId}' part is added to manage state separately. E.g. cluster wide cache key = 'ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key = 'ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities' strategy.org.apache.nifi.distributed.cache.client.DistributedMapCacheClientorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalselisting-strategyListing Strategyentitieset-initial-listing-targetEntity Tracking Initial Listing TargetSpecify how initial listing should be handled. Used by 'Tracking Entities' strategy.allTracking Time WindowwindowIgnore entities having timestamp older than the specified 'Tracking Time Window' at the initial listing activity.All AvailableallRegardless of entities timestamp, all existing entities will be listed at the initial listing activity.falsefalsefalseNONEfalsefalselisting-strategyListing Strategyentitieset-time-windowEntity Tracking Time WindowSpecify how long this processor should track already-listed entities. 'Tracking Entities' strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by 'Tracking Entities' strategy.3 hourstruefalsetrueENVIRONMENTfalsefalserecord-writerRecord WriterSpecifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles.org.apache.nifi.serialization.RecordSetWriterFactoryorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalsegcs-use-generationsUse GenerationsSpecifies whether to use GCS Generations, if applicable. If false, only the latest version of each object will be returned.falsetruetruefalsefalsetruefalsefalseNONEfalsefalsegcp-retry-countNumber of retriesHow many retry attempts should be made before routing to the failure relationship.6truefalsefalseNONEfalsefalsestorage-api-urlStorage API URLOverrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections.falsefalsetrueENVIRONMENTfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalsesuccessFlowFiles are routed to this relationship after a successful Google Cloud Storage operation.falsefilenameThe name of the filegcs.bucketBucket of the object.gcs.keyName of the object.gcs.sizeSize of the object.gcs.cache.controlData cache control of the object.gcs.component.countThe number of components which make up the object.gcs.content.dispositionThe data content disposition of the object.gcs.content.encodingThe content encoding of the object.gcs.content.languageThe content language of the object.mime.typeThe MIME/Content-Type of the objectgcs.crc32cThe CRC32C checksum of object's data, encoded in base64 in big-endian order.gcs.create.timeThe creation time of the object (milliseconds)gcs.update.timeThe last modification time of the object (milliseconds)gcs.encryption.algorithmThe algorithm used to encrypt the object.gcs.encryption.sha256The SHA256 hash of the key used to encrypt the objectgcs.etagThe HTTP 1.1 Entity tag for the object.gcs.generated.idThe service-generated for the objectgcs.generationThe data generation of the object.gcs.md5The MD5 hash of the object's data encoded in base64.gcs.media.linkThe media download link to the object.gcs.metagenerationThe metageneration of the object.gcs.ownerThe owner (uploader) of the object.gcs.owner.typeThe ACL entity type of the uploader of the object.gcs.acl.ownerA comma-delimited list of ACL entities that have owner access to the object. Entities will be either email addresses, domains, or project IDs.gcs.acl.writerA comma-delimited list of ACL entities that have write access to the object. Entities will be either email addresses, domains, or project IDs.gcs.acl.readerA comma-delimited list of ACL entities that have read access to the object. Entities will be either email addresses, domains, or project IDs.gcs.uriThe URI of the object as a string.truetruetrueAfter performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.CLUSTERINPUT_FORBIDDENorg.apache.nifi.processors.gcp.storage.PutGCSObjectorg.apache.nifi.processors.gcp.storage.DeleteGCSObjectorg.apache.nifi.processors.gcp.storage.FetchGCSObjectTIMER_DRIVEN1 min1org.apache.nifi.processors.gcp.storage.PutGCSObjectPROCESSORWrites the contents of a FlowFile as an object in a Google Cloud Storage.googlegoogle cloudgcsarchiveputGCP Credentials Provider ServiceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDfalsefalsetrueENVIRONMENTfalsefalsegcs-bucketBucketBucket of the object.${gcs.bucket}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-keyKeyName of the object.${filename}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalseResource Transfer SourceResource Transfer SourceThe source of the content to be transferredFLOWFILE_CONTENTFlowFile ContentFLOWFILE_CONTENTThe content of the incoming FlowFile provides the source for transferFile Resource ServiceFILE_RESOURCE_SERVICEThe File Resource Service provides the source for transfertruefalsefalseNONEfalsefalseFile Resource ServiceFile Resource ServiceFile Resource Service providing access to the local resource to be transferredorg.apache.nifi.fileresource.service.api.FileResourceServiceorg.apache.nifinifi-standard-services-api-nar2.5.0truefalsefalseNONEfalsefalseResource Transfer SourceResource Transfer SourceFILE_RESOURCE_SERVICEgcs-content-typeContent TypeContent Type for the file, i.e. text/plain${mime.type}falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-object-crc32cCRC32C ChecksumCRC32C Checksum (encoded in Base64, big-Endian order) of the file for server-side validation.falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-object-aclObject ACLAccess Control to be attached to the object uploaded. Not providing this will revert to bucket defaults.All Authenticated UsersALL_AUTHENTICATED_USERSGives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER and WRITER permissions. All other permissions are removed.Authenticated ReadAUTHENTICATED_READGives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER permission. All other permissions are removed.Bucket Owner Full ControlBUCKET_OWNER_FULL_CONTROLGives the object and bucket owners OWNER permission. All other permissions are removed.Bucket Owner Read OnlyBUCKET_OWNER_READGives the object owner OWNER permission, and gives the bucket owner READER permission. All other permissions are removed.PrivatePRIVATEGives the bucket or object owner OWNER permission for a bucket or object, and removes all other access permissions.Project PrivatePROJECT_PRIVATEGives permission to the project team based on their roles. Anyone who is part of the team has READER permission. Project owners and project editors have OWNER permission. This is the default ACL for newly created buckets. This is also the default ACL for newly created objects unless the default object ACL for that bucket has been changed.Public Read OnlyPUBLIC_READGives the bucket or object owner OWNER permission, and gives all users, both authenticated and anonymous, READER permission. When you apply this to an object, anyone on the Internet can read the object without authenticating.falsefalsefalseNONEfalsefalsegcs-server-side-encryption-keyServer Side Encryption KeyAn AES256 Encryption Key (encoded in base64) for server-side encryption of the object.falsetruetrueFLOWFILE_ATTRIBUTESfalsefalsegcs-overwrite-objectOverwrite ObjectIf false, the upload to GCS will succeed only if the object does not exist.truetruetruefalsefalsetruefalsefalseNONEfalsefalsegcs-content-disposition-typeContent Disposition TypeType of RFC-6266 Content Disposition to be attached to the objectInlineinlineIndicates that the object should be loaded and rendered within the browser.AttachmentattachmentIndicates that the object should be saved (using a Save As... dialog) rather than opened directly within the browserfalsefalsefalseNONEfalsefalsegzip.content.enabledGZIP Compression EnabledSignals to the GCS Blob Writer whether GZIP compression during transfer is desired. False means do not gzip and can boost performance in many cases.truetruetruefalsefalsefalsefalsefalseNONEfalsefalsestorage-api-urlStorage API URLOverrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections.falsefalsetrueENVIRONMENTfalsefalsegcp-retry-countNumber of retriesHow many retry attempts should be made before routing to the failure relationship.6truefalsefalseNONEfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseThe name of a User-Defined Metadata field to add to the GCS ObjectThe value of a User-Defined Metadata field to add to the GCS ObjectAllows user-defined metadata to be added to the GCS object as key/value pairsFLOWFILE_ATTRIBUTESfailureFlowFiles are routed to this relationship if the Google Cloud Storage operation fails.falsesuccessFlowFiles are routed to this relationship after a successful Google Cloud Storage operation.falsefilenameUses the FlowFile's filename as the filename for the GCS objectmime.typeUses the FlowFile's MIME type as the content-type for the GCS objectgcs.bucketBucket of the object.gcs.keyName of the object.gcs.sizeSize of the object.gcs.cache.controlData cache control of the object.gcs.component.countThe number of components which make up the object.gcs.content.dispositionThe data content disposition of the object.gcs.content.encodingThe content encoding of the object.gcs.content.languageThe content language of the object.mime.typeThe MIME/Content-Type of the objectgcs.crc32cThe CRC32C checksum of object's data, encoded in base64 in big-endian order.gcs.create.timeThe creation time of the object (milliseconds)gcs.update.timeThe last modification time of the object (milliseconds)gcs.encryption.algorithmThe algorithm used to encrypt the object.gcs.encryption.sha256The SHA256 hash of the key used to encrypt the objectgcs.etagThe HTTP 1.1 Entity tag for the object.gcs.generated.idThe service-generated for the objectgcs.generationThe data generation of the object.gcs.md5The MD5 hash of the object's data encoded in base64.gcs.media.linkThe media download link to the object.gcs.metagenerationThe metageneration of the object.gcs.ownerThe owner (uploader) of the object.gcs.owner.typeThe ACL entity type of the uploader of the object.gcs.uriThe URI of the object as a string.INPUT_REQUIREDorg.apache.nifi.processors.gcp.storage.FetchGCSObjectorg.apache.nifi.processors.gcp.storage.DeleteGCSObjectorg.apache.nifi.processors.gcp.storage.ListGCSBucketorg.apache.nifi.processors.gcp.vision.GetGcpVisionAnnotateFilesOperationStatusPROCESSORRetrieves the current status of an Google Vision operation.GoogleCloudVisionMachine Learninggcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalseoperationKeyGCP Operation KeyThe unique identifier of the Vision operation.${operationKey}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsefailureFlowFiles are routed to failure relationshipfalseoriginalUpon successful completion, the original FlowFile will be routed to this relationship.truerunningThe job is currently still being processedfalsesuccessFlowFiles are routed to success relationshipfalseoperationKeyA unique identifier of the operation designated by the Vision server.org.apache.nifi.processors.gcp.vision.StartGcpVisionAnnotateFilesOperationorg.apache.nifi.processors.gcp.vision.GetGcpVisionAnnotateImagesOperationStatusPROCESSORRetrieves the current status of an Google Vision operation.GoogleCloudVisionMachine Learninggcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalseoperationKeyGCP Operation KeyThe unique identifier of the Vision operation.${operationKey}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsefailureFlowFiles are routed to failure relationshipfalseoriginalUpon successful completion, the original FlowFile will be routed to this relationship.truerunningThe job is currently still being processedfalsesuccessFlowFiles are routed to success relationshipfalseoperationKeyA unique identifier of the operation designated by the Vision server.org.apache.nifi.processors.gcp.vision.StartGcpVisionAnnotateImagesOperationorg.apache.nifi.processors.gcp.vision.StartGcpVisionAnnotateFilesOperationPROCESSORTrigger a Vision operation on file input. It should be followed by GetGcpVisionAnnotateFilesOperationStatus processor in order to monitor operation status.GoogleCloudMachine LearningVisionjson-payloadJSON PayloadJSON request for AWS Machine Learning services. The Processor will use FlowFile content for the request when this property is not specified.{ "requests": [ { "inputConfig": { "gcsSource": { "uri": "gs://${gcs.bucket}/${filename}" }, "mimeType": "application/pdf" }, "features": [{ "type": "${vision-feature-type}", "maxResults": 4 }], "outputConfig": { "gcsDestination": { "uri": "gs://${output-bucket}/${filename}/" }, "batchSize": 2 } }] }falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalseoutput-bucketOutput BucketName of the GCS bucket where the output of the Vision job will be persisted. The value of this property applies when the JSON Payload property is configured. The JSON Payload property value can use Expression Language to reference the value of ${output-bucket}falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsevision-feature-typeVision Feature TypeType of GCP Vision Feature. The value of this property applies when the JSON Payload property is configured. The JSON Payload property value can use Expression Language to reference the value of ${vision-feature-type}TEXT_DETECTIONfalsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsefailureFlowFiles are routed to failure relationshipfalsesuccessFlowFiles are routed to success relationshipfalseoperationKeyA unique identifier of the operation returned by the Vision server.org.apache.nifi.processors.gcp.vision.GetGcpVisionAnnotateFilesOperationStatusorg.apache.nifi.processors.gcp.vision.StartGcpVisionAnnotateImagesOperationPROCESSORTrigger a Vision operation on image input. It should be followed by GetGcpVisionAnnotateImagesOperationStatus processor in order to monitor operation status.GoogleCloudMachine LearningVisionjson-payloadJSON PayloadJSON request for AWS Machine Learning services. The Processor will use FlowFile content for the request when this property is not specified.{ "requests": [{ "image": { "source": { "imageUri": "gs://${gcs.bucket}/${filename}" } }, "features": [{ "type": "${vision-feature-type}", "maxResults": 4 }] }], "outputConfig": { "gcsDestination": { "uri": "gs://${output-bucket}/${filename}/" }, "batchSize": 2 } }falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalseoutput-bucketOutput BucketName of the GCS bucket where the output of the Vision job will be persisted. The value of this property applies when the JSON Payload property is configured. The JSON Payload property value can use Expression Language to reference the value of ${output-bucket}falsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsevision-feature-typeVision Feature TypeType of GCP Vision Feature. The value of this property applies when the JSON Payload property is configured. The JSON Payload property value can use Expression Language to reference the value of ${vision-feature-type}TEXT_DETECTIONfalsefalsetrueFLOWFILE_ATTRIBUTESfalsefalsefailureFlowFiles are routed to failure relationshipfalsesuccessFlowFiles are routed to success relationshipfalseoperationKeyA unique identifier of the operation returned by the Vision server.org.apache.nifi.processors.gcp.vision.GetGcpVisionAnnotateImagesOperationStatusorg.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerServiceCONTROLLER_SERVICEDefines credentials for Google Cloud Platform processors. Uses Application Default credentials without configuration. Application Default credentials support environmental variable (GOOGLE_APPLICATION_CREDENTIALS) pointing to a credential file, the config generated by `gcloud auth application-default login`, AppEngine/Compute Engine service accounts, etc.gcpcredentialsproviderapplication-default-credentialsUse Application Default CredentialsIf true, uses Google Application Default Credentials, which checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filepath to a service account JSON key, the config generated by the gcloud sdk, the App Engine service account, and the Compute Engine service account.falsetruetruefalsefalsefalsefalsefalseNONEfalsefalsecompute-engine-credentialsUse Compute Engine CredentialsIf true, uses Google Compute Engine Credentials of the Compute Engine VM Instance which NiFi is running on.falsetruetruefalsefalsefalsefalsefalseNONEfalsefalseservice-account-json-fileService Account JSON FilePath to a file containing a Service Account key file in JSON format.falsefalsefalseNONEfalsefalseSINGLEFILEservice-account-jsonService Account JSONThe raw JSON containing a Service Account keyfile.falsetruetrueENVIRONMENTfalsefalseproxy-configuration-serviceProxy Configuration ServiceSpecifies the Proxy Configuration Controller Service to proxy network requests.org.apache.nifi.proxy.ProxyConfigurationServiceorg.apache.nifinifi-standard-services-api-nar2.5.0falsefalsefalseNONEfalsefalseDelegation StrategyDelegation StrategyThe Delegation Strategy determines which account is used when calls are made with the GCP Credential.Service AccountService AccountService AccountThe service account should access data using it's own identity and permissions.Delegated AccountDelegated AccountThe service account should access data on behalf of a specified user account. This setting requires domain-wide delgation to be enabled for the service account for the scopes that it will be used in.truefalsefalseNONEfalsefalseDelegation UserDelegation UserThis user will be impersonated by the service account for api calls. API calls made using this credential will appear as if they are coming from delegate user with the delegate user's access. Any scopes supplied from processors to this credential must have domain-wide delegation setup with the service account.truefalsetrueENVIRONMENTfalsefalseDelegation StrategyDelegation StrategyDelegated Accountaccess environment credentialsThe default configuration can read environment variables and system properties for credentialsorg.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0org.apache.nifi.processors.gcp.storage.GCSFileResourceServiceCONTROLLER_SERVICEProvides a Google Compute Storage (GCS) file resource for other components.fileresourcegcsBucketBucketBucket of the object.${gcs.bucket}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalseNameNameName of the object.${filename}truefalsetrueFLOWFILE_ATTRIBUTESfalsefalsegcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalseFetch a specific file from GCS. The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.INPUT_ALLOWED"Bucket" = "${gcs.bucket}" "Name" = "${filename}" The "GCP Credentials Provider Service" property should specify an instance of the GCPCredentialsService in order to provide credentials for accessing the bucket. org.apache.nifi.processors.gcp.storage.FetchGCSObjectorg.apache.nifi.fileresource.service.api.FileResourceServiceorg.apache.nifinifi-standard-services-api-nar2.5.0org.apache.nifi.parameter.gcp.GcpSecretManagerParameterProviderPARAMETER_PROVIDERFetches parameters from GCP Secret Manager. Each secret becomes a Parameter, which can be mapped to a Parameter Group by adding a GCP label named 'group-name'.gcpsecretmanagergroup-name-patternGroup Name PatternA Regular Expression matching on the 'group-name' label value that identifies Secrets whose parameters should be fetched. Any secrets without a 'group-name' label value that matches this Regex will not be fetched..*truefalsefalseNONEfalsefalsegcp-project-idProject IDGoogle Cloud Project IDtruefalsefalseNONEfalsefalsegcp-credentials-provider-serviceGCP Credentials Provider ServiceThe Controller Service used to obtain Google Cloud Platform credentials.org.apache.nifi.gcp.credentials.service.GCPCredentialsServiceorg.apache.nifinifi-gcp-services-api-nar2.5.0truefalsefalseNONEfalsefalse