org.finra.herd.dao.helper

Class EmrHelper

java.lang.Object

org.finra.herd.dao.helper.AwsHelper

org.finra.herd.dao.helper.EmrHelper

@Component
public class EmrHelper
extends AwsHelper

A helper class that provides EMR functions.

Field Summary

Fields inherited from class org.finra.herd.dao.helper.AwsHelper

configurationHelper, herdStringHelper

Constructor Summary

Constructors

Constructor and Description
EmrHelper()

Method Summary

Modifier and Type	Method and Description
String	buildEmrClusterName(String namespaceCd, String emrDefinitionName, String clusterName) Returns EMR cluster name constructed according to the template defined.
String	getActiveEmrClusterId(String emrClusterId, String emrClusterName, String accountId) Gets the ID of an active EMR cluster which matches the given criteria.
AwsParamsDto	getAwsparamsDtoByAcccountId(String accountId) Get the AWS Params DTO for the account Id if no account id is specified, use the default
org.apache.oozie.client.WorkflowAction	getClientWorkflowAction(org.apache.oozie.client.WorkflowJob wrapperWorkflowJob) Retrieves the workflow action for the client workflow.
EmrDao	getEmrDao()
com.amazonaws.services.elasticmapreduce.model.StepConfig	getEmrHadoopJarStepConfig(String stepName, String jarLocation, String mainClass, List<String> scriptArguments, Boolean isContinueOnError) Builds the StepConfig for the Hadoop jar step.
ConfigurationValue	getEmrOozieHerdWorkflowS3LocationConfiguration() Gets the Oozie herd wrapper Workflow S3 Location ConfigurationValue.
org.apache.oozie.client.WorkflowAction	getFirstWorkflowActionInError(org.apache.oozie.client.WorkflowJob workflowJob) Retrieves the first workflow action that is in error.
String	getS3HdfsCopyScriptName() Gets the S3 to HDFS copy script name.
String	getS3StagingLocation() Get the S3_STAGING_RESOURCE full path from the bucket name as well as other details.
boolean	isActiveEmrState(String status)
void	setEmrDao(EmrDao emrDao)

Methods inherited from class org.finra.herd.dao.helper.AwsHelper

getAwsParamsDto, getClientConfiguration, getTransferRateInKilobytesPerSecond, getTransferRateInMegabitsPerSecond

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmrHelper

public EmrHelper()

Method Detail

buildEmrClusterName

public String buildEmrClusterName(String namespaceCd,
                                  String emrDefinitionName,
                                  String clusterName)

Returns EMR cluster name constructed according to the template defined.

Parameters:

namespaceCd - the namespace code value.

emrDefinitionName - the EMR definition name value.

clusterName - the cluster name value.

Returns:

the cluster name.

getS3StagingLocation

public String getS3StagingLocation()

Get the S3_STAGING_RESOURCE full path from the bucket name as well as other details.

Returns:

the s3 managed location.

getS3HdfsCopyScriptName

public String getS3HdfsCopyScriptName()
                               throws IllegalStateException

Gets the S3 to HDFS copy script name.

Returns:

the S3 to HDFS copy script name.

Throws:

IllegalStateException - if the S3 to HDGFS copy script name is not configured.

getEmrOozieHerdWorkflowS3LocationConfiguration

public ConfigurationValue getEmrOozieHerdWorkflowS3LocationConfiguration()
                                                                  throws IllegalStateException

Gets the Oozie herd wrapper Workflow S3 Location ConfigurationValue.

Returns:

the ConfigurationValue of S3 location of herd oozie wrapper workflow.

Throws:

IllegalStateException - if the S3 location of herd wrapper workflow is not configured.

getClientWorkflowAction

public org.apache.oozie.client.WorkflowAction getClientWorkflowAction(org.apache.oozie.client.WorkflowJob wrapperWorkflowJob)

Retrieves the workflow action for the client workflow. This is the sub workflow with the name OozieDaoImpl.ACTION_NAME_CLIENT_WORKFLOW. Returns null if not found.

Parameters:

wrapperWorkflowJob - the herd wrapper workflow job.

Returns:

the client workflow action.

getFirstWorkflowActionInError

public org.apache.oozie.client.WorkflowAction getFirstWorkflowActionInError(org.apache.oozie.client.WorkflowJob workflowJob)

Retrieves the first workflow action that is in error. Returns null if not found.

Parameters:

workflowJob - the oozie workflow job.

Returns:

the workflow action that has errors.

isActiveEmrState

public boolean isActiveEmrState(String status)

getEmrHadoopJarStepConfig

public com.amazonaws.services.elasticmapreduce.model.StepConfig getEmrHadoopJarStepConfig(String stepName,
                                                                                          String jarLocation,
                                                                                          String mainClass,
                                                                                          List<String> scriptArguments,
                                                                                          Boolean isContinueOnError)

Builds the StepConfig for the Hadoop jar step.

Parameters:

stepName - the step name.

jarLocation - the location of jar.

mainClass - the main class.

scriptArguments - the arguments.

isContinueOnError - indicate what to do on error.

Returns:

the stepConfig.

getActiveEmrClusterId

public String getActiveEmrClusterId(String emrClusterId,
                                    String emrClusterName,
                                    String accountId)

Gets the ID of an active EMR cluster which matches the given criteria. If both cluster ID and cluster name is specified, the name of the actual cluster with the given ID must match the specified name. For cases where the cluster is not found (does not exists or not active), the method fails. All parameters are case-insensitive and whitespace trimmed. Blank parameters are equal to null.

Parameters:

emrClusterId - EMR cluster ID

emrClusterName - EMR cluster name

accountId - the account Id that EMR cluster is running under

Returns:

The cluster ID

getEmrDao

public EmrDao getEmrDao()

setEmrDao

public void setEmrDao(EmrDao emrDao)

getAwsparamsDtoByAcccountId

public AwsParamsDto getAwsparamsDtoByAcccountId(String accountId)

Get the AWS Params DTO for the account Id if no account id is specified, use the default

Parameters:

accountId - account Id

Returns:

AwsParamsDto