org.finra.herd.dao.impl

Class EmrDaoImpl

java.lang.Object

org.finra.herd.dao.impl.EmrDaoImpl

All Implemented Interfaces:

EmrDao

@Repository
public class EmrDaoImpl
extends Object
implements EmrDao

The EMR DAO implementation.

Constructor Summary

Constructors

Constructor and Description
EmrDaoImpl()

Method Summary

Modifier and Type	Method and Description
List<String>	addEmrMasterSecurityGroups(String clusterId, List<String> securityGroups, AwsParamsDto awsParams) Add Security groups to the master node of EMR cluster.
String	addEmrStep(String clusterId, com.amazonaws.services.elasticmapreduce.model.StepConfig emrStepConfig, AwsParamsDto awsParamsDto) Add an EMR Step.
String	createEmrCluster(String clusterName, EmrClusterDefinition emrClusterDefinition, AwsParamsDto awsParams) Create the EMR cluster.
com.amazonaws.services.elasticmapreduce.model.ClusterSummary	getActiveEmrClusterByName(String clusterName, AwsParamsDto awsParams) Get an Active EMR cluster by the cluster name.
List<com.amazonaws.services.elasticmapreduce.model.Application>	getApplications(List<EmrClusterDefinitionApplication> emrClusterDefinitionApplications) Converts the given list of EmrClusterDefinitionApplication into a list of Application
com.amazonaws.services.elasticmapreduce.model.StepSummary	getClusterActiveStep(String clusterId, AwsParamsDto awsParamsDto) Gets the active step on the cluster if any.
com.amazonaws.services.elasticmapreduce.model.Step	getClusterStep(String clusterId, String stepId, AwsParamsDto awsParamsDto) Gets the step on the cluster.
List<com.amazonaws.services.elasticmapreduce.model.Configuration>	getConfigurations(List<EmrClusterDefinitionConfiguration> emrClusterDefinitionConfigurations) Converts the given list of EmrClusterDefinitionConfiguration into a list of Configuration.
com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient	getEmrClient(AwsParamsDto awsParamsDto) Create the EMR client with the given proxy and access key details.
com.amazonaws.services.elasticmapreduce.model.Cluster	getEmrClusterById(String clusterId, AwsParamsDto awsParams) Get EMR cluster by cluster Id.
String	getEmrClusterStatusById(String clusterId, AwsParamsDto awsParams) Get EMR cluster status by cluster Id.
com.amazonaws.services.elasticmapreduce.model.Instance	getEmrMasterInstance(String clusterId, AwsParamsDto awsParams) Gets the master instance of the EMR cluster.
Map<String,String>	getMap(List<Parameter> parameters) Converts the given list of Parameter into a Map of String, String
void	terminateEmrCluster(String clusterId, boolean overrideTerminationProtection, AwsParamsDto awsParams) Terminates the EMR cluster.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmrDaoImpl

public EmrDaoImpl()

Method Detail

addEmrStep

public String addEmrStep(String clusterId,
                         com.amazonaws.services.elasticmapreduce.model.StepConfig emrStepConfig,
                         AwsParamsDto awsParamsDto)
                  throws Exception

Add an EMR Step. This method adds the step to EMR cluster based on the input.

Specified by:

addEmrStep in interface EmrDao

Parameters:

clusterId - EMR cluster ID.

emrStepConfig - the EMR step config to be added.

awsParamsDto - the proxy details.

There are five serializable objects supported currently. They are 1: ShellStep - For shell scripts 2: OozieStep - For Oozie workflow xml files 3: HiveStep - For hive scripts 4: HadoopJarStep - For Custom Map Reduce Jar files and 5: PigStep - For Pig scripts.

Returns:

the step id

Throws:

Exception

addEmrMasterSecurityGroups

public List<String> addEmrMasterSecurityGroups(String clusterId,
                                               List<String> securityGroups,
                                               AwsParamsDto awsParams)
                                        throws Exception

Add Security groups to the master node of EMR cluster.

Specified by:

addEmrMasterSecurityGroups in interface EmrDao

Parameters:

clusterId - EMR cluster Id.

securityGroups - the security groups list.

awsParams - the proxy details.

Returns:

the security groups that were added.

Throws:

Exception

getEmrMasterInstance

public com.amazonaws.services.elasticmapreduce.model.Instance getEmrMasterInstance(String clusterId,
                                                                                   AwsParamsDto awsParams)
                                                                            throws Exception

Gets the master instance of the EMR cluster.

Specified by:

getEmrMasterInstance in interface EmrDao

Parameters:

clusterId - EMR cluster id.

awsParams - the proxy details.

Returns:

the master instance of the cluster.

Throws:

Exception

createEmrCluster

public String createEmrCluster(String clusterName,
                               EmrClusterDefinition emrClusterDefinition,
                               AwsParamsDto awsParams)

Create the EMR cluster.

Specified by:

createEmrCluster in interface EmrDao

Parameters:

awsParams - AWS related parameters for access/secret keys and proxy details.

emrClusterDefinition - the EMR cluster definition that contains all the EMR parameters.

clusterName - the cluster name value.

Returns:

the cluster Id.

terminateEmrCluster

public void terminateEmrCluster(String clusterId,
                                boolean overrideTerminationProtection,
                                AwsParamsDto awsParams)

Terminates the EMR cluster.

Specified by:

terminateEmrCluster in interface EmrDao

Parameters:

clusterId - the cluster Id.

awsParams - AWS related parameters for access/secret keys and proxy details.

getEmrClusterById

public com.amazonaws.services.elasticmapreduce.model.Cluster getEmrClusterById(String clusterId,
                                                                               AwsParamsDto awsParams)

Get EMR cluster by cluster Id.

Specified by:

getEmrClusterById in interface EmrDao

Parameters:

clusterId - the job Id returned by EMR for the cluster.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

the cluster status.

getEmrClusterStatusById

public String getEmrClusterStatusById(String clusterId,
                                      AwsParamsDto awsParams)

Get EMR cluster status by cluster Id.

Specified by:

getEmrClusterStatusById in interface EmrDao

Parameters:

clusterId - the job Id returned by EMR for the cluster.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

the cluster status.

getActiveEmrClusterByName

public com.amazonaws.services.elasticmapreduce.model.ClusterSummary getActiveEmrClusterByName(String clusterName,
                                                                                              AwsParamsDto awsParams)

Get an Active EMR cluster by the cluster name. Cluster only in following states are returned: ClusterState.BOOTSTRAPPING, ClusterState.RUNNING, ClusterState.STARTING, ClusterState.WAITING

Specified by:

getActiveEmrClusterByName in interface EmrDao

Parameters:

awsParams - AWS related parameters for access/secret keys and proxy details.

clusterName - the cluster name value.

Returns:

the ClusterSummary object.

getClusterActiveStep

public com.amazonaws.services.elasticmapreduce.model.StepSummary getClusterActiveStep(String clusterId,
                                                                                      AwsParamsDto awsParamsDto)

Gets the active step on the cluster if any.

Specified by:

getClusterActiveStep in interface EmrDao

Parameters:

clusterId - the cluster id.

awsParamsDto - AWS related parameters for access/secret keys and proxy details.

Returns:

the step summary object.

getClusterStep

public com.amazonaws.services.elasticmapreduce.model.Step getClusterStep(String clusterId,
                                                                         String stepId,
                                                                         AwsParamsDto awsParamsDto)

Gets the step on the cluster.

Specified by:

getClusterStep in interface EmrDao

Parameters:

clusterId - the cluster id.

stepId - the step id to get details of.

awsParamsDto - AWS related parameters for access/secret keys and proxy details.

Returns:

the step object.

getEmrClient

public com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient getEmrClient(AwsParamsDto awsParamsDto)

Create the EMR client with the given proxy and access key details.

Specified by:

getEmrClient in interface EmrDao

Parameters:

awsParamsDto - AWS related parameters for access/secret keys and proxy details.

Returns:

the AmazonElasticMapReduceClient object.

getApplications

public List<com.amazonaws.services.elasticmapreduce.model.Application> getApplications(List<EmrClusterDefinitionApplication> emrClusterDefinitionApplications)

Converts the given list of EmrClusterDefinitionApplication into a list of Application

Parameters:

emrClusterDefinitionApplications - list of EmrClusterDefinitionApplication

Returns:

list Application

getMap

public Map<String,String> getMap(List<Parameter> parameters)

Converts the given list of Parameter into a Map of String, String

Parameters:

parameters - List of Parameter

Returns:

Map

getConfigurations

public List<com.amazonaws.services.elasticmapreduce.model.Configuration> getConfigurations(List<EmrClusterDefinitionConfiguration> emrClusterDefinitionConfigurations)

Converts the given list of EmrClusterDefinitionConfiguration into a list of Configuration.

Parameters:

emrClusterDefinitionConfigurations - list of EmrClusterDefinitionConfiguration

Returns:

list of Configuration