org.finra.herd.service.impl

Class EmrServiceImpl

java.lang.Object

org.finra.herd.service.impl.EmrServiceImpl

All Implemented Interfaces:

EmrService

@Service
 @Transactional(value="herdTransactionManager")
public class EmrServiceImpl
extends Object
implements EmrService

The EMR service implementation.

Constructor Summary

Constructors

Constructor and Description
EmrServiceImpl()

Method Summary

Modifier and Type	Method and Description
EmrMasterSecurityGroup	addSecurityGroupsToClusterMaster(EmrMasterSecurityGroupAddRequest request) Adds security groups to the master node of an existing EMR Cluster.
protected EmrMasterSecurityGroup	addSecurityGroupsToClusterMasterImpl(EmrMasterSecurityGroupAddRequest request) Adds security groups to the master node of an existing EMR Cluster.
Object	addStepToCluster(Object request) Adds step to an existing EMR Cluster.
protected Object	addStepToClusterImpl(Object request) Adds step to an existing EMR Cluster.
EmrCluster	createCluster(EmrClusterCreateRequest request) Creates a new EMR Cluster.
protected EmrCluster	createClusterImpl(EmrClusterCreateRequest request) Creates a new EMR cluster based on the given request if the cluster with the given name does not already exist.
EmrCluster	getCluster(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto, String emrClusterId, String emrStepId, boolean verbose, boolean retrieveOozieJobs, String accountId) Gets details of an existing EMR Cluster.
protected EmrCluster	getClusterImpl(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto, String emrClusterId, String emrStepId, boolean verbose, boolean retrieveOozieJobs, String accountId) Gets details of an existing EMR Cluster.
OozieWorkflowJob	getEmrOozieWorkflowJob(String namespace, String emrClusterDefinitionName, String emrClusterName, String oozieWorkflowJobId, Boolean verbose, String emrClusterId, String accountId) Get the oozie workflow.
protected OozieWorkflowJob	getEmrOozieWorkflowJobImpl(String namespace, String emrClusterDefinitionName, String emrClusterName, String oozieWorkflowJobId, Boolean verbose, String emrClusterId, String accountId) Get the oozie workflow.
OozieWorkflowJob	runOozieWorkflow(RunOozieWorkflowRequest request) Runs oozie job on an existing EMR Cluster.
protected OozieWorkflowJob	runOozieWorkflowImpl(RunOozieWorkflowRequest request) Runs oozie job on an existing EMR Cluster.
EmrCluster	terminateCluster(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto, boolean overrideTerminationProtection, String emrClusterId, String accountId) Terminates the EMR Cluster.
protected EmrCluster	terminateClusterImpl(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto, boolean overrideTerminationProtection, String emrClusterId, String accountId) Terminates the EMR Cluster.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmrServiceImpl

public EmrServiceImpl()

Method Detail

getCluster

@NamespacePermission(fields="#emrClusterAlternateKeyDto?.namespace",
                     permissions=READ)
 @Transactional(propagation=REQUIRES_NEW)
public EmrCluster getCluster(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto,
                                                                                                                                                                                   String emrClusterId,
                                                                                                                                                                                   String emrStepId,
                                                                                                                                                                                   boolean verbose,
                                                                                                                                                                                   boolean retrieveOozieJobs,
                                                                                                                                                                                   String accountId)
                                                                                                                                                                            throws Exception

Gets details of an existing EMR Cluster. Creates its own transaction.

Specified by:

getCluster in interface EmrService

Parameters:

emrClusterId - the cluster id of the cluster to get details

emrStepId - the step id of the step to get details

verbose - parameter for whether to return detailed information

retrieveOozieJobs - parameter for whether to retrieve oozie job information

emrClusterAlternateKeyDto - the EMR cluster alternate key

Returns:

the EMR Cluster object with details.

Throws:

Exception

getClusterImpl

protected EmrCluster getClusterImpl(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto,
                                    String emrClusterId,
                                    String emrStepId,
                                    boolean verbose,
                                    boolean retrieveOozieJobs,
                                    String accountId)
                             throws Exception

Gets details of an existing EMR Cluster.

Parameters:

emrClusterAlternateKeyDto - the EMR cluster alternate key

emrClusterId - the cluster id of the cluster to get details

emrStepId - the step id of the step to get details

verbose - parameter for whether to return detailed information

retrieveOozieJobs - parameter for whether to retrieve oozie job information

accountId - the optional AWS account that EMR cluster is running in

Returns:

the EMR Cluster object with details.

Throws:

Exception - if an error occurred while getting the cluster.

createCluster

@NamespacePermission(fields="#request?.namespace",
                     permissions=EXECUTE)
 @Transactional(propagation=REQUIRES_NEW)
public EmrCluster createCluster(EmrClusterCreateRequest request)
                                                                                                                                                                throws Exception

Creates a new EMR Cluster. Creates its own transaction.

Specified by:

createCluster in interface EmrService

Parameters:

request - the EMR cluster create request

Returns:

the created EMR cluster object

Throws:

Exception - if there were any errors while creating the cluster.

createClusterImpl

protected EmrCluster createClusterImpl(EmrClusterCreateRequest request)
                                throws Exception

Creates a new EMR cluster based on the given request if the cluster with the given name does not already exist. If the cluster already exist, returns the information about the existing cluster.

The request must contain:

• A namespace and definition name which refer to an existing EMR cluster definition.
• A valid cluster name to create.

The request may optionally contain:

• A "dry run" flag, which when set to true, no calls to AWS will occur, but validations and override will. Defaults to false.
• An override parameter, which when set, overrides the given parameters in the cluster definition before creating the cluster. Defaults to no override.

A successful response will contain:

• The ID of the cluster that was created, or if the cluster already exists, the ID of the cluster that exists. This field will be null when dry run flag is true.
• The status of the cluster that was created or already exists. The status will normally be "Starting" on successful creations. This field will be null when dry run flag is true
• The namespace, definition name, and cluster name of the cluster.
• The dry run flag, if given in the request.
• An indicator whether the cluster was created or not. If the cluster already exists, the cluster will not be created and this flag will be set to false.
• The definition which was used to create the cluster. If any overrides were given, this definition's values will be the values after the override. This field will be null if the cluster was not created.

Notes:

• At any point of the execution, if there are validation errors, the method will immediately throw an exception.
• Even if the validations pass, AWS may still reject the request, which will cause this method to throw an exception.
• Dry runs do not make any calls to AWS, therefore AWS may still reject the creation request even when a dry run succeeds.

Parameters:

request - - EmrClusterCreateRequest The EMR cluster create request

Returns:

EmrCluster the created EMR cluster object

Throws:

Exception - when the original EMR cluster definition XML is malformed

terminateCluster

@NamespacePermission(fields="#emrClusterAlternateKeyDto?.namespace",
                     permissions=EXECUTE)
 @Transactional(propagation=REQUIRES_NEW)
public EmrCluster terminateCluster(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto,
                                                                                                                                                                                            boolean overrideTerminationProtection,
                                                                                                                                                                                            String emrClusterId,
                                                                                                                                                                                            String accountId)
                                                                                                                                                                                     throws Exception

Terminates the EMR Cluster. Creates its own transaction.

Specified by:

terminateCluster in interface EmrService

Parameters:

emrClusterAlternateKeyDto - the EMR cluster alternate key

overrideTerminationProtection - parameter for whether to override termination protection

Returns:

the terminated EMR cluster object

Throws:

Exception - if there were any errors while terminating the cluster.

terminateClusterImpl

protected EmrCluster terminateClusterImpl(EmrClusterAlternateKeyDto emrClusterAlternateKeyDto,
                                          boolean overrideTerminationProtection,
                                          String emrClusterId,
                                          String accountId)
                                   throws Exception

Terminates the EMR Cluster.

Parameters:

emrClusterAlternateKeyDto - the EMR cluster alternate key

overrideTerminationProtection - parameter for whether to override termination protection

emrClusterId - The EMR cluster ID

accountId - The account Id

Returns:

the terminated EMR cluster object

Throws:

Exception - if there were any errors while terminating the cluster.

addStepToCluster

@NamespacePermission(fields="#request?.namespace",
                     permissions=EXECUTE)
 @Transactional(propagation=REQUIRES_NEW)
public Object addStepToCluster(Object request)
                                                                                                                                                               throws Exception

Adds step to an existing EMR Cluster. Creates its own transaction.

There are five serializable objects supported currently. They are 1: ShellStep - For shell scripts 2: OozieStep - For Oozie workflow xml files 3: HiveStep - For hive scripts 4: HadoopJarStep - For Custom Map Reduce Jar files and 5: PigStep - For Pig scripts.

Specified by:

addStepToCluster in interface EmrService

Parameters:

request - the EMR steps add request

Returns:

the EMR steps add object with added steps

Throws:

Exception - if there were any errors while adding a step to the cluster.

addStepToClusterImpl

protected Object addStepToClusterImpl(Object request)
                               throws Exception

Adds step to an existing EMR Cluster.

Parameters:

request - the EMR steps add request

Returns:

the EMR step add object with added steps

Throws:

Exception - if there were any errors while adding a step to the cluster.

addSecurityGroupsToClusterMaster

@NamespacePermission(fields="#request?.namespace",
                     permissions=WRITE)
 @Transactional(propagation=REQUIRES_NEW)
public EmrMasterSecurityGroup addSecurityGroupsToClusterMaster(EmrMasterSecurityGroupAddRequest request)
                                                                                                                                                                                             throws Exception

Adds security groups to the master node of an existing EMR Cluster. Creates its own transaction.

Specified by:

addSecurityGroupsToClusterMaster in interface EmrService

Parameters:

request - the EMR master security group add request

Returns:

the added EMR master security groups

Throws:

Exception - if there were any errors adding the security groups to the cluster master.

addSecurityGroupsToClusterMasterImpl

protected EmrMasterSecurityGroup addSecurityGroupsToClusterMasterImpl(EmrMasterSecurityGroupAddRequest request)
                                                               throws Exception

Adds security groups to the master node of an existing EMR Cluster.

Parameters:

request - the EMR master security group add request

Returns:

the added EMR master security groups

Throws:

Exception - if there were any errors adding the security groups to the cluster master.

runOozieWorkflow

@NamespacePermission(fields="#request?.namespace",
                     permissions=EXECUTE)
 @Transactional(propagation=REQUIRES_NEW)
public OozieWorkflowJob runOozieWorkflow(RunOozieWorkflowRequest request)
                                                                                                                                                                         throws Exception

Runs oozie job on an existing EMR Cluster. Creates its own transaction.

Specified by:

runOozieWorkflow in interface EmrService

Parameters:

request - the Run oozie workflow request

Returns:

the oozie workflow job that was submitted.

Throws:

Exception - if there were any errors while submitting the job.

runOozieWorkflowImpl

protected OozieWorkflowJob runOozieWorkflowImpl(RunOozieWorkflowRequest request)
                                         throws Exception

Runs oozie job on an existing EMR Cluster.

Parameters:

request - the Run oozie workflow request

Returns:

the oozie workflow job that was submitted.

Throws:

Exception - if there were any errors while submitting the job.

getEmrOozieWorkflowJob

@NamespacePermission(fields="#namespace",
                     permissions=READ)
 @Transactional(propagation=REQUIRES_NEW)
public OozieWorkflowJob getEmrOozieWorkflowJob(String namespace,
                                                                                                                                                                          String emrClusterDefinitionName,
                                                                                                                                                                          String emrClusterName,
                                                                                                                                                                          String oozieWorkflowJobId,
                                                                                                                                                                          Boolean verbose,
                                                                                                                                                                          String emrClusterId,
                                                                                                                                                                          String accountId)
                                                                                                                                                                   throws Exception

Get the oozie workflow. Starts a new transaction.

Specified by:

getEmrOozieWorkflowJob in interface EmrService

Parameters:

namespace - the namespace

emrClusterDefinitionName - the EMR cluster definition name

emrClusterName - the EMR cluster name

oozieWorkflowJobId - the ooxie workflow Id.

verbose - the flag to indicate whether to return verbose information

emrClusterId - The EMR cluster ID

accountId - account Id

Returns:

OozieWorkflowJob OozieWorkflowJob

Throws:

Exception - Exception

getEmrOozieWorkflowJobImpl

protected OozieWorkflowJob getEmrOozieWorkflowJobImpl(String namespace,
                                                      String emrClusterDefinitionName,
                                                      String emrClusterName,
                                                      String oozieWorkflowJobId,
                                                      Boolean verbose,
                                                      String emrClusterId,
                                                      String accountId)
                                               throws Exception

Get the oozie workflow.

Parameters:

namespace - the namespace

emrClusterDefinitionName - the EMR cluster definition name

emrClusterName - the EMR cluster name

oozieWorkflowJobId - the ooxie workflow Id.

verbose - the flag to indicate whether to return verbose information

emrClusterId - The EMR cluster ID

accountId - The account Id

Returns:

OozieWorkflowJob OozieWorkflowJob

Throws:

Exception - Exception