Package org.rcsb.mmtf.spark.utils

Class SparkUtils

  • public class SparkUtils
extends java.lang.Object
    A class of Spark utility methods
    Author:
    Anthony Bradley

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String CHANGELOG_LOCATION
      The URL to find the Changelog for MMTF.
      static java.lang.String FULL_LOCATION
      The URL to find the Hadoop sequence file for the full version of MMTF.
      static java.lang.String REDUCED_LOCATION
      The URL to find the Hadoop sequence file for the reduced version of MMTF.

    • Constructor Summary

      Constructors 
      Constructor Description
      SparkUtils()  

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static void combineDirToFile​(java.io.File dir, java.lang.String header)
      Join a Spark output dir of text files into a single text file.
      static <T> org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> convertToDataFrame​(org.apache.spark.api.java.JavaRDD<T> javaRDD, java.lang.Class<T> clazz)
      Convert a JavaRDD to a Dataframe (Spark 2.0.0 Dataset(Row) = Dataframe) of the same type.
      static <T> org.apache.spark.sql.Dataset<T> convertToDataset​(org.apache.spark.api.java.JavaRDD<T> javaRDD, java.lang.Class<T> clazz)
      Convert a JavaRDD to a Dataset of the same type.
      static void downloadPdb​(java.lang.String inputPath)
      Function to download the PDB and place it on the file system.
      static void filePath​(java.lang.String filePath)
      Set the file path of the Hadoop file to read.
      static java.util.Map<java.lang.String,​java.util.Set<java.lang.String>> findUpdates​(java.lang.String changelogLocation, java.util.Date lastUpdatedDate)
      Find the updates from the change logs using.
      AtomSelectObject generateAtomSelectObject​(java.util.List<java.lang.String> atomNameList, java.util.List<java.lang.String> groupNameList, boolean charged, java.util.List<java.lang.String> elementNameList, java.lang.String groupType)
      Utility function to generate an AtomSelectObject.
      static javax.vecmath.Point3d getCalpha​(org.rcsb.mmtf.api.StructureDataInterface structureDataInterface, int groupType, int atomCounter)
      Get the calpha as a Point3d.
      static SegmentDataRDD getCalphaChains​(java.lang.String[] inputIds)
      Get the Calpha chains for a few structures as a SegmentDataRDD.
      static org.apache.spark.api.java.JavaPairRDD<java.lang.Integer,​java.lang.Integer> getComparisonMatrix​(int numMembers)
      Get a JavaPairRDD of Integers to do a half matrix comparison.
      static org.apache.spark.SparkConf getConf()
      Get the SparkConf for this run.
      static byte[] getDataAsByteArray​(java.lang.String pdbCode)
      Helper function to get the data for a PDB id as an gzip compressed byte array.
      static java.lang.String getFilePath()
      Get the path to the Hadoop sequence file to read.
      static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​org.rcsb.mmtf.api.StructureDataInterface> getFromList​(java.util.List<java.lang.String> pdbIdList)
      Generate a JavaPairRDD of String (PDB ID) and Value StructureDataInterface from a list of PDB ids
      static java.lang.String getFullPdbFile()
      Get the path for where the full PDB file will be stored.
      static java.lang.String getHadoopBase()
      Get the base path of where to store Hadoop data.
      static <K extends java.lang.Comparable<K>,​V>
      org.apache.spark.api.java.JavaPairRDD<scala.Tuple2<K,​V>,​scala.Tuple2<K,​V>>      		 getHalfCartesian​(org.apache.spark.api.java.JavaPairRDD<K,​V> inputRDD)
      Get a half cartesian - using the first part of the input RDD as a string key.
      static <K extends java.lang.Comparable<K>,​V>
      org.apache.spark.api.java.JavaPairRDD<scala.Tuple2<K,​V>,​scala.Tuple2<K,​V>>      		 getHalfCartesian​(org.apache.spark.api.java.JavaPairRDD<K,​V> inputRDD, int numPartitions)
      Get a half cartesian - using the first part of the input RDD as a string key.
      static <T> org.apache.spark.api.java.JavaRDD<T> getJavaRdd​(org.apache.spark.sql.Dataset<T> atomDataset, java.lang.Class<T> class1)
      Get a JavaRDD from a Dataset.
      static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​org.rcsb.mmtf.dataholders.MmtfStructure> getMmtfStructureRdd​(java.lang.String filePath)
      Get a JavaPairRDD of String MmtfStructure from a file path
      static org.apache.spark.api.java.JavaSparkContext getSparkContext()
      Get the JavaSparkContext for this run.
      static org.apache.spark.api.java.JavaSparkContext getSparkContext​(org.apache.spark.SparkConf conf)
      Get the JavaSparkContext for this run.
      static org.apache.spark.sql.SparkSession getSparkSession()
      Get the SparkSession for this run
      static org.apache.spark.sql.SQLContext getSqlContext()
      Get the SQLContext for producing datasets and data frames.
      static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​org.rcsb.mmtf.api.StructureDataInterface> getStructureDataRdd​(java.lang.String filePath)
      Get an JavaPairRDD of String StructureDataInterface from a file path.
      static StructureDataRDD getStructureObj​(java.lang.String filePath)
      Get the StructureDataRDD from a file path.
      static byte[] gzipCompress​(byte[] byteArray)
      Compress a byte array using Gzip.
      static void joinAndSave​(org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetOne, org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetTwo, java.lang.String outputUri)
      Join two RDDs together into one larger one and save it as a new file
      static org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> joinDatasets​(org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetOne, org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetTwo)
      Join two RDDs together into one larger one.
      static java.util.Date parseDate​(java.lang.String releaseDate)
      Parse the date as it is stored in the String.
      static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​byte[]> readHadoopFile​(java.lang.String filePath)
      Read a hadoop sequence file to a Strign byte[] pair.
      StructureDataRDD selectSubset​(java.lang.String filePath, float subsetToChose)
      Select a subset of the PDB in a fast and efficient way.
      StructureDataRDD selectSubset​(java.lang.String filePath, java.util.List<java.lang.String> subsetToChose)
      Select a subset of the PDB in a fast and efficient way.
      static void shutdown()
      Gently shutdown at the end of a run.
      static void updatePdb​(java.lang.String filePath)
      Update the PDB data from when it was last updated.
      static java.lang.String writeDate​(java.util.Date releaseDate)
      Write the date out as a string using the format we are using.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • FULL_LOCATION

        public static final java.lang.String FULL_LOCATION
        The URL to find the Hadoop sequence file for the full version of MMTF.
        See Also:
        Constant Field Values

      • REDUCED_LOCATION

        public static final java.lang.String REDUCED_LOCATION
        The URL to find the Hadoop sequence file for the reduced version of MMTF.
        See Also:
        Constant Field Values

      • CHANGELOG_LOCATION

        public static final java.lang.String CHANGELOG_LOCATION
        The URL to find the Changelog for MMTF.
        See Also:
        Constant Field Values

    • Constructor Detail

      • SparkUtils

        public SparkUtils()

    • Method Detail

      • convertToDataset

        public static <T> org.apache.spark.sql.Dataset<T> convertToDataset​(org.apache.spark.api.java.JavaRDD<T> javaRDD,
                                                                   java.lang.Class<T> clazz)
        Convert a JavaRDD to a Dataset of the same type.
        Type Parameters:
        T - the type of the RDD
        Parameters:
        javaRDD - the input JavaRDD
        clazz - the input class of the RDD
        Returns:
        the converted dataset

      • convertToDataFrame

        public static <T> org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> convertToDataFrame​(org.apache.spark.api.java.JavaRDD<T> javaRDD,
                                                                                            java.lang.Class<T> clazz)
        Convert a JavaRDD to a Dataframe (Spark 2.0.0 Dataset(Row) = Dataframe) of the same type.
        Type Parameters:
        T - the type of the RDD
        Parameters:
        javaRDD - the input JavaRDD
        clazz - the input class of the RDD
        Returns:
        the converted dataframe

      • getStructureDataRdd

        public static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​org.rcsb.mmtf.api.StructureDataInterface> getStructureDataRdd​(java.lang.String filePath)
        Get an JavaPairRDD of String StructureDataInterface from a file path.
        Parameters:
        filePath - the input path to the hadoop sequence file
        Returns:
        the JavaPairRDD of String StructureDataInterface

      • getMmtfStructureRdd

        public static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​org.rcsb.mmtf.dataholders.MmtfStructure> getMmtfStructureRdd​(java.lang.String filePath)
        Get a JavaPairRDD of String MmtfStructure from a file path
        Parameters:
        filePath - the input path to the hadoop sequence file
        Returns:
        the JavaPairRDD of String MmtfStructure

      • selectSubset

        public StructureDataRDD selectSubset​(java.lang.String filePath,
                                     java.util.List<java.lang.String> subsetToChose)
        Select a subset of the PDB in a fast and efficient way.
        Parameters:
        filePath - the input Hadoop file to read
        subsetToChose - the list of PDB ids to chose
        Returns:
        the StructureDataRDD of the selected data

      • selectSubset

        public StructureDataRDD selectSubset​(java.lang.String filePath,
                                     float subsetToChose)
        Select a subset of the PDB in a fast and efficient way.
        Parameters:
        filePath - the input Hadoop file to read
        subsetToChose - the fraction to chose
        Returns:
        the StructureDataRDD of the selected data

      • readHadoopFile

        public static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​byte[]> readHadoopFile​(java.lang.String filePath)
        Read a hadoop sequence file to a Strign byte[] pair.
        Parameters:
        filePath - the input path of the hadoop file
        Returns:
        the JavaPairRDD of the PDB code and the byte array of the data.

      • getConf

        public static org.apache.spark.SparkConf getConf()
        Get the SparkConf for this run.
        Returns:
        the SparkConf for this run

      • getSparkContext

        public static org.apache.spark.api.java.JavaSparkContext getSparkContext()
        Get the JavaSparkContext for this run.
        Returns:
        the JavaSparkContext for this run

      • getSqlContext

        public static org.apache.spark.sql.SQLContext getSqlContext()
        Get the SQLContext for producing datasets and data frames.
        Returns:
        the SQLContext.

      • getSparkSession

        public static org.apache.spark.sql.SparkSession getSparkSession()
        Get the SparkSession for this run
        Returns:
        the SparkSession for this run

      • getSparkContext

        public static org.apache.spark.api.java.JavaSparkContext getSparkContext​(org.apache.spark.SparkConf conf)
        Get the JavaSparkContext for this run.
        Parameters:
        conf - the SparkConf to use to setup the context
        Returns:
        the JavaSparkContext for this run

      • shutdown

        public static void shutdown()
        Gently shutdown at the end of a run.

      • filePath

        public static void filePath​(java.lang.String filePath)
        Set the file path of the Hadoop file to read.
        Parameters:
        filePath - the input file path to read

      • getCalpha

        public static javax.vecmath.Point3d getCalpha​(org.rcsb.mmtf.api.StructureDataInterface structureDataInterface,
                                              int groupType,
                                              int atomCounter)
        Get the calpha as a Point3d.
        Parameters:
        structureDataInterface - the StructureDataInterface to read
        groupType - the integer specifying the grouptype
        atomCounter - the atom count at the start of this group
        Returns:
        the point3d object specifying the calpha of this point

      • gzipCompress

        public static byte[] gzipCompress​(byte[] byteArray)
                           throws java.io.IOException
        Compress a byte array using Gzip.
        Parameters:
        byteArray - the input byte array
        Returns:
        the compressed byte array
        Throws:
        java.io.IOException - an error reading from the URL

      • getFilePath

        public static java.lang.String getFilePath()
        Get the path to the Hadoop sequence file to read.
        Returns:
        the String path of the Hadoop sequence file to read.

      • downloadPdb

        public static void downloadPdb​(java.lang.String inputPath)
                        throws java.io.FileNotFoundException,
                               java.io.IOException
        Function to download the PDB and place it on the file system.
        Parameters:
        inputPath - the input path to read from
        Throws:
        java.io.IOException - an error reading from the URL
        java.io.FileNotFoundException - an error transferring files

      • getFullPdbFile

        public static java.lang.String getFullPdbFile()
        Get the path for where the full PDB file will be stored.
        Returns:
        the String describing where the full PDB data is

      • getHadoopBase

        public static java.lang.String getHadoopBase()
        Get the base path of where to store Hadoop data.
        Returns:
        the String of the path of where Hadoop data should be

      • getCalphaChains

        public static SegmentDataRDD getCalphaChains​(java.lang.String[] inputIds)
                                      throws java.io.IOException
        Get the Calpha chains for a few structures as a SegmentDataRDD.
        Parameters:
        inputIds - the list of input ids as strings
        Returns:
        the SegmentDataRDD of the calpha chains given the ids.
        Throws:
        java.io.IOException - due to reading from the MMTF url

      • getDataAsByteArray

        public static byte[] getDataAsByteArray​(java.lang.String pdbCode)
                                 throws java.io.IOException
        Helper function to get the data for a PDB id as an gzip compressed byte array. Data is retrieved from the REST service. This should be moved to mmtf for the next release.
        Parameters:
        pdbCode - the input PDB id
        Returns:
        the gzip compressed byte array for this structure
        Throws:
        java.io.IOException - due to retrieving data from the URL

      • generateAtomSelectObject

        public AtomSelectObject generateAtomSelectObject​(java.util.List<java.lang.String> atomNameList,
                                                 java.util.List<java.lang.String> groupNameList,
                                                 boolean charged,
                                                 java.util.List<java.lang.String> elementNameList,
                                                 java.lang.String groupType)
        Utility function to generate an AtomSelectObject. Mainly for application to the Python API.
        Parameters:
        atomNameList - the list of atoms to consider
        groupNameList - the list of groups to consider (e.g. LYS)
        charged - whether to consider charged atoms only (true)
        elementNameList - the list of elements to consider
        groupType - a string defining the type of group
        Returns:
        an atom select object of the appropriate type.

      • getComparisonMatrix

        public static org.apache.spark.api.java.JavaPairRDD<java.lang.Integer,​java.lang.Integer> getComparisonMatrix​(int numMembers)
        Get a JavaPairRDD of Integers to do a half matrix comparison. i.e. all comparisons where i!=j and i is greather than j
        Parameters:
        numMembers - the total number of members to compare
        Returns:
        the JavaPairRDD of the comparisons

      • combineDirToFile

        public static void combineDirToFile​(java.io.File dir,
                                    java.lang.String header)
                             throws java.io.IOException
        Join a Spark output dir of text files into a single text file.
        Parameters:
        dir - the file the partitions are written to
        header - the header for the top of the file
        Throws:
        java.io.IOException - an error joining the files

      • getHalfCartesian

        public static <K extends java.lang.Comparable<K>,​V> org.apache.spark.api.java.JavaPairRDD<scala.Tuple2<K,​V>,​scala.Tuple2<K,​V>> getHalfCartesian​(org.apache.spark.api.java.JavaPairRDD<K,​V> inputRDD)
        Get a half cartesian - using the first part of the input RDD as a string key.
        Type Parameters:
        K - the type of the value
        V - the type of the value
        Parameters:
        inputRDD - the input rdd - keys are
        Returns:
        the RDD of all non-repated comparisons

      • getHalfCartesian

        public static <K extends java.lang.Comparable<K>,​V> org.apache.spark.api.java.JavaPairRDD<scala.Tuple2<K,​V>,​scala.Tuple2<K,​V>> getHalfCartesian​(org.apache.spark.api.java.JavaPairRDD<K,​V> inputRDD,
                                                                                                                                                                        int numPartitions)
        Get a half cartesian - using the first part of the input RDD as a string key.
        Type Parameters:
        K - the type of the value
        V - the type of the value
        Parameters:
        inputRDD - the input rdd - keys are
        numPartitions - the number of partitions to repartition
        Returns:
        the RDD of all non-repated comparisons

      • getFromList

        public static org.apache.spark.api.java.JavaPairRDD<java.lang.String,​org.rcsb.mmtf.api.StructureDataInterface> getFromList​(java.util.List<java.lang.String> pdbIdList)
                                                                                                                          throws java.io.IOException
        Generate a JavaPairRDD of String (PDB ID) and Value StructureDataInterface from a list of PDB ids
        Parameters:
        pdbIdList - the list of PDB ids
        Returns:
        the JavaPairRDD of String and StructureDataInterface
        Throws:
        java.io.IOException - due to reading from the URL

      • getJavaRdd

        public static <T> org.apache.spark.api.java.JavaRDD<T> getJavaRdd​(org.apache.spark.sql.Dataset<T> atomDataset,
                                                                  java.lang.Class<T> class1)
        Get a JavaRDD from a Dataset.
        Type Parameters:
        T - the type of the dataset
        Parameters:
        atomDataset - the dataset to convert
        class1 - the class of the dataset
        Returns:
        the JavaRDD fromn the dataset

      • parseDate

        public static java.util.Date parseDate​(java.lang.String releaseDate)
                                throws java.text.ParseException
        Parse the date as it is stored in the String.
        Parameters:
        releaseDate - the input date as a string
        Returns:
        the parsed date
        Throws:
        java.text.ParseException - an error in parsing the date

      • writeDate

        public static java.lang.String writeDate​(java.util.Date releaseDate)
        Write the date out as a string using the format we are using.
        Parameters:
        releaseDate - the Date being considered
        Returns:
        the String representation of that date

      • findUpdates

        public static java.util.Map<java.lang.String,​java.util.Set<java.lang.String>> findUpdates​(java.lang.String changelogLocation,
                                                                                                java.util.Date lastUpdatedDate)
        Find the updates from the change logs using. Only those after the date of the initial input will be used.
        Parameters:
        changelogLocation - the location of the change log information
        lastUpdatedDate - the date of the last update
        Returns:
        the REMOVED and ADDED sets of PDB codes to be used in the update

      • updatePdb

        public static void updatePdb​(java.lang.String filePath)
                      throws java.io.IOException
        Update the PDB data from when it was last updated.
        Parameters:
        filePath - the input Hadoop sequence folder path
        Throws:
        java.io.IOException - an error writing the final output file

      • joinAndSave

        public static void joinAndSave​(org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetOne,
                               org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetTwo,
                               java.lang.String outputUri)
        Join two RDDs together into one larger one and save it as a new file
        Parameters:
        datasetOne - the first JavaPairRDD to join
        datasetTwo - the second JavaPairRDD to join
        outputUri - the output URI to write to

      • joinDatasets

        public static org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> joinDatasets​(org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetOne,
                                                                                                                                     org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.Text,​org.apache.hadoop.io.BytesWritable> datasetTwo)
        Join two RDDs together into one larger one.
        Parameters:
        datasetOne - the first JavaPairRDD to join
        datasetTwo - the second JavaPairRDD to join
        Returns:
        the JavaPairRDD of the joined datasets