Class

org.opencypher.spark.api.io.fs

FSGraphSource

Related Doc: package fs

Permalink

class FSGraphSource extends AbstractPropertyGraphDataSource with JsonSerialization

Data source implementation that handles the writing of files and tables to a filesystem.

By default Spark is used to write tables and the Hadoop filesystem configured in Spark is used to write files. The file/folder/table structure into which the graphs are stored is defined in DefaultGraphDirectoryStructure.

Linear Supertypes
JsonSerialization, AbstractPropertyGraphDataSource, CAPSPropertyGraphDataSource, PropertyGraphDataSource, AnyRef, Any
Known Subclasses
Neo4jBulkCSVDataSink
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FSGraphSource
  2. JsonSerialization
  3. AbstractPropertyGraphDataSource
  4. CAPSPropertyGraphDataSource
  5. PropertyGraphDataSource
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FSGraphSource(rootPath: String, tableStorageFormat: String, customFileSystem: Option[CAPSFileSystem] = None, filesPerTable: Option[Int] = None)(implicit caps: CAPSSession)

    Permalink

    rootPath

    path where the graphs are stored

    tableStorageFormat

    Spark configuration parameter for the table format

    customFileSystem

    optional alternative filesystem to use for writing files

    filesPerTable

    optional parameter that specifies how many files a table is coalesced into, by default 1

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. implicit val caps: CAPSSession

    Permalink
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  6. def checkStorable(name: GraphName): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    CAPSPropertyGraphDataSource

  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. val customFileSystem: Option[CAPSFileSystem]

    Permalink

    optional alternative filesystem to use for writing files

  9. def delete(graphName: GraphName): Unit

    Permalink
    Definition Classes
    AbstractPropertyGraphDataSource → PropertyGraphDataSource

  10. def deleteDirectory(path: String): Unit

    Permalink
    Attributes
    protected

  11. def deleteGraph(graphName: GraphName): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  12. val directoryStructure: DefaultGraphDirectoryStructure

    Permalink
    Attributes
    protected

  13. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  15. lazy val fileSystem: CAPSFileSystem

    Permalink
    Attributes
    protected

  16. val filesPerTable: Option[Int]

    Permalink

    optional parameter that specifies how many files a table is coalesced into, by default 1

  17. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  18. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  19. def graph(name: GraphName): PropertyGraph

    Permalink
    Definition Classes
    AbstractPropertyGraphDataSource → PropertyGraphDataSource

  20. var graphNameCache: Set[GraphName]

    Permalink
    Attributes
    protected
    Definition Classes
    AbstractPropertyGraphDataSource

  21. def graphNames: Set[GraphName]

    Permalink
    Definition Classes
    AbstractPropertyGraphDataSource → PropertyGraphDataSource

  22. def hasGraph(graphName: GraphName): Boolean

    Permalink
    Definition Classes
    AbstractPropertyGraphDataSource → PropertyGraphDataSource

  23. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  24. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  25. def listDirectories(path: String): List[String]

    Permalink
    Attributes
    protected

  26. def listGraphNames: List[String]

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  27. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  28. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  29. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  30. def readCAPSGraphMetaData(graphName: GraphName): CAPSGraphMetaData

    Permalink
    Attributes
    protected
    Definition Classes
    JsonSerialization

  31. def readFile(path: String): String

    Permalink
    Attributes
    protected

  32. def readJsonCAPSGraphMetaData(graphName: GraphName): String

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → JsonSerialization

  33. def readJsonSchema(graphName: GraphName): String

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → JsonSerialization

  34. def readNodeTable(graphName: GraphName, labels: Set[String], sparkSchema: StructType): DataFrame

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  35. def readRelationshipTable(graphName: GraphName, relKey: String, sparkSchema: StructType): DataFrame

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  36. def readSchema(graphName: GraphName): CAPSSchema

    Permalink
    Attributes
    protected
    Definition Classes
    JsonSerialization

  37. def readTable(path: String, schema: StructType): DataFrame

    Permalink
    Attributes
    protected

  38. val rootPath: String

    Permalink

    path where the graphs are stored

  39. def schema(graphName: GraphName): Option[CAPSSchema]

    Permalink
    Definition Classes
    AbstractPropertyGraphDataSource → PropertyGraphDataSource

  40. var schemaCache: Map[GraphName, CAPSSchema]

    Permalink
    Attributes
    protected
    Definition Classes
    AbstractPropertyGraphDataSource

  41. def store(graphName: GraphName, graph: PropertyGraph): Unit

    Permalink
    Definition Classes
    AbstractPropertyGraphDataSource → PropertyGraphDataSource

  42. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  43. val tableStorageFormat: String

    Permalink

    Spark configuration parameter for the table format

    Spark configuration parameter for the table format

    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  44. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  45. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  46. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  47. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  48. def waitForWriteCompletion(writeFutures: Set[Future[Unit]])(implicit ec: ExecutionContext): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AbstractPropertyGraphDataSource

  49. def writeCAPSGraphMetaData(graphName: GraphName, capsGraphMetaData: CAPSGraphMetaData): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    JsonSerialization

  50. def writeFile(path: String, content: String): Unit

    Permalink
    Attributes
    protected

  51. def writeJsonCAPSGraphMetaData(graphName: GraphName, capsGraphMetaData: String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → JsonSerialization

  52. def writeJsonSchema(graphName: GraphName, schema: String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → JsonSerialization

  53. def writeNodeTable(graphName: GraphName, labels: Set[String], table: DataFrame): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  54. def writeRelationshipTable(graphName: GraphName, relKey: String, table: DataFrame): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    FSGraphSource → AbstractPropertyGraphDataSource

  55. def writeSchema(graphName: GraphName, schema: CAPSSchema): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    JsonSerialization

  56. def writeTable(path: String, table: DataFrame): Unit

    Permalink
    Attributes
    protected

Inherited from JsonSerialization

Inherited from AbstractPropertyGraphDataSource

Inherited from CAPSPropertyGraphDataSource

Inherited from PropertyGraphDataSource

Inherited from AnyRef

Inherited from Any

Ungrouped