Packages

o

org.apache.spark.sql

QualitySparkUtils

object QualitySparkUtils

Set of utilities to reach in to private functions

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. QualitySparkUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class Batch(name: String, strategy: Strategy, rules: Rule[LogicalPlan]*) extends Product with Serializable
  2. case class FakePlan(expr: Expression, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
  3. case class Strategy(maxIterations: Int, errorOnExceed: Boolean = false, maxIterationsSetting: String = null) extends Product with Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def drop_field(update: Column, fieldNames: String*): Column

    Drops a field from a structure

    Drops a field from a structure

    fieldNames

    may be nested

  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def execute(logicalPlan: LogicalPlan, batch: Batch): LogicalPlan
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def resolution(analyzer: Analyzer, sparkSession: SparkSession, plan: LogicalPlan): Batch
  18. def resolveExpression(dataFrame: DataFrame, expr: Expression): Expression

    Resolves expressions against a dataframe, this allows them to be swapped out after name checking - spark cannot then simply optimise the tree so certain things like constant folding etc.

    Resolves expressions against a dataframe, this allows them to be swapped out after name checking - spark cannot then simply optimise the tree so certain things like constant folding etc. won't show up.

    dataFrame

    resolve's must be against a given dataframe to keep names matching

    expr

    the expression to resolve

  19. def resolveWithOverride(orig: Option[DataFrame]): Option[DataFrame]

    Where resolveWith is not possible (e.g.

    Where resolveWith is not possible (e.g. 10.x DBRs) it is disabled here. This is, in the 10.x DBR case, due to the class files for UnaryNode (FakePlan) being radically different and causing an IncompatibleClassChangeError: Implementing class

  20. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. def transformFields(exp: Expression): Expression
    Attributes
    protected
  23. def tryResolveReferences(sparkSession: SparkSession)(expr: Expression, child: LogicalPlan): Expression
    Attributes
    protected
  24. def update_field(update: Column, transformations: (String, Column)*): Column

    Adds fields, in order, for each field path it's paired transformation is applied to the update column

    Adds fields, in order, for each field path it's paired transformation is applied to the update column

    returns

    a new copy of update with the changes applied

  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped