Packages

o

org.apache.spark.sql

QualitySparkUtils

object QualitySparkUtils

Set of utilities to reach in to private functions

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. QualitySparkUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class Batch(name: String, strategy: Strategy, rules: Rule[LogicalPlan]*) extends Product with Serializable
  2. type DatasetBase[F] = Dataset[F]
  3. case class EvaluableExpressions(plan: LogicalPlan) extends PredicateHelperPlus with Product with Serializable
  4. case class FakePlan(expr: Expression, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
  5. case class Strategy(maxIterations: Int, errorOnExceed: Boolean = false, maxIterationsSetting: String = null) extends Product with Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def drop_field(update: Column, fieldNames: String*): Column

    Drops a field from a structure

    Drops a field from a structure

    fieldNames

    may be nested

  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. def execute(logicalPlan: LogicalPlan, batch: Batch): LogicalPlan
  10. def funNRewrite(plan: LogicalPlan, expressionToExpression: PartialFunction[Expression, Expression]): LogicalPlan
  11. def genParams(ctx: CodegenContext, child: Expression): (String, String, String)

    Spark >3.1 supports the very useful getLocalInputVariableValues, 2.4 needs the previous approach

    Spark >3.1 supports the very useful getLocalInputVariableValues, 2.4 needs the previous approach

    returns

    (parameters for function decleration, parmaters for calling, code that must be before fungroup)

  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  18. lazy val optimizerBatches: Seq[Rule[LogicalPlan]]
  19. def registerLambdaFunctions(functions: Seq[LambdaFunction]): Unit
  20. def resolution(analyzer: Analyzer, sparkSession: SparkSession, plan: LogicalPlan): Batch
  21. def resolveExpression(dataFrame: DataFrame, expr: Expression): Expression

    Resolves expressions against a dataframe, this allows them to be swapped out after name checking - spark cannot then simply optimise the tree so certain things like constant folding etc.

    Resolves expressions against a dataframe, this allows them to be swapped out after name checking - spark cannot then simply optimise the tree so certain things like constant folding etc. won't show up.

    dataFrame

    resolve's must be against a given dataframe to keep names matching

    expr

    the expression to resolve

  22. def resolveExpressions[T, R](encFrom: Encoder[T], embeddedTypeCorrection: EmbeddedTypeCorrection, dataFrameF: (DataFrame) => DataFrame)(implicit arg0: Encoder[R]): (Seq[Expression], Expression)

    Provides a starting plan for a dataframe, resolves the

    Provides a starting plan for a dataframe, resolves the

    encFrom

    starting data type to encode from

  23. def resolveExpressions(fields: StructType, dataFrameF: (DataFrame) => DataFrame): Seq[Expression]

    Provides a starting plan for a dataframe, resolves the

    Provides a starting plan for a dataframe, resolves the

    fields

    input types

  24. def resolveWithOverride(orig: Option[DataFrame]): Option[DataFrame]

    Where resolveWith is not possible (e.g.

    Where resolveWith is not possible (e.g. 10.x DBRs) it is disabled here. This is, in the 10.x DBR case, due to the class files for UnaryNode (FakePlan) being radically different and causing an IncompatibleClassChangeError: Implementing class

  25. def rowProcessor(exprs: Seq[Expression], compile: Boolean = true): Projection

    Creates a projection from InputRow to InputRow.

    Creates a projection from InputRow to InputRow.

    exprs

    expressions from resolveExpressions, already resolved without

    returns

    typically a mutable projection, callers must ensure partition is set and the target row is provided

  26. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  27. def toString(): String
    Definition Classes
    AnyRef → Any
  28. def transformFields(exp: Expression): Expression
    Attributes
    protected
  29. def tryResolveReferences(sparkSession: SparkSession)(expr: Expression, child: LogicalPlan): Expression
    Attributes
    protected
  30. def update_field(update: Column, transformations: (String, Column)*): Column

    Adds fields, in order, for each field path it's paired transformation is applied to the update column

    Adds fields, in order, for each field path it's paired transformation is applied to the update column

    returns

    a new copy of update with the changes applied

  31. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  32. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  33. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped