Packages

o

org.apache.spark.sql

ClassicQualitySparkUtils

object ClassicQualitySparkUtils

Set of utilities to reach in to private functions

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ClassicQualitySparkUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class Batch(name: String, strategy: Strategy, rules: Rule[LogicalPlan]*) extends Product with Serializable
  2. type DatasetBase[F] = Dataset[F]
  3. case class EvaluableExpressions(plan: LogicalPlan) extends PredicateHelperPlus with Product with Serializable
  4. case class FakePlan(expr: Expression, child: LogicalPlan) extends LogicalPlan with UnaryNode with Product with Serializable
  5. case class Strategy(maxIterations: Int, errorOnExceed: Boolean = false, maxIterationsSetting: String = null) extends Product with Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def aggregator[I, B, O](agg: expressions.Aggregator[I, B, O], exps: Seq[Expression])(implicit arg0: Encoder[I]): ScalaAggregator[I, B, O]
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. def drop_field(update: Column, fieldNames: String*): Column

    Drops a field from a structure

    Drops a field from a structure

    fieldNames

    may be nested

  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def execute(logicalPlan: LogicalPlan, batch: Batch): LogicalPlan
  11. def funNRewrite(plan: LogicalPlan, expressionToExpression: PartialFunction[Expression, Expression]): LogicalPlan
  12. def genParams(ctx: CodegenContext, child: Expression, additional: Seq[ExprValue] = Seq.empty): ParameterInformation

    Spark >3.1 supports the very useful getLocalInputVariableValues, 2.4 needs the previous approach

    Spark >3.1 supports the very useful getLocalInputVariableValues, 2.4 needs the previous approach

    returns

    (parameters for function declaration, parameters for calling, code that must be before fungroup)

  13. def genParamsForNested(ctx: CodegenContext, children: Seq[Expression], additional: Seq[ExprValue]): ParameterInformation

    Only evaluates against subexpressions

    Only evaluates against subexpressions

    returns

    (parameters for function declaration, parameters for calling, code that must be before fungroup)

  14. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  15. def getLocalInputVariableValues(ctx: CodegenContext, expr: Seq[Expression], subExprs: Map[ExpressionEquals, SubExprEliminationState]): (Set[VariableValue], Set[ExprCode])
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  21. lazy val optimizerBatches: Seq[Rule[LogicalPlan]]
  22. def resolution(analyzer: Analyzer, sparkSession: SparkSession, plan: LogicalPlan): Batch
  23. def resolveExpression(dataFrame: DataFrame, expr: Expression): Expression

    Resolves expressions against a dataframe, this allows them to be swapped out after name checking - spark cannot then simply optimise the tree so certain things like constant folding etc.

    Resolves expressions against a dataframe, this allows them to be swapped out after name checking - spark cannot then simply optimise the tree so certain things like constant folding etc. won't show up.

    dataFrame

    resolve's must be against a given dataframe to keep names matching

    expr

    the expression to resolve

  24. def resolveExpressions[T, R](encFrom: Encoder[T], embeddedTypeCorrection: EmbeddedTypeCorrection, dataFrameF: (DataFrame) => DataFrame)(implicit arg0: Encoder[R]): (Seq[Expression], Expression)

    Provides a starting plan for a dataframe, resolves the

    Provides a starting plan for a dataframe, resolves the

    encFrom

    starting data type to encode from

  25. def resolveExpressions(fields: StructType, dataFrameF: (DataFrame) => DataFrame): Seq[Expression]

    Provides a starting plan for a dataframe, resolves the

    Provides a starting plan for a dataframe, resolves the

    fields

    input types

  26. def resolveWithOverride(orig: Option[DataFrame]): Option[DataFrame]

    Where resolveWith is not possible (e.g.

    Where resolveWith is not possible (e.g. 10.x DBRs) it is disabled here. This is, in the 10.x DBR case, due to the class files for UnaryNode (FakePlan) being radically different and causing an IncompatibleClassChangeError: Implementing class

  27. def rowProcessor(exprs: Seq[Expression], compile: Boolean = true): Projection

    Creates a projection from InputRow to InputRow.

    Creates a projection from InputRow to InputRow.

    exprs

    expressions from resolveExpressions, already resolved without

    returns

    typically a mutable projection, callers must ensure partition is set and the target row is provided

  28. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  29. def toString(): String
    Definition Classes
    AnyRef → Any
  30. def transformFields(exp: Expression): Expression
    Attributes
    protected
  31. def tryResolveReferences(sparkSession: SparkSession)(expr: Expression, child: LogicalPlan): Expression
    Attributes
    protected
  32. def update_field(update: Column, transformations: (String, Column)*): Column

    Adds fields, in order, for each field path it's paired transformation is applied to the update column

    Adds fields, in order, for each field path it's paired transformation is applied to the update column

    returns

    a new copy of update with the changes applied

  33. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  34. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  35. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped