package extension
- Alphabetic
- Public
- Protected
Type Members
- abstract class AsymmetricFilterExpressions extends org.apache.spark.sql.catalyst.rules.Rule[LogicalPlan] with PredicateHelperPlus with ConstraintHelper
Replaces any attributes aliased by an expression which is then used in select filters / constraints wth constituent predicates that can be pushed down.
Replaces any attributes aliased by an expression which is then used in select filters / constraints wth constituent predicates that can be pushed down. A simple example is:
select as_uuid(lower, higher) as context where context = 'string'this would be replaced with:
select as_uuid(lower, higher) as context where (lower = longPairFromUUID('string').lower and higher = longPairFromUUID('string').higher)
in the case of the string being a constant, constant folding should replace both of these with simple lookups that are easy to push down.
*NB.* Pushdown predicates based on constant folding only work via the SparkExtension approach, the use of experimental.extraOptimizations does provide this as it happens too late in the plans
Only In and BinaryComparison's are supported currently
- case class ConnectCommandParsers(sparkSession: SparkSession, delegate: ParserInterface) extends AbstractInjectableParser with Logging with Product with Serializable
- trait FunNRewriteBase extends org.apache.spark.sql.catalyst.rules.Rule[LogicalPlan]
- class LambdaEnablers extends AnyRef
- case class QualityFunctionParser(sparkSession: SparkSession, delegate: ParserInterface) extends AbstractInjectableParser with Logging with Product with Serializable
- class QualitySparkExtension extends (SparkSessionExtensions) => Unit with Logging
Registers Quality sql functions using the defaults for registerQualityFunctions, these can be overridden without having to subclass DriverPlugin.
Registers Quality sql functions using the defaults for registerQualityFunctions, these can be overridden without having to subclass DriverPlugin. Functions are registered via FunctionRegistry.builtIn making them available for view creation, if they should be registered via extensions only then use the quality_force_inject_function = true configuration.
It also registers plan optimiser rule's such as AsUUIDFilter, which rewrites filters with variables backed by as_uuid.
Optimiser rules can be disabled via the quality_disable_optimiser_rules system environment variable. "*" disables all rules, otherwise a comma separated list of fqn class names may be used.
- case class ZeroCodeGen(child: Expression, realChild: Expression, on32: Boolean = false, wrapped: Boolean = false) extends UnaryExpression with Logging with Product with Serializable
Cannot be used as part of the initial expression trees, but can be used to swap out a runner to reduce compilation time and size.
Cannot be used as part of the initial expression trees, but can be used to swap out a runner to reduce compilation time and size. Added as part of #129 as Databricks could not generate a 20k rulesuite, despite #131's separate compilation, the common subexpressions were already too much for databricks.
Value Members
- object AsUUIDFilter extends AsymmetricFilterExpressions
Optimises as_uuid usage for filters on the generated value, swapping out for long lookups
- object AsymmetricFilterExpressions
- object ConnectCommandParsers extends Serializable
- object FunNRewrite extends org.apache.spark.sql.catalyst.rules.Rule[LogicalPlan] with FunNRewriteBase
Rewrites FunN's when not used as LambdaFunctions (e.g.
Rewrites FunN's when not used as LambdaFunctions (e.g. a parent HigherOrderFunction or HigherOrderFunctionLike). The application is a simple replacement of the resolved input expression with all occurrences of the matching LambdaVariable.
This happens early enough to allow for user functions to be subexpr eliminated (as of Spark 4 not possible with either CodegenFallback or an expression containing any LambdaVariables).
The use of USED_AS_LAMBDA and any Spark HoF with a registered LambdaCompilationHandler will keep the expression as a lambda. The LambdaCompilationHandler case is a trade-off between overhead of tree copies/compilation in processors, the underlying hof eval vs compile and sub expression elimination.
IMPORTANT: This only works on 3.2 and above as it introduced transformDownWithPruning
- object IDBase64Filter extends AsymmetricFilterExpressions
Optimises id_base64 usage for filters on the generated base64 value, swapping out for int / long lookups
- object QualitySparkExtension
- object ZeroCodeGenWrap