package impl
- Alphabetic
- By Inheritance
- impl
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Package Members
Type Members
- trait CollectRunnerBase[T] extends Expression with NonSQLExpression with SplitCompilation with HasOutput
Children will be rewritten by the plan, it's then re-incorporated into ruleSuite expressionOffsets.length is the length of the trigger expressions in realChildren, realChildren(expressionOffsets.length + expressionOffsets(x)) will be the correct OutputExpression
- case class CollectRunnerRunner(ruleSuite: RuleSuite, children: Seq[Expression], resultDataType: Option[DataType], variablesPerFunc: Int, variableFuncGroup: Int, expressionOffsets: Array[Int], triggerCount: Int, flatten: Boolean, includeNulls: Boolean, canUnroll: Array[Int], isInPlace: Array[Boolean], unroll: Boolean, unrollOutputArraySize: Int, extraConfig: Map[String, String], audited: Boolean = false, alreadyZero: Boolean = false) extends Expression with CollectRunnerBase[CollectRunnerRunner] with Product with Serializable
Children will be rewritten by the plan, it's then re-incorporated into ruleSuite expressionOffsets.length is the length of the trigger expressions in realChildren, realChildren(expressionOffsets.length + expressionOffsets(x)) will be the correct OutputExpression
- case class DataFrameSyntaxError(error: String) extends SyntaxError with Product with Serializable
- trait DefaultProcessor extends quality.DefaultProcessor with Serializable
- case class DefaultProcessorImpl(id: Id, rule: String, outputExpression: OutputExprLogic) extends DefaultProcessor with Serializable with Product
Only run for folder and collector to provide defaults (engine can just use a low prio rule)
Only run for folder and collector to provide defaults (engine can just use a low prio rule)
- Annotations
- @SerialVersionUID()
- case class DefaultTriggerGrouper() extends AbstractFunction10[CodegenContext, Runner, String, Seq[(VariableValue, Boolean)], Seq[(Trigger, (CodegenContext, ParameterInformation, Expression, Boolean) => Block)], ParameterInformation, String, () => Block, (String) => Block, Boolean, TriggerResult] with TriggerGrouper with Product with Serializable
- trait ExprLogic extends quality.ExpressionRule with RuleLogic[ExprLogic] with HasExpr
- trait ExpressionCompiler extends HasExpr
- case class ExpressionRule(rule: String) extends quality.ExpressionRule with ExprLogic with HasRuleText[ExpressionRule] with Product with Serializable
The result of serializing or loading rules
- case class ExpressionRuleExpr(rule: String, expr: Expression) extends ExprLogic with HasRuleText[ExpressionRuleExpr] with Product with Serializable
Used in load postprocessing, e.g.
Used in load postprocessing, e.g. coalesce removal, to keep the rule text around
- Annotations
- @SerialVersionUID()
- trait ExpressionRunnerBase[T] extends Expression with NonSQLExpression with SplitCompilation with TriggerOnly
Creates an extensible wrapper result column for aggregate expressions, storing the results as yaml
- case class ExpressionRunnerCompiled(ruleSuite: RuleSuite, children: Seq[Expression], ddlType: DataType, compileEvals: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, extraConfig: Map[String, String], alreadyZero: Boolean = false) extends Expression with ExpressionRunnerBase[ExpressionRunnerCompiled] with Product with Serializable
Creates an extensible wrapper result column for aggregate expressions, storing the results as yaml
- case class ExpressionRunnerEval(ruleSuite: RuleSuite, children: Seq[Expression], ddlType: DataType, compileEvals: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, extraConfig: Map[String, String], alreadyZero: Boolean = false) extends Expression with ExpressionRunnerBase[ExpressionRunnerEval] with CodegenFallback with Product with Serializable
Creates an extensible wrapper result column for aggregate expressions, storing the results as yaml
- case class ExpressionWrapper(expr: Expression, compileEval: Boolean = true) extends ExprLogic with ExpressionCompiler with Product with Serializable
Rewritten children at eval will be swapped out
Rewritten children at eval will be swapped out
- Annotations
- @SerialVersionUID()
- case class ExtraDocParameter(id: Id, name: String) extends SyntaxNameWarning with LambdaRelevant with Product with Serializable
- case class FlattenFolderResultsExpression(child: Expression, deserializer: Expression) extends UnaryExpression with CodegenFallback with NullIntolerant with Product with Serializable
NB this doesn't make much sense for folder unless in debug
NB this doesn't make much sense for folder unless in debug
- Annotations
- @ExpressionDescription()
- case class FlattenResultsExpression(child: Expression, deserializer: Expression) extends UnaryExpression with CodegenFallback with NullIntolerant with InputTypeChecks with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class FlattenRulesResultsExpression(child: Expression, deserializer: Expression) extends UnaryExpression with CodegenFallback with NullIntolerant with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class Group(groupFilter: Expression, lowestSalience: Int, payload: GroupOr) extends LowestSalience with Product with Serializable
- case class GroupAudit(children: Seq[Expression]) extends Expression with GroupAuditBase with Product with Serializable
- trait GroupAuditBase extends Expression with NonSQLExpression with CodegenFallback
- trait GroupBasedGrouper extends AbstractFunction10[CodegenContext, Runner, String, Seq[(VariableValue, Boolean)], Seq[(Trigger, (CodegenContext, ParameterInformation, Expression, Boolean) => Block)], ParameterInformation, String, () => Block, (String) => Block, Boolean, TriggerResult] with TriggerGrouper
Default grouping approach, implementations may call performGrouping with their own Groups.
- sealed trait GroupOr extends LowestSalience
- case class GroupResults(children: Seq[Expression]) extends Expression with GroupResultsBase with Product with Serializable
- trait GroupResultsBase extends Expression with NonSQLExpression with CodegenFallback
- case class Groups(groups: Seq[Group]) extends GroupOr with Product with Serializable
- trait HasExpr extends AnyRef
- sealed trait HasId extends AnyRef
- sealed trait HasNonIdText extends AnyRef
- trait HasOutput extends Expression with Runner
Base trait for runners that use triggers_ collector, engine and folder
- sealed trait HasOutputText extends AnyRef
- trait HasRuleText[T <: HasRuleText[T]] extends HasExpr with quality.HasRuleText
- case class IfRelevantExpr(left: Expression, right: Expression) extends BinaryExpression with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class InPlaceArray(children: Seq[Expression]) extends Expression with Product with Serializable
Replacement for CreateArray when flatten is true, eval creates new arrays, but gencode does not and only replaces the array location
- case class InPlaceOffsets(offsets: Seq[(String) => Block], beforeProcessing: Block, resultRowPrep: Block, runner: String, runnerClazz: Class[_]) extends Product with Serializable
Used for compilation
Used for compilation
- offsets
each trigger offset
- beforeProcessing
must be placed in the generated code after the output row is copied and sets up the class variables for each ruleSet
- resultRowPrep
must be called after processing and re-integrates the overallResult's from each set and suite
- trait LambdaFunction extends quality.LambdaFunction with HasRuleText[LambdaFunction] with HasExpr
Lambda functions are for re-use across rules.
Lambda functions are for re-use across rules. (param: Type, paramN: Type) -> logicResult .
- case class LambdaFunctionImpl(name: String, rule: String, id: Id) extends LambdaFunction with Product with Serializable
- Annotations
- @SerialVersionUID()
- case class LambdaFunctionParsed(name: String, rule: String, id: Id, expr: Expression) extends LambdaFunction with Product with Serializable
- Annotations
- @SerialVersionUID()
- case class LambdaMultipleImplementationWithSameArityError(name: String, count: Int, argLength: Int, ids: Set[Id]) extends SyntaxError with LambdaRelevant with Product with Serializable
- case class LambdaNameError(name: String, id: Id) extends NameMissingError with LambdaRelevant with Product with Serializable
- case class LambdaPossibleSOE(id: Id) extends RuleWarning with LambdaRelevant with Product with Serializable
- sealed trait LambdaRelevant extends AnyRef
- case class LambdaSparkFunctionNameError(name: String, id: Id) extends NameMissingError with LambdaRelevant with Product with Serializable
- case class LambdaStackOverflowError(id: Id) extends SyntaxError with LambdaRelevant with Product with Serializable
- case class LambdaSyntaxError(id: Id, error: String) extends SyntaxError with LambdaRelevant with Product with Serializable
- case class LambdaViewError(name: String, id: Id) extends ViewMissingError with LambdaRelevant with Product with Serializable
- case class LazyRuleSuiteResultDetailsImpl(row: Row) extends LazyRuleSuiteResultDetails with Serializable with Product
- case class LazyRuleSuiteResultDetailsProxyImpl(_ruleSuiteResultDetails: RuleSuiteResultDetails) extends LazyRuleSuiteResultDetails with Serializable with Product
- case class LazyRuleSuiteResultImpl(row: Row) extends LazyRuleSuiteResult with Serializable with Product
- trait LowestSalience extends AnyRef
- trait MergeGroups[T] extends Serializable
- sealed trait NameMissingError extends RuleError
- case class NonLambdaDocParameters(id: Id) extends SyntaxWarning with Product with Serializable
- abstract class OfX[T, E] extends AnyRef
Ignores extra output expressions
- case class OuputSparkFunctionNameError(name: String, id: Id) extends NameMissingError with OutputExpressionRelevant with Product with Serializable
- trait OutputExprLogic extends quality.OutputExpression with HasExpr
- case class OutputExpression(rule: String) extends quality.OutputExpression with OutputExprLogic with HasRuleText[OutputExpression] with Logging with Product with Serializable
Used as a result of serializing
- case class OutputExpressionExpr(rule: String, expr: Expression) extends OutputExprLogic with HasRuleText[OutputExpressionExpr] with Product with Serializable
Used in post serializing processing to keep the rule around, processCoalesce only
Used in post serializing processing to keep the rule around, processCoalesce only
- Annotations
- @SerialVersionUID()
- sealed trait OutputExpressionRelevant extends AnyRef
- case class OutputExpressionWrapper(expr: Expression, compileEval: Boolean = true) extends OutputExprLogic with ExpressionCompiler with Product with Serializable
- case class OutputRuleNameError(name: String, id: Id) extends NameMissingError with OutputExpressionRelevant with Product with Serializable
- case class OutputRuleSyntaxError(id: Id, error: String) extends SyntaxError with OutputExpressionRelevant with Product with Serializable
- case class OutputRuleViewError(name: String, id: Id) extends ViewMissingError with OutputExpressionRelevant with Product with Serializable
- case class PackExpression(left: Expression, right: Expression) extends BinaryExpression with NullIntolerant with InputTypeChecks with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class PassedTestExpr(child: Expression) extends UnaryExpression with Product with Serializable
When the value is compatible with PassedInt (or TRUE_INT) true, otherwise false
- case class ProbabilityExpr(child: Expression) extends UnaryExpression with NullIntolerant with InputTypeChecks with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class RngUUIDExpression(child: Expression) extends UnaryExpression with InputTypeChecks with Product with Serializable
Creates a uuid from byte arrays or two longs
Creates a uuid from byte arrays or two longs
- child
the expression to produce a BinaryType
- case class RuleEngineRunner(ruleSuite: RuleSuite, children: Seq[Expression], userResultDataType: Option[DataType], compileEvals: Boolean, debugMode: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, expressionOffsets: Array[Int], forceTriggerEval: Boolean, triggerCount: Int, extraConfig: Map[String, String], audited: Boolean = false, alreadyZero: Boolean = false) extends Expression with RuleEngineRunnerBase[RuleEngineRunner] with Product with Serializable
- trait RuleEngineRunnerBase[T] extends Expression with NonSQLExpression with SplitCompilation with HasOutput
Children will be rewritten by the plan, it's then re-incorporated into ruleSuite expressionOffsets.length is the length of the trigger expressions in realChildren, realChildren(expressionOffsets.length + expressionOffsets(x)) will be the correct OutputExpression
- case class RuleEngineRunnerEval(ruleSuite: RuleSuite, children: Seq[Expression], userResultDataType: Option[DataType], compileEvals: Boolean, debugMode: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, expressionOffsets: Array[Int], forceTriggerEval: Boolean, triggerCount: Int, extraConfig: Map[String, String], audited: Boolean = false, alreadyZero: Boolean = false) extends Expression with RuleEngineRunnerBase[RuleEngineRunnerEval] with CodegenFallback with Product with Serializable
- sealed trait RuleError extends HasId with HasOutputText with HasNonIdText
Base for RuleErrors
- case class RuleFolderRunner(ruleSuite: RuleSuite, children: Seq[Expression], resultDataType: () => DataType, compileEvals: Boolean, debugMode: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, expressionOffsets: Array[Int], dataRef: AtomicReference[DataType], forceTriggerEval: Boolean, triggerCount: Int, extraConfig: Map[String, String], audited: Boolean = false, alreadyZero: Boolean = false) extends Expression with RuleFolderRunnerBase[RuleFolderRunner] with Product with Serializable
Children will be rewritten by the plan, it's then re-incorporated into ruleSuite expressionOffsets.length is the length of the trigger expressions in realChildren, realChildren(expressionOffsets.length + expressionOffsets(x)) will be the correct OutputExpression
- trait RuleFolderRunnerBase[T] extends Expression with NonSQLExpression with SplitCompilation with HasOutput
Children will be rewritten by the plan, it's then re-incorporated into ruleSuite expressionOffsets.length is the length of the trigger expressions in realChildren, realChildren(expressionOffsets.length + expressionOffsets(x)) will be the correct OutputExpression
- case class RuleFolderRunnerEval(ruleSuite: RuleSuite, children: Seq[Expression], resultDataType: () => DataType, compileEvals: Boolean, debugMode: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, expressionOffsets: Array[Int], dataRef: AtomicReference[DataType], forceTriggerEval: Boolean, triggerCount: Int, extraConfig: Map[String, String], audited: Boolean = false, alreadyZero: Boolean = false) extends Expression with RuleFolderRunnerBase[RuleFolderRunnerEval] with CodegenFallback with Product with Serializable
Children will be rewritten by the plan, it's then re-incorporated into ruleSuite expressionOffsets.length is the length of the trigger expressions in realChildren, realChildren(expressionOffsets.length + expressionOffsets(x)) will be the correct OutputExpression
- trait RuleLogic[T <: RuleLogic[T]] extends quality.ExpressionRule with Serializable
- case class RuleNameError(name: String, id: Id) extends NameMissingError with RuleRelevant with Product with Serializable
- sealed trait RuleRelevant extends AnyRef
- case class RuleResultExpression(children: Seq[Expression]) extends Expression with InputTypeChecks with Product with Serializable
- case class RuleRunner(ruleSuite: RuleSuite, children: Seq[Expression], compileEvals: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, extraConfig: Map[String, String], alreadyZero: Boolean = false) extends Expression with RuleRunnerBase[RuleRunner] with Product with Serializable
- trait RuleRunnerBase[T] extends Expression with NonSQLExpression with SplitCompilation with TriggerOnly
Children will be rewritten by the plan, either by resolveWith, it's then re-incorporated into ruleSuite
Children will be rewritten by the plan, either by resolveWith, it's then re-incorporated into ruleSuite
The variablePerFunc (40) and variableFuncGroup (20) parameters are chosen due to 64k method limitation, is null for all of a 1600 column row blasts the 64k jvm limit. You may customise these values if a given rulesuite requires it.
- case class RuleRunnerEval(ruleSuite: RuleSuite, children: Seq[Expression], compileEvals: Boolean, variablesPerFunc: Int, variableFuncGroup: Int, extraConfig: Map[String, String], alreadyZero: Boolean = false) extends Expression with RuleRunnerBase[RuleRunnerEval] with CodegenFallback with Product with Serializable
- case class RuleSuiteResultDetailsExpr(child: Expression) extends UnaryExpression with NonSQLExpression with InputTypeChecks with Product with Serializable
Consumes a RuleSuiteResult and returns RuleSuiteDetails
- case class RuleSyntaxError(id: Id, error: String) extends SyntaxError with RuleRelevant with Product with Serializable
- case class RuleViewError(name: String, id: Id) extends ViewMissingError with RuleRelevant with Product with Serializable
- sealed trait RuleWarning extends HasId with HasOutputText with HasNonIdText
Base for RuleWarnings
- trait RunOnPassProcessor extends quality.RunOnPassProcessor with Serializable
- case class RunOnPassProcessorImpl(salience: Int, id: Id, rule: String, returnIfPassed: OutputExprLogic) extends RunOnPassProcessor with Serializable with Product
Generates a result upon a pass, it is not evaluated otherwise and only evaluated if no other rule has higher salience.
Generates a result upon a pass, it is not evaluated otherwise and only evaluated if no other rule has higher salience. It is not possible to have more than one rule evaluate the returnIfPassed.
- Annotations
- @SerialVersionUID()
- trait Runner extends Expression
Shared trait for all runners
- case class SoftFailExpr(child: Expression) extends UnaryExpression with NullIntolerant with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class SparkFunctionNameError(name: String, id: Id) extends NameMissingError with RuleRelevant with Product with Serializable
- trait SplitCompilation extends Expression with Runner
Handles state management for separate compilation, the generated source code is cached locally and then compiled on demand
- case class StripResultTypes(child: Expression) extends UnaryExpression with CodegenFallback with InputTypeChecks with Product with Serializable
- sealed trait SyntaxError extends RuleError
- sealed trait SyntaxNameWarning extends SyntaxWarning
- sealed trait SyntaxWarning extends RuleWarning
- trait ThreeOnlyNonFoldable extends AnyRef
- case class TopLevelBooleanGrouper() extends AbstractFunction10[CodegenContext, Runner, String, Seq[(VariableValue, Boolean)], Seq[(Trigger, (CodegenContext, ParameterInformation, Expression, Boolean) => Block)], ParameterInformation, String, () => Block, (String) => Block, Boolean, TriggerResult] with GroupBasedGrouper with Product with Serializable
Groups by common top level Boolean And and EqualTo expressions with Literals, using buckets of hashes on the literal values.
Groups by common top level Boolean And and EqualTo expressions with Literals, using buckets of hashes on the literal values. Using this approach can lead to a 10x spread increase over the default grouper for very large truth table style rules (tested against the 20k_rule_suite.csv in the BigRules testsuite).
Only supported with Spark 3.2 and above
- case class Trigger(expression: Expression, index: Int, salience: Int, outputExpression: Option[Expression] = None) extends Product with Serializable
- trait TriggerGrouper extends AbstractFunction10[CodegenContext, Runner, String, Seq[(VariableValue, Boolean)], Seq[(Trigger, (CodegenContext, ParameterInformation, Expression, Boolean) => Block)], ParameterInformation, String, () => Block, (String) => Block, Boolean, TriggerResult]
Allow customised grouping of runner triggers, DQ and ExpressionRunner should evaluate all so the default implementation is sufficient.
Allow customised grouping of runner triggers, DQ and ExpressionRunner should evaluate all so the default implementation is sufficient. This abstraction was added as part of #129 due to 20k trigger rules.
The return type is the list of function names to call and any extra common subexpressions needed to group
- trait TriggerOnly extends Expression with Runner
Base trait for dq and expression
- case class TriggerResult(groupCalls: Iterator[String], subExpressions: String, extraClasses: Seq[(Int, CodeAndComment)], usedParameters: ParameterInformation) extends Product with Serializable
- case class Triggers(triggers: Seq[Trigger], lowestSalience: Int) extends GroupOr with Product with Serializable
- type TypedRowFunction[A, R] = (TypedColumn[Row, A]) => TypedColumn[Row, R]
Easier type to work with for UserDefinedFunction A -> R on a Row
- case class UUIDToLongsExpression(child: Expression) extends UnaryExpression with InputTypeChecks with Product with Serializable
Pulls Longs out from a uuid
Pulls Longs out from a uuid
- child
the UUID expression
- case class UnPackExpression(child: Expression) extends UnaryExpression with NullIntolerant with InputTypeChecks with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class UnPackIdTripleExpression(child: Expression) extends UnaryExpression with InputTypeChecks with Product with Serializable
- Annotations
- @ExpressionDescription()
- case class UnifyResult(child: Expression) extends UnaryExpression with CodegenFallback with NonSQLExpression with Product with Serializable
- sealed trait ViewMissingError extends RuleError
Value Members
- object CollectRunner
- object DefaultProcessorImpl extends Serializable
- object ExpressionCompiler
- object ExpressionRuleExpr extends Serializable
- object ExpressionRunner
- object FlattenStruct
- object GenerateDecoderOpEncoderProjection extends CodeGenerator[Seq[Expression], DecoderOpEncoderProjection[_, _]]
CODE is based on MutableProjection and generates a transformation between two encoders over a middle operation.
CODE is based on MutableProjection and generates a transformation between two encoders over a middle operation. The generated class itself can create new instances directly, unlike Spark projections that need to go through the codegen source generation cycle, only the compilation is cached.
If the expression tree contains stateful expressions with codegenfallback the code must be regenerated against a fresh tree.
- object GenerateDecoderOpEncoderVarProjection extends CodeGenerator[Seq[Expression], DecoderOpEncoderProjection[_, _]]
CODE is based on MutableProjection, but uses currentVars and wholestage elimination and generates a transformation between two encoders over a middle operation.
CODE is based on MutableProjection, but uses currentVars and wholestage elimination and generates a transformation between two encoders over a middle operation. The generated class itself can create new instances directly, unlike Spark projections that need to go through the codegen source generation cycle, only the compilation is cached.
If the expression tree contains stateful expressions with codegenfallback the code must be regenerated against a fresh tree.
- object GetRealChildren
- object GroupAudit extends Serializable
- object GroupResults extends Serializable
- object LambdaFunctionImpl extends Serializable
- object LazyRuleSuiteResultDetailsUtils
- object LazyRuleSuiteResultUtils
- object MergeGroups extends Serializable
- object OfRuleOutputSuite
Requires output expressions
- object OfRuleSuite extends OfX[RuleSuite, CombinedRuleSuiteRows]
- object OfRuleSuiteGroup extends OfX[RuleSuiteGroup, Seq[CombinedRuleSuiteRows]]
- object Pack
- object PassedTestExpr extends Serializable
- object ProcessDisableIfMissing extends RuleRunnerImports
- object RngUUIDExpression extends Serializable
- object RowDeserializing
- object RuleEngineRunnerImpl
- object RuleLogicUtils
- object RuleRegistrationFunctions
- object RuleResultExpression extends Serializable
- object RuleRunnerImpl
- Attributes
- protected[quality]
- object RuleSuiteFunctions
- object RuleSuiteResultDetailsExpr extends Serializable
- object RunOnPassProcessorImpl extends Serializable
- object Runners
forwards to the server side implementations, when used in server, where stub is dropped, jar resolveWith can work
- object SoftFailedUtils
- object Triggers extends Serializable
- object UUIDToLongsExpression extends Serializable
- object UnPack
- object UnPackIdTriple
- object UnifyResult extends Serializable
- object UpdateFolderExpression
- object Validation
- object VariableProcessIfMissingFunctions
- object YamlDecoder
2.0 (14.0dbr) needs security checks for allowed types.
2.0 (14.0dbr) needs security checks for allowed types. Forcing it on 3.5 so we can debug