Those are some Quality flavours
Quality has three main flavours with sprinklings of other Quality ingredients like the sql function suite.
These flavours are provided by three "runners" which add a Column to a Spark Dataset/Dataframe.
Quality / QualityData - ruleRunner¶
Execute SQL based data validation rules, capture all the results and store them with your data for easy and fast access.
Example Usage: Validating in-bound data or the results of a calculation.
What is stored:
QualityRules - ruleEngineRunner¶
QualityRules extends the base Quality framework to provide the ability to generate output based on a single SQL rule matching the input data. Effectively an auditable large scale SQL case statement.
Conceptually trigger rules are the when and Output rules are the then ordered by salience.
Example Usage: Derivation Logic.
What is stored:
QualityFolder - ruleFolderRunner¶
QualityFolder extends QualityRules providing the ability to change values of attributes based on any number of SQL rules matching the input data.
Unlike QualityRules which uses salience to select only one Output expression, Folder uses salience to order the execution of all the matching Trigger's paired Output Expressions - folding the results as it goes.
Example Usage: Correction of in-bound data to enable subsequent calculators to process, defaulting etc.
What is stored:
Created: March 27, 2023 09:08:01