Workflow

Overview and terms¶

QualityRules is a matching engine which applies match/trigger rules to a Dataframe and, when these rules evaluate to passed (i.e. they match or trigger) output sql is run.

Only one trigger rule may produce output, so salience is used as a tie-breaker, the lowest salience wins.

An alternative way to think of this is the trigger rules are your if and the output expressions are the when, from a logic perspective it may be helpful to think of them as output verbs - when this is true do that.

Suggested approach to QualityRules management¶

• Keep unrelated rules in their own RuleSuites, making things easier to reason about
• Make commonly used lambdas or output expressions global
• Use descriptive verbs for your output expressions
  • Keep duplication or complexity in lambdas
  • Only use fields that change as parameters to those lambdas
• Always start with test data you want to match against and your expected output
• Run all test cases for your RuleSuite for any change, don't assume because your rule worked that others won't stop working
• Use the validation and documentation functionality to document your lambdas and verify you've not made simple mistakes - Spark errors aren't always easy to understand

This could be visualised as such: