Quality - 0.3.0-lazyCSE¶

Run complex data quality and transformation rules using simple SQL in a batch or streaming Spark application at scale.¶

Write rules using simple SQL or create re-usable functions via SQL Lambdas.

Your rules are just versioned data, store them wherever convenient, use them by simply defining a column.

• Rule suite templates - derive and override rules to simplify repetitive configuration
• Named projector - use rule suites to define projections
• Lazy CSE - an experimental optimisation for large rule suites with a high number of common sub-expressions

Rules are evaluated lazily during Spark actions, such as writing a row, with results saved in a single predictable column.

Enhanced Spark Functionality¶

• Lambda Functions - user provided re-usable sql functions over late bound columns
• Map lookup expressions for exact lookups and contains tests, using broadcast variables on Classic and Variables on Connect under the hood they are a great fit for small reference data sets

• View loading - manage the use of session views in your application through configuration and a pluggable DataFrameLoader

• Aggregate functions over Maps expandable with simple SQL Lambdas

• Row ID expressions including guaranteed unique row IDs (based on MAC address guarantees)

• Fast PRNG's exposing RandomSource allowing pluggable and stable generation across the cluster

Plus a collection of handy functions to integrate it all.