Quality - 0.1.3.1-RC1¶

Run complex data quality rules using simple SQL in a batch or streaming Spark application at scale.¶

Write rules using simple SQL or create re-usable functions via SQL Lambdas.

Your rules are just versioned data, store them wherever convenient, use them by simply defining a column.

• typed expression runner - test packages for Fabric

Rules are evaluated lazily during Spark actions, such as writing a row, with results saved in a single predictable column.

Enhanced Spark Functionality¶

Lookup Functions are distributed across the Spark cluster and held in memory, as such no shuffling is required where the shuffling introduced by joins may be too expensive:

• Support for massive Bloom Filters while retaining FPP (i.e. several billion items at 0.001 would not fit into a normal 2gb byte array)
• Map lookup expressions for exact lookups and contains tests, using broadcast variables under the hood they are a great fit for small reference data sets

• View loading - manage the use of session views in your application through configuration and a pluggable DataFrameLoader

• Lambda Functions - user provided re-usable sql functions over late bound columns

• Fast PRNG's exposing RandomSource allowing pluggable and stable generation across the cluster

• Aggregate functions over Maps expandable with simple SQL Lambdas

• Row ID expressions including guaranteed unique row IDs (based on MAC address guarantees)

Plus a collection of handy functions to integrate it all.