Creates an aggregate by applying filter to rows, calling sum with a starting value (provided by zero) and finally calls result to process the sum and count values for a final result.

Note, when working with lambda's in the dsl it's often required to use the dataframes col function as the scope is incorrect in the lambda.

Efficiently converts the map column to struct for comparison, unioning, sorting etc.

The default_rule value

Converts columns into a digest via the MessageDigest digestImpl

Converts columns into a digest via the MessageDigest digestImpl

The disabled_rule value

Drops a field from a structure

The failed value

Creates an id from fields using MessageDigests

Flattens folder results, unpacking the nested structure into a simple relation

Flattens DQ results, unpacking the nested structure into a simple relation

Flattens rule results, unpacking the nested structure into a simple relation

Converts yaml expressions to spark native types

Groups runnerResults which represent the result column from an array of rule runners, expressionRunner excluded, or an array of RuleSuiteGroupResults. Note many of the results will contain nullable fields, see GroupResultsTest for example result types with Scala Encoders. The allowed input types are:

- array(ruleRunner DQ results) which returns RuleSuiteGroupResults - array(ruleEngineResults) which returns either (RuleSuiteGroupResults, array((salientRule, result))) or (RuleSuiteGroupResults, array((salientRule, array((salience,result)))) for debug - array(ruleFolderResults) which returns either (RuleSuiteGroupResults, array(result)) or (RuleSuiteGroupResults, array(array((salience,result))) for debug - array(collectRunner results) which returns (RuleSuiteGroupResults, array(result)), you can also use processResult to flatten - array( group_results( array( group_results( collect_runner ... will return (RuleSuiteGroupResults, array(array(array(X))))

Creates an id from fields using Guava Hashers

Converts columns into a digest using Guava Hashers

Converts columns into a digest using Guava Hashers

Converts either a single ID or individual field parts into base64. The parts must be provided in the correct order, base, i0, i1.. iN

Given a base64 string convert to an ID, use id_size to understand how large IDs could be.

Returns the underlying raw type of an id (base, i0, i1 etc.) without prefixes

Returns the size of an underlying ID, a unique_id will have 2, other id's may have more, each further increment is another 64bits

The ignored_rule value

Adds incrementWith to the sum value

Adds 1L to the sum value

creates a (lower, higher) struct

creates a (lower, higher) struct from a uuid's least and most significant bits

Tests if there is a stored value from a map via the name mapLookupName and 'key' lookupKey. Implementation is map_lookup.isNotNull

Retrieves the stored value from a map via the name mapLookupName and 'key' lookupKey

Creates an entry in a map sum with id and the result of 'sum' with the previous sum at that id as it's parameter.

Provides the mean (summed value / count of filtered rows)

Murmur3 hash

Packs two integers into a long, typically used for versioned ids.

Packs two integers into a long, typically used for versioned ids.

Packs two integers into a long, typically used for versioned ids.

The passed value

Converts a prefixed long pair to lower, higher

Returns the probability from a given rule result

Creates a hash based ID based on an upstream compatible long generator

Produces an aggregate result

returns both the count and sum

returns the sum, ignoring the count

Efficiently converts the mapStruct column to it's original Map type

Creates a default randomRNG based on RandomSource.XO_RO_SHI_RO_128_PP

Creates a random number generator using a given commons-rng source

Creates a randomRNG ID based on randomSource with a given seed

Creates a uuid from byte arrays or two longs, use with the rng() function to generate random uuids.

Retrieves the rule result for a given id, the result type is dependent on ruleSuiteResults's type. Integer is returned for DQ checks and either String or (ruleResult: String, resultDDL: String) for ExpressionResults.

Retrieves the rule result for a given id, the result type is dependent on ruleSuiteResults's type. Integer is returned for DQ checks and either String or (ruleResult: String, resultDDL: String) for ExpressionResults.

Retrieves the rule result for a given id, the result type is dependent on ruleSuiteResults's type. Integer is returned for DQ checks and either String or (ruleResult: String, resultDDL: String) for ExpressionResults.

Consumes a RuleSuiteResult and returns RuleSuiteDetails

Aggregates over RuleSuiteResults columns and returns a RuleSuiteGroupStatistics row

The soft_failed value

Stores only the ruleResult, removing the structure including the resultDDL column

Given the current sum, produce the next sum, for example by incrementing 1 on the sum to count filtered rows

Converts spark expressions to yaml using snakeyml.

Converts non-debug engine results into a single result column, allowing engine, collector and folder results to be nested and combined via group_results within the same nested rule suite calls. Similarly, non-debug RuleSuiteGroupResults can be converted.

Creates a uniqueID backed by the GuaranteedUniqueID Spark Snowflake ID approach

Takes a packedId long and unpacks to id, version

Unpacks an IdTriple column into it's six constituent integers

Adds fields, in order, for each field path it's paired transformation is applied to the update column

Creates an id from fields using ZeroAllocation LongHashFactory (64bit)

Converts columns into a digest via ZeroAllocation LongTuple Factory (128-bit)

Converts columns into a digest via ZeroAllocation LongTuple Factory (128-bit)

Converts columns into a digest via ZeroAllocation LongHashFactory (64bit)

Converts columns into a digest via ZeroAllocation LongHashFactory (64bit)

Creates an id from fields using ZeroAllocation LongTuple Factory (128-bit)

Creates an id from fields using MessageDigests, in line with SQL naming please use field_based_id

Creates a hash based ID based on an upstream compatible long generator, in line with sql functions please migrate to provided_id

Prefer rule_suite_statistics, this function is only provided as a fallback should there be issues in the rule_suite_statistics implementation, which is over 6% faster.

Aggregates over RuleSuiteResults columns and returns a RuleSuiteGroupStatistics row using a pre Spark 4 unified API Aggregator, per https://github.com/sparkutils/quality/issues/117 this does not run on Databricks Shared Clusters. *