Shim - 0.3.0-RC1¶

Provides shims over private Apache Spark APIs¶

Using the private Spark APIs can allow for improved functionality (e.g. Frameless encoder derivation) and performance (e.g. Quality HOF compilation or Plan rules) but it comes at a cost: lots of workarounds to support different runtimes.

Shim aims to bring those workarounds in a separate runtime library shim and let the library authors focus on useful functionality.

• Support for Spark 4, 4.1, 4.2 and DBR 17.3, 18.3 (has been tested on 19)
• Shims enabling Connect handling and AgnosticEncoder handling by default on Spark 4
• beanEncoder shim allowing correction of field name usage in encoding
• tryCastCompat ansi disabled version of cast leaving consistent casts for all versions