Apache Spark

This post explains how Spark registers native functions internally and the public facing APIs for you to register your own functions. Registering native functions is important if you want to […]

Migrating Scala Projects to Spark 3

This post explains how to migrate your Scala projects to Spark 3. It covers the high level steps and doesn’t get into all the details. Migrating PySpark projects is easier. […]