Apache Spark

Migrating Scala Projects to Spark 3

This post explains how to migrate your Scala projects to Spark 3. It covers the high level steps and doesn’t get into all the details. Migrating PySpark projects is easier. […]

The term “column equality” refers to two different things in Spark: When a column is equal to a particular value (typically when filtering) When all the values in two columns […]

Spark DataFrame columns support maps, which are great for key / value pairs with an arbitrary length. This blog post describes how to create MapType columns, demonstrates built-in functions to […]