The term “column equality” refers to two different things in Spark: When a column is equal to a particular value (typically when filtering) When all the values in two columns […]

Testing Scala with MUnit

MUnit is a test library with a collection of features that are not available in other Scala testing libraries. MUnit draws inspiration from Scalatest, utest, and ava (JavaScript testing library). […]

Testing Scala with Scalatest

Scalatest makes it easy to test your Scala code. This blog post shows how to add Scalatest to a sbt project and write some basic tests. Writing a simple test […]

This post explains the best practices Scala libraries should follow. Here are the most important best practices to follow: Great README Clearly defined API Accessible API documentation Clean JAR file […]

Spark DataFrame columns support maps, which are great for key / value pairs with an arbitrary length. This blog post describes how to create MapType columns, demonstrates built-in functions to […]