Testing Scala with MUnit
MUnit is a test library with a collection of features that are not available in other Scala testing libraries. MUnit draws inspiration from Scalatest, utest, and ava (JavaScript testing library). […]
Testing Scala with Scalatest
Scalatest makes it easy to test your Scala code. This blog post shows how to add Scalatest to a sbt project and write some basic tests. Writing a simple test […]
Scala Library Best Practices
This post explains the best practices Scala libraries should follow. Here are the most important best practices to follow: Great README Clearly defined API Accessible API documentation Clean JAR file […]
Working with Spark MapType Columns
Spark DataFrame columns support maps, which are great for key / value pairs with an arbitrary length. This blog post describes how to create MapType columns, demonstrates built-in functions to […]
Learning Emacs Keybindings After Multiple Failed Attempts
Have you ever tried Emacs, found the keybindings to be highly unergonomic, and wondered why so many great programmers are passionate about such a weird editor? You’re not alone – […]
Creating an IT Ticketing System with GitHub and Slack
GitHub is great for an IT ticketing system – it’s easy to create issues, set assignees, and prioritize the ticket with labels. The GitHub Slack integration makes it easy to […]
Designing Scala Packages and Imports for Readable Spark Code
This blog post explains how to import core Spark and Scala libraries like spark-daria into your projects. It’s important for library developers to organize package namespaces so it’s easy for […]
Best Apache Spark Books
Apache Spark is a big data engine that has quickly become one of the biggest distributed processing frameworks in the world. It’s used by all the big financial institutions and […]
Using HyperLogLog for count distinct computations with Spark
This blog post explains how to use the HyperLogLog algorithm to perform fast count distinct operations. HyperLogLog sketches can be generated with spark-alchemy, loaded into Postgres databases, and queried with […]
Delta Lake schema enforcement and evolution with mergeSchema and overwriteSchema
Delta lakes prevent data with incompatible schema from being written, unlike Parquet lakes which allow for any data to get written. Let’s demonstrate how Parquet allows for files with incompatible […]