Working with Spark MapType Columns
Spark DataFrame columns support maps, which are great for key / value pairs with an arbitrary length. This blog post describes how to create MapType columns, demonstrates built-in functions to […]
mrpowers
Learning Emacs Keybindings After Multiple Failed Attempts
Have you ever tried Emacs, found the keybindings to be highly unergonomic, and wondered why so many great programmers are passionate about such a weird editor? You’re not alone – […]
Creating an IT Ticketing System with GitHub and Slack
GitHub is great for an IT ticketing system – it’s easy to create issues, set assignees, and prioritize the ticket with labels. The GitHub Slack integration makes it easy to […]
Designing Scala Packages and Imports for Readable Spark Code
This blog post explains how to import core Spark and Scala libraries like spark-daria into your projects. It’s important for library developers to organize package namespaces so it’s easy for […]
Best Apache Spark Books
Apache Spark is a big data engine that has quickly become one of the biggest distributed processing frameworks in the world. It’s used by all the big financial institutions and […]
Using HyperLogLog for count distinct computations with Spark
This blog post explains how to use the HyperLogLog algorithm to perform fast count distinct operations. HyperLogLog sketches can be generated with spark-alchemy, loaded into Postgres databases, and queried with […]
Delta Lake schema enforcement and evolution with mergeSchema and overwriteSchema
Delta lakes prevent data with incompatible schema from being written, unlike Parquet lakes which allow for any data to get written. Let’s demonstrate how Parquet allows for files with incompatible […]
Selectively updating Delta partitions with replaceWhere
Delta makes it easy to update certain disk partitions with the replaceWhere option. Selectively applying updates to certain partitions isn’t always possible (sometimes the entire lake needs the update), but […]
Partitioning on Disk with partitionBy
Spark writers allow for data to be partitioned on disk with partitionBy. Some queries can run 50 to 100 times faster on a partitioned data lake, so partitioning is vital […]
Using Delta lake merge to update columns and perform upserts
This blog posts explains how to update a table column and perform upserts with the merge command. We explain how to use the merge command and what the command does […]