This blog post explains how to import core Spark and Scala libraries like spark-daria into your projects. It’s important for library developers to organize package namespaces so it’s easy for […]
Apache Spark is a big data engine that has quickly become one of the biggest distributed processing frameworks in the world. It’s used by all the big financial institutions and […]
This blog post explains how to use the HyperLogLog algorithm to perform fast count distinct operations. HyperLogLog sketches can be generated with spark-alchemy, loaded into Postgres databases, and queried with […]
Spark writers allow for data to be partitioned on disk with partitionBy. Some queries can run 50 to 100 times faster on a partitioned data lake, so partitioning is vital […]
This post describes how to programatically compact Parquet files in a folder. Incremental updates frequently result in lots of small files that can be slow to read. It’s best to […]
Spark makes it easy to broadcast maps and perform hash lookups in a cluster computing environment. This post explains how to broadcast maps and how to use these broadcasted variables […]
Spark Structured Streaming and Trigger.Once make it easy to run incremental updates. Spark uses a checkpoint directory to identify the data that’s already been processed and only analyzes the new […]
Spark can use the disk partitioning of files to greatly speed up certain filtering operations. This post explains the difference between memory and disk partitioning, describes how to analyze physical […]
The Spark console is a great way to run Spark code on your local machine. You can easily create a DataFrame and play around with code in the Spark console […]
The SparkSession is used to create and read DataFrames. It’s used whenever you create a DataFrame in your test suite or whenever you read a Parquet / CSV data lake […]