select and add columns in PySpark
This post shows you how to select a subset of the columns in a DataFrame with select. It also shows how select can be used to add and rename columns. […]
Filtering PySpark Arrays and DataFrame Array Columns
This post explains how to filter values from a PySpark array column. It also explains how to filter DataFrames with array columns (i.e. reduce the number of rows in a […]
Combining PySpark DataFrames with union and unionByName
Multiple PySpark DataFrames can be combined into a single DataFrame with union and unionByName. union works when the columns of both DataFrames being joined are in the same order. It […]
Combining PySpark arrays with concat, union, except and intersect
This post shows the different ways to combine multiple PySpark arrays into a single array. These operations were difficult prior to Spark 2.4, but now there are built-in functions that […]
exists and forall PySpark array functions
This blog post demonstrates how to find if any element in a PySpark array meets a condition with exists or if all elements in an array meet a condition with […]
Exploring DataFrames with summary and describe
The summary and describe methods make it easy to explore the contents of a DataFrame at a high level. This post shows you how to use these methods. TL;DR – […]
Calculating Percentile, Approximate Percentile, and Median with Spark
This blog post explains how to compute the percentile, approximate percentile and median of a column in Spark. There are a variety of different ways to perform these computations and […]
Scala is a Maintenance Nightmare
This post explains why Scala projects are difficult to maintain. Scala is a powerful programming language that can make certain small teams hyper-productive. Scala can also slow productivity by drowning […]
Scala Spark vs Python PySpark: Which is better?
Apache Spark code can be written with the Scala, Java, Python, or R APIs. Scala and Python are the most popular APIs. This blog post performs a detailed comparison of […]
Type 2 Slowly Changing Dimension Upserts with Delta Lake
This post explains how to perform type 2 upserts for slowly changing dimension tables with Delta Lake. We’ll start out by covering the basics of type 2 SCDs and when […]