Registering Native Spark Functions
This post explains how Spark registers native functions internally and the public facing APIs for you to register your own functions. Registering native functions is important if you want to […]
2021
select and add columns in PySpark
This post shows you how to select a subset of the columns in a DataFrame with select. It also shows how select can be used to add and rename columns. […]
Filtering PySpark Arrays and DataFrame Array Columns
This post explains how to filter values from a PySpark array column. It also explains how to filter DataFrames with array columns (i.e. reduce the number of rows in a […]
Combining PySpark DataFrames with union and unionByName
Multiple PySpark DataFrames can be combined into a single DataFrame with union and unionByName. union works when the columns of both DataFrames being joined are in the same order. It […]
Combining PySpark arrays with concat, union, except and intersect
This post shows the different ways to combine multiple PySpark arrays into a single array. These operations were difficult prior to Spark 2.4, but now there are built-in functions that […]
exists and forall PySpark array functions
This blog post demonstrates how to find if any element in a PySpark array meets a condition with exists or if all elements in an array meet a condition with […]
Exploring DataFrames with summary and describe
The summary and describe methods make it easy to explore the contents of a DataFrame at a high level. This post shows you how to use these methods. TL;DR – […]
Calculating Percentile, Approximate Percentile, and Median with Spark
This blog post explains how to compute the percentile, approximate percentile and median of a column in Spark. There are a variety of different ways to perform these computations and […]
Scala is a Maintenance Nightmare
This post explains why Scala projects are difficult to maintain. Scala is a powerful programming language that can make certain small teams hyper-productive. Scala can also slow productivity by drowning […]
Scala Spark vs Python PySpark: Which is better?
Apache Spark code can be written with the Scala, Java, Python, or R APIs. Scala and Python are the most popular APIs. This blog post performs a detailed comparison of […]