Renaming Columns in Pandas DataFrames
This article explains how to rename a single or multiple columns in a Pandas DataFrame. There are multiple different ways to rename columns and you’ll often want to perform this […]
mrpowers
Adding constant columns with lit and typedLit to PySpark DataFrames
This post explains how to add constant columns to PySpark DataFrames with lit and typedLit. You’ll see examples where these functions are useful and when these functions are invoked implicitly. […]
Navigating None and null in PySpark
This blog post shows you how to gracefully handle null in PySpark and how to avoid null input errors. Mismanaging the null case is a common source of errors and […]
Creating and reusing the SparkSession with PySpark
This post explains how to create a SparkSession with getOrCreate and how to reuse the SparkSession with getActiveSession. You need a SparkSession to read data stored in files, when manually […]
Different ways to write CSV files with Dask
This post explains how to write a Dask DataFrame to CSV files. You’ll see how to write CSV files, customize the filename, change the compression, and append files to an […]
Writing Dask DataFrame to a Single CSV File
Dask DataFrames are composed of multiple partitions and are outputted as multiple files, one per partition, by default. This post explains the different approaches to write a Dask DataFrame to […]
Registering Native Spark Functions
This post explains how Spark registers native functions internally and the public facing APIs for you to register your own functions. Registering native functions is important if you want to […]
select and add columns in PySpark
This post shows you how to select a subset of the columns in a DataFrame with select. It also shows how select can be used to add and rename columns. […]
Filtering PySpark Arrays and DataFrame Array Columns
This post explains how to filter values from a PySpark array column. It also explains how to filter DataFrames with array columns (i.e. reduce the number of rows in a […]
Combining PySpark DataFrames with union and unionByName
Multiple PySpark DataFrames can be combined into a single DataFrame with union and unionByName. union works when the columns of both DataFrames being joined are in the same order. It […]