Adding executables to your PATH for fun
Adding executables to your PATH is fun, easy, and a great way to learn about how your machine works. This pattern is especially useful for long commands that you need […]
Important Considerations when filtering in Spark with filter and where
This blog post explains how to filter in Spark and discusses the vital factors to consider when filtering. Poorly executed filtering operations are a common bottleneck in Spark analyses. You […]
Deep dive into how pyenv actually works by leveraging the shim design pattern
pyenv lets you manage multiple versions of Python on your computer. This blog post focuses on how pyenv uses the shim design pattern to provide a wonderful user experience (it […]
New Spark 3 Array Functions (exists, forall, transform, aggregate, zip_with)
Spark 3 has new array functions that make working with ArrayType columns much easier. Spark developers previously needed to use UDFs to perform complicated array functions. The new Spark functions […]
DataFrames in Go with gota, qframe, and dataframe-go
Go has great DataFrame libraries that let you easily manipulate data that’s stored in CSV files and databases. Working with CSV files directly can be burdensome. DataFrames are easier because […]
Writing Parquet Files in Python with Pandas, PySpark, and Koalas
This blog post shows how to convert a CSV file to Parquet with Pandas and Spark. It discusses the pros and cons of each approach and explains how both approaches […]
Converting CSV files to Parquet with Go
This blog post explains how to read data from a CSV file and write it out as a Parquet file. The Parquet file format is better than CSV for a […]
Executing Spark code with expr and eval
You can execute Spark column functions with a genius combination of expr and eval(). This technique lets you execute Spark functions without having to create a DataFrame. This makes it […]
Spark Column Equality
The term “column equality” refers to two different things in Spark: When a column is equal to a particular value (typically when filtering) When all the values in two columns […]
Testing Scala with MUnit
MUnit is a test library with a collection of features that are not available in other Scala testing libraries. MUnit draws inspiration from Scalatest, utest, and ava (JavaScript testing library). […]