DataFrames in Go with gota, qframe, and dataframe-go
Go has great DataFrame libraries that let you easily manipulate data that’s stored in CSV files and databases. Working with CSV files directly can be burdensome. DataFrames are easier because […]
mrpowers
Writing Parquet Files in Python with Pandas, PySpark, and Koalas
This blog post shows how to convert a CSV file to Parquet with Pandas and Spark. It discusses the pros and cons of each approach and explains how both approaches […]
Converting CSV files to Parquet with Go
This blog post explains how to read data from a CSV file and write it out as a Parquet file. The Parquet file format is better than CSV for a […]
Executing Spark code with expr and eval
You can execute Spark column functions with a genius combination of expr and eval(). This technique lets you execute Spark functions without having to create a DataFrame. This makes it […]
Spark Column Equality
The term “column equality” refers to two different things in Spark: When a column is equal to a particular value (typically when filtering) When all the values in two columns […]
Testing Scala with MUnit
MUnit is a test library with a collection of features that are not available in other Scala testing libraries. MUnit draws inspiration from Scalatest, utest, and ava (JavaScript testing library). […]
Testing Scala with Scalatest
Scalatest makes it easy to test your Scala code. This blog post shows how to add Scalatest to a sbt project and write some basic tests. Writing a simple test […]
Scala Library Best Practices
This post explains the best practices Scala libraries should follow. Here are the most important best practices to follow: Great README Clearly defined API Accessible API documentation Clean JAR file […]
Working with Spark MapType Columns
Spark DataFrame columns support maps, which are great for key / value pairs with an arbitrary length. This blog post describes how to create MapType columns, demonstrates built-in functions to […]
Learning Emacs Keybindings After Multiple Failed Attempts
Have you ever tried Emacs, found the keybindings to be highly unergonomic, and wondered why so many great programmers are passionate about such a weird editor? You’re not alone – […]