Migrating Scala Projects to Spark 3
This post explains how to migrate your Scala projects to Spark 3. It covers the high level steps and doesn’t get into all the details. Migrating PySpark projects is easier. […]
mrpowers
Reading data from Google Sheets to Spark DataFrames
This blog post explains how to read a Google Sheet into a Spark DataFrame with the spark-google-spreadsheets library. Google Sheets is not a good place to store a lot of […]
Managing Multiple Java, SBT, and Scala Versions with SDKMAN
SDKMAN is an amazing project that makes it easy to manage multiple versions of Java and Scala on a single computer. It also provides support for other Java ecosystem projects […]
Running Multiple Versions of Java on MacOS with jenv
jenv makes it easy to run multiple versions of Java on a Mac computer. It also makes it easy to seamlessly switch between Java versions when you switch projects. Running […]
Scala Templates with Scalate, Mustache, and SSP
The scalate library makes it easy to use Mustache or SSP templates with Scala. This blog post will show how to use Mustache and SSP templates and compares the different […]
Expressively Typed Spark Datasets with Frameless
frameless is a great library for writing Datasets with expressive types. The library helps users write correct code with descriptive compile time errors instead of runtime errors with long stack […]
Write sqlite tables to CSV / Parquet files
This blog post explains how to write sqlite tables to CSV and Parquet files. It’ll also show how to output SQL queries to CSV files. It’ll even show how to […]
Creating a sqlite database from CSVs with Python
This blog post demonstrates how to build a sqlite database from CSV files. Python is perfect language for this task because it has great libraries for sqlite and CSV DataFrames. […]
Amazing Python Data Workflow with Poetry, Pandas, and Jupyter
Poetry makes it easy to install Pandas and Jupyter to perform data analyses. Poetry is a robust dependency management system and makes it easy to make Python libraries accessible in […]
Writing Custom Metadata to Parquet Files and Columns with PyArrow
Metadata can be written to Parquet files or columns. This blog post explains how to write Parquet files with metadata using PyArrow. Here are some powerful features that Parquet files […]