Notes on Spark: What, Why, How
These are orientation notes, put together from reading rather than from using Spark: its own documentation, and courses on DataCamp and Udemy. The posts planned after this one move from description to use, working through PySpark and Spark SQL in Databricks.
What Spark is
Apache Spark, or just Spark, is an open-source engine for working with data that a single machine cannot hold, or cannot process in reasonable time. It is one of many projects held by the Apache Software Foundation, a non-profit that maintains open-source software.
A Spark job splits a dataset into pieces and gives each piece to a different machine, so that all of them work at the same time. The machines are called nodes, and three parts divide the job between them.
- Driver holds the plan and hands out the work. It runs on one machine, separate from the ones running the executors.
- Executors carry out the work and hold the data between steps. They run on the worker nodes.
- Cluster manager decides how many machines the job gets. It is part of the cluster rather than of the job.
The cluster manager is not always the same software. Spark comes with one called Standalone, and Hadoop’s YARN or Kubernetes can fill the role instead. A cluster runs one of them.
The diagram names three more things, inside those parts. A task is the work done on one piece of the split data. If the data is split into two hundred pieces, each step has two hundred tasks. The cache is memory where an executor can keep a piece, so that a later step uses it again instead of redoing the work. Spark does not cache anything by default. The code has to say so, with a call named cache or persist.
SparkContext sits inside the driver and stands for the connection to the cluster. Code written against the DataFrame API reaches it through a SparkSession.
The splitting, the machines and the parts that hand out the work do not appear in the code. Code works on a DataFrame, and Spark plans the rest.
Why Spark is needed
The case for Spark starts where a single machine runs out: a dataset larger than the memory available, or a job that takes too long on one machine. Spreading the work over several machines answers both, with more memory between them and more processors working at once. That is what Spark does.
Distributing work was not new when Spark arrived. What was new is where the data sits between steps: Spark keeps it in memory. That is what the label “in-memory” means.
Older large-scale processing engines, like Hadoop’s MapReduce, worked the other way around: a job was broken into a series of steps, and the results were written to disk after each one. That approach made sense at the time. Memory used to be small and expensive relative to the size of the datasets being processed, and writing to disk made it easier to recover if something went wrong partway through a job.
As memory became cheaper and more widely available, keeping data in memory during processing became a realistic option. That is much of why Spark can be faster than the older engines on many jobs.
The saving is in the reading. When the same data is used more than once, as when a model is trained or one dataset is queried several times, MapReduce reads it back from disk for every pass. Spark can keep it in memory, so the slow read happens once at the start.
Spreading work over machines costs something too. Starting a job, planning it and moving data between machines all take time. A program running on one machine does none of that. So while the data still fits in memory, pandas, Polars or DuckDB do the same work without that cost. Spark pays off when the data no longer fits.
How Spark is used
Spark can be installed and run without any platform around it. Spark and Java both go on the machine directly. Java is there because Spark’s engine runs on the JVM, the Java Virtual Machine. The Spark job itself is then written in a script, an interactive shell or a notebook.
There is no separate server that has to be running ahead of time. Spark starts up only while the code is running. It runs on that one machine, driver and executors together, or spreads to others if they have been set up as workers under one of the cluster managers.
A managed platform is the other route: Databricks, Amazon EMR, Google Cloud Dataproc, Azure Synapse Analytics or Microsoft Fabric, its successor. Databricks is a cloud-based platform founded by the creators of Spark. It provisions the cluster and adds tooling on top, such as notebooks, job scheduling and autoscaling. It also lets several languages be mixed side by side in the same notebook. In its settings the machine running the driver is the driver node and the rest are workers.
Several languages reach Spark. Whether a language runs on the JVM decides how it connects to the engine. Scala and Java run on the JVM, where the engine is, so they reach it directly. Python and R run outside it and pass work and data in. SQL is a case of its own: not a program to run, but a query handed to the engine.
Scala
Spark’s engine is written in Scala (a general-purpose programming language), and Scala compiles to run on the JVM. There is nothing standing between Scala code and the engine.
Java
Java compiles to run on the JVM as well, so it reaches the engine by the same route as Scala. It appears less often in Spark code than Scala or Python, and the reason usually given is that the same operation takes more lines to write, not that it can do less.
Python
Used through a library called pyspark. Python runs in a process of its own, outside the JVM, so the two pass work and data to each other. An operation Spark already knows, like a filter or a join, runs entirely on the Java side. A function written in Python and applied row by row is different: every row is handed over and back. Python is still the most used of the five, because so many data tools are written for it, pandas and scikit-learn among them.
R
R reaches Spark in two ways, and one of them is closing. SparkR came with Spark and is deprecated as of Spark 4.0, to be removed in Spark 5. sparklyr, maintained by Posit rather than by the Spark project, is the one that stays. Like Python, R runs in a process of its own, so it works the same way.
SQL
SQL also reaches Spark in two ways. It can be written by itself, in a notebook cell or at Spark’s spark-sql command line. The other way is inside code in another language, as a string, as in spark.sql("...") in Python or Scala. That spark is the SparkSession named earlier. Either way the query goes to the same engine underneath, the part of Spark called Spark SQL.
What comes next
The notes planned next take up Python (via PySpark) and Spark SQL, in Databricks.