Apache Spark is one of the most widely used engines for large scale data processing.
It exists to make distributed processing feel close to working with a single table.
It brings batch jobs, SQL and streaming together in one engine.
A Spark application has a driver, which plans the work, and executors, which run it on the cluster's machines.
Data is held in DataFrames that are split into partitions across the executors.
Transformations, such as filter and groupBy, describe steps without running them.
Actions, such as count or writing a file, trigger the planned steps to run.
The optimiser rewrites the plan before running it, for example applying filters early so less data is read.
PySpark is the Python interface, and much of its DataFrame code looks similar to pandas.
Spark SQL lets the same data be queried with ordinary SQL.
Structured Streaming applies the same DataFrame ideas to continuous streams.
Caching keeps a DataFrame in memory when several steps reuse it, avoiding recomputation.
Small data runs slower in Spark than in pandas, because planning and coordination have a fixed cost.
In practice, most Spark tuning is about partitions, shuffles and memory rather than code style.
The usual mistake is collecting a huge DataFrame back to the driver, which overloads that 1 machine.