DEV Community: Nicoda-27

How to for developers: Mastering your corporate MacBook Setup

Nicoda-27 — Sat, 17 May 2025 09:51:05 +0000

Starting with a fresh MacBook can be exciting, but navigating corporate IT requirements can feel daunting. This article demystifies the process, offering a step-by-step guide to ensure a smooth and efficient setup that aligns with your company's policies and empowers your productivity from day one.

This article will focus on a persona of a Python Developer but is transferable for any developers.

A corporate MacBook

Depending on your company, the Macbook provided to you as a developer workstation can pose certain difficulties when compared with a privately own MacBook.

Namely:

You will not be sudo of your workstation
A proxy could be implemented as company policy

Not being sudo will be the main difficulty and we will see how to navigate around that and still comply with your company policies, meaning we will not hack the system but leverage the MacOs potential.

Homebrew

Homebrew is the go to for developer using MacOs to be able to install applications. It's the equivalent of Aptitude in Ubuntu.

The official documentation requires you to be sudo to install it. But you can also install it for your specific user with

mkdir $HOME/homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew

This will create a directory Homebrew in your $HOME where brew will be installed.

You can now add it to your path permanently, in the rest of this article, it's assumed that zsh is your favorite shell (but feel free to use any other shells that you prefer)

echo 'export PATH=$HOME/homebrew/bin:$PATH' >> ~/.zshrc

Let's check that Homebrew is detected and works:

The following command will make sure that the above export happened on your current terminal

source ~/.zshrc

The following command should display a version, and proves that Homebrew was properly installed and is usable by your system.

brew --version

Cli tools

Then let's give a try and install something

brew install git

Now you are able to install all your favorite developer tools if packaged in Homebrew. They would appear for your user and you can install them without sudo permissions, yay !

Rich applications

Rich application will require an Applications directory for it to be installed; by default brew choose the /Applications directory which is located at the root of your file system and will not be accessible for you.

For instance, let's try to install OpenLens, it's a developer tool that showcase an UI to observe your Kubernetes cluster, it's proposed as a tool package in homebrew

If you try to install it with

brew install openlens

This installation will fail because homebrew will not be able to use /Applications.

Instead, let's create our own Applications directory for our user.

mkdir $HOME/Applications

Then you can leverage it when using Homebrew

brew install openlens --appdir $HOME/Applications

You can now try to open OpenLens as an application from the Spotlight Search.

You can now install rich applications without sudo, yay !

Rosetta

There might be circonstances where you will need a specific target architecture for your application to run (namely either arm or intel).

The native architecture for M1, M2, M3, M4 MacBook is now arm, but Rosetta provides a way to emulate x86_64 architecture. It's very useful when librairies have been compiled for only one specific architecture, x86_64 being older there are higher chances to be available for this rather than for arm.

You can implement aliases to switch the Rosetta emulator on or off. You can add these two aliases to your .zshrc

alias arm="env /usr/bin/arch -arm64 /bin/zsh --login"
alias intel="env /usr/bin/arch -x86_64 /bin/zsh --login"

To make sure of which target you are own, you can type

arch

It should either displays i386 (x86_64/Intel architectures) or arm64 (arm/M* architectures

Installing language specific tools

To help you manage different versions of python, node, awscli, cargo the usage of mise will be demonstrated.

We will install two versions of mise so it can handle environments with both architecture as advised in here.

Just follow the installation guide above and should be able to have mise for x64 available with

mise-x64 --version

And you can also have mise available for arm if you're sure you don't need x86_64 specifics.

Just follow the usual installation with Homebrew:

brew install mise

You can now install python, node or many tools with mise.

The following command will install python for a specific x64_86 target:

mise-x64 use python@3.10

Containerization tools

When we speak about containerization tool, there are multiple candidates.

The most used would be [docker](https://www.docker.com/- but open source alternatives like podman also exists.

Docker

You will need support of your IT team to be able to install docker as it requires sudo access.

Podman

It can act as docker but does not requires elevated privileges, you can easily install it with Homebrew or follow the guide:

brew install podman

You can alias podman to docker to make it even more transparent. Add the alias in your .zshrc.

alias docker=podman

As the podman cli is compliant with the docker cli, it should work transparently.

Give it a try:

docker run hello-world

and it should be working. You will first have to run the following commands to initialise podman:

podman machine init
podman machine start

There is a dedicated troubleshoot section to help you out.

Disclaimer:
Some tools like testcontainers relies entirely on pure docker therefore you will have issue using podman to replace it.

Though it can be solved thanks to podman-mac-helper but it requires elevated privileges from your IT team.

Proxy and certificates

Your company might put in place a proxy that is using a certificate.

When using python tools like az-cli, you will be blocked and stumble upon similar issues like this.

To solve it, as documented you will need to create an environment variables called REQUESTS_CA_BUNDLE to indicate the path to your certificate.

This is the case also when you will be installing dependencies and that a python tool requires internet access.

Finding the certificate

You will need to find the name of the certificate, so we can export it. To find the name, you can open Keychain Access application, on the certificates section, open the certificate added by your proxy team and copy the "Common name", let's assume the certificate is called "corporate_proxy_certificate".

security find-certificate -p -c "corporate_proxy_certificate" /Library/Keychains/System.keychain

This should display the raw certificate value. If not you can look at other usual places for certificates, it can also be under /System/Library/Keychains/SystemRootCertificates.keychain.

Export the certificate

Now you can export it to a pem file like so:

security find-certificate -p -c "corporate_proxy_certificate" /Library/Keychains/System.keychain >$HOME/corporate_proxy_certificate.pem

See the content of the .pem file with:

cat $HOME/corporate_proxy_certificate.pem

Leverage the certificate

You can now export the environment variable and add it in the .zshrc

echo 'export REQUESTS_CA_BUNDLE=/$HOME/corporate_proxy_certificate.pem' >> ~/.zshrc

And now, under the proxy any call to az cli tool should be working like a charm.

It also means that when disabling the proxy, you will need to unset the REQUESTS_CA_BUNDLE because it's part of your .zshrc like so:

unset REQUESTS_CA_BUNDLE
az login

Conclusion

Hoping this article help navigating through corporate compliancy and still maintain high developer productivity.

Don't hesitate to reach out in comments if further explanation are necessary, or if there are missing topics.

How to be Test Driven with Spark: Chapter 5: Leverage spark in a container

Nicoda-27 — Sat, 15 Mar 2025 07:33:58 +0000

This goal of this tutorial is to provide a way to easily be test driven with spark on your local setup without using cloud resources.

This is a series of tutorials and the initial chapters can be found in:

In chapter 3, it was demonstrated that the current testing approach rely on Java being available on the developer setup. As mentioned, this is not ideal as there is limited control and unexpected behavior can happen. A good testing practice is to have reproducible and idempotent tests, this means:

Launching the tests an infinite number of times should always have the same results
A test should leave a clean plate after it has run, there should be no side effect to a test running (no files written, no change of environment variables, no database with remaining data etc)

The reasons why it's so important, is because otherwise you will spend most of your time relaunching the tests due to false positive, you would never be sure if you actually broke something or if the test is randomly failing. At the end, you will not trust the tests anymore and skip some of them, which defeats the purpose.

Why using a container?

If you are unfamiliar with the concept of containers and docker images, I suggest you have a look at docker. It will be leveraged here to start the Spark server for the tests; it's important to mention there are other opensource alternatives like podman or nerdctl to allow containerization.

Docker will be used thereafter as it has become the defacto standards for most companies, and it's available in the Github ci runner. It will be assumed that you have enough knowledge about the technology to use it.

Container with spark connect

There is a small subtlety that needs to be understood. Previously, the Java Virtual Machine (JVM) was used to communicate with the python spark implementation (through the spark_session), it was using the java binary to create a swarm of workers that were handling the data processing. At the end, all the results were collected and communicated to the spark_session which was exposing it in the python code.

If you start a container with this, the spark_session will never be able to find the JVM inside the container as it's a binary. The container you want to create needs a way to communicate outside with the spark_session through the network. Luckily, Spark connect is providing a solution and the documentation is a must known. This is the chosen approach to containerize the Spark server and the worker creation.

Spark is already providing a docker image that you will leverage. If you don't have docker available on your setup, you will need to install it, see the official documentation.

Let's uninstall openjdk to make sure spark_session will use the new setup, it will require elevation of privileges:

apt-get autoremove openjdk-8-jre

You can now relaunch the tests, it's expected that they fail with the following error:

ERROR tests/test_minimal_transfo.py::test_minimal_transfo - pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.
ERROR tests/test_minimal_transfo.py::test_transfo_w_synthetic_data - pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

Start the container

You will need to start the container with spark connect, you can launch

docker run -p 8081:8081 -e SPARK_NO_DAEMONIZE=True --name spark_connect apache/spark /opt/spark/sbin/start-connect-server.sh org.apache.spark.deploy.master.Master --packages org.apache.spark:spark-connect_2.12:3.5.2,io.delta:delta-core_2.12:2.3.0 --conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp -Divy.home=/tmp' --conf spark.connect.grpc.binding.port=8081

It will print a lot in the terminal and at the end you should have:

24/12/27 14:04:27 INFO SparkConnectServer: Spark Connect server started at: 0:0:0:0:0:0:0:0%0:8081

This shows that the Spark server is up and running.

Each argument in the above command has a meaning and its importance:

docker run is the docker command to start a container
-p 8081:8081 is an arguments to docker run that enables to use port 8081 to communicate with the created container
-e SPARK_NO_DAEMONIZE=True is an environment variable that is passed to the container creation, it's necessary to use it for the server to be created as a foreground process
--name spark_connect allows to name the created container
apache/spark is the docker image that is used, if you never used it, it will be downloaded from Docker Hub

The rest of the command is what is called an entrypoint, it's the command that will be executed inside the container. In here it contains multiple elements:

/opt/spark/sbin/start-connect-server.sh is the binary of the spark server
org.apache.spark.deploy.master.Master is an argument to the binary, in here the binary is asked to deploy a Master server, the same binary can be used to deploy a Worker
--packages org.apache.spark:spark-connect_2.12:3.5.2,io.delta:delta-core_2.12:2.3.0 is an optional argument to pass specific versions of spark, and delta dependencies
--conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp -Divy.home=/tmp' is extra argument to ask the server to write to /tmp inside the container, it's not a mandatory argument
--conf spark.connect.grpc.binding.port=8081 is an extra argument to start the server on the port 8081 on the localhost of the container

The last argument is where the magic happens, the server is started on port 8081, and docker is exposing the port of this container to the port of the docker host. Meaning, a spark server is now available on http://localhost:8081

Use the container

Keep the previous terminal opened to keep the server running and open a new terminal. Now run:

pytest -k test_transfo_w_synthetic_data -s

The same error should appear, indeed the spark_session needs to be adapted to connect to the server you have just created. In test/conftest.py:

@pytest.fixture(scope="session")
def spark_session() -> Generator[SparkSession, Any, Any]:
    yield (
        SparkSession.builder.remote("sc://localhost:8081")  # type: ignore
        .appName("Testing PySpark Example")
        .getOrCreate()
    )

Basically, it indicates the Spark connect server url to the Spark session.

And you need to add an extra dependency, which is mandatory to communicate with the spark connect server. It's worth pointing to the usage of extras in uv:

uv add pyspark --extra connect

As this project is in Python 3.12, another error will appear related to distutils as it was removed from the latest python version, yet some dependencies still requires it. You will have to add:

uv add setuptools

Now you can run:

pytest -k test_minimal_transfo -s

And it should run successfully, you should also see logs in the spark server in the docker run terminal.

Improve the container usage

As mentioned at the beginning of this chapter, the tests need to leave a clean plate. In the previous approach, a container is still running eventhough the tests are done, it's not ideal.

To improve this, you will leverage testcontainers which empower you with easy docker creation and removal at the test level.

uv add testcontainers --dev

Now, the docker can be started at the session fixture level, in tests/conftest.py, you can add an extra fixture:

from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import wait_for_logs

@pytest.fixture(scope="session")
def spark_connect_start():
    kwargs = {
        "entrypoint": "/opt/spark/sbin/start-connect-server.sh org.apache.spark.deploy.master.Master --packages org.apache.spark:spark-connect_2.12:3.5.2,io.delta:delta-core_2.12:2.3.0 --conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp -Divy.home=/tmp' --conf spark.connect.grpc.binding.port=8081",
    }
    with (
        DockerContainer(
            "apache/spark",
        )
        .with_bind_ports(8081, 8081)
        .with_env("SPARK_NO_DAEMONIZE", "True")
        .with_kwargs(**kwargs) as container
    ):
        _ = wait_for_logs(
            container, "SparkConnectServer: Spark Connect server started at"
        )
        yield container

This will create a container with the previously described argument, the great thing with fixtures is that will kill the container at the end of the test execution. There is an extra step with:

        _ = wait_for_logs(
            container, "SparkConnectServer: Spark Connect server started at"
        )

This enforces to yield the container only when the SparkConnectServer: Spark Connect server started at appeared in the container logs. It's necessary to wait for the server to be ready until it can be called.

The value that is yielded is the container which also contains the server url, you need to reuse in the spark_session fixture:

@pytest.fixture(scope="session")
def spark_session(spark_connect_start: DockerContainer) -> Generator[SparkSession, Any, Any]:
    ip = spark_connect_start.get_container_host_ip()
    yield (
        SparkSession.builder.remote(f"sc://{ip}:8081")  # type: ignore
        .appName("Testing PySpark Example")
        .getOrCreate()
    )

You can now stop the container you started before

docker stop spark_connect

And run the tests:

pytest

You will notice all the tests are passing, and at the end of the test session there is no running containers.

The following command will show what remaining containers are still running. The spark container should not appear.

docker ps -a

Conclusion

You are now able to run local tests using spark and you can quickly iterate on your codebase and implement new features. You are no more depending on spark server to be launched for you on the cloud and waiting for it to process the data for you.

The feedback loop is quicker, you are no more giving money to cloud provider for testing purposes and you provide an easy setup for developers to iterate on your project.

They can launch pytest and will be transparent; this also means less documentation for you to write to describe the expected developer setup.

You can find the original materials in spark_tdd. This repository exposes what's the expected repository layout at the end of each chapter in each branch:

What's next

Several ideas come to mind on how to improve our very small codebase

Leverage devcontainer to improve ci and local development
Templatize the repository for easier reusage with the help of ffizer
Explore ibis to handle multiple transformation backends transparently

How to be Test Driven with Spark: Chapter 4 - Leaning into Property Based Testing

Nicoda-27 — Sun, 09 Mar 2025 08:38:56 +0000

This goal of this tutorial is to provide a way to easily be test driven with spark on your local setup without using cloud resources.

This is a series of tutorials and the initial chapters can be found in:

The test that you implemented in Chapter 3 is great, yet not complete as it takes only a limited amount of data. As spark is used to process data at scale, you have to test at scale too.

There are several solutions, the first one being taking a snapshot of production data and reusing at the test level (meaning integration test or local test). The second one is to generate synthetic data based on the data schema. With the second approach, you will be leaning into a property based testing approach.

The second approach will be leveraged here as the test case generation is deported to automated generation.

The python ecosystem provides Hypothesis for proper property based testing, or Faker for fake data generation. Hypothesis is way more powerful than Faker in the sense that it will generate test cases for you based on data property (being a string, being an integer etc) and shrink the test cases when unexpected behavior happen. Faker will be used here to generate synthetic data based on business property.

A data driven test

You need two new fixtures similar to persons and employments that will generate synthetic data. First you need to install faker as a dev dependency:

uv add faker --dev

You can create persons_synthetic in tests/conftest.py like so:

@pytest.fixture(scope="session")
def persons_synthetic(spark_session: SparkSession) -> Generator[DataFrame, Any, Any]:
    fake = Faker()

    nb_elem = fake.pyint(1, 100_000)
    data = [
        (i, fake.first_name(), fake.last_name(), fake.date()) for i in range(nb_elem)
    ]
    yield spark_session.createDataFrame(
        data,
        ["id", "PersonalityName", "PersonalitySurname", "birth"],
    )

In the above, a data frame of 100 000 rows is generated, feel free to increase the size to generate larger data frames. Fake names, surnames and date are generated on the fly according to business needs.

You can also create employments_synthetic in tests/conftest.py, there is a dependency on foreign_key from persons_synthetic that needs to be handled:

@pytest.fixture(scope="session")
def employments_synthetic(
    spark_session: SparkSession, persons_synthetic: DataFrame
) -> Generator[DataFrame, Any, Any]:
    fake = Faker()
    persons_sample = persons_synthetic.sample(0.8)
    person_ids_sample = persons_sample.select(collect_list("id")).first()[0]

    data = [(idx, id_fk, fake.job()) for idx, id_fk in enumerate(person_ids_sample)]
    yield spark_session.createDataFrame(
        data,
        ["id", "person_fk", "Employment"],
    )

The foreign_key is reused from a sample of persons_synthetic and job name are generated on the fly.

The test can now be created:

def test_transfo_w_synthetic_data(
    persons_synthetic: DataFrame, employments_synthetic: DataFrame, spark_session
):
    processor = DataProcessor(spark_session)
    df_out: DataFrame = processor.run(persons_synthetic, employments_synthetic)

    assert not df_out.isEmpty()
    assert set(df_out.columns) == set(
        ["name", "surname", "date_of_birth", "employment"]
    )

And you can launch pytest -k test_transfo_w_synthetic_data -s that should pass.

How to handle slow tests

You might notice that test_transfo_w_synthetic_data is a bit slow, indeed it's generating a decent amount of data (even though far from a big data scale), modifying the data frames and joining two together.

In a test driven approach, it's necessary to have a quick feedback loop to iterate quickly on your local setup. Yet, this tests needs to be launched anyway as they validate behavior with decent amount of data.

A solution is to add tags to tests like so:

import pytest
...

@pytest.mark.slow
def test_transfo_w_synthetic_data(
    persons_synthetic: DataFrame, employments_synthetic: DataFrame, spark_session
):
...

This tag can be leveraged by pytest to filter out tests at execution time, see documentation.

and add to pyproject.toml the expected markers for Pytest

[tool.pytest.ini_options]
pythonpath = ["src"]
markers = ["slow"]

Pytest is now aware of this new marker when launching:

pytest --markers

You can now launch:

pytest -m "not slow"

It will validate only the tests not marked as slow.

In the ci, there is nothing to change as by default Pytest will launch all the test.

What's next?

On the next chapter, the next chapter will focus on test repeatability by improving how java is used for Spark at the test level.

You can find the original materials in spark_tdd. This repository exposes what's the expected repository layout at the end of each chapter in each branch:

[15/03/25 UPDATE]: Chapter 5 has been released

How to be Test Driven with Spark: Chapter 3 - First Spark test

Nicoda-27 — Sat, 01 Mar 2025 08:01:00 +0000

This goal of this tutorial is to provide a way to easily be test driven with spark on your local setup without using cloud resources.

This is a series of tutorials and the initial chapters can be found in:

Chapter 3: Implement a first test with spark

This chapter will focus on implementing a first spark data manipulation with an associated test. It will go through the issues that will be encountered and how to solve them.

The data

A dummy use case is used to demonstrate the workflow.

The scenario is that production data is made of two tables persons and employments with the following schema and data types. Here is a sample of the data.

Persons

id: int	PersonalityName: str	PersonalitySurname: str	birth: datetime(str)
1	George	Washington	1732-02-22
2	Henry	Ford	1863-06-30
3	Benjamin	Franklin	1706-01-17
4	Martin	Luther King Jr.	1929-01-15

Employments

id: int	person_fk: int	Employment
1	1	president
2	2	industrialist
3	3	inventor
4	4	minister

The goal is to change the names of the columns and to join the data. The data here is just a sample, it's overkill to use spark to process data like this. Yet, in a big data context, you need to foresee that the data will contains more lines and more complex joins. The sample is just here as a demonstration.

The dummy test

First, you need to add spark dependencies

uv add pyspark

Before diving into the implementation, you need to make sure you can reproduce a very simple use case. It's not worth diving into complex data manipulation if you are not able to reproduce simple documentation snippet.

You will write your first test test_minimal_transfo.py. You will try first to use pyspark to do simple data frame creation.

from pyspark.sql import SparkSession


def test_minimal_transfo():
    spark: SparkSession = (
        SparkSession.builder.master("local")
        .appName("Testing PySpark Example")
        .getOrCreate()
    )

    df = spark.createDataFrame([(3, 4), (1, 2)], ["col1", "col2"])
    df.show()

The first part with the session create or fetch a local spark session, the second part leverages the session to create a data frame.

Then you can launch:

pytest -k test_minimal_transfo -s

If you have a minimal developer setup, it should not work because it's trying to use Java which you might be missing and the following error will be displayed:

FAILED tests/test_minimal_transfo.py::test_minimal_transfo - pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number

It's a bit annoying, because you need to have Java installed on our dev setup, the ci setup and all your collaborators setup. On the future chapters, a better alternative will be described.

There are different flavors of Java, you can simply install the openjdk one. It will require elevation of privileges:

apt-get install openjdk-8-jre

You can now relaunch:

pytest -k test_minimal_transfo -s

and it should display

+----+----+                                                                     
|col1|col2|
+----+----+
|   3|   4|
|   1|   2|
+----+----+

This is a small victory, but you can now use a local spark session to manage data frames, yay !

The real test case - version 0

On the previous sample, it shows that the spark session plays a pivotal role, it will be instantiated differently in the tests context than in the production context.

This means we can leverage a pytest fixture to be reused for all tests later on; it can be created at the session level so there is only one spark session for the whole test suite. Meaning, you can create a tests/conftest.py to factorize common behavior. If you are not familiar with pytest and fixtures, it's advised to have a look at documentation.

In tests/conftest.py:

from typing import Any, Generator

import pytest
from pyspark.sql import SparkSession


@pytest.fixture(scope="session")
def spark_session() -> Generator[SparkSession, Any, Any]:
    yield (
        SparkSession.builder.master("local")
        .appName("Testing PySpark Example")
        .getOrCreate()
    )

Then, it can be reused in tests/test_minimal_transo.py:

from pyspark.sql import SparkSession

def test_minimal_transfo(spark_session: SparkSession):

    df = spark_session.createDataFrame([(3, 4), (1, 2)], ["col1", "col2"])
    df.show()

You can again run pytest -k test_minimal_transfo -s to check the behavior has not changed. It's important in a test driven approach to keep launching the tests after code modification to ensure nothing was broken.

To be closer to the business context, you can implement a data transformation object. There will be a clear separation between data generation and data transformation. You can do so in src/data_transform.py

from pyspark.sql import DataFrame, SparkSession


class DataProcessor:
    def __init__(self, spark_session: SparkSession):
        self.spark_session = spark_session

    def run(self, persons: DataFrame, employments: DataFrame) -> DataFrame:
        raise NotImplementedError

Now, there is a prototype for DataProcessor, the tests can be improved to actually assert on elements like so in test_minimal_transfo.py

from pyspark.sql import DataFrame, SparkSession

from pyspark_tdd.data_processor import DataProcessor


def test_minimal_transfo(spark_session: SparkSession):
    persons = spark_session.createDataFrame(
        [
            (1, "George", "Washington", "1732-02-22"),
            (2, "Henry", "Ford", "1863-06-30"),
            (3, "Benjamin", "Franklin", "1706-01-17"),
            (4, "Martin", "Luther King Jr.", "1929-01-15"),
        ],
        ["id", "PersonalityName", "PersonalitySurname", "birth"],
    )
    employments = spark_session.createDataFrame(
        [
            (1, 1, "president"),
            (2, 2, "industrialist"),
            (3, 3, "inventor"),
            (4, 4, "minister"),
        ],
        ["id", "person_fk", "Employment"],
    )
    processor = DataProcessor(spark_session)
    df_out: DataFrame = processor.run(persons, employments)

    assert not df_out.isEmpty()
    assert set(df_out.columns) == set(
        ["name", "surname", "date_of_birth", "employment"]
    )

The example above will ensure that the data frame fits some criteria, but it will raise an NotImplementedError as you have to implement the actual data processing. It's intended, the actual processing code can be created after testing is properly setup.

The actual test is still not ideal as test case generation is part of the test itself. Pytest parametrization can be leveraged:

import pytest 

from pyspark.sql import SparkSession

@pytest.mark.parametrize(
    "persons,employments",
    [
        (
            (
                [
                    (1, "George", "Washington", "1732-02-22"),
                    (2, "Henry", "Ford", "1863-06-30"),
                    (3, "Benjamin", "Franklin", "1706-01-17"),
                    (4, "Martin", "Luther King Jr.", "1929-01-15"),
                ],
                ["id", "PersonalityName", "PersonalitySurname", "birth"],
            ),
            (
                [
                    (1, 1, "president"),
                    (2, 2, "industrialist"),
                    (3, 3, "inventor"),
                    (4, 4, "minister"),
                ],
                ["id", "person_fk", "Employment"],
            ),
        )
    ],
)
def test_minimal_transfo(spark_session: SparkSession, persons, employments):
    persons = spark_session.createDataFrame(*persons)
    employments = spark_session.createDataFrame(*employments)

    processor = DataProcessor(spark_session)
    df_out: DataFrame = processor.run(persons, employments)

    assert not df_out.isEmpty()
    assert set(df_out.columns) == set(
        ["name", "surname", "date_of_birth", "employment"]
    )

The above example show how test cases generation can be separated from test runs. It allows to see at first glance what this test is about without noise about test data. Most likely, the test data frames could be reused in another test, it needs to be refactored again. The test part becomes:

from pyspark.sql import DataFrame, SparkSession

from pyspark_tdd.data_processor import DataProcessor


def test_minimal_transfo(spark_session: SparkSession, persons: DataFrame, employments: DataFrame):
    processor = DataProcessor(spark_session)
    df_out: DataFrame = processor.run(persons, employments)

    assert not df_out.isEmpty()
    assert set(df_out.columns) == set(
        ["name", "surname", "date_of_birth", "employment"]
    )

and two fixtures persons and employments are created in tests/conftest.py:

@pytest.fixture(scope="session")
def persons(spark_session: SparkSession) -> Generator[DataFrame, Any, Any]:
    yield spark_session.createDataFrame(
        [
            (1, "George", "Washington", "1732-02-22"),
            (2, "Henry", "Ford", "1863-06-30"),
            (3, "Benjamin", "Franklin", "1706-01-17"),
            (4, "Martin", "Luther King Jr.", "1929-01-15"),
        ],
        ["id", "PersonalityName", "PersonalitySurname", "birth"],
    )


@pytest.fixture(scope="session")
def employments(spark_session: SparkSession) -> Generator[DataFrame, Any, Any]:
    yield spark_session.createDataFrame(
        [
            (1, 1, "president"),
            (2, 2, "industrialist"),
            (3, 3, "inventor"),
            (4, 4, "minister"),
        ],
        ["id", "person_fk", "Employment"],
    )

You can now relaunch pytest -k test_minimal_transfo -s and notice the NotImplementedError being raised; which is a good thing. The code has changed 3 times, yet the behavior remains the same, and the tests confirm it.

The real test case - version 1

Now that there is a proper testing in place, source code can be implemented. There could be variations of this, the intent here is not to provide the best source code, but the best way to test:

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql.functions import to_date


class DataProcessor:
    def __init__(self, spark_session: SparkSession):
        self.spark_session = spark_session
        self.persons_rename = {
            "PersonalityName": "name",
            "PersonalitySurname": "surname",
            "birth": "date_of_birth",
        }
        self.employments_rename = {"Employment": "employment"}

    def run(self, persons: DataFrame, employments: DataFrame) -> DataFrame:
        persons = persons.withColumn(
            "birth", to_date(persons.birth)
        ).withColumnsRenamed(colsMap=self.persons_rename)

        employments = employments.withColumnRenamed(colsMap=self.employments_rename)
        joined = persons.join(
            employments, persons.id == employments.person_fk, how="left"
        )
        joined = joined.drop("id", "person_fk")
        return joined

If you rerun pytest -k test_minimal_transfo -s, then the test is successful.

What about ci?

A strong dependency to Java is now in place, running the tests in ci will depend on the ci having Java installed or not. This is an issue because it requires the developer to have a defined dev setup outside of the python ecosystem, there are extra steps for anyone to launch the tests.

Keep in mind, there is limited control over the developer setup, what if the Java already installed in the developer setup is not spark compliant? It will then be frustrating for the developer to investigate and most likely reinstall another Java version which might impact other projects. See the mess

Luckily, the ci runner on Github has Java installed for us; so the ci should run.

Clean up

You can now also clean up the repository to have a clean plate. For instance, src/pyspark_tdd/multiply.py and tests/test_dummy.py can be removed.

What's next

Now, you have a comfortable setup to modify and tweak the code. You can run the tests and be sure to reproduce.

In the next chapter, a more data driven approach to test case generation will be explored.

You can find the original materials in spark_tdd. This repository exposes what's the expected repository layout at the end of each chapter in each branch:

[09/03/25 UPDATE]: Chapter 4 has been released
[15/03/25 UPDATE]: Chapter 5 has been released

How to be Test Driven with Spark: Chapter 2 - CI

Nicoda-27 — Sun, 23 Feb 2025 10:21:54 +0000

This goal of this tutorial is to provide a way to easily be test driven with spark on your local setup without using cloud resources.

This is a series of tutorials and the initial chapters can be found in:

Chapter 0 and 1

Chapter 2: Continuous Integration (ci)

Having a ci is mandatory for any project that aims at having multiple contributors. In the following chapter, a proposal ci will be implemented.

As ci implementation is specific to a collaborative platform being Github, Gitlab, Bitbucket, Azure Devops etc. The following chapter will try to provide a technology agnostic ci as much as possible.

Similar concepts are available in all ci, you will have to transpose the concepts that will be used here.

Content of the ci

The ci here will be very minimal but showcases concepts that you implemented in Chapter 1, namely:

Python setup
Project setup
Code Formatting
Test automation

There are many more addition to the continuous integration that will not be tackled here. A minimal ci is required to guarantee non regressions in terms of:

code styling rules to guarantee no indivual contributors diverge from the coding style
tests, namely all tests must be passing

Implementation

Github provides extensive documentation for you to tweak your ci.

Github is expecting ci files to be provided at a specific location, you can therefore create a file in .github/workflows/ci.yaml.

In this file, you can add

name: Continuous Integration
run-name: Continuous Integration
on: [push]
jobs:
  Continuous-Integration:
    runs-on: ubuntu-latest

The name and run-name define the names of the pipeline that will run.
The on defines the event that will trigger the pipeline to run, push means that for every commit the pipeline will run.
The jobs defines a list of jobs, the ci is made of one job with multiple steps for the sake of simplicity.
The runs-on defined the docker image used to run (the runner) the environment against, it's a list of docker images maintained by Github.

Now into the steps section we can add:

steps:
  - name: Check out repository code
    uses: actions/checkout@v4
  - uses: jdx/mise-action@v2
  - name: Run Formatting
    run: |
      uv run ruff check
  - name: Run Tests
    run: |
      uv run pytest

The actions/checkout@v4 is the Github action that checkout the current branch of the repository.
The jdx/mise-action@v2 is the Github action that will read the mise.toml and install everything for us.
The Run Formatting step will install the dependencies and run the formatting. It there is an error, the command will fail and the pipeline too.
The Run Tests step will run the tests. It there is an error, the command will fail and the pipeline too.

Ci as documentation

As it was stated, the ci is the only source of truth. If it passes on ci, it should pass on your local setup. If not, it means there are discrepancies between the ci setup and yours.

Going through the ci implementation will help you on reproducibility. Maybe you're not using the same way to install python version, or the same dependency management tool. You need to align your tools and the ones presented in chapter 1 help not to conflict with your local setup. You might have installed python package globally or you might have manually changed PYTHON_HOME or your PATH and this can easily be a mess.

To help on reproducibility, a dev container approach can be used. It means, the ci will run inside a container and this container can be reused as a developer environment. This will not be implemented for the moment.

A better ci structure

To improve readability and segregates between code formatting and testing, Github actions can be implemented as job with interdependencies. Then, the workflow becomes:

name: Continuous Integration
run-name: Continuous Integration
on: [push]
jobs:
  Formatting:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - uses: jdx/mise-action@v2
      - name: Run Formatting
        run: |
          uv run ruff check
  Tests:
    runs-on: ubuntu-latest
    needs: [Formatting]
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - uses: jdx/mise-action@v2
      - name: Run Tests
        run: |
          uv run pytest

In here we added the needs: [Formatting] to create dependencies between ci job. It means, we will not run the tests until the code style is compliant; this will save some time and resources. Indeed, if the code is not formatted, don't even bother running the tests. The execution graph will be like:

We can see here some duplication, which is not ideal as for future code improvements, you will have to do it at two places at the same time. This is technical debt that one would have to tackle using composite action. We will consider it's ok for now.

Caching dependency resolution

You will see additional steps in the ci.yaml, namely related to cache

    - name: Restore uv cache
        uses: actions/cache@v4
        with:
          path: /tmp/.uv-cache
          key: uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
          restore-keys: |
            uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
            uv-${{ runner.os }}

These steps aim at caching the .venv when there are no changes on the uv.lock and reusing it. The intent is to speed up the ci execution as dependency resolution and installation can be time consuming.

An extra step to minimize caching size is added as mise proposes such feature, namely an extra step and an environment variable is added to configure the location of the cache.

      - name: Minimize uv cache
        run: uv cache prune --ci
    env:
      UV_CACHE_DIR: /tmp/.uv-cache

What's next

On the next chapter, you will implement your first spark code and implement a way to guarantee test automation of it. This is long overdue as we spent 3 chapters on setup...

You can find the original materials in spark_tdd. This repository exposes what's the expected repository layout at the end of each chapter in each branch:

[03/03/25 UPDATE]: Chapter 3 has been released
[09/03/25 UPDATE]: Chapter 4 has been released
[15/03/25 UPDATE]: Chapter 5 has been released

How to be Test Driven with Spark: Chapter 0 and 1 - Modern Python Setup

Nicoda-27 — Sat, 15 Feb 2025 09:24:09 +0000

Chapter 0: Why this tutorial

This goal of this tutorial is to provide a way to easily be test driven with spark on your local setup without using cloud resources.

Before deep diving into spark and how, we must first align on our setup environment to ease reproducibility; this will be the focus of this article.

The official documentation describes how to create tests with pyspark.

It requires to have spark server with a spark connect support for it to work as described in the documentation.

As a reminder, this is how spark connect works:

Namely, a specific server needs to be created so your tests can connect to this server and process the data as intended.

Why it is not enough?

Launching the server requires some extra requirements on your machine, namely a java virtual machine.
Launching the server requires a specific script called start-connect-server.sh which is to be found

Some data engineers might argue they can just use a spark server already deployed to be able to test; but there are several drawbacks to this approach:

You are being charged to launch simple tests or run experiments keeping cloud providers very happy
You slow down the developer feedback loop which is the time necessary to implement a feature and validates that no regression has been introduced. A developer is more confident to have no regression when tests are all executed
You create external dependencies that you have no control off. You might encounter issues with testing when the cloud provider is down, or you don't have internet access or someone changes the configuration of the server by accident.

The goal is to have a test environment that is self descriptive, quick to setup, quick to start and reliable.

Chapter 1: Setup

In this chapter, multiples tool will be introduced and setup. The intent is to have a clean python environment to reproduce the code. This is a very opinionated section, but it might be useful to challenge your existing tools with this section.

Python version management

Mise will be leveraged to handle python versions. It claims to be the The front-end to your dev env and it will be used to install specific versions of languages and tools.

It can be used for much more, and it is strongly advised to look at the documentation to understand the true power of this tool not limited to python developement.

Mise first needs to be installed, see documentation for further instructions. You can launch the following:

curl https://mise.run | sh

Once installed, you will have to customize your .bashrc or your .zhsrc (or other terminal support) to activate mise on your terminal.

echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc

Mise can now be used to install python at a specific version with the following command:

mise install python@3.12

It will download a pre-compiled version of python and make it available globally.

Let's now use it, you first need to position yourself at the root of the project and launch:

mise use python@3.12

It will create a mise.toml file with the following section

[tools]
python = "3.12"

And a .python-version with the indication

3.12

With the help of these files, mise will be able to activate when located at the root of your project. It's also a great way to document other contributors of the requirements to launch this project without relying on README that becomes easily outdated.

Python dependency management

A tool to help us add, remove and download dependencies is necessary. Uv, will be used later on as it's very fast and easy to use.

To install it, the official documentation; but in this tutorial mise will be leveraged:

mise use uv

This will both install and setup uv for the project. See how [mise.toml]((https://github.com/Nicoda-27/spark_tdd/blob/doc/chapter_1/mise.toml) has been modified with the addition of:

[tools]
python = "3.12"
uv = "latest"

Now it can be used to initialize the project, namely:

uv init

This will create a folder structure for you and a hello.py. In this project, we have customized it a bit to add a tests section a pyspark_tdd package as part of src so it looks like:

.
├── src
│   ├── hello.py
├── tests
├── .python-version
├── .mise.toml
└── pyproject.toml

Ignoring files

Every repository needs a set of files to ignore before adding them to a commit. This is done via a .gitignore file and anyone can leverage existing templates for your language of preference.

If you start a project from scratch, you will need to first setup git

git init

Github maintains ignore files template for each language. You can leverage it with:

curl -L -o .gitignore https://raw.githubusercontent.com/github/gitignore/refs/heads/main/Python.gitignore

The chosen language for gitignore is in this project the python template.

Adding formatting and linting

Python

Linters and formatters are powerful tools to enforce code writing rules among developers. It takes away the pain of having to care how the code is written at the syntax level.

Ruff will be leveraged to format our python code as it's very powerful and can be run at file saves without latency.

Ruff will be added as a project dev dependency. A dev dependency is one that the project does not need to run, it can be related to tests, experimentation, formatting etc. Everything that is not meant to be shipped to production must be retained as a dev dependency to keep your python package as self contained as possible.

We can add ruff like so:

uv add ruff --dev

This will add a dev dependency in the pyproject.toml with

[dependency-groups]
dev = [
    "ruff>=0.8.4",
]

This will also create a .venv at the current working directory. You might notice that the .venv is ignored from git which is intended. Indeed, you don't want to commit your .venv directory as it's a copy of the dependencies of your project and can be quite extensive.

It will also create an uv.lock that documents your direct dependencies version and the indirect dependencies (the dependencies of your dependencies). This mechanism allows to segregates dependencies of your project from the rest.

Your project should now look like

.
├── .venv
├── src
│   ├── hello.py
├── tests
├── .python-version
├── .gitignore
├── .mise.toml
├── pyproject.toml
└── uv.lock

Other languages

As a project is not just python files, but also configuration, pipelines, documentation etc, formatting these files too is also necessary.

Documenting how these files will be formatted is done using editorconfig.
We will use the one from the editorconfig website.

Your Integrated Development Environment (IDE)

Whichever IDE will be used, it's very important that you setup formatting at file saves to save you time and remove the pain from handling it by hand.

If you are using VSCode, you can install the ruff extension and adjust the following to your settings.json

"editor.formatOnSave": true,
"[python]": {
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
        "source.fixAll": "explicit",
        "source.organizeImports": "explicit"
    },
    "editor.defaultFormatter": "charliermarsh.ruff"
},

The first test

To see if everything works as expected, you will write a very simple unit test. In a test driven approach, the test is written before the source code.

A test framework is required to launch the test automation, pytest will be used. You need to add it as a dev dependency

uv add pytest --dev

You can create a tests/test_dummy.py with the following code:

from your_python_package.multiply import multiply


def test_my_dummy_function():
    assert multiply(1, 2) == 2

This requires a function multiply that can be defined as in src/your_python_package/multiply.py:

 def multiply(a: int, b: int) -> int:
    return a * b

You can now run the tests, make sure you're using the right python from the .venv

which python

should display something like /$HOME/somepath/your_project/.venv/bin/python. If not, you can restart a new terminal, mise should be able to resolve.

Then run

pytest

Then it will display an error:

tests/test_dummy.py:1: in <module>
    from your_python_package.multiply import multiply
E   ModuleNotFoundError: No module named 'your_python_package'

You need to add an extra entry for pytest to detect the src layout. In pyproject.toml, you can add:

[tool.pytest.ini_options]
pythonpath = ["src"]

Now

pytest

should display

============================================================= test session starts ===============================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/somepath/src/your_project
configfile: pyproject.toml
collected 1 item                                                                                                                                 

tests/test_dummy.py .                                                                                                                      [100%]

=============================================================== 1 passed in 0.01s ================================================================

You can do some housekeeping and remove the unnecessary src/your_python_package/hello.py.

You now have a proper setup to start working.

What's next

Now that one test is implemented, the continuous integration (ci) must be setup. In a collaborative way of working, the ci is the only source of truth to guarantee if everything is broken or not.

Notice we still have not touched upon any spark components, it's very important to have a clean reproducible codebase before diving.

That will be the topic of the next chapter.

You can find the original materials in spark_tdd. This repository exposes what's the expected repository layout at the end of each chapter in each branch:

[23/02/25 UPDATE]: Chapter 2 has been released
[03/03/25 UPDATE]: Chapter 3 has been released
[09/03/25 UPDATE]: Chapter 4 has been released
[15/03/25 UPDATE]: Chapter 5 has been released