How I bypassed Docker's build abstraction to recover intermediate artifacts and rescue a CI/CD pipeline.
The problem
A few years ago, I was tasked with migrating an entire Atlassian Bamboo CI/CD infrastructure. And I don't mean just upgrading a few packages: the existing Bamboo version had been out of support for several years, the underlying hardware platform was being decommissioned, and to make things even more interesting, the Bamboo agents were running Red Hat Enterprise Linux 6.
Yes. RHEL 6.
The objective was to migrate the infrastructure to a modern, supported environment while keeping the existing build pipelines operational.
And one of those pipelines belonged to a bu$ine$$-critical internal application.
The build process involved multiple stages, including Docker builds, C++ compilation, and static code analysis.
Somewhere during the migration, we discovered that certain files generated inside Docker's intermediate build stages were needed by subsequent pipeline steps, but were not available in the final image.
To make things more entertaining, the Dockerfile wasn't entirely under my control.
I tried the conventional approaches. I even experimented with modifying parts of the build process from the pipeline itself.
Nothing provided a satisfactory solution in that environment.
And the requirement was simple:
Make it work.
Docker multi-stage builds: now you see it, now you don't
Multi-stage builds are wonderful for producing smaller, cleaner runtime images.
You compile your application in one stage, copy what you need into another stage, and discard the build environment.
Conceptually:
FROM build-environment AS builder
WORKDIR /workspace
COPY . .
RUN ./build.sh
FROM runtime-environment
COPY --from=builder \
/workspace/bin/application \
/app/application
Perfect.
Unless you need something else from the builder stage, and your existing build process doesn't provide an accessible way to retrieve it.
In my case, part of the challenge involved C++ build outputs and information required by static analysis.
One particularly interesting file was compile_commands.json, a compilation database describing how source files were compiled.
The problem wasn't that Docker couldn't support artifact extraction in general.
The problem was that I couldn't get the specific existing pipeline to give me what I needed.
I even tried to modify the application's build process myself, despite it being completely outside my area of responsibility.
When I brought the idea up with the application's development lead, he looked at me as if I had completely lost my mind.
Apparently, suggesting changes to their build system was a crazier idea than digging through Docker's internal filesystem.
Well, challenge accepted.
Knowing what's under the hood
I've been working with Linux long enough to have developed a healthy distrust of abstractions.
Containers aren't virtual machines.
Underneath the packaging, namespaces, cgroups and runtime machinery, we're still dealing with Linux processes and filesystems.
At the time, the Docker agents used the overlay2 storage driver.
Which meant that Docker's internal filesystem data lived under:
/var/lib/docker/overlay2/
And here's where things got interesting.
Just because a file wasn't available in the final Docker image didn't necessarily mean its data had already disappeared from the host.
Docker's storage could still contain files from intermediate build layers.
So I had a thought:
Why ask Docker for the files when I could ask Linux?
The first hack: finding artifacts directly
I introduced a configurable variable into the CI/CD pipeline:
DOCKER_EXTRACT_FILES="*.so"
The pattern above is illustrative, not the actual proprietary filename or value used in the pipeline.
The actual extraction mechanism was wonderfully uncomplicated.
A simplified, anonymized version of the original approach:
OVERLAY_PATH="/var/lib/docker/overlay2"
OUTPUT_DIR="${WORKSPACE}/out"
mkdir -p "${OUTPUT_DIR}"
sudo find "${OVERLAY_PATH}" \
-type f \
-mmin -5 \
-name "${DOCKER_EXTRACT_FILES}" \
-exec cp -v {} "${OUTPUT_DIR}/" \;
Yes.
That's a find.
Running against Docker's internal storage.
From a production CI/CD pipeline.
The -mmin -5 filter was intentional. I wanted recently modified files, not arbitrary historical leftovers from previous builds.
The idea was to run the extraction immediately after docker build, while the relevant filesystem data was still present.
No Docker API.
No additional container.
No registry round trip.
Just Linux filesystem access.
And it worked.
The second hack: recovering an entire build workspace
The next challenge was more interesting.
For C++ static analysis, I needed build information that had been generated inside Docker.
The solution started by modifying CMake configuration before the Docker build:
find "${APP_PATH}" \
-type f \
-name "CMakeLists.txt" \
-exec sed -i \
'1 i set(CMAKE_EXPORT_COMPILE_COMMANDS ON)' \
{} \;
This enabled generation of compile_commands.json in the relevant CMake builds.
Once Docker finished building the image, I searched its storage for that file:
COMPILE_COMMANDS_FILE="$(
sudo find /var/lib/docker/overlay2 \
-type f \
-mmin -4 \
-name "compile_commands.json"
)"
That gave me more than a filename.
It gave me a way to locate the build workspace.
In that particular environment, I knew the workspace directory layout.
By using the path of compile_commands.json, I could identify the corresponding workspace location within Docker's storage and synchronize its contents back into Bamboo's working directory.
Conceptually:
sudo rsync -a \
"${RECOVERED_WORKSPACE}/" \
"${BAMBOO_WORKSPACE}/"
Now Bamboo had access to build information that had previously been trapped inside Docker's intermediate filesystem data.
I had effectively turned Docker's internal storage into an artifact extraction interface.
An undocumented, implementation-dependent, absolutely unsupported interface.
But an interface nonetheless.
Integrating the hack into the pipeline
The beautiful part was that I didn't need a separate manual recovery procedure.
The extraction became part of the ordinary build sequence:
Prepare source files
|
v
Patch CMake configuration
|
v
docker build
|
v
Recover C++ build workspace
|
v
Extract selected build files
|
v
Continue the normal CI/CD pipeline
|
v
Publish artifacts
The scripts ran on the Bamboo agent itself, with the permissions required to inspect the Docker storage.
Everything was automated.
No engineer manually copying files.
No SSH session to rescue a broken build.
The pipeline simply executed the recovery steps and continued.
Did it actually work?
Yes.
Not just once.
Not just during a proof of concept.
I tested the approach repeatedly, integrated it into the pipeline, and deployed it into production.
And it worked.
Of course, there were caveats:
- Direct dependency on Docker's internal storage implementation.
- Reliance on the
overlay2storage driver. - No guarantees about intermediate layer persistence.
- Potential race conditions during extraction.
- Possible collisions between concurrent builds.
- Risk of retrieving stale files from previous builds.
- Dependency on file modification timestamps.
- Elevated filesystem permissions required.
- Potential interference from Docker garbage collection and cleanup operations.
- No official Docker API or support for this approach.
- No portability across different Docker storage drivers or build backends.
- Potential breakage after Docker upgrades.
Did I care?
No.
Did it work?
Every. Single. Time.
This wasn't a general-purpose artifact extraction framework.
It was a deliberately narrow solution for a known build environment, tested extensively and deployed into production.
The requirement wasn't to make Docker happy. The requirement was to make the pipeline work.
And it did.
The real lesson
There's a reason I still believe that understanding Linux internals matters, even in an era of containers, orchestration, infrastructure as code and increasingly sophisticated developer platforms.
Abstractions are useful.
They're supposed to make our lives easier.
But when an abstraction fails to provide what you need, understanding the layer underneath can make the difference between being stuck and finding a solution.
Docker saw build layers.
Bamboo saw a pipeline.
I saw files on a Linux filesystem.
And sometimes, that's all you need.
The requirement wasn't to produce an elegant solution. The requirement was to make the pipeline work.
So I did.
The real lesson
There's a reason I believe that understanding Linux internals matters THE MOST, even in an era of containers, orchestration, infrastructure as code, and increasingly sophisticated developer platforms.
Abstractions are useful.
They're supposed to make our lives easier.
But they also hide what's really happening underneath.
And that's where things get interesting.
What some companies call "expected behavior", the rest of the world might call "unexpected behavior".
And what most people see as unexpected behavior, some of us see as a whole new world of possibilities.
Just as shown in this article. ;-)
- Docker saw build layers.
- Bamboo saw a pipeline.
- I saw files on a Linux filesystem.
And sometimes, that's all you need.
The requirement wasn't to produce an elegant solution. The requirement was to make the pipeline work.
So I did.
Top comments (1)
can’t touch the Dockerfile, a pragmatic hack that ships beats an elegant one that doesn’t.
If anyone has to maintain this, a few ways to keep the same idea with fewer failure modes:
Skip the find -mmin guessing. docker inspect on the image or container exposes the exact layer paths under GraphDriver.Data (UpperDir, MergedDir), so you copy from a known directory instead of whatever changed in the last five minutes. That also removes the stale-file and concurrent-build races.
Keep the intermediate containers instead of digging for them. With the classic builder, docker build --rm=false leaves the intermediate containers, and docker cp :/path extracts files with no overlay2 access. It’s still unusual, but it uses the Docker CLI.
If the stages are named, build only the builder stage. docker build --target builder -t tmp ., then docker create plus docker cp. That needs no Dockerfile change, only a stage name.
Shared agents add a leak risk. A sudo find over all of overlay2 can match other projects’ builds, so artifacts from one pipeline could land in another’s workspace. A per-build label or unique tag avoids that.
Was the Dockerfile’s stage structure something you could rely on (named stages), or was --target ruled out?