DEV Community: dodger

Docker Wouldn't Give Me My Build Artifacts. So I Stole Them from OverlayFS.

dodger — Fri, 16 Oct 2026 08:11:43 +0000

How I bypassed Docker's build abstraction to recover intermediate artifacts and rescue a CI/CD pipeline.

The problem

A few years ago, I was tasked with migrating an entire Atlassian Bamboo CI/CD infrastructure. And I don't mean just upgrading a few packages: the existing Bamboo version had been out of support for several years, the underlying hardware platform was being decommissioned, and to make things even more interesting, the Bamboo agents were running Red Hat Enterprise Linux 6.

Yes. RHEL 6.

The objective was to migrate the infrastructure to a modern, supported environment while keeping the existing build pipelines operational.

And one of those pipelines belonged to a bu$ine$$-critical internal application.

The build process involved multiple stages, including Docker builds, C++ compilation, and static code analysis.

Somewhere during the migration, we discovered that certain files generated inside Docker's intermediate build stages were needed by subsequent pipeline steps, but were not available in the final image.

To make things more entertaining, the Dockerfile wasn't entirely under my control.

I tried the conventional approaches. I even experimented with modifying parts of the build process from the pipeline itself.

Nothing provided a satisfactory solution in that environment.

And the requirement was simple:

Make it work.

Docker multi-stage builds: now you see it, now you don't

Multi-stage builds are wonderful for producing smaller, cleaner runtime images.

You compile your application in one stage, copy what you need into another stage, and discard the build environment.

Conceptually:

FROM build-environment AS builder

WORKDIR /workspace
COPY . .

RUN ./build.sh

FROM runtime-environment

COPY --from=builder \
    /workspace/bin/application \
    /app/application

Perfect.

Unless you need something else from the builder stage, and your existing build process doesn't provide an accessible way to retrieve it.

In my case, part of the challenge involved C++ build outputs and information required by static analysis.

One particularly interesting file was compile_commands.json, a compilation database describing how source files were compiled.

The problem wasn't that Docker couldn't support artifact extraction in general.

The problem was that I couldn't get the specific existing pipeline to give me what I needed.

I even tried to modify the application's build process myself, despite it being completely outside my area of responsibility.

When I brought the idea up with the application's development lead, he looked at me as if I had completely lost my mind.

Apparently, suggesting changes to their build system was a crazier idea than digging through Docker's internal filesystem.

Well, challenge accepted.

Knowing what's under the hood

I've been working with Linux long enough to have developed a healthy distrust of abstractions.

Containers aren't virtual machines.

Underneath the packaging, namespaces, cgroups and runtime machinery, we're still dealing with Linux processes and filesystems.

At the time, the Docker agents used the overlay2 storage driver.

Which meant that Docker's internal filesystem data lived under:

/var/lib/docker/overlay2/

And here's where things got interesting.

Just because a file wasn't available in the final Docker image didn't necessarily mean its data had already disappeared from the host.

Docker's storage could still contain files from intermediate build layers.

So I had a thought:

Why ask Docker for the files when I could ask Linux?

The first hack: finding artifacts directly

I introduced a configurable variable into the CI/CD pipeline:

DOCKER_EXTRACT_FILES="*.so"

The pattern above is illustrative, not the actual proprietary filename or value used in the pipeline.

The actual extraction mechanism was wonderfully uncomplicated.

A simplified, anonymized version of the original approach:

OVERLAY_PATH="/var/lib/docker/overlay2"
OUTPUT_DIR="${WORKSPACE}/out"

mkdir -p "${OUTPUT_DIR}"

sudo find "${OVERLAY_PATH}" \
    -type f \
    -mmin -5 \
    -name "${DOCKER_EXTRACT_FILES}" \
    -exec cp -v {} "${OUTPUT_DIR}/" \;

Yes.

That's a find.

Running against Docker's internal storage.

From a production CI/CD pipeline.

The -mmin -5 filter was intentional. I wanted recently modified files, not arbitrary historical leftovers from previous builds.

The idea was to run the extraction immediately after docker build, while the relevant filesystem data was still present.

No Docker API.

No additional container.

No registry round trip.

Just Linux filesystem access.

And it worked.

The second hack: recovering an entire build workspace

The next challenge was more interesting.

For C++ static analysis, I needed build information that had been generated inside Docker.

The solution started by modifying CMake configuration before the Docker build:

find "${APP_PATH}" \
    -type f \
    -name "CMakeLists.txt" \
    -exec sed -i \
    '1 i set(CMAKE_EXPORT_COMPILE_COMMANDS ON)' \
    {} \;

This enabled generation of compile_commands.json in the relevant CMake builds.

Once Docker finished building the image, I searched its storage for that file:

COMPILE_COMMANDS_FILE="$(
    sudo find /var/lib/docker/overlay2 \
        -type f \
        -mmin -4 \
        -name "compile_commands.json"
)"

That gave me more than a filename.

It gave me a way to locate the build workspace.

In that particular environment, I knew the workspace directory layout.

By using the path of compile_commands.json, I could identify the corresponding workspace location within Docker's storage and synchronize its contents back into Bamboo's working directory.

Conceptually:

sudo rsync -a \
    "${RECOVERED_WORKSPACE}/" \
    "${BAMBOO_WORKSPACE}/"

Now Bamboo had access to build information that had previously been trapped inside Docker's intermediate filesystem data.

I had effectively turned Docker's internal storage into an artifact extraction interface.

An undocumented, implementation-dependent, absolutely unsupported interface.

But an interface nonetheless.

Integrating the hack into the pipeline

The beautiful part was that I didn't need a separate manual recovery procedure.

The extraction became part of the ordinary build sequence:

Prepare source files
       |
       v
Patch CMake configuration
       |
       v
docker build
       |
       v
Recover C++ build workspace
       |
       v
Extract selected build files
       |
       v
Continue the normal CI/CD pipeline
       |
       v
Publish artifacts

The scripts ran on the Bamboo agent itself, with the permissions required to inspect the Docker storage.

Everything was automated.

No engineer manually copying files.

No SSH session to rescue a broken build.

The pipeline simply executed the recovery steps and continued.

Did it actually work?

Yes.

Not just once.

Not just during a proof of concept.

I tested the approach repeatedly, integrated it into the pipeline, and deployed it into production.

And it worked.

Of course, there were caveats:

Direct dependency on Docker's internal storage implementation.
Reliance on the overlay2 storage driver.
No guarantees about intermediate layer persistence.
Potential race conditions during extraction.
Possible collisions between concurrent builds.
Risk of retrieving stale files from previous builds.
Dependency on file modification timestamps.
Elevated filesystem permissions required.
Potential interference from Docker garbage collection and cleanup operations.
No official Docker API or support for this approach.
No portability across different Docker storage drivers or build backends.
Potential breakage after Docker upgrades.

Did I care?

No.

Did it work?

Every. Single. Time.

This wasn't a general-purpose artifact extraction framework.

It was a deliberately narrow solution for a known build environment, tested extensively and deployed into production.

The requirement wasn't to make Docker happy. The requirement was to make the pipeline work.

And it did.

The real lesson

There's a reason I still believe that understanding Linux internals matters, even in an era of containers, orchestration, infrastructure as code and increasingly sophisticated developer platforms.

Abstractions are useful.

They're supposed to make our lives easier.

But when an abstraction fails to provide what you need, understanding the layer underneath can make the difference between being stuck and finding a solution.

Docker saw build layers.

Bamboo saw a pipeline.

I saw files on a Linux filesystem.

And sometimes, that's all you need.

The requirement wasn't to produce an elegant solution. The requirement was to make the pipeline work.

So I did.

The real lesson

There's a reason I believe that understanding Linux internals matters THE MOST, even in an era of containers, orchestration, infrastructure as code, and increasingly sophisticated developer platforms.

Abstractions are useful.

They're supposed to make our lives easier.

But they also hide what's really happening underneath.

And that's where things get interesting.

What some companies call "expected behavior", the rest of the world might call "unexpected behavior".

And what most people see as unexpected behavior, some of us see as a whole new world of possibilities.

Just as shown in this article. ;-)

Docker saw build layers.
Bamboo saw a pipeline.
I saw files on a Linux filesystem.

And sometimes, that's all you need.

The requirement wasn't to produce an elegant solution. The requirement was to make the pipeline work.

So I did.

Open-Source Oracle Database Performance History with PostgreSQL and Grafana

dodger — Wed, 09 Sep 2026 10:47:06 +0000

Oracle databases expose a huge amount of performance information through dynamic performance views. The difficult part is not obtaining a metric at this exact moment. It is retaining enough connected evidence to investigate what happened after the workload has already changed.

That is the problem I built Harry Performance Scraper for Oracle Database to solve.

Harry is an open-source performance data pipeline. It periodically collects SQL statistics, session activity, waits, blocking information, execution plans and system metrics from Oracle Database, stores that history in PostgreSQL, and makes it available through Grafana or plain SQL.

Grafana is the interface. The PostgreSQL repository is the evidence.

Performance monitoring needs more than current metrics

A conventional monitoring stack is excellent at answering operational questions:

Is the database available?
How much CPU is it using?
Is a tablespace filling up?
How many sessions are connected?
Has a counter crossed an alerting threshold?

These are important questions, and tools such as Prometheus are extremely good at answering them.

Performance investigations require a different kind of information. If database activity increased twenty minutes ago, I want to know:

which sessions were active;
which SQL statements they were executing;
which events they were waiting for;
which modules and programs created the workload;
whether sessions were blocking one another;
which execution plans were involved;
and how those facts changed during the same time window.

A graph showing that active sessions increased is a metric. The identities of those sessions, their SQL, waits, plans and relationships are the evidence required to explain the graph.

Once the sessions disconnect or begin doing something else, much of that evidence is gone unless something collected it.

The architecture

Harry separates the collection, storage and presentation layers:

The collectors run scheduled queries against Oracle dynamic performance views. Different types of information are collected at different intervals because not every fact changes at the same rate or has the same cost.

The default schedule currently includes:

Data	Default interval
Database activity history	2 seconds
SQL statistics, sessions and blocking	15 seconds
Operational metrics	1 minute
SQL text and execution-plan discovery	2 minutes

Every interval is configurable per collector. A small development database and a busy production estate do not need to use the same sampling policy.

Harry writes the collected information into a relational PostgreSQL schema. High-frequency samples are kept separate from SQL text, execution plans and other slowly changing data. Time-based data can be partitioned and retention is configurable.

This matters because repeatedly copying the full SQL text or execution plan into every sample would waste storage and make the repository harder to query. Harry instead stores different kinds of evidence according to their own lifecycle and connects them through relational keys.

Why PostgreSQL instead of Prometheus?

I use Prometheus extensively. It is an excellent choice for numeric time-series data, operational monitoring and alerting. Harry does not exist because Prometheus is bad at those jobs.

It exists because an Oracle performance incident is not only a collection of numeric time series.

The information involved includes high-cardinality identifiers and relationships:

database and instance;
session ID and serial number;
SQL ID and plan hash value;
module, action and program;
wait class and wait event;
blocking and blocked sessions;
SQL text;
execution-plan operations.

Some of these values make reasonable dimensions for a metric. Treating all of them as labels, however, can create a large number of series. SQL text and execution plans are not natural metric labels at all.

The questions asked during an investigation are relational as well:

Show the SQL executed by the sessions contributing to this wait class.
Compare the plans used by the same SQL during two different periods.
Find the blockers behind the sessions visible in this activity spike.
Group activity by module, program, event, SQL or session without deciding every future grouping when the data is ingested.

PostgreSQL gives Harry a natural representation for that information. It also means the collected data is not locked inside Harry: DBAs can query it directly, join it with their own information, build new Grafana panels or export it using standard tools.

Prometheus can continue monitoring the infrastructure and handling conventional metrics. Harry adds the detailed, relational history needed for Oracle performance analysis.

Harry's Database Activity History

One of Harry's main collectors samples active database work every two seconds and builds Database Activity History (DAH).

DAH can be explored by:

wait class;
wait event;
SQL ID;
session;
module;
program;
instance;
and other collected dimensions.

The corresponding Grafana dashboard presents the samples as an activity timeline with supporting SQL, session, wait, module and program breakdowns. Selecting a time range makes it possible to move from an overall workload spike to the SQL, sessions or waits that formed it.

This is not a summary generated after the incident. It is the sequence of samples retained while the activity was occurring.

Harry can then correlate that window with SQL statistics and session history instead of stopping at “database activity was high.”

From activity to the SQL responsible

The following demo workload shows how the Top Consumers dashboard connects a selected SQL statement with its workload, I/O, plan history, full text and execution plan.

From there, an investigation can continue into:

executions and elapsed time;
CPU and wait contribution;
buffer activity and reads;
SQL text;
plan hash values;
execution-plan contents;
plan changes across time.

Grafana is useful here because the time range and variables provide a fast way to navigate the repository. It is not mandatory. The same information remains accessible as relational data through PostgreSQL.

That distinction is deliberate: dashboards should accelerate an investigation, not define the limits of what can be investigated.

ASH, AWR and Oracle licensing

Harry's Database Activity History is not Oracle Active Session History.

By default, Harry builds its own sampled history from non-ASH dynamic performance views. It does not present external sampling as a reimplementation of every semantic or capability provided by Oracle ASH or AWR.

Optional ASH collection can be enabled in environments that are appropriately licensed and configured to use it. Because Oracle feature licensing depends on the environment and contract, operators must verify their own licensing position before enabling optional collectors that query licensed sources.

Keeping this boundary explicit is important. The objective is to provide useful and transparent performance history, not to hide where the information originated.

Operating the collector

Performance history only has value if the collector is available when the incident occurs. Harry therefore supports multiple collector instances with leader election and automatic failover.

The PostgreSQL repository supports partitioned storage and configurable retention, allowing deployments to choose how much history to preserve. Grafana can also create native alerts directly from Harry's PostgreSQL data.

The components remain independent:

Oracle Database is the performance source.
Harry performs scheduled collection.
PostgreSQL retains and relates the evidence.
Grafana provides dashboards and alerting.
Standard SQL provides unrestricted direct access to the repository.

This makes the deployment self-hosted and avoids a proprietary performance-data backend.

Try Harry

Harry Performance Scraper for Oracle Database is open source and under active development.

Project and documentation: harryperformance.com
Live Grafana demo: demo.harryperformance.com
Source code: GitHub repository
Published Grafana dashboard: Harry – Database Activity History for Oracle Database (DAH)

I am particularly interested in feedback from DBAs operating real Oracle workloads:

Which piece of evidence do you most often discover was missing only after a performance incident had already ended?