DEV Community: Bryn Mailer

Blueprints, Not Prompts: Using TLA+ to Guide AI-Written Code

Bryn Mailer — Wed, 21 Oct 2026 02:29:27 +0000

Code used to be where a team's understanding of a system lived. Now that LLMs write most of it, that understanding is quietly eroding. Here's why we're exploring formal methods at Mulga, specifically TLA+, to build better systems with AI without losing our grip on what we're building, with a real AWS EBS spec as an example.

The real contribution isn't code

It is my sincere belief that shared understanding and curated mental models are the primary artifact produced by a great engineering team.

For many, the act of sitting down to write the code was the critical step where their understanding of the problem space would SLOWLY be formalized, picked apart, and rearranged. The codebase simply provided a consistent and objective way to distribute the fruits of individual reasoning efforts to the wider team. The design discussions, frantic bug hunts, and PR reviews were core activities that served to deepen a team's understanding of how their particular problem could, and should, be solved.

At the end of it all, yes, the codebase "did the thing" (ideally). Hooray. The more important outcome was that there was now a formalized solution to the original problem, and likely many adjacent issues. Such a repository of distilled technical intent enabled later generations of engineers within the same company/group/project/community to leverage the hard-won mental models of their forebears, and ultimately drive forward the collective understanding.

Where it all went wrong

Since AI-written code became "good enough" to feasibly reach production in early 2026, many of us have been aware of the slow decline of this feedback loop that sustains our industries, livelihoods, and passions. For most of us, days now consist of prompting an LLM to produce code that satisfies a particular business need. Usually via written English. I can't speak for the entire development community, but I can personally attest to this having degraded my ability to write code by hand.

One could play devil's advocate and say something along the lines of: if LLMs can write all the code now, then why does it matter if we lose the ability to do so ourselves? To which I would say: it shouldn't really matter, provided we are able to maintain the cycle of capturing individual understanding in a manner that may be easily consumed by peers.

At last, we arrive at the core of the problem. By degrading/losing the ability to write code, we become disconnected from the act of translating ideas and intent into verifiable formalized logic that is useful to others.

The problem with natural language

How many of us regularly chat to our LLMs using the most horrendous, broken English we can possibly conceive out of sheer laziness? I'm definitely guilty! Even if you are one of the few who maintain a high quality of communication in your prompts, written/spoken language is a famously imprecise method of communication (cough synonyms cough cough).

I challenge you to try to write a system specification in your language of choice that makes explicit all assumptions and ignored details, yet leaves no room for alternate interpretations of the details that are captured. It is unlikely that you will succeed to a degree that allows you to hand that specification to an LLM and get a result that satisfies your intent.

If your system is simple enough that you do achieve an initially satisfactory result, you will inevitably discover some edge case that needs addressing. At that point, you could either adjust the written specification and have AI regenerate the implementation from scratch, which would likely resolve the issue, but also leave you with an entirely different system (unless your written spec is absurdly long and detailed).

Or you could simply ask the LLM to patch the change into the existing implementation via a quick prompt. A prompt whose core ideas are never properly understood, refined, and formalized into the collective human understanding of the project. Even if you do take the time to update the specification, fixing the edge case in isolation from the rest of the system introduces drift that severely degrades (even destroys) the spec's trustworthiness.

To illustrate the hilarity of the current prevailing approach to software specification, imagine an architect giving a construction team nothing more than a written description of the building! If natural language were precise enough to facilitate proper specifications, we wouldn't need lawyers to disambiguate the legal system for us average folk.

Formal methods and TLA+

So what do we do? How do we plug the AI-shaped hole in the bottom of the bucket that used to hold the collective knowledge of our field? Honestly, I don't know. However, my desire for a solution to this conundrum has led me deep into the formal methods rabbit hole over the past year. On my journey, I've picked up a few techniques/tools that have begun to help me address my concerns with LLM-assisted development.

One such tool that has been especially beneficial is TLA+. It was originally created by Leslie Lamport, and proves once again just how far ahead of his time he was. I'm not going to dive into an explanation of TLA+ here as it's already been done far better than I ever could, by the man himself. I highly recommend taking a moment to read Lamport's high-level view of TLA+ over on his website before proceeding. For those of you who will not heed my advice (shame), TLA+ is a language based on formal mathematical notation that allows one to create precise models of anything that can be represented as a state machine (everything, with a bit of creativity). It ships alongside TLC, which can be used to exhaustively explore logical paths within your models, and find design issues.

TLA+ is NOT code; it is mathematics. Before you recoil in disgust, here's an excerpt from Lamport's website that explains why this is so:

TLA+ teaches you that math is inherently more expressive than programming languages because it can describe a value without having to describe how the value is computed. For example, it can describe the greatest common divisor (GCD) of two numbers as the largest positive integer that divides both numbers. This makes it possible to write a model for a specific purpose, abstracting away irrelevant details such as how to calculate the GCD. (Putting code in a procedure doesn't abstract it away; it just makes you go elsewhere to read the code, requiring that you understand the semantics of procedure call as well as the code.)

In the context of a world in which AI writes the code, this is enormously important. It gives us a language in which we can elegantly capture our intent, without getting bogged down in implementation details that ultimately don't matter to the use case. A well-written TLA+ spec is a human-readable artifact that forces the author to reason through the system they are designing in a constrained and logical format, is easy for peers to digest and iterate upon, and is objectively machine checkable (via TLC) so that LLMs are given a concrete benchmark that implementations may be evaluated against.

The TLA+ model is the architectural blueprint that the design team argues over, iterates upon, and hands to the next generation as learning material, just as the codebase used to be. The code itself becomes a concern of the construction team (coding agents). The details of the implementation can be tied to the spec, preventing drift, with the use of strategies such as trace-checking and refinement mapping. I liken these processes to the surveyor who measures a new build against the numbers in the plans to ensure conformance.

Many readers might be thinking something along the lines of "if TLA+ is so great, why haven't I heard of it before?". TLA+, and the discipline of formal methods as a whole, has largely flown under the radar of the wider software engineering community, with the exception of niche teams within resource-rich industry titans such as Google and AWS. As far as I can tell, this is mostly due to the barrier to entry.

Engineers are trained to write code. Most gave their discrete mathematics classes only the bare minimum attention needed to pass their exams, then promptly forgot the skills once they entered the workforce. From the perspective of an engineering executive: why would I have my team spend time writing a model in TLA+ (or another specification language), when I could have them build the real solution in [insert programming language of choice]? Someone would have to write the code eventually anyway...

At the time, such arguments were perfectly reasonable. No matter how precise your formal spec, someone still has to write the executable code. Or rather, someTHING! Enter LLMs. On February 5th of this year (2026), Anthropic released Opus 4.6. It is widely considered the turning point for AI-assisted development; models could finally write code that was at least somewhat reasonable. We live in a world where, if we can produce testable specifications that sufficiently constrain the details of a system that matter to the use case, we can give AI an "abstract" model and receive a working implementation with very little human babysitting.

This is MUCH easier said than done, but TLA+ is an incredible step in the right direction. And given that so many engineers are no longer writing code themselves, one could argue that improving the quality and accuracy of our specifications is a core part of the job, in a way that it never was before. I have a hunch that this mentality shift will go a long way towards breaking down the perceived "barrier to entry" that has so limited the usefulness of formal methods to the wider software engineering community.

A TLA+ specification of the AWS Elastic Block Store API

My attention at Mulga is largely focused on the storage components of our stack, Predastore (S3-compatible object storage) and Viperblock (EBS-like virtual disks). In such projects, details are extraordinarily important, and ambiguity is to be avoided at all costs. When a user engages with something claiming to be a storage service, they assume that their data is in good hands and won't be carelessly lost. On top of this responsibility, the systems are complex and easy to lose track of under the tidal wave of AI-generated contributions.

I recently used TLA+ to write a specification of the observable behavior of a subset of AWS's EBS API. The intent was to solidify my understanding of the EBS volume lifecycle state machine, so that I can ensure Spinifex (our AWS-compatible control plane) is properly adhering to the behaviors expected by the AWS CLI for the EBS operations it exposes.

For the curious reader, the specification is open source and is available to read in the Viperblock GitHub repository. It's only ~400 lines (as of now), across 4 files! This is rather notable, considering the relevant (English) technical documentation published by AWS for these operations extends across 13 separate web pages, and still manages to leave a vast amount of detail up to the reader to discern. The specification's brevity makes sense as, naturally, a written description of a building would take up far more space than a blueprint capturing the relevant details! Some request/response fields, possible error codes, and features have been abstracted out of this initial version, of course, but the core volume/attachment status state machines are captured very clearly. You do not need to be a career mathematician to make fantastic use of TLA+!

A common question I receive is "can't AI just write the TLA+ too?". Well, yes... technically. However, I'd argue there's not really any value in doing things that way. If an AI writes it, you're right back to relying on prompts that nobody has rigorously thought through, and that are invisible to colleagues and mentees.

TLA+ is ultimately a tool for the communication of ideas, in the same way that the legal system uses a constrained version of natural language to avoid confusion and mathematicians use formal notation. The power in a specification language built upon said mathematical notation is its innate ability to be applied effectively at any level of abstraction. If all you have is a vague idea of how the system you wish to build should appear to a user, that is enough to write a solid specification. There is plenty of room later to refine, and add detail that was missed (or deemed unimportant) on the first pass. This can extend as far down as the electrons themselves, if such detail is relevant to your problem space.

To illustrate this idea, here is the part of my EBS specification that captures the subset of the observable behavior of the CreateVolume operation that I was particularly interested in:

\* A client requested a new volume.
CreateVolume(req, res) ==
  /\ req \in CreateVolumeRequests
  /\ LET PossibleErrors(db) == Errors({})
         PossibleRecords    == [status             : {"creating"},
                                \* After enabling encryption by default, CreateVolume may still
                                \* see a stale value from another replica for a short window.
                                encrypted          : {db.encryptedByDefault : db \in VisibleDBs},
                                multiAttachEnabled : {FALSE}]
        \* The request was accepted.
     IN \/ /\ PossibleErrors(CurrentDB) = {}
           /\ \E vId \in VolumeIds \ usedIds, record \in PossibleRecords :
                /\ Commit([CurrentDB EXCEPT !.volumes = InsertKey(@, vId, record)])
                /\ res = [vId |-> vId] @@ record
        \* The request was rejected, possibly by a stale replica.
        \/ /\ \E db \in VisibleDBs : res \in PossibleErrors(db)
           /\ UNCHANGED vars

Some information to aid in reading this excerpt: /\ and \/ can be read as AND and OR respectively; curly braces represent a set; == is a definition; = is a comparison; the lines that begin with \E are checking whether an element exists in a target set that satisfies the condition after the :. The comments give enough of the gist that the rest of the syntax can be glossed over. Specifics can be found at learntla.com. It helps to think of a TLA+ spec as one big condition in an if statement that you would find in a traditional programming language.

This operator (an action in TLA+ terms) contains very little detail about how exactly a volume is created. In fact, throughout my entire spec, the concept of a "volume" being a virtual disk that bytes can be read from and written to is not captured anywhere! This is because it was not relevant to the control plane operations that I was concerned with at the time. I simply abstracted it away.

A final anecdote: while writing the EBS control plane model, I found myself unable to glean all of the information required to get a good picture of the possible volume status transitions from the AWS docs. This drove me to run some tests myself, and led to the discovery that a volume whose status is "creating" can immediately transition to "deleting" without first passing through "error" in certain cases. This is undocumented behavior that is extremely valuable to me in the design of Viperblock.

What's next?

At the time of writing, the spec only captures a few particularly important EBS API endpoints. I'm currently working on using TLC to trace-check my model against the real EBS API to verify that it is correct. The plan is to then trace-check Spinifex's EBS API implementation against the verified model, and fix the divergences that will inevitably be surfaced! From there I will extend the model to capture all EBS API operations that Spinifex supports, and repeat the trace-checking process.

This was not the article I initially set out to write. I'd intended to cover far more technical detail and dive deeper into the structure of the spec itself, the trace-checking methodology, and my ideas around how formal specification may be feasibly brought into the industry as a standard practice. However, in explaining my experiences to friends and colleagues, I realized that such content would not be very useful to the engineering community if I didn't first attempt to give the "why" behind the methods. Alas, the rest will have to wait for future articles.

If I have sufficiently piqued your interest, rest assured that you will not be waiting long for the next installment in this saga. So stay tuned. Until next time!

Bryn

Migrating from MinIO to Predastore

Bryn Mailer — Tue, 29 Sep 2026 04:58:47 +0000

If you're reading this, you're probably already aware of MinIO's fall from grace. The open source community's favorite object storage system was officially archived back in April (2026, for future readers). The impending deficit of security patches and bug fixes has since sent many sysadmins looking for a maintained alternative.

If you're one of the affected individuals, I'd like to take the opportunity to introduce you to Predastore!

My team and I have been working on this project for around a year and a half now. Long before the crescendo of the MinIO saga. The original motivation was to build an object storage system capable of operating in resource constrained edge environments with an unreliable network fabric.

Although there is still plenty of work to be done to soundly meet that goal, the project in its current state happens to fit the more standard use cases (that MinIO excelled in) rather well!

By the end of this article, you will hopefully have a clear understanding of the process of migrating from a MinIO deployment to a brand new Predastore cluster.

An architectural overview

Note: Everything in this section is subject to change. Predastore is a fairly young project, and certain design decisions have not been set in stone yet. For example, I'm not particularly sold on the "blob" terminology yet. If anyone has any better ideas, please raise a GitHub issue! It will make my day.

In order to start building a mental model of how MinIO concepts map to Predastore, a rudimentary of understanding of Predastore itself is required. This will be rather brief. If you want the meatier details, check out the design doc located here.

A Predastore cluster is organized into two levels; hosts, and nodes. A host is just a single Predastore process, invoked via s3d. Each host can contain any number of nodes, where each node has a role that is one of:

gate - Exposes the S3 HTTP API. There may be at most one gate node per host process.
meta - A Raft replica holding bucket and object records, including where every object's shards live.
blob - A key-value store that writes encrypted, erasure-coded shards to append-only segment files. The directory to which a blob node reads/writes is set using the data_dir field in the config file.

Each host gets its own IP address, and each node within a single host binds to a separate port under the host IP address. This ensures that every node is individually addressable.

A cluster must contain at least one of each type of node. Additionally, it is highly recommended that each blob node is pointed at a separate disk to ensure proper isolation.

Before you begin: what Predastore doesn't currently do

MinIO was a mature project, with many years spent adding support for all manner of features provided by S3. Predastore is rather young as of yet, and while we support the main body of features required by most deployments, there are a number of things still on the TODO list.

Before proceeding with the following steps, take a moment to skim the S3 API coverage page here to ensure Predastore isn't missing an operation your use case depends on.

If something you're after is absent, please notify my team and me by raising a GitHub issue. We're pretty responsive. Depending on the request, there's a good chance we'll be able to slip it into a minor release.

1. Pick a mode: standalone or under Spinifex

The first step in the process is to decide whether your new Predastore cluster is going to be deployed standalone, or via Spinifex. This is usually a fairly simple choice, depending on your desired authentication structure. The general capabilities of each method:

Standalone - A fixed set of service accounts, one tenant, no external dependencies. Manual distribution of keys and certificates to nodes during setup.
Via Spinifex - IAM users/groups/roles, STS, multiple tenants. Automatic distribution of keys and certificates to member nodes.

I highly recommend taking a moment to peruse the "Standalone or Spinifex?" section of Predastore's README for guidance on which path to take.

2. Stand up Predastore

The exact installation steps you should follow depend on the structure of your pre-existing MinIO cluster. Expand the section below that matches your deployment topology and chosen mode of installation.

Single-Node Single/Multi-Drive - Standalone

This is the simplest path, and the closest equivalent to a single MinIO server. You'll need a Linux machine running systemd, along with Go (1.27 or newer), git, make, and openssl in order to build Predastore from source. Most distro packages of Go are a few versions behind, so you may need to grab it from go.dev. You'll also want the AWS CLI for checking Predastore once it's up.

Build and install

First up, clone the repository and build the s3d binary:

git clone --branch v1.21.0 https://github.com/mulgadc/predastore.git
cd predastore
make build

Note: This guide was written against v1.21.0. The install tooling used below (make install, predastore-keygen, and the example config) first shipped in v1.20.0, so don't go any older than that.

Install it, along with its systemd service:

sudo make install

Then create the dedicated predastore user, and the config and data directories (/etc/predastore and /var/lib/predastore respectively):

sudo systemd-sysusers && sudo systemd-tmpfiles --create

Share MinIO's drives

There's no need for new drives. Predastore can live on the ones MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Each drive needs roughly as much free space again as MinIO is using on it, plus some headroom for re-syncs, which df -h will tell you.

The steps below assume MinIO's drives are mounted at /mnt/disk1, /mnt/disk2, and so on. If they're pooled with RAID or ZFS, treat the pool as a single drive.

For each drive, create a directory for Predastore. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o predastore -g predastore -m 0700 /mnt/disk1/.predastore

The service isn't permitted to write anywhere outside /var/lib/predastore, so bind-mount each directory into it. The extra options make sure the service waits for the mount, and refuses to start without it:

sudo install -d -m 0755 /var/lib/predastore/disk1
echo '/mnt/disk1/.predastore /var/lib/predastore/disk1 none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=predastore.service,x-systemd.before=predastore.service 0 0' | sudo tee -a /etc/fstab

Once every drive has its line in /etc/fstab, mount them:

sudo systemctl daemon-reload
sudo mount -a

Note: Leave /var/lib/predastore/disk1 and friends owned by root. If a mount ever goes missing, that's what stops Predastore from quietly writing to the system disk in its place.

Match your MinIO setup

Predastore ships with an example config that already describes a working single-drive host, so there isn't a great deal to change. Copy it into place, and open it up in your editor of choice:

sudo cp /etc/predastore/predastore.toml.example /etc/predastore/predastore.toml
sudo $EDITOR /etc/predastore/predastore.toml

For the purposes of the migration, there are only two fields you need to set. region needs to match whatever your MinIO clients are configured with, as S3 clients sign every request for a specific region. [[auth]] holds the access key and secret your applications will use.

Keep in mind that these are root credentials, with full access to every bucket in the cluster (as mentioned in the standalone vs Spinifex discussion).

region = "us-east-1"

[[auth]]
access_key_id     = "<access-key>"
secret_access_key = "<secret-key>"
account_id        = "100000000001"

The example config also includes an example-bucket entry, which you can go ahead and delete.

Next, point the blob nodes at the drives. If MinIO only had one drive, set data_dir = "/var/lib/predastore/disk1" on the example's blob node, and the config is done.

Otherwise, replace the example's single blob node with one blob node per drive. Give each one its own id and port, and point its own data_dir at the drive's bind mount.

Leave the host's data_dir as it is, since the meta node doesn't name a directory of its own and keeps its state under /var/lib/predastore on the system disk. With three drives, the [[host]] block ends up looking like this:

[[host]]
id = 1
addr = "127.0.0.1"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 1
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id = 3
  role = "blob"
  port = 9991
  data_dir = "/var/lib/predastore/disk1"

  [[host.node]]
  id = 4
  role = "blob"
  port = 9992
  data_dir = "/var/lib/predastore/disk2"

  [[host.node]]
  id = 5
  role = "blob"
  port = 9993
  data_dir = "/var/lib/predastore/disk3"

Finally, set [rs] to match. Each object is split into data shards plus parity shards, each on a different drive, so parity is the number of drives you can lose, and data + parity can't exceed the number of drives.

RS(2,1) across three drives will survive losing any one of them (for two drives, use RS(1,1)). Your data survives, but the service won't start while a drive's bind mount is missing, so mount the replacement drive at the same path, recreate its .predastore directory, and run sudo mount -a before restarting. [rs] can't be changed once objects have been written, so make sure you're happy with it before moving any data across.

The configuration reference covers the details.

[rs]
data   = 2
parity = 1

Generate keys and a certificate

Generate the key used to encrypt data at rest, along with a self-signed TLS certificate. PREDA_SAN should list every hostname and IP address your clients will use to reach Predastore, otherwise they'll refuse to trust the certificate when you come to copy data across. It replaces the default list rather than adding to it, so include DNS:localhost and IP:127.0.0.1 if you'll also be connecting from the machine itself.

sudo PREDA_SAN="DNS:s3.example.com,IP:192.0.2.10" predastore-keygen /etc/predastore

Note: Make sure to back up /etc/predastore/master.key somewhere separate from your data backups. Without it your data is unreadable, but storing it alongside the data backups defeats the purpose of encrypting at all.

Start the service

Enable and start the service, then follow its logs to make sure it comes up without any errors:

sudo systemctl daemon-reload
sudo systemctl enable --now predastore
sudo journalctl -u predastore -f

Note: A few warnings while the nodes start up are normal, such as the gate reporting it isn't ready yet. It's errors, and the service restarting, that you're looking out for.

Check it's up

With the service running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region.

The certificate is only readable by the predastore user, so copy it somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/predastore/server.pem ~/predastore-ca.pem

Point the AWS CLI at the credentials from your [[auth]] entry, and at the certificate:

export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-key>
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

If all three commands complete without any errors, your Predastore cluster is ready for data. If you see AuthorizationHeaderMalformed ... incorrect region, the region your client is signing for doesn't match the one Predastore was set up with.

Multi-Node Multi-Drive - Standalone

This path is the equivalent of a distributed MinIO deployment. Every machine runs the same config file, and shares a single TLS certificate and encryption key.

The steps below assume three machines at 10.0.0.1, 10.0.0.2, and 10.0.0.3, but the process is the same for any number of them. You'll need Go (1.27 or newer), git, make, and openssl on each machine, as well as the AWS CLI on at least one of them for checking the cluster once it's up.

Build and install

On every machine, clone the repository and build the s3d binary:

git clone --branch v1.21.0 https://github.com/mulgadc/predastore.git
cd predastore
make build

Note: This guide was written against v1.21.0. The install tooling used below (make install, predastore-keygen, and the example config) first shipped in v1.20.0, so don't go any older than that.

Install it, along with its systemd service:

sudo make install

Create the dedicated predastore user, and the config and data directories:

sudo systemd-sysusers && sudo systemd-tmpfiles --create

make install also ships some kernel network settings. Hosts on separate machines talk to each other over QUIC, and need larger socket buffers than most distros provide by default. Without them, writes will start failing under load, with no obvious indication as to why. Apply them:

sudo sysctl --system

Share MinIO's drives

The steps below assume each machine's MinIO drives are mounted at /mnt/disk1, /mnt/disk2, and so on. If they're pooled with RAID or ZFS, treat the pool as a single drive. The pool covers a failed drive, and Predastore covers a failed machine.

On every machine, create a directory for Predastore on each drive. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o predastore -g predastore -m 0700 /mnt/disk1/.predastore

sudo install -d -m 0755 /var/lib/predastore/disk1
echo '/mnt/disk1/.predastore /var/lib/predastore/disk1 none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=predastore.service,x-systemd.before=predastore.service 0 0' | sudo tee -a /etc/fstab

Once every drive has its line in /etc/fstab, mount them:

sudo systemctl daemon-reload
sudo mount -a

Note: Leave /var/lib/predastore/disk1 and friends owned by root. If a mount ever goes missing, that's what stops Predastore from quietly writing to the system disk in its place.

Describe the cluster

A single config file describes the entire cluster, with one [[host]] block per machine. Each host gets its own address, its data directory, key, and certificate paths, and its own gate, meta, and blob nodes. Node ids must be unique across the whole file, not just within a host.

region needs to match whatever your MinIO clients are configured with, as S3 clients sign every request for a specific region. [[auth]] holds the access key and secret your applications will use.

Keep in mind that these are root credentials, with full access to every bucket in the cluster (as mentioned in the standalone vs Spinifex discussion).

[rs] determines how objects are spread across the machines. Each object is split into data shards plus parity shards, each on a different blob node. RS(2,1) across three machines will survive losing any one of them.

Check out the configuration reference for the full format.

version = 1
region  = "us-east-1"

[rs]
data   = 2
parity = 1

[[host]]
id   = 1
addr = "10.0.0.1"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 1
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id = 3
  role = "blob"
  port = 9991

[[host]]
id   = 2
addr = "10.0.0.2"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 4
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 5
  role = "meta"
  port = 6660

  [[host.node]]
  id = 6
  role = "blob"
  port = 9991

[[host]]
id   = 3
addr = "10.0.0.3"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 7
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 8
  role = "meta"
  port = 6660

  [[host.node]]
  id = 9
  role = "blob"
  port = 9991

[[auth]]
access_key_id     = "<access-key>"
secret_access_key = "<secret-key>"
account_id        = "100000000001"

Save it as /etc/predastore/predastore.toml on every machine.

Since every machine shares the same config file, each one needs telling which [[host]] block describes it. That lives in /etc/predastore/predastore.env. On each machine, copy the example into place, and open it up:

sudo cp /etc/predastore/predastore.env.example /etc/predastore/predastore.env
sudo $EDITOR /etc/predastore/predastore.env

Set PREDA_HOST_ID to the id of the [[host]] block that describes this machine, so 1 on 10.0.0.1, 2 on 10.0.0.2, and so on. The rest of the file can be left as it is.

PREDA_HOST_ID=1

Next, point the blob nodes at the drives. If each machine only has one drive, add data_dir = "/var/lib/predastore/disk1" to every host's blob node, and the config is done.

Otherwise, give each host one blob node per drive, each with its own id, port, and data_dir pointing at the drive's bind mount. Leave each host's data_dir as it is, since the meta node keeps its state there.

With two drives per machine, host 1 looks like this, and hosts 2 and 3 follow the same pattern with node ids 5 to 8 and 9 to 12:

[[host]]
id   = 1
addr = "10.0.0.1"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 1
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id = 3
  role = "blob"
  port = 9991
  data_dir = "/var/lib/predastore/disk1"

  [[host.node]]
  id = 4
  role = "blob"
  port = 9992
  data_dir = "/var/lib/predastore/disk2"

Finally, set [rs] to match, which needs some care. Predastore spreads an object's shards across distinct blob nodes, but doesn't take into account which machine each blob node lives on, so two shards of the same object can land on two drives in the same machine.

To survive losing a whole machine, parity needs to be at least the number of drives per machine, and data + parity can't exceed the total number of drives. For three machines with two drives each, RS(2,2) survives losing any one machine. [rs] can't be changed once objects have been written, so make sure you're happy with it before moving any data across.

[rs]
data   = 2
parity = 2

Generate keys and a certificate

Every machine in the cluster needs to share the same TLS certificate and the same encryption key. This one is a bit of a trap; machines with their own certificates never manage to form a cluster.

So rather than running predastore-keygen on each machine straight away, generate the files once on a single machine and distribute them yourself.

On one machine, generate a self-signed certificate. Make sure subjectAltName lists every machine's address, as well as the hostname your clients will use:

sudo openssl req -x509 -newkey rsa:2048 -nodes -days 825 -subj '/CN=predastore' \
  -addext "subjectAltName=DNS:s3.example.com,IP:10.0.0.1,IP:10.0.0.2,IP:10.0.0.3" \
  -keyout /etc/predastore/server.key -out /etc/predastore/server.pem

On the same machine, generate the encryption key:

sudo sh -c 'umask 0177 && openssl rand -out /etc/predastore/master.key 32'

Copy server.pem, server.key and master.key into /etc/predastore on every other machine, using something that preserves file permissions, such as scp -p. Predastore refuses to start if either key is readable by anyone other than its owner, so if in doubt, tighten them up on each machine once they've landed:

sudo chmod 0600 /etc/predastore/master.key /etc/predastore/server.key

Then, on every machine, run predastore-keygen. It keeps the copied files, and sets their ownership:

sudo predastore-keygen /etc/predastore

Still on every machine, add the certificate to the system trust store, so each host trusts the others:

sudo cp /etc/predastore/server.pem /usr/local/share/ca-certificates/predastore.crt
sudo update-ca-certificates

Note: The trust store commands above are for Debian and Ubuntu. For RHEL and Fedora, see the Standalone TLS Trust section of the README. As with the single node setup, make sure to back up master.key somewhere separate from your data backups.

Start the service

On every machine, enable and start the service, then follow its logs:

sudo systemctl daemon-reload
sudo systemctl enable --now predastore
sudo journalctl -u predastore -f

Until every machine is up, the ones that started first will log errors about not being able to reach the others, which is expected. Once the service is running on every machine, the hosts will find each other and elect a leader, and those errors should stop. From then on, it's errors and the service restarting that you're looking out for.

Check it's up

With the cluster running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region. Run the following from any one of the machines.

s3.example.com should point at every machine's gate, not just one, either with a DNS record per machine or a load balancer in front of them. Clients only talk to the gates that name resolves to, so if it names a single machine, losing that machine takes Predastore offline for your applications, even though the rest of the cluster carries on.

sudo install -m 0644 -o "$USER" /etc/predastore/server.pem ~/predastore-ca.pem

Point the AWS CLI at the credentials from your [[auth]] entry, and at the certificate:

export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-key>
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

Single-Node Single/Multi-Drive - Via Spinifex

A caveat before you start: Spinifex currently keeps all of Predastore's data on a single drive. While MinIO is still running, that means picking one of MinIO's drives to hold everything, which limits Predastore to that drive's capacity, with no protection against losing it. If MinIO's drives are already pooled with RAID or ZFS, the pool counts as one drive and keeps its redundancy. Support for spreading Predastore across multiple drives under Spinifex is on its way. Until then, if you need to survive a failed drive and can live without IAM, the standalone path is the better fit.

If you've gone with Spinifex, Predastore comes along for the ride. It's installed and configured as part of Spinifex, so there's no need to set it up separately.

On a single node, Spinifex runs Predastore with one blob node and RS(1,0), which means a single copy of each object with no parity.

Run the installer

Before installing, make sure the machine has a Linux bridge configured for VM networking. The bridge setup guide walks through setting one up.

Run the installer:

curl -fsSL https://install.mulgadc.com | bash

Then set up OVN networking. --management also starts OVN's central services, which a single node needs to run itself:

sudo /usr/local/share/spinifex/setup-ovn.sh --management

Initialize the node

Initializing the node generates Predastore's config, keys, and certificates for you. Just like the standalone setup, --region needs to match your MinIO clients, as Spinifex defaults to ap-southeast-2.

sudo spx admin init --node node1 --nodes 1 --region us-east-1

Share one of MinIO's drives

Predastore can live on one of the drives MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Pick the drive with the most free space, as it needs enough room for a full copy of your data, plus some headroom for re-syncs.

The steps below assume that drive is mounted at /mnt/disk1. Create a directory for Predastore on it. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o spinifex-storage -g spinifex -m 0700 /mnt/disk1/.spinifex-predastore

Predastore's service isn't permitted to write anywhere outside /var/lib/spinifex/predastore, so bind-mount the directory over it. The extra options make sure the service waits for the mount, and won't write to the system disk without it:

sudo chown root:root /var/lib/spinifex/predastore
sudo chmod 0755 /var/lib/spinifex/predastore
echo '/mnt/disk1/.spinifex-predastore /var/lib/spinifex/predastore none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=spinifex-predastore.service,x-systemd.before=spinifex-predastore.service 0 0' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount -a

Start the services

Start Spinifex's services, Predastore included:

sudo systemctl start spinifex.target

Spinifex's certificate only covers the machine's own hostname and addresses. If your clients will reach Predastore by any other name, such as s3.example.com, add it to the certificate, and restart the services to pick it up:

sudo spx admin cert renew --extra-dns s3.example.com
sudo systemctl restart spinifex.target

Check it's up

With Spinifex running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region.

Copy Spinifex's CA certificate somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/spinifex/ca.pem ~/predastore-ca.pem

Point the AWS CLI at the spinifex profile created during setup, which takes care of the credentials, and at your copy of the certificate:

export AWS_PROFILE=spinifex
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

Multi-Node Multi-Drive - Via Spinifex

A caveat before you start: Spinifex currently keeps all of each machine's Predastore data on a single drive. While MinIO is still running, that means picking one of MinIO's drives on each machine, which limits each machine's share of Predastore to that drive's capacity. Predastore's erasure coding still covers a failed machine, and it treats a failed drive as a failed machine, but there's no protection against losing a drive beyond that. If MinIO's drives are already pooled with RAID or ZFS, the pool counts as one drive and keeps its redundancy. Support for spreading Predastore across multiple drives under Spinifex is on its way. Until then, if you need to survive more failed drives than failed machines and can live without IAM, the standalone path is the better fit.

Setting up a multi-node Spinifex cluster involves a fair bit more than Predastore alone, including configuring OVN networking on each server.

Rather than repeating all of it here, I'll point you to Spinifex's multi-node install guide for the full procedure, and only call out the parts that matter for the migration.

Spinifex runs one Predastore blob node per machine, so its erasure coding protects you against losing a machine.

Follow the install guide

Work through steps 1 to 3 of the multi-node guide, which cover installing Spinifex, setting the node IP variables, and configuring OVN networking on each server.

Form the cluster

Step 4 of the guide forms the cluster, with one change; --region needs to match your MinIO clients, on both the init and every join. Spinifex picks the erasure coding for you, using RS(1,1) for two machines and RS(2,1) for three or more.

On server 1, initialize the cluster:

sudo spx admin init --force --node node1 --nodes 3 \
  --bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
  --port 4432 --region us-east-1 --az us-east-1a

While init is still running, join each other server, using the token from init's output. Change --node and the bind addresses to match each server:

sudo spx admin join --force --node node2 \
  --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region us-east-1 --az us-east-1a

Share one of MinIO's drives

On each machine, Predastore can live on one of the drives MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Pick the drive with the most free space, as it needs roughly as much room as MinIO is using across all of that machine's drives combined, plus some headroom for re-syncs.

The steps below assume that drive is mounted at /mnt/disk1. On every server, create a directory for Predastore on it. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o spinifex-storage -g spinifex -m 0700 /mnt/disk1/.spinifex-predastore

Predastore's service isn't permitted to write anywhere outside /var/lib/spinifex/predastore, so on every server, bind-mount the directory over it. The extra options make sure the service waits for the mount, and won't write to the system disk without it:

sudo chown root:root /var/lib/spinifex/predastore
sudo chmod 0755 /var/lib/spinifex/predastore
echo '/mnt/disk1/.spinifex-predastore /var/lib/spinifex/predastore none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=spinifex-predastore.service,x-systemd.before=spinifex-predastore.service 0 0' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount -a

Start the services

Finish off with steps 5 and 6 of the multi-node guide to start the services and verify the cluster.

As with a single node, Spinifex's certificates only cover each server's own hostname and addresses. If your clients will reach Predastore by any other name, such as s3.example.com, add it on every server, and restart the services to pick it up:

sudo spx admin cert renew --extra-dns s3.example.com
sudo systemctl restart spinifex.target

Check it's up

With the cluster running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region. Run the following on server 1.

Copy Spinifex's CA certificate somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/spinifex/ca.pem ~/predastore-ca.pem

Point the AWS CLI at the spinifex profile created during setup, which takes care of the credentials, and at your copy of the certificate:

export AWS_PROFILE=spinifex
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

3. Recreate identities and access

With Predastore up and running, the next step is to give your applications a way in. How you go about this depends on the mode you picked back in step 1.

Standalone

Standalone Predastore has no concept of users or policies. Access is controlled entirely by the [[auth]] entries in predastore.toml, and every one of them is a root credential. So the job here is fairly simple.

Add an [[auth]] entry for each access key your applications currently use against MinIO. You're free to reuse the same key and secret values, which means your applications won't need new credentials when you switch them over.

On a multi-node cluster, make the same change to the config on every machine.

[[auth]]
access_key_id     = "<existing-minio-access-key>"
secret_access_key = "<existing-minio-secret-key>"
account_id        = "100000000001"

Then restart the service (on each machine, for a multi-node cluster) to pick it up:

sudo systemctl restart predastore

Note: If you relied on MinIO policies to restrict what each application could access, there's no standalone equivalent. That's what Spinifex is for.

Via Spinifex

Spinifex implements AWS-compatible IAM, so MinIO's users, groups, and policies map across fairly directly. All of the commands below use the standard AWS CLI, pointed at Spinifex via the profile created during setup.

For more detail on any of them, check out Spinifex's IAM guide.

export AWS_PROFILE=spinifex

Export your MinIO policies

MinIO policies are written in the same JSON format as AWS IAM policies, so in most cases they can be carried over as-is.

List the policies on your MinIO deployment:

mc admin policy list myminio

Then export each of your custom policies to a file:

mc admin policy info myminio <policy-name> --policy-file <policy-name>.json

Before importing them, there are a few things worth checking for, as Spinifex only supports a subset of the IAM policy language:

MinIO-specific actions such as admin:* and kms:* mean nothing to Predastore, and should be removed. Spinifex doesn't check action names, so it will accept them without complaint. MinIO's admin:* statements usually have no Resource, though, which Spinifex does reject, with Resource is required.
NotAction and NotResource aren't supported yet. Use Action and Resource with an explicit list instead. A statement with only a NotAction or NotResource is rejected with Action is required or Resource is required.
Condition keys are limited to aws:SourceIp, aws:SecureTransport, aws:username, aws:userid, aws:PrincipalAccount, aws:PrincipalType, and s3:prefix. The only supported operators are StringEquals, StringLike, IpAddress, and Bool, so conditions using the likes of StringNotEquals will need rewriting.
Policy variables are limited to ${aws:username}, ${aws:userid}, and ${aws:PrincipalAccount}. MinIO's ${jwt:*} and ${ldap:*} variables won't carry over.

Create the policies

Create each policy from its exported file:

aws iam create-policy --policy-name <policy-name> --policy-document file://<policy-name>.json

The good news is that Spinifex loudly refuses a policy that uses an unsupported element, condition, or variable. So if one of those slips through, create-policy will fail with a MalformedPolicyDocument error telling you what needs changing. Unrecognised actions are the exception, as mentioned above.

Recreate your users and groups

Each MinIO user and group gets an IAM equivalent, with the same policies attached. You'll need your account ID for the policy ARNs, which this will tell you:

aws sts get-caller-identity

Create each user, and attach their policies:

aws iam create-user --user-name alice
aws iam attach-user-policy --user-name alice \
  --policy-arn arn:aws:iam::<account-id>:policy/<policy-name>

Create each group, attach its policies, and add its members:

aws iam create-group --group-name developers
aws iam attach-group-policy --group-name developers \
  --policy-arn arn:aws:iam::<account-id>:policy/<policy-name>
aws iam add-user-to-group --group-name developers --user-name alice

Issue new access keys

MinIO won't hand over existing secret keys, and Spinifex generates its own, so each user will need a fresh key pair. The secret is only shown once, so make sure to save it somewhere safe.

aws iam create-access-key --user-name alice

Note: Buckets belong to the account that creates them. When you come to copy data across in the next step, use a key belonging to the account you want to own the buckets.

4. Move the data

Now for the main event. To copy your data from MinIO to Predastore, I recommend using rclone.

You might be tempted to reach for mc mirror, and it does work, but with MinIO archived, mc is no longer maintained and its official downloads have been taken down. rclone on the other hand is actively maintained, packaged for pretty much every distro, and lets you set the region explicitly on both ends.

Install rclone

Install it on a machine that can reach both MinIO and Predastore:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Configure both ends

rclone needs a remote for each side. Add the following to ~/.config/rclone/rclone.conf, filling in your own endpoints and keys. force_path_style is required, as Predastore only supports path-style requests, and region needs to match the one you set up Predastore with.

If you went with Spinifex, use a key belonging to the account you want to own the buckets. The spinifex profile's own key, in ~/.aws/credentials, belongs to the account created during setup.

[minio]
type = s3
provider = Minio
endpoint = https://minio.example.com:9000
access_key_id = <minio-access-key>
secret_access_key = <minio-secret-key>
region = us-east-1

[predastore]
type = s3
provider = Other
endpoint = https://s3.example.com:8443
access_key_id = <predastore-access-key>
secret_access_key = <predastore-secret-key>
region = us-east-1
force_path_style = true

Confirm both remotes work by listing their buckets:

rclone lsd minio:
rclone lsd predastore: --ca-cert ~/predastore-ca.pem

Copy each bucket

Copy a bucket by creating it on the Predastore side, then filling it with rclone sync. rclone only creates a bucket when it has something to put in it, so without the mkdir an empty bucket would quietly get left behind.

--checksum compares objects by their hash rather than their modification time, and --metadata carries across user metadata (x-amz-meta-*) and Content-Type.

rclone's --ca-cert flag applies to both remotes at once, not just Predastore. If your MinIO is served over HTTPS, passing Predastore's certificate on its own will stop rclone from trusting MinIO, so pass the system's CA bundle alongside it (on RHEL and Fedora, that's /etc/pki/tls/certs/ca-bundle.crt):

rclone mkdir predastore:<bucket> --ca-cert ~/predastore-ca.pem
rclone sync minio:<bucket> predastore:<bucket> \
  --checksum --metadata --progress \
  --ca-cert /etc/ssl/certs/ca-certificates.crt --ca-cert ~/predastore-ca.pem

Note: rclone sync makes the destination match the source, which means deleting anything in the destination that isn't in the source. Always name the bucket on both sides, and never run it against the root of the remotes, as that will clear out anything in Predastore that MinIO doesn't know about. If in doubt, add --dry-run first to see what it would do.

To copy every bucket in one go:

for bucket in $(rclone lsf --dirs-only minio: | tr -d /); do
  rclone mkdir "predastore:$bucket" --ca-cert ~/predastore-ca.pem
  rclone sync "minio:$bucket" "predastore:$bucket" \
    --checksum --metadata --progress \
    --ca-cert /etc/ssl/certs/ca-certificates.crt --ca-cert ~/predastore-ca.pem
done

Check the copy

Check each bucket with rclone check. --download reads every object from both sides and compares the actual contents. It's slow for large buckets, but it's the most reliable way to be sure.

rclone check minio:<bucket> predastore:<bucket> --download \
  --ca-cert /etc/ssl/certs/ca-certificates.crt --ca-cert ~/predastore-ca.pem

A successful check reports 0 differences found.

5. Switch clients over and check the result

With the data across, all that's left is to point your applications at Predastore. It's worth doing this in a sensible order, so that nothing written to MinIO in the meantime gets left behind.

Stop writes to MinIO, and run a final sync

Stop your applications from writing to MinIO (or put them into maintenance mode), then run the copy from step 4 one last time to pick up anything that changed since.

Compare the two sides

Before switching anything over, check that both sides agree on the number of objects and the total size of each bucket:

rclone size minio:<bucket>
rclone size predastore:<bucket> --ca-cert ~/predastore-ca.pem

Update your clients

For most clients, switching over comes down to a handful of settings:

Endpoint - https://<predastore-host>:8443. Predastore only speaks HTTPS, so any http:// endpoints will need updating along with the host and port.
Region - Whatever you set during setup in step 2. A mismatch shows up as an AuthorizationHeaderMalformed error on every bucket and object request.
Credentials - Unchanged if you reused your MinIO keys on a standalone deployment. Otherwise, the new keys from step 3.
Addressing style - Predastore only supports path-style requests (https://host:8443/bucket/key), not virtual-hosted ones (https://bucket.host:8443/key). Most SDKs have an option for this, such as addressing_style in boto3, forcePathStyle in the JavaScript SDK, and UsePathStyle in the Go SDK.
Certificate trust - If you're using a self-signed certificate, clients need to trust it. That's the ~/predastore-ca.pem copy made when you checked the cluster was up in step 2.

For example, an AWS CLI profile for Predastore in ~/.aws/config looks something like this:

[profile predastore]
region = us-east-1
endpoint_url = https://s3.example.com:8443
ca_bundle = /home/<you>/predastore-ca.pem
aws_access_key_id = <access-key>
aws_secret_access_key = <secret-key>
s3 =
  addressing_style = path

Note: Any presigned URLs generated before the switch point at MinIO, and won't carry over. Your applications will need to generate new ones against Predastore.

Keep MinIO around for a little while

Once your applications are happily talking to Predastore, I'd recommend leaving MinIO running in a read-only state for a while before decommissioning it. That way, if anything turns out to be missing, the original is still there to fall back on.

Retire MinIO

Once you're confident nothing is missing, stop MinIO for good, and delete its data to hand the space over to Predastore. Predastore carries on running throughout, and there's nothing to restart. Under Spinifex, only the drive holding Predastore's data gains anything for now, and the rest are free for other uses.

If MinIO was pointed at a directory on each drive, such as /mnt/disk1/minio, delete that directory on every drive:

sudo rm -r /mnt/disk1/minio

If MinIO was pointed at the drive itself, delete its .minio.sys directory, along with a directory for each of your buckets, naming them explicitly:

cd /mnt/disk1 && sudo rm -r .minio.sys <bucket> <bucket> ...

Note: Resist the temptation to reach for rm -r /mnt/disk1/*. It misses .minio.sys, and if Predastore's directory is ever missing its leading ., it'll take your freshly migrated data with it.

Closing thoughts

Hurray! Hopefully that went smoothly for you. If it didn't, raise a GitHub issue so we can help you out.

We've got plenty of features and improvements planned for Predastore, so be sure to keep your eyes on the release page.

I've got other articles in the works as well, focusing more on the design rationale behind the system. Stay tuned for those. That's all for this one!