DEV Community

Michael Oteng
Michael Oteng

Posted on

Decommissioning a Laravel SaaS Without Losing the Ability to Restore It

#laravel #devops #mysql #disasterrecovery

Recently, I decommissioned the infrastructure behind Scholent, a multi-tenant Laravel SaaS application I built.

Deleting the infrastructure itself was easy.

The more important question was:

If I needed Scholent again six months or two years from now, could I actually recover it?

I didn't want the answer to be:

I have a backup somewhere.

I wanted evidence that the critical recovery material was intact and, particularly for the database, actually restorable.

So I treated this less as a server deletion exercise and more as a small disaster-recovery exercise.

The process eventually looked like this:

Inventory
   ↓
Archive
   ↓
Encrypt
   ↓
Transfer off-server
   ↓
Verify integrity
   ↓
Restore independently
   ↓
Inspect recovered data
   ↓
Document reconstruction
   ↓
Decommission infrastructure
Enter fullscreen mode Exit fullscreen mode

The distinction between having a backup and having tested recovery material became the most important part of the exercise.

Note: Commands in this article have been sanitised where necessary. Credentials, encryption secrets, application keys and other sensitive infrastructure information are deliberately excluded or represented using placeholders.

What was being decommissioned?

Scholent wasn't simply a Laravel directory sitting on a VPS.

The production environment included:

  • Ubuntu 22.04
  • Laravel 12
  • PHP 8.3
  • MySQL 8
  • Nginx
  • Redis
  • Supervisor
  • Let's Encrypt
  • Stancl-based multi-tenancy
  • DigitalOcean compute
  • DigitalOcean Managed MySQL

The deployed Laravel application lived under:

/var/www/school-platform
Enter fullscreen mode Exit fullscreen mode

There were also existing infrastructure snapshots.

This matters because preserving a Git repository alone would not necessarily preserve enough information to reconstruct the application.

A working Laravel SaaS can depend on considerably more:

Application source
        +
Composer dependencies
        +
Database
        +
Environment configuration
        +
APP_KEY / encryption material
        +
Web server configuration
        +
Queue workers
        +
Redis
        +
Scheduled jobs
        +
DNS
        +
TLS
        +
Tenant configuration
        +
External services
Enter fullscreen mode Exit fullscreen mode

So the first job was not creating a .zip file.

It was determining what constituted the system.

1. Inventory the environment before touching anything

Before removing infrastructure, I documented the environment and the components required to reconstruct it.

For a PHP application, even something as basic as recording versions matters.

Typical checks include:

php -v
Enter fullscreen mode Exit fullscreen mode 
mysql --version
Enter fullscreen mode Exit fullscreen mode

and for Laravel:

php artisan --version
Enter fullscreen mode Exit fullscreen mode

Along with the obvious questions:

Which PHP version?
Which MySQL version?
Which Laravel version?
Where is Redis?
How are queues processed?
How is Nginx configured?
How is TLS handled?
How does tenancy work?
Which scheduled jobs exist?
Which external services does the application depend on?
Enter fullscreen mode Exit fullscreen mode

This information is easy to obtain while the server exists.

It becomes considerably less convenient after the server has been deleted.

2. Create independent recovery artefacts

I separated the recovery material into broad categories rather than treating the existing server as the archive.

That included:

Application
Database
Server configuration
Recovery documentation
Enter fullscreen mode Exit fullscreen mode

The application and relevant configuration were archived independently.

A simplified example of the approach is:

tar -czf application.tar.gz /var/www/school-platform
Enter fullscreen mode Exit fullscreen mode

with relevant server configuration archived separately.

The actual application archive was approximately 16 MB, while the server-configuration archive was approximately 121 KB.

Those numbers aren't particularly important by themselves.

What mattered was that the recovery set contained the material I expected it to contain.

The database was handled separately because I wanted to test it independently later.

3. Encrypt sensitive recovery material

Application archives and database exports can contain information that should not simply be copied into general storage unprotected.

Depending on the application, this may include:

  • environment configuration
  • database connection information
  • API credentials
  • application encryption material
  • user data
  • internal infrastructure information

I encrypted the sensitive archives using AES-256-CBC with PBKDF2-based key derivation and a deliberately high iteration count.

The general form was:

openssl enc \
  -aes-256-cbc \
  -salt \
  -pbkdf2 \
  -iter <iteration-count> \
  -in <archive> \
  -out <archive>.enc
Enter fullscreen mode Exit fullscreen mode

The real password and sensitive parameters obviously do not belong in a public article.

The same applies to Laravel's APP_KEY.

For some Laravel applications, losing the original APP_KEY can mean losing the ability to decrypt data that was encrypted using it.

So preserving recovery secrets matters.

But preserving them inside an unprotected archive rather defeats the point.

4. Get the recovery material off the infrastructure

This is an obvious rule, but an important one:

A backup that exists only on the infrastructure you are about to destroy is not an acceptable long-term backup.

Once the archives were prepared, I copied them away from the DigitalOcean infrastructure.

I retained independent copies outside the server environment, including local storage and Google Drive.

At this point, however, I still only knew that I had files.

I didn't yet know that those files would recover anything.

5. Verify the transferred files

After transferring the archives, I calculated SHA-256 hashes.

On Linux, that can be done with:

sha256sum <archive>
Enter fullscreen mode Exit fullscreen mode

The same archive can then be hashed again after transfer.

Conceptually:

Server archive SHA-256
        ↓
        == ?
        ↑
Downloaded archive SHA-256
Enter fullscreen mode Exit fullscreen mode

The hashes matched.

That established that the transferred files were byte-for-byte identical to the files produced on the server.

Useful?

Absolutely.

Proof that the application could be recovered?

No.

A checksum answers:

Has this file changed?

It doesn't answer:

Is this file actually useful?

That required a restore test.

6. Test the database backup independently

This was the most important technical test in the process.

I didn't want to verify the database backup by importing it into the original production database.

That would still leave too much dependency on the existing environment.

Instead, I created a disposable MySQL environment using Docker.

The idea was straightforward:

Production database
        │
        │ export
        ▼
Encrypted database archive
        │
        │ copy elsewhere
        ▼
Independent machine
        │
        │ decrypt / decompress
        ▼
Disposable MySQL container
        │
        │ import
        ▼
Inspect recovered database
Enter fullscreen mode Exit fullscreen mode

If the archive failed here, I still had the live infrastructure available to investigate and create another backup.

That's exactly when I wanted failure to occur.

Before deletion.

7. Start a clean MySQL container

I used Docker to provide an isolated MySQL 8 environment.

A simplified version of the command is:

docker run \
  --name scholent-restore-test \
  -e MYSQL_ROOT_PASSWORD='<temporary-test-password>' \
  -e MYSQL_DATABASE=scholent_restore \
  -d \
  mysql:8.0
Enter fullscreen mode Exit fullscreen mode

The restore environment was running:

MySQL 8.0.46
Enter fullscreen mode Exit fullscreen mode

Using a disposable container had several advantages.

It didn't require installing another permanent MySQL instance.

It kept the restore isolated from anything important.

And when the verification was finished, the entire test environment could simply be removed.

Before restoring anything, the container could be checked with:

docker ps
Enter fullscreen mode Exit fullscreen mode

and the MySQL instance interrogated independently.

The important point was that this database server had no dependency on the original Scholent database infrastructure.

8. Restore the backup, not the production database

The encrypted database archive then had to survive the reverse process.

Conceptually:

Encrypted archive
      ↓
Decrypt
      ↓
Decompress
      ↓
MySQL import
Enter fullscreen mode Exit fullscreen mode

The exact recovery pipeline depends on how the original database archive was packaged.

The important thing was that I was feeding the archived database into the disposable MySQL environment rather than querying the original production database.

A conventional import into the container would look like:

docker exec -i scholent-restore-test \
  mysql \
  -uroot \
  -p'<temporary-test-password>' \
  scholent_restore < scholent.sql
Enter fullscreen mode Exit fullscreen mode

In my case, the archived database was decrypted and decompressed as part of the recovery process before being imported.

That exercised several parts of the recovery chain at once:

Can I access the encrypted archive?
            ↓
Can it be decrypted?
            ↓
Can the database payload be recovered?
            ↓
Will MySQL accept it?
            ↓
Does the restored database contain what I expect?
Enter fullscreen mode Exit fullscreen mode

The import completed successfully.

But I deliberately didn't stop there.

9. "Import succeeded" isn't enough either

A database import returning without an obvious error is encouraging.

It isn't the same as checking the restored database.

So I queried the restored environment.

For example:

docker exec -it scholent-restore-test \
  mysql -uroot -p
Enter fullscreen mode Exit fullscreen mode

Then:

USE scholent_restore;
SHOW TABLES;
Enter fullscreen mode Exit fullscreen mode

The restored Scholent database contained:

82 tables
Enter fullscreen mode Exit fullscreen mode

That was the first useful structural verification.

The schema had actually been reconstructed inside the independent MySQL environment.

But I wanted to go one level further.

10. Check representative application data

Rather than attempting to manually inspect all 82 tables, I selected representative areas of the application.

Queries equivalent to the following were used:

SELECT COUNT(*) FROM tenants;
SELECT COUNT(*) FROM students;
SELECT COUNT(*) FROM users;
SELECT COUNT(*) FROM assessments;
SELECT COUNT(*) FROM behaviour_records;
Enter fullscreen mode Exit fullscreen mode

The restored database returned representative counts including:

tenants               4
students              69
users                 29
assessments           11
behaviour_records      4
Enter fullscreen mode Exit fullscreen mode

This was much more useful than simply knowing that mysqldump had produced a file.

At this point I knew that:

✓ The archive existed
✓ The transferred archive matched the original
✓ The protected database material could be recovered
✓ MySQL could import it independently
✓ The database schema had been reconstructed
✓ 82 tables were present
✓ Representative application data was present
Enter fullscreen mode Exit fullscreen mode

That's a substantially stronger recovery position.

11. What exactly had I proved?

It's important not to overstate the result.

I had not rebuilt the complete Scholent production environment from zero and run a full end-to-end application acceptance test.

That would have been a different exercise.

What I had independently tested was the critical database recovery path.

The distinction matters.

I had evidence for:

Archive
   ↓
Transfer
   ↓
Integrity verification
   ↓
Decryption
   ↓
Database recovery
   ↓
Independent MySQL import
   ↓
Schema verification
   ↓
Representative data verification
Enter fullscreen mode Exit fullscreen mode

The reconstruction of the broader Laravel stack was documented in the recovery runbook rather than fully executed during this decommissioning exercise.

I think that distinction is important whenever we talk about disaster recovery.

"Backup tested" can mean many different things.

It is better to state precisely what was actually tested.

12. Create the recovery runbook

Once I knew the critical data could be recovered, I documented how the wider system would be reconstructed.

The runbook covered the major dependencies around:

Ubuntu
PHP
Laravel
MySQL
Nginx
Redis
Supervisor
SSL
DNS
Tenant routing
Application configuration
Encryption material
Post-restore verification
Enter fullscreen mode Exit fullscreen mode

The purpose wasn't to create documentation that could rebuild everything automatically.

It was to reduce ambiguity.

Future me shouldn't have to ask:

How did I configure the workers again?

or:

Was Redis actually required?

or:

How were tenant domains routed?

or, considerably worse:

Where is the encryption key for this data?

A good recovery archive should contain both the material and enough knowledge to make sense of that material.

13. Snapshots are useful, but I didn't want them to be the archive

The DigitalOcean environment also had existing snapshots.

It would have been easy to leave them there and consider the system archived.

I chose not to make that the long-term recovery strategy.

Snapshots are useful operational tools, but for long-term archival I preferred independent recovery artefacts.

That gave me:

Application archive
Database archive
Configuration archive
Recovery documentation
Enter fullscreen mode Exit fullscreen mode

rather than:

"Hopefully that old provider snapshot still gives me everything."
Enter fullscreen mode Exit fullscreen mode

The independent approach is also more portable.

If I eventually rebuild Scholent somewhere else, the recovery strategy isn't inherently tied to the original compute instance.

14. Only then did I destroy the infrastructure

At this point, I had moved through several different verification stages.

Application and configuration 

✓ Archived
✓ Encrypted where required
✓ Copied off-server
✓ Integrity checked
Enter fullscreen mode Exit fullscreen mode

Database 

✓ Exported
✓ Protected
✓ Copied independently
✓ Decrypted successfully
✓ Restored into disposable MySQL
✓ 82 tables recovered
✓ Representative records verified
Enter fullscreen mode Exit fullscreen mode

Recovery knowledge 

✓ Infrastructure documented
✓ Reconstruction procedure documented
✓ Critical configuration requirements identified
Enter fullscreen mode Exit fullscreen mode

Only after those checks did I proceed with removing the paid DigitalOcean infrastructure.

That included the relevant:

Droplet
Managed MySQL instance
Infrastructure snapshots
Enter fullscreen mode Exit fullscreen mode

Obsolete DNS records were also removed.

The domain and email did not need to disappear simply because the compute infrastructure was being retired, so they were retained.

That's another distinction I think is useful:

Decommissioning infrastructure doesn't necessarily mean destroying the application identity or intellectual property associated with it.

My objective was to stop paying for infrastructure I wasn't currently using without turning that cost-saving decision into irreversible data loss.

15. Clean up the disposable recovery environment too

One advantage of using Docker for this kind of verification is that the recovery environment itself doesn't need to become another permanent piece of infrastructure.

Once verification was complete, the test container could be removed:

docker stop scholent-restore-test
docker rm scholent-restore-test
Enter fullscreen mode Exit fullscreen mode

If a dedicated Docker volume had been created for the exercise, that could also be removed once it was no longer required.

The restore environment had served its purpose:

prove that the archived database could reconstruct usable data independently of production.

Then it could disappear too.

The different levels of backup confidence

This exercise made me think about backup verification as a progression.

Level 1: The backup command completed 

I probably have a backup.
Enter fullscreen mode Exit fullscreen mode

Level 2: The file exists 

I definitely have a file.
Enter fullscreen mode Exit fullscreen mode

Level 3: The checksum matches after transfer 

I know the file I downloaded is the same file I created.
Enter fullscreen mode Exit fullscreen mode

Level 4: The encrypted archive can be recovered 

I can still access the protected contents.
Enter fullscreen mode Exit fullscreen mode

Level 5: The database imports successfully elsewhere 

The backup can reconstruct a database independently.
Enter fullscreen mode Exit fullscreen mode

Level 6: Schema and representative data are verified 

The reconstructed database resembles the application data I intended to preserve.
Enter fullscreen mode Exit fullscreen mode

Level 7: Full application recovery is tested 

The complete service can be reconstructed and validated.
Enter fullscreen mode Exit fullscreen mode

For this decommissioning exercise, I independently tested through the database-recovery and data-verification stages.

I documented the remaining application reconstruction requirements rather than claiming a complete end-to-end disaster-recovery test that I hadn't performed.

A practical Laravel SaaS decommissioning checklist

If I were retiring another Laravel application tomorrow, my high-level checklist would look something like this.

Understand what exists 

[ ] Application/version inventory
[ ] PHP version
[ ] Database/version
[ ] Redis/cache dependencies
[ ] Queue configuration
[ ] Scheduled jobs
[ ] Web server configuration
[ ] SSL configuration
[ ] DNS requirements
[ ] External services
[ ] Tenant-specific architecture
Enter fullscreen mode Exit fullscreen mode

Preserve it 

[ ] Application/source
[ ] Database export
[ ] Relevant environment configuration
[ ] APP_KEY and other required secrets
[ ] Nginx/server configuration
[ ] Queue/worker configuration
[ ] Relevant storage
[ ] Recovery documentation
Enter fullscreen mode Exit fullscreen mode

Protect it 

[ ] Encrypt sensitive recovery material
[ ] Store recovery credentials separately
[ ] Copy archives away from production
Enter fullscreen mode Exit fullscreen mode

Verify it 

[ ] Generate checksums
[ ] Verify transferred copies
[ ] Decrypt the archive
[ ] Restore the database independently
[ ] Inspect schema
[ ] Query representative records
Enter fullscreen mode Exit fullscreen mode

Document it 

[ ] Write recovery sequence
[ ] Record software versions
[ ] Record infrastructure dependencies
[ ] Document DNS/TLS requirements
[ ] Document tenancy
[ ] Record post-restoration verification
Enter fullscreen mode Exit fullscreen mode

Only then 

[ ] Remove compute
[ ] Remove managed databases
[ ] Remove unnecessary snapshots
[ ] Remove obsolete DNS
[ ] Verify billing/resources
[ ] Retain domains/services that still have a purpose
Enter fullscreen mode Exit fullscreen mode

What I'd change next time

The biggest improvement I'd make is to treat the recovery runbook as a living part of the application rather than something created primarily when the application is being retired.

The same applies to restore testing.

Backups are easy to automate.

Restore tests are easier to postpone.

But the restore is the part that actually tells you whether the backup strategy works.

I'd also consider automating more of the verification process for a system that remained operational long-term.

For example, a controlled process could periodically:

Create backup
      ↓
Verify checksum
      ↓
Create disposable database
      ↓
Restore
      ↓
Run verification queries
      ↓
Report result
      ↓
Destroy test environment
Enter fullscreen mode Exit fullscreen mode

Docker makes that kind of disposable recovery testing considerably easier.

Final thought

Deleting the Scholent infrastructure took very little time.

Getting to the point where I was comfortable deleting it took considerably longer.

And that's how I think it should be.

The process ultimately became:

Inventory
   ↓
Archive
   ↓
Encrypt
   ↓
Transfer
   ↓
Hash
   ↓
Restore
   ↓
Inspect
   ↓
Document
   ↓
Decommission
Enter fullscreen mode Exit fullscreen mode

The most important lesson wasn't really about DigitalOcean, Docker, MySQL or Laravel.

It was this:

Don't test whether you can create a backup. Test whether you can restore from it.

A backup file tells me that I saved something.

Seeing 82 tables and the expected application data inside a clean MySQL container tells me considerably more.

Top comments (1)

Collapse
 
saqrelfirgany profile image
Ahmed ElFirgany •

"Having a backup" versus "having tested recovery material" is the whole point, and most teams only learn it during an incident.

One more thing that tends to break two years later: composer install itself. A package gets abandoned, a private repo token expires, or a PHP extension changes. Archiving vendor/ next to composer.lock, or a built image, removes that risk from the restore.

Did your independent restore run on the same PHP version as production, or a newer one?