DEV Community

Cover image for We Crashed Lioran S3 5 Times While Writing 100 GiB. Here Are the Raw Logs.
Swaraj Puppalwar
Swaraj Puppalwar

Posted on

We Crashed Lioran S3 5 Times While Writing 100 GiB. Here Are the Raw Logs.

#rust #liorans3 #lioranbastion #lds

We Crashed Lioran S3 5 Times While Writing 100 GiB. Here Are the Raw Logs.

Building storage infrastructure is easy when everything works.

The interesting part starts when you kill the server in the middle of a write.

So before launching Lioran S3 / Lioran Bastion V1 Pre-Alpha, I wanted something more useful than a screenshot of a benchmark terminal.

I wanted the raw evidence.

On September 30, 2026, I ran a 100 GiB storage workload and chaos test against Lioran S3.

The test:

  • wrote more than 100 GiB of objects,
  • distributed them across 15 buckets,
  • used 32 concurrent writers,
  • deliberately terminated the server 5 times,
  • restarted the same storage engine against the same dataset,
  • continued writing,
  • read the complete dataset back,
  • and finally performed SHA-256 integrity verification.

The complete logs from that run are publicly available.

No cropped benchmark screenshots.

No manually rewritten numbers.

The raw logs, manifests, metrics, crash events, recovery events, and integrity verification are linked at the bottom of this article.

What is Lioran S3?

Lioran S3, also known internally as Lioran Bastion, is an object-storage engine written primarily in Rust.

It is being developed by Lioran Developer Solutions (LDS) under Lioran Group.

I'm Swaraj Puppalwar, Founder & CTO of Lioran Group.

Lioran S3 V1 Pre-Alpha launched on October 1, 2026.

The goal of the project is not to produce another wrapper around an existing object-storage service.

The storage engine itself handles physical object placement, streaming I/O, metadata, integrity checks, multipart uploads, range reads, durability modes, media workloads, access control, and recovery behavior.

This test was designed to attack one of the most important questions:

What happens when the storage server disappears while data is actively being written?

Test configuration

The benchmark configuration recorded in the test artifact was:

Target dataset:       100 GiB
Buckets:              15
Write concurrency:    32
Read concurrency:     32
Scheduled crashes:    5
Chaos mode:           enabled
Upload timeout:       120 seconds
Write target:         500 MB/s
Read target:          500 MB/s
Enter fullscreen mode Exit fullscreen mode

The workload used the same source asset repeatedly with unique object keys.

The source object was:

Size:     17,839,845 bytes
SHA-256:  d6617a009c0c6c9aebf7398d43cad6d1985ddc1b9ab0479e2ea977362b8af5b0
Enter fullscreen mode Exit fullscreen mode

The final benchmark ran for approximately:

1631.7 seconds
≈ 27 minutes
Enter fullscreen mode Exit fullscreen mode

from workload start through the final integrity phase.

Phase 1: Write 100 GiB

The write phase eventually stored:

100.52 GiB
107,931,062,250 bytes
6,050 successful objects
Enter fullscreen mode Exit fullscreen mode

with:

Write concurrency:        32
Write duration:            1236.1 seconds
Average throughput:        87.3 MB/s
Healthy throughput:        88.5 MB/s
5-second peak throughput:  663.2 MB/s
Enter fullscreen mode Exit fullscreen mode

Latency distribution:

p50:   4316.5 ms
p90:  12823.8 ms
p95:  13386.9 ms
p99:  15016.0 ms
Enter fullscreen mode Exit fullscreen mode

The important number here is not the 663 MB/s peak.

A short peak is not sustained throughput.

The sustained average for the complete write phase was approximately 87.3 MB/s.

That distinction matters when publishing storage benchmarks.

Then we started killing the server

Chaos mode scheduled five abrupt server terminations at different points in the write workload.

The crash log records them at approximately:

Crash 1: 10.00 GB acknowledged progress
Crash 2: 27.02 GB acknowledged progress
Crash 3: 45.01 GB acknowledged progress
Crash 4: 68.00 GB acknowledged progress
Crash 5: 85.00 GB acknowledged progress
Enter fullscreen mode Exit fullscreen mode

These were not graceful shutdowns.

The harness terminated the exact Bastion server process.

For example:

[CHAOS] Triggering scheduled server crash 1/5 at 10.00 GB acknowledged progress
[CHAOS] Executing abrupt server termination on exact PID 28980...
Enter fullscreen mode Exit fullscreen mode

The same process was repeated five times during the workload.

Crash #1

The first server termination happened at approximately:

10.00 GB
Enter fullscreen mode Exit fullscreen mode

After termination, the harness restarted Bastion against the existing data directory.

The recovery log recorded:

Server restored and healthy after 163.2ms
Total downtime: 2372ms
Enter fullscreen mode Exit fullscreen mode

The SDK transport was then rebuilt and authenticated again.

serverGen=1
clientGen=1
rebuilds=1
authenticated=YES
Enter fullscreen mode Exit fullscreen mode

The workload continued.

Crash #2

The second termination happened at:

27.02 GB
Enter fullscreen mode Exit fullscreen mode

The storage engine restarted against the same dataset.

Recorded server recovery:

155.7ms
Enter fullscreen mode Exit fullscreen mode

with total recorded downtime of approximately:

2339ms
Enter fullscreen mode Exit fullscreen mode

The client was rebuilt again and the workers were released.

Crash #3

The third crash happened at:

45.01 GB
Enter fullscreen mode Exit fullscreen mode

This recovery was slower.

The server health recovery measurement was:

3413.8ms
Enter fullscreen mode Exit fullscreen mode

with total recorded downtime around:

6117ms
Enter fullscreen mode Exit fullscreen mode

This is exactly why publishing only the fastest recovery number would be misleading.

Recovery time varied between crashes.

Crash #4

The fourth forced termination happened at:

68.00 GB
Enter fullscreen mode Exit fullscreen mode

The server was restarted again against the existing storage directory.

The workload continued rather than starting from an empty dataset.

Crash #5

The final forced crash occurred at:

85.00 GB
Enter fullscreen mode Exit fullscreen mode

After the fifth recovery, the workload continued until the target dataset was reached.

The final chaos result was:

Crashes requested:             5
Crashes completed:             5
Crash-related write drops:     155
Unexpected errors:             0
Recovery status:               ALL CRASHES RECOVERED
Enter fullscreen mode Exit fullscreen mode

Those 155 dropped writes were expected consequences of deliberately killing the process while requests were in flight.

They were classified separately from unexpected write failures.

The final dataset

After all five crashes, Lioran S3 reached:

100.52 GiB
Enter fullscreen mode Exit fullscreen mode

stored across:

15 buckets
6,050 successful objects
Enter fullscreen mode Exit fullscreen mode

The workload recorded:

Unexpected write failures: 0
Retries:                   0
Enter fullscreen mode Exit fullscreen mode

This does not mean requests were magically unaffected by crashes.

The logs explicitly record:

155 chaos-expected write drops
Enter fullscreen mode Exit fullscreen mode

The distinction is important.

Expected failures caused by intentionally terminating the server are not the same thing as unexplained storage-engine failures.

Phase 2: Read everything back

Writing data is only half of a storage benchmark.

The next phase read the stored dataset.

Total:

Data read:          100.52 GiB
Read operations:    6,050
Concurrency:        32
Duration:           383.1 seconds
Enter fullscreen mode Exit fullscreen mode

Measured throughput:

Average:            281.7 MB/s
5-second peak:      414.1 MB/s
Enter fullscreen mode Exit fullscreen mode

Read latency:

p50:  1879.5 ms
p90:  2488.8 ms
p95:  2738.2 ms
p99:  8779.2 ms
Enter fullscreen mode Exit fullscreen mode

And:

Unexpected read failures: 0
Retries:                  0
Enter fullscreen mode Exit fullscreen mode

We did not hit the configured 500 MB/s read target

This is worth stating directly.

The test configuration specified:

Read target: 500 MB/s
Enter fullscreen mode Exit fullscreen mode

The recorded sustained read throughput was:

281.7 MB/s
Enter fullscreen mode Exit fullscreen mode

and the recorded 5-second peak was:

414.1 MB/s
Enter fullscreen mode Exit fullscreen mode

Therefore this run did not achieve the configured 500 MB/s read target.

The machine, filesystem, workload, implementation, and test harness all influence these numbers.

Pre-alpha benchmarking should expose that instead of hiding it.

There is more optimization work to do.

Memory usage

The benchmark also tracked process memory.

The recorded values were:

Peak RSS:   924.2 MB
Final RSS:  231.7 MB
Enter fullscreen mode Exit fullscreen mode

while operating on a dataset larger than:

100 GiB
Enter fullscreen mode Exit fullscreen mode

The object engine uses streaming I/O rather than loading the complete dataset or complete large objects into memory.

Dataset size and process memory therefore do not need to scale together linearly.

The most important phase: integrity verification

Performance is irrelevant if the recovered data is wrong.

After the write and read workloads completed, the test harness performed SHA-256 verification against sampled stored objects.

The integrity phase verified:

188 objects
Enter fullscreen mode Exit fullscreen mode

and reported:

Checksum failures:  0
Missing objects:    0
Enter fullscreen mode Exit fullscreen mode

The final integrity log ends with:

Final verification complete:
188 verified,
0 checksum failures,
0 missing objects.
Result: PASS
Enter fullscreen mode Exit fullscreen mode

The benchmark summary reports:

Integrity Verification: PASS
Missing Objects:        0
Corrupt Objects:        0
SHA-256 Mismatches:     0
Overall Result:         PASS
Enter fullscreen mode Exit fullscreen mode

That is the result I care about more than the peak throughput graph.

Final result

The complete run finished with:

Dataset:                   100.52 GiB
Objects:                   6,050
Buckets:                   15

Scheduled crashes:         5
Completed crashes:         5
Crash write drops:         155
Unexpected write errors:   0
Unexpected read errors:    0

Average write:             87.3 MB/s
5s write peak:             663.2 MB/s

Average read:              281.7 MB/s
5s read peak:              414.1 MB/s

Peak RSS:                  924.2 MB
Final RSS:                 231.7 MB

Integrity samples:         188
Missing objects:           0
Corrupt objects:           0
SHA-256 mismatches:        0

Overall result:            PASS
Enter fullscreen mode Exit fullscreen mode

Don't trust the summary. Inspect the evidence.

A benchmark summary is still produced by the benchmark harness.

So I'm publishing the underlying artifacts too.

You can inspect the crash timestamps, server output, individual writes, reads, recovery events, metrics, object manifest, and SHA-256 verification yourself.

Raw proof files

Human-readable benchmark summary

https://liorans3.sbs/proof/summary.txt

This is the quickest artifact to inspect.

Machine-readable benchmark summary

https://liorans3.sbs/proof/summary.json

Contains the structured configuration, throughput, latency percentiles, crash recovery measurements, memory statistics, integrity results, and correctness flags.

Crash log

https://liorans3.sbs/proof/crashes.log

Contains all five scheduled server terminations and the acknowledged dataset progress at which each crash occurred.

Recovery log

https://liorans3.sbs/proof/recovery.log

Contains server restart events, recovery timing, SDK transport reconstruction, authentication restoration, and workload release after each crash.

Integrity verification log

https://liorans3.sbs/proof/integrity.log

Contains individual SHA-256 verification operations and the final integrity result.

Write log

https://liorans3.sbs/proof/writes.log

Contains the write workload events used to build the dataset.

Read log

https://liorans3.sbs/proof/reads.log

Contains the read workload events for the completed dataset.

Server log

https://liorans3.sbs/proof/server.log

Raw server-side output captured during the benchmark and crash/recovery workload.

Test harness log

https://liorans3.sbs/proof/test.log

Contains the orchestration and lifecycle output from the benchmark harness.

Errors log

https://liorans3.sbs/proof/errors.log

The provided artifact from this run is empty.

That file is intentionally still published as part of the evidence bundle.

Object manifest

https://liorans3.sbs/proof/manifest.jsonl

The object-level JSONL manifest generated by the workload.

This is one of the larger artifacts and allows the run to be inspected at a much lower level than the summary.

Raw metrics

https://liorans3.sbs/proof/metrics.jsonl

Machine-readable JSONL metrics collected throughout the workload.

This is the largest proof artifact in the bundle and contains the time-series benchmark measurements used during the run.

Why publish all of this?

Because infrastructure benchmarks are extremely easy to make impressive.

Choose the fastest five seconds.

Use warm cache.

Ignore failed requests.

Disable durability.

Hide recovery behavior.

Crop the terminal.

Put a giant number on a landing page.

Done.

That is not what I want Lioran S3 engineering to become.

If I publish:

87.3 MB/s average writes
Enter fullscreen mode Exit fullscreen mode

you should be able to inspect where that number came from.

If I say:

5 server crashes
Enter fullscreen mode Exit fullscreen mode

you should be able to see all five crash timestamps.

If I say:

0 SHA-256 mismatches
Enter fullscreen mode Exit fullscreen mode

you should be able to open the integrity log.

And when a configured target is missed, such as the 500 MB/s read target in this run, that should be visible too.

What this test proves, and what it does not

This run provides evidence that this specific Lioran S3 pre-alpha build, under this specific workload and environment, completed a 100.52 GiB write/read chaos test with five scheduled server terminations and finished the test harness's integrity checks without a detected missing or corrupted object.

It does not prove that Lioran S3 can never lose data.

It does not prove that every hardware configuration will achieve these throughput numbers.

It does not prove that every possible crash point has been tested.

It does not prove production readiness.

This is V1 Pre-Alpha.

The point of publishing the evidence is precisely to avoid turning one successful chaos run into a universal reliability claim.

What's next

This test exposed both strengths and obvious optimization targets.

The engine recovered from all five scheduled crashes and completed the integrity phase successfully.

At the same time, sustained throughput remains below the long-term targets, particularly on the read side of this run.

The next versions will continue focusing on:

  • storage-engine correctness,
  • crash consistency,
  • recovery,
  • throughput,
  • tail latency,
  • multipart workloads,
  • larger datasets,
  • longer-running workloads,
  • and more aggressive failure injection.

Lioran S3 V1 Pre-Alpha launched on October 1, 2026.

The next planned version is scheduled for November 17, 2026.

This is still early infrastructure.

But now the benchmark claims come with receipts.

Links

Lioran S3:

https://liorans3.sbs

Documentation:

https://docs.liorans3.sbs

Lioran Group:

https://lioran.group

Lioran Developer Solutions:

https://lioransolutions.com

GitHub:

https://github.com/LioranGroupOfficial/LioranBastion-Rust

Complete benchmark evidence

https://liorans3.sbs/proof/summary.txt

https://liorans3.sbs/proof/summary.json

https://liorans3.sbs/proof/crashes.log

https://liorans3.sbs/proof/recovery.log

https://liorans3.sbs/proof/integrity.log

https://liorans3.sbs/proof/writes.log

https://liorans3.sbs/proof/reads.log

https://liorans3.sbs/proof/server.log

https://liorans3.sbs/proof/test.log

https://liorans3.sbs/proof/errors.log

https://liorans3.sbs/proof/manifest.jsonl

https://liorans3.sbs/proof/metrics.jsonl

Swaraj Puppalwar

Founder & CTO, Lioran Group

Lioran Developer Solutions

Building developer infrastructure in India.
Enter fullscreen mode Exit fullscreen mode

Top comments (0)