2 posts tagged with "chaos-testing"

In a world that's moving faster every day, where software verification is essential to keep up with that pace, building increasingly specialized harnesses, simulators, and test suites has become imperative.

A few weeks ago, we introduced Caraxes, our test suite for simulating extreme scenarios and pushing the database to its limits under all kinds of conditions: node failures, network issues, disk contention, and more.

The results have been extremely valuable, helping us uncover issues in both performance and correctness.

It's similar to putting an F1 car through different configurations in a simulator before those failures happen during a race. It allows us to expose the system to hundreds of hours of testing and exceptional conditions, compressed into just a few hours.

This post is the second part. It tells what the dragon found, what we fixed, and how the database is more robust today.

A database earns your trust on its worst day, not on its best one.

When everything is healthy, almost any database looks correct. You insert rows, you read them back, and the numbers add up. The real question is what happens when a server dies in the middle of a commit, when the network starts losing packets, or when a disk fills up while a transaction is still writing.

Those moments are hard to create by hand. They are also the moments a distributed database exists to survive.

So we built a tool that creates them whenever we want. We named it Caraxes, after a dragon, because its only job is to attack our own cluster while we watch what happens.