CamusDB is a young database, so you have a fair question: how much evidence stands behind it? Our answer has three parts: about 5.4 million operations per night, 23 to 39 million operations in each daily two-hour fault soak, and more than 21,000 automated tests. This post shows where these numbers come from and what they do not prove.
Your database can return the wrong rows and report no error. A hand-written test catches only the cases its author thought of. A SQL fuzzer catches cases nobody thought of, and it runs every night on its own. This post shows how the CamusDB fuzzer works, which defects it can catch, and what it cannot see. You need no knowledge of fuzzers or of database internals.
A ten-minute test finds the bugs that happen in ten minutes.
A distributed database is hard to build, and the code is the smaller half of the work. The larger half is verification. You check the results again and again, you dig into the system until you know what it really does, and you push it past the point where it should break. A guarantee on a documentation page is worth exactly what the evidence behind it can show, and that evidence has to cover speed and correctness at the same time.
Part one introduced Caraxes, the harness that attacks a real CamusDB cluster in Docker. Part two told what it found in its first month: a dropped transfer leg, a cluster that said healthy while it was dead, scans that lost rows under load.
This part covers the three weeks after that. Most of it is about one test. It runs for two hours, it injects a fault every eight minutes, and it took eight attempts before it passed. Every failed attempt found something, and the runs after the first pass kept finding things.
In a world that's moving faster every day, where software verification is essential to keep up with that pace, building increasingly specialized harnesses, simulators, and test suites has become imperative.
A few weeks ago, we introduced Caraxes, our test suite for simulating extreme scenarios and pushing the database to its limits under all kinds of conditions: node failures, network issues, disk contention, and more.
The results have been extremely valuable, helping us uncover issues in both performance and correctness.
It's similar to putting an F1 car through different configurations in a simulator before those failures happen during a race. It allows us to expose the system to hundreds of hours of testing and exceptional conditions, compressed into just a few hours.
This post is the second part. It tells what the dragon found, what we fixed, and how the database is more robust today.
A database earns your trust on its worst day, not on its best one.
When everything is healthy, almost any database looks correct. You insert rows, you read them back, and the numbers add up. The real question is what happens when a server dies in the middle of a commit, when the network starts losing packets, or when a disk fills up while a transaction is still writing.
Those moments are hard to create by hand. They are also the moments a distributed database exists to survive.
So we built a tool that creates them whenever we want. We named it Caraxes, after a dragon, because its only job is to attack our own cluster while we watch what happens.