4 posts tagged with "distributed-systems"

If you're investigating database throughput, our experience offers a practical place to look: the work repeated around every query and commit. Here's what we removed, how we measured the gains, and what the resulting benchmark can tell you about CamusDB.

Many months ago, when we first started measuring database throughput in cluster mode, we discovered it could only handle around 400 ops/sec. After a long journey of benchmarking, investigating bottlenecks, and optimizing the system, we've reached 10K ops/sec.

On October 12, a three-node cluster sustained 10,284 durable operations per second for 45 minutes, completing 27.8 million operations after a two-minute warm-up. Every measured minute stayed above 9,700 ops/sec.

All three nodes ran on one laptop, alongside the load generator and metrics collector. And the storage was memory-backed. Both details matter when interpreting the number.

A ten-minute test finds the bugs that happen in ten minutes.

A distributed database is hard to build, and the code is the smaller half of the work. The larger half is verification. You check the results again and again, you dig into the system until you know what it really does, and you push it past the point where it should break. A guarantee on a documentation page is worth exactly what the evidence behind it can show, and that evidence has to cover speed and correctness at the same time.

Part one introduced Caraxes, the harness that attacks a real CamusDB cluster in Docker. Part two told what it found in its first month: a dropped transfer leg, a cluster that said healthy while it was dead, scans that lost rows under load.

This part covers the three weeks after that. Most of it is about one test. It runs for two hours, it injects a fault every eight minutes, and it took eight attempts before it passed. Every failed attempt found something, and the runs after the first pass kept finding things.

In a world that's moving faster every day, where software verification is essential to keep up with that pace, building increasingly specialized harnesses, simulators, and test suites has become imperative.

A few weeks ago, we introduced Caraxes, our test suite for simulating extreme scenarios and pushing the database to its limits under all kinds of conditions: node failures, network issues, disk contention, and more.

The results have been extremely valuable, helping us uncover issues in both performance and correctness.

It's similar to putting an F1 car through different configurations in a simulator before those failures happen during a race. It allows us to expose the system to hundreds of hours of testing and exceptional conditions, compressed into just a few hours.

This post is the second part. It tells what the dragon found, what we fixed, and how the database is more robust today.

A database earns your trust on its worst day, not on its best one.

When everything is healthy, almost any database looks correct. You insert rows, you read them back, and the numbers add up. The real question is what happens when a server dies in the middle of a commit, when the network starts losing packets, or when a disk fills up while a transaction is still writing.

Those moments are hard to create by hand. They are also the moments a distributed database exists to survive.

So we built a tool that creates them whenever we want. We named it Caraxes, after a dragon, because its only job is to attack our own cluster while we watch what happens.