Skip to main content

How much we test CamusDB

· 9 min read
Andres Gutierrez
Creator of CamusDB

A dark pyramid with glowing blue cracks under a night sky and a large planet

CamusDB is a young database, so you have a fair question: how much evidence stands behind it? Our answer has three parts: about 5.4 million operations per night, 23 to 39 million operations in each daily two-hour fault soak, and more than 21,000 automated tests. This post shows where these numbers come from and what they do not prove.

What a SQL fuzzer catches

· 8 min read
Andres Gutierrez
Creator of CamusDB

SQLFuzz

Your database can return the wrong rows and report no error. A hand-written test catches only the cases its author thought of. A SQL fuzzer catches cases nobody thought of, and it runs every night on its own. This post shows how the CamusDB fuzzer works, which defects it can catch, and what it cannot see. You need no knowledge of fuzzers or of database internals.

Introducing SolariumDB: Taking Control of the Storage Layer in .NET

· 7 min read
Andres Gutierrez
Creator of CamusDB

SolariumDB

What if you could debug your database's storage engine with the same tools you use to debug your .NET applications? Or optimize how it writes to disk for your specific workload?

That's the motivation behind SolariumDB, an embedded key-value database written entirely in C#.

SolariumDB brings us one step closer to building a database entirely in .NET, from the SQL layer down to the code responsible for organizing files and writing data to disk.

From 400 to 10,000 operations per second in CamusDB

· 7 min read
Andres Gutierrez
Creator of CamusDB

Pyramid benchmark

If you're investigating database throughput, our experience offers a practical place to look: the work repeated around every query and commit. Here's what we removed, how we measured the gains, and what the resulting benchmark can tell you about CamusDB.

Many months ago, when we first started measuring database throughput in cluster mode, we discovered it could only handle around 400 ops/sec. After a long journey of benchmarking, investigating bottlenecks, and optimizing the system, we've reached 10K ops/sec.

On October 12, a three-node cluster sustained 10,284 durable operations per second for 45 minutes, completing 27.8 million operations after a two-minute warm-up. Every measured minute stayed above 9,700 ops/sec.

All three nodes ran on one laptop, alongside the load generator and metrics collector. And the storage was memory-backed. Both details matter when interpreting the number.

A young database meets the Join Order Benchmark

· 9 min read
Andres Gutierrez
Creator of CamusDB

Data temple order

If your queries join five, ten or fifteen tables, CamusDB now plans them better and runs them faster. This post explains what changed in plain terms, so you can see what to expect from your own queries.

A join query names its tables and its conditions. It does not say in which order to join the tables. The database picks the order, and the order decides how much data the query reads. A good order starts from the few rows that survive the filters. A bad order can build millions of rows and then throw most of them away.

The Join Order Benchmark (JOB) is a set of 113 queries from a 2015 paper by Leis and five co-authors. The queries run over a movie database of 21 tables, and each one joins 4 to 17 tables. The paper tested mature databases and found that their estimates of how many rows a join returns were often wrong by orders of magnitude. The error grows with each join, and a wrong estimate leads to a slow plan. The benchmark exists to show that weakness.

What the dragon found, part three

· 28 min read
Andres Gutierrez
Creator of CamusDB

A dragon breaking one artic city of glowing pyramids

A ten-minute test finds the bugs that happen in ten minutes.

A distributed database is hard to build, and the code is the smaller half of the work. The larger half is verification. You check the results again and again, you dig into the system until you know what it really does, and you push it past the point where it should break. A guarantee on a documentation page is worth exactly what the evidence behind it can show, and that evidence has to cover speed and correctness at the same time.

Part one introduced Caraxes, the harness that attacks a real CamusDB cluster in Docker. Part two told what it found in its first month: a dropped transfer leg, a cluster that said healthy while it was dead, scans that lost rows under load.

This part covers the three weeks after that. Most of it is about one test. It runs for two hours, it injects a fault every eight minutes, and it took eight attempts before it passed. Every failed attempt found something, and the runs after the first pass kept finding things.

What the dragon found

· 19 min read
Andres Gutierrez
Creator of CamusDB

A dragon breaking one pyramid in a city of glowing pyramids

In a world that's moving faster every day, where software verification is essential to keep up with that pace, building increasingly specialized harnesses, simulators, and test suites has become imperative.

A few weeks ago, we introduced Caraxes, our test suite for simulating extreme scenarios and pushing the database to its limits under all kinds of conditions: node failures, network issues, disk contention, and more.

The results have been extremely valuable, helping us uncover issues in both performance and correctness.

It's similar to putting an F1 car through different configurations in a simulator before those failures happen during a race. It allows us to expose the system to hundreds of hours of testing and exceptional conditions, compressed into just a few hours.

This post is the second part. It tells what the dragon found, what we fixed, and how the database is more robust today.

Player inventories are bank accounts

· 12 min read
Andres Gutierrez
Creator of CamusDB

Cyberpunk game economy scene

A player spends forty hours to earn a sword. Another player pays real money for a bundle of skins. A guild saves gold for three weeks to buy a fortress.

None of that is "game data" to the person who did it. It is time, money, and trust. When a backend loses an item, duplicates a currency, or shows two owners for one mount, the player does not read it as a bug. The player reads it as disrespect.

We run game backends on CamusDB today. This post explains why we treat every inventory and every wallet as a small bank account. It also explains why that decision leads to distributed transactions, not away from them.

We built a dragon to attack our own database

· 11 min read
Andres Gutierrez
Creator of CamusDB

A dragon burning a city of glowing pyramids at night

A database earns your trust on its worst day, not on its best one.

When everything is healthy, almost any database looks correct. You insert rows, you read them back, and the numbers add up. The real question is what happens when a server dies in the middle of a commit, when the network starts losing packets, or when a disk fills up while a transaction is still writing.

Those moments are hard to create by hand. They are also the moments a distributed database exists to survive.

So we built a tool that creates them whenever we want. We named it Caraxes, after a dragon, because its only job is to attack our own cluster while we watch what happens.

When every experiment needs its own database

· 8 min read
Andres Gutierrez
Creator of CamusDB

Software is starting to work at a different rhythm.

A developer may test one idea at a time, but an AI agent can explore several possible solutions in parallel. One path may try a schema migration. Another may rewrite some data. A third may reproduce a bug using a different sequence of events.

Each path needs a database that feels real, but none of them should be allowed to damage the shared starting point.

This is where database branching becomes more than a convenient development feature. It becomes a practical way to give every experiment a safe place to work.