> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.
Why would anyone believe you when you say this? From the report:
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."
How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?
"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.
Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.
> we're working on benchmarking now as well.
But you already said you were 1000x faster than Postgres??
No one in their right mind will use this product ever.
Alas! It turns out it's really hard to do it right.
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).
What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.
That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.
https://web.archive.org/web/20260518201315/https://particled...
https://web.archive.org/web/20260518200305/https://particled...
Please try to resist the urge of being a meat proxy.
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
I mean did you see the "primary key equality" thing? Holy shit.
You can't be sure until they test it again
No. Wrong. False. All performance numbers are plainly invalid. As my coworker says, performance numbers don't count if the code is broken.
... executing SELECT * FROM releases WHERE 4 = 5 -- it's not just a slopdb, but TLA+ load bearing and 100% Jepsen ready
ParticleDB 1.1.4 ParticleDB 1.1.18.11.rc-22
2 rows returned
What a waste of CO₂.