Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.
The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.