Distributed, high-performance blob storage on SQLite
https://zerodha.tech/blog/high-performance-distributed-blob-storage/I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.
sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.
Or is there a better way?
Why sqlite instead of RocksDB ?
What you need at the end of the day for a blob storage is a transactional key value store.
Which RocksDB will give you with much better throughput (and latency) and solves your transaction problem.
An alternative to faster blob storage is to represent the blob's schema in SQL directly.
If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.
One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.
These are user preferences for UI setup, and they have a lot of write traffic. But probably each user changes their preferences only by a little bit each time. Their 1MB writes would go down to just the small diffs.
They could store the diffs and transparently apply them before sending to the client. After some timeout or maybe after a certain number of diffs have accumulated, squash them all and write back.
They’d still need a server and custom code to handle this but it would probably reduce the database load and SSD writes by a lot. And let them just use something off-the-shelf for storage.
Bad fault tolerance, no words on replication or backup. Offline rebalance... meh. I am not sure how I feel about their choice of Redis protocol.