Hacker News new | past | comments | ask | show | jobs | submit

OpenZFS deduplication is good now and you shouldn't use it

https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/
I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately.

Because:

> When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool.

To me, this is "obviously" the wrong approach in most cases. When I'm writing data, I want that write to complete as fast as possible, even at the cost of disk space. That's why I don't save files I'm actively working on in 7zip archives.

But later on, when the system is quiet, I would love for ZFS to go back and figure out which data is duplicated, and use the BRT or whatever to reclaim space. This could be part of a normal scrub operation.

loading story #42002643
loading story #42001483
loading story #42002155
The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.
You need block pointer rewrite for this?
You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.
loading story #42005923
loading story #42003486
loading story #42001903
loading story #42001330
loading story #42001923
loading story #42001970
loading story #42002290
loading story #42031327
loading story #42001870
loading story #42004780
loading story #42001154
loading story #42001245
loading story #42004636
loading story #42005394
loading story #42001336
loading story #42001757
loading story #42002952
loading story #42001317
loading story #42003801
loading story #42002186
loading story #42007816
loading story #42005112
loading story #42004925
loading story #42003907
loading story #42007311
loading story #42004547
loading story #42002321
loading story #42002231
loading story #42001922
loading story #42009482
loading story #42014928
loading story #42002615
loading story #42001853
loading story #42001926
loading story #42002009
loading story #42001219
loading story #42004927
loading story #42001425
loading story #42001497
loading story #42005940