Customers who actively clean up their database encounter a situation when the scavenge.db file grows uncontrollably because it keeps the information about stuff that has been deleted.
Eventually, it might lead to a situation when the scavenge.db file will be larger than the database itself, and it will be very confusing.
If ESDB allows deleting streams without leaving any trace, it won't need to track those streams in scavenge.db, and it will save space. It might also help with compliance if someone has things like username in the stream name, and we keep it, although inaccessible (not entirely true as $streams will show it).
I believe that we need to elaborate on use cases. For example, one thing I heard about requires streams to be deleted completely so the new stream with the same name can be created with
NoStream
state right away. When usingUnsafeIgnoreHardDeletes
it would be necessary to wait until the stream and the meta gets scavenged. Unless I don't fully understand how it works.We do currently have
UnsafeIgnoreHardDeletes
which does this, but as the name implies it is unsafe - the user has to take care to completely remove a stream across the cluster before re-creating it (or alternatively, do not recreate it). It would be good if we could make it a better user experience.On the downside, if streams are recreated, then of course a two reads of event 5 in stream A could return different events at different times, breaking immutability.