EventStoreDB Ideas Portal
Ideas for improvements and new features in EventStoreDB, client libraries, and database extensions.
Ideas for improvements and new features in EventStoreDB, client libraries, and database extensions.
Customers who actively clean up their database encounter a situation when the scavenge.db file grows uncontrollably because it keeps the information about stuff that has been deleted.
Eventually, it might lead to a situation when the scavenge.db file will be larger than the database itself, and it will be very confusing.
If ESDB allows deleting streams without leaving any trace, it won't need to track those streams in scavenge.db, and it will save space. It might also help with compliance if someone has things like username in the stream name, and we keep it, although inaccessible (not entirely true as $streams will show it).
Having the ability to quickly whack a stream without adding load on scavenging will open up a a category of new use cases where having "transient streams" will be very useful. For example, "backend-for-the-frontend" session state; durable execution streams; and scratch space for the data engineering pipelines we are seeing more and more of on ESDB.
I believe that we need to elaborate on use cases. For example, one thing I heard about requires streams to be deleted completely so the new stream with the same name can be created with
NoStreamstate right away. When usingUnsafeIgnoreHardDeletesit would be necessary to wait until the stream and the meta gets scavenged. Unless I don't fully understand how it works.We do currently have
UnsafeIgnoreHardDeleteswhich does this, but as the name implies it is unsafe - the user has to take care to completely remove a stream across the cluster before re-creating it (or alternatively, do not recreate it). It would be good if we could make it a better user experience.On the downside, if streams are recreated, then of course a two reads of event 5 in stream A could return different events at different times, breaking immutability.