Ideas for using it as a backup/restore/deploy technology

[krystalcode]

Hello!

I find the idea of dvol quite exciting as a potential and I'm trying to find use cases. There may be projects with small data sets that you may want to switch between different states in which case dvol could be incredibly useful - I guess that's what it was originally designed for. In most projects I've worked on, and with databases such as MySQL, Redis, MongoDB, ElasticSearch etc. I have found the following two reasons for deploying, or reverting to, a specific state:

1. Recover a backup, usually after something goes wrong (disaster recovery).
2. Take a copy and deploy it on another environment for non-real time usage (such as analytics or troubleshooting).

I haven't actually played with dvol yet, but I can see how it could be a pretty cool and easy to use solution in such use cases, but I think there would be a couple of blocker issues. I don't know the internals of how dvol work, so I might be wrong in some points.

1. If dvol keeps all commits on the same Docker volume, the data usage can be quite high over time, much more than the actual data of the current state. That's especially if dvol cannot keep diffs but it actually stores all data separately for each commit.
2. Having to stop the container in order to make a commit may not be an option for many systems. That would mean downtime for an application using the database. Downtime when reverting to or deploying a commit for the first time would probably be less of an issue in most cases.

Potential resolutions for these issues? I'm not sure how technically feasible the following ideas may be:

1. Having a remote "repository" that dvol could move the data to in the background without disrupting the service, and so that it can locally keep only the data for the current state. There could be different drivers on where to store the "repository" e.g. local storage, AWS, Google Cloud, Rackspace etc. When you would want to revert to a specific commit, it would be able to pull the data from the "repository" and then discard the previous commit's data.
2. I'm not sure what would be the best solution for the second issue. Maybe by providing specific integrations with various databases and their ways of taking backups while the database is running. Or in a similar way that cloud solutions allow you to take a snapshot of a VM or even move it while it's running.

If these issues could be addressed, this would have a pretty awesome and easy data backup/restore/deploy solution: all with just dvol commit, dvol push, dvol pull, dvol checkout etc.

[Stephenitis]

This is wonderful feedback!

1. there is building feedback that echos this feature request. In it's initial implementation storing commits of data separately was intentional as its thought development volumes and databases would be small in comparison to production datasets.
2. a fair number of assumptions would have to be made about the running process in the docker container for this to work. I wonder if having a few strategies would work
3. stop container and commit
4. set a cron like commit schedule
5. allow the service to trigger dvol commits when it has idle moment?
6. do it in realtime with some magic

[wallnerryan]

I think @lukemarsden would find this conversation very interesting, thanks!

[chrisLeeTW]

+1 for continuous integration purpose, such as unit test or functional test, which need rebuild/restore compelete clear minimal database.