Hello Self-Hosters,
What is the best practice for backing up data from docker as a self-hoster looking for ease of maintenance and foolproof backups? (pick only one :D )
Assume directories with user data are mapped to a NAS share via NFS and backups are handled separately.
My bigger concern here is how do you handle all the other stuff that is stored locally on the server, like caches, databases, etc. The backup target will eventually be the NAS and then from there it’ll be double-backed up to externals.
-
Is it better to run
cp /var/lib/docker/volumes/* /backupLocationevery once in a while, or is it preferable to define mountpoints for everything inside of/home/user/Containersand then use a script to sync it to wherever you keep backups? What pros and cons have you seen or experienced with these approaches? -
How do you test your backups? I’m thinking about digging up an old PC to use to test backups. I assume I can just edit the ip addresses in the docker compose, mount my NFS dirs, and failover to see if it runs.
-
I started documenting my system in my notes and making a checklist for what I need to backup and where it’s stored. Currently trying to figure out if I want to move some directories for consistency. Can I just do
docker-compose downedit the mountpoints indocker-compose.ymland rundocker-compose upto get a working system?
I’m lucky enough to run a business that needs a datacenter presence. So most my home-lab (including Lemmy) is actually hosted on a Dell PowerEdge R740xd in the DC. I can then use the small rack I have at home as off-site backups and some local services.
I treat the entirety of
/var/lib/dockeras expendable. When creating containers, I make sure any persistent data is mounted from a directory made just to host the persistent data. It meansdocker compose down --rmi all --volumesisn’t destructive.When a container needs a database, I make sure to add an extra read-only user. And all databases have their container and persistent volume directory named so scripts can identify them.
The backup strategy is then to backup all non-database persistent directories and dump all SQL databases, including permissions and user accounts. This gets run 4 times a day and the backup target is an NFS share elsewhere.
This is on top of daily backuppc backups of critical folders, automated Proxmox snapshots for docker hosts every 20 minutes, daily VM backups via Proxmox Backup Server and replication to another PBS at home.
I also try and use S3 where possible (seafile and lemmy are the 2 main uses) which is hosted in a container on a Synology RS2423RP+. Synology HyperBackup then performs a backup overnight to the Synology RS822+ I have at home.
Years ago I fucked up, didn’t have backups, and lost all the photos of my sons early years. Backups are super important.
That’s a heart breaking way to learn your lesson about backups. It must have taken a long time to accept.
Everyone I know that actually keeps backups has the same kind of story. It’s sad that no matter how many other people talk about keeping backups, it always takes a tragic loss like this to get people to buy hardware/subscriptions.
I hear you. I worked for an msp where some customers would refuse to invest in backup solutions and we either declined to renew their contract or they suffered an event and we were then setting up backups.
I was in the middle of a migration from OVH to Hetzner. I knew I had good backups at home so the plan was to blow away OVH and restore from backup to Hetzner. This was the mistake.
Mid migration I get an alert from the raid system that a drive has failed and had been marked as offline. I had a spare disk ready, as I planned for this type of event. So I swapped the disk. Mistake number 2.
I pulled the wrong disk. The Adaptec card shit a brick, kicked the whole array out. Couldn’t bring it back together. I was too poor to afford recovery. This was my lesson.
Now I only use ZFS or MDRAID, and have multiple copies of data at all times.