History

HF f7352a49c7 set -e in backup rsync		2022-09-16 13:12:40 +02:00
..
backupSync.sh	set -e in backup rsync	2022-09-16 13:12:40 +02:00
compressBackup.sh	full path commands for compressBackup	2021-09-13 13:11:10 +02:00
README.md	add scripts for compression on backup server	2021-07-08 17:23:58 +02:00

README.md

Scripts and notes for backup (historical view) server

backupSync.sh

shell script that can be launched with backup.js to sync to a storage server after every backup. It uses rsync which is much faster than ftp, sftp or any other methode.

Notes

Historical view and backups are best to be stored on a different server. Backups do generate lots of files. Every day one full backup is made with png files for every single tile/chunks - and incremential backups all 15min into png files with just the changed pixels since the last full backup set. If the filesystem has a limit on inodes, this is very likely to be hit within a year or so by the amount of small files.

We can mitigate this issue by turning duplicates into hardlinks (a file with multiple filenames/paths) and by compressing every month into a squshfs image. squashfs also compresses inodes. However, mksquashfs needs to parse all inodes, which takes lots of RAM (at least 256 bytes per inode - we got millions of files). We use the arguments -b 8192 -no-xattrs -no-exports -progress, where no-export is neccessary in order to not hit memory limit when mounting multiple images later. We do all of this in:

compressBackup.sh

Shell script that compresses the size of the backups with hardlink and squashfs. It is supposed to run as a daily cron script. Look into it's source for a comment what it does and set BACKUPROOT within it.

This uses the hardlink_0.3 util from https://jak-linux.org/projects/hardlink/ which ships with current debian and ubuntu, but in a different version on other distributions.