2022-09-16 13:12:40 +02:00
backupSync.sh set -e in backup rsync 2022-09-16 13:12:40 +02:00
compressBackup.sh full path commands for compressBackup 2021-09-13 13:11:10 +02:00
README.md add scripts for compression on backup server 2021-07-08 17:23:58 +02:00

Scripts and notes for backup (historical view) server


shell script that can be launched with backup.js to sync to a storage server after every backup. It uses rsync which is much faster than ftp, sftp or any other methode.


Historical view and backups are best to be stored on a different server. Backups do generate lots of files. Every day one full backup is made with png files for every single tile/chunks - and incremential backups all 15min into png files with just the changed pixels since the last full backup set. If the filesystem has a limit on inodes, this is very likely to be hit within a year or so by the amount of small files.

We can mitigate this issue by turning duplicates into hardlinks (a file with multiple filenames/paths) and by compressing every month into a squshfs image. squashfs also compresses inodes. However, mksquashfs needs to parse all inodes, which takes lots of RAM (at least 256 bytes per inode - we got millions of files). We use the arguments -b 8192 -no-xattrs -no-exports -progress, where no-export is neccessary in order to not hit memory limit when mounting multiple images later. We do all of this in:


Shell script that compresses the size of the backups with hardlink and squashfs. It is supposed to run as a daily cron script. Look into it's source for a comment what it does and set BACKUPROOT within it.

This uses the hardlink_0.3 util from https://jak-linux.org/projects/hardlink/ which ships with current debian and ubuntu, but in a different version on other distributions.