rsyncing rsnapshots takes up too much space

[silbro]

Hi everybody

I use rsnapshot for my backups. I don't directly back up to an external disk. I backup my data locally via rsnapshot and then rsync the rsnapshot to the usb disk. The usb disk is always full though. I think it doesn't back up the rsnapshots like they are, instead backs up all of the data every time. Is that possible ? If yes, what could I do instead? I don't want to rsnapshot to the usb disk directly.

Thanks for all answers

cheers

[gyzer]

Why don't you want to do the rsnapshot to the drive directly? I've been using rsnapshot for several years now to a USB connected drive without issues.

[silbro]

The reason I don't want to do that is because I will be switching the USB disk and I want the Backup of my data to be local as well.

[silbro]

Ok I believe this was my problem (yes I'm ashamed of myself):

* 04 * * * /root/backup/rsynctousb.sh

instead of

0 04 * * * /root/backup/rsynctousb.sh



If it works now (which I believe I will close the thread)

[silbro]

rsyncing still fills up all the space on my usb disk. Anyone have any ideas ?

[LHammonds]

I have to ask....how large is the total amount of space you are wanting to sync to the USB and what size is the USB...and is it dedicated for just that backup?

[silbro]

Hi LHammonds,

Thanks for your reply.

The rsnapshot backup is 213GB. My usb disk is 1.8 TB and yes it is only used for backups. df -T outputs this:

/dev/sdi1 fuseblk 1953511420 1952793420 718000 100% /usb_backup

Could it be that the problem is the type? (fuseblk). I think i formatted it as ntfs so I could just plug it into either linux or windows if I need a backup.

[LHammonds]

fuseblk is how Linux reports NTFS mounted drives so that is probably no problem at all. However, a big part of the puzzle is exactly how much space is being consumed. The block size being used and the amount/size of the files being backed up may not be matching up very well (e.g. very large block size but tons of super tiny files)

Type the following so we can see how big the folder is:

du -sh /path/to/usb

Then list the contents of the drive where the snapshot files are being stored in order to get a feel for the size of each file over time.

ls -lh /path/to/usb

or if all the files are over a GB,

ls -l --blocksize=g /path/to/usb

[silbro]

Hi LHammonds

/data1/backup/rsnapshot is where the rsnapshot writes to locally

sudo du -sh /data1/backup/rsnapshot/
213G    /data1/backup/rsnapshot/

du -sh /data1/backup/rsnapshot/daily.0
201G    /data1/backup/rsnapshot/daily.0

du -sh /data1/backup/rsnapshot/daily.1
203G    /data1/backup/rsnapshot/daily.1

du -sh /data1/backup/rsnapshot/daily.2
203G    /data1/backup/rsnapshot/daily.2

du -sh /data1/backup/rsnapshot/daily.3
203G    /data1/backup/rsnapshot/daily.3

du -sh /data1/backup/rsnapshot/daily.4
203G    /data1/backup/rsnapshot/daily.4

du -sh /data1/backup/rsnapshot/daily.5
203G    /data1/backup/rsnapshot/daily.5

du -sh /data1/backup/rsnapshot/daily.6
203G    /data1/backup/rsnapshot/daily.6

ls -lh /data1/backup/rsnapshot/
total 208K
drwxr-xr-x 6 root root 4.0K 2012-09-11 03:30 daily.0
drwxr-xr-x 6 root root 4.0K 2012-09-10 03:30 daily.1
drwxr-xr-x 6 root root 4.0K 2012-09-09 03:30 daily.2
drwxr-xr-x 6 root root 4.0K 2012-09-08 03:30 daily.3
drwxr-xr-x 6 root root 4.0K 2012-09-07 03:30 daily.4
drwxr-xr-x 6 root root 4.0K 2012-09-06 03:30 daily.5
drwxr-xr-x 6 root root 4.0K 2012-09-05 03:30 daily.6
drwxr-xr-x 6 root root 4.0K 2012-07-30 03:30 monthly.0
-rw-r--r-- 1 root root 156K 2012-09-11 03:30 rsnapshot.log
drwxr-xr-x 6 root root 4.0K 2012-09-03 03:30 weekly.0
drwxr-xr-x 6 root root 4.0K 2012-08-27 03:30 weekly.1
drwxr-xr-x 6 root root 4.0K 2012-08-20 03:32 weekly.2
drwxr-xr-x 6 root root 4.0K 2012-08-10 03:30 weekly.3

/usb_backup/backup is where the local rsnapshots I made are being rsynced to

du -sh /usb_backup/backup/rsnapshot/
1.6T    /usb_backup/backup/rsnapshot/

ls -lh /usb_backup/backup/rsnapshot/
total 160K
drwxrwxrwx 1 root root    0 2012-09-11 03:30 daily.0
drwxrwxrwx 1 root root    0 2012-09-10 03:30 daily.1
drwxrwxrwx 1 root root    0 2012-09-09 03:30 daily.2
drwxrwxrwx 1 root root    0 2012-09-08 03:30 daily.3
drwxrwxrwx 1 root root    0 2012-09-07 03:30 daily.4
drwxrwxrwx 1 root root    0 2012-09-06 03:30 daily.5
drwxrwxrwx 1 root root    0 2012-09-05 03:30 daily.6
drwxrwxrwx 1 root root 4.0K 2012-07-30 03:30 monthly.0
-rwxrwxrwx 1 root root 156K 2012-09-11 03:30 rsnapshot.log
drwxrwxrwx 1 root root    0 2012-09-04 04:00 weekly.0
drwxrwxrwx 1 root root    0 2012-09-04 04:00 weekly.1
drwxrwxrwx 1 root root    0 2012-09-04 04:00 weekly.2
drwxrwxrwx 1 root root    0 2012-09-04 04:00 weekly.3

du -sh /usb_backup/backup/rsnapshot/daily.0/
201G    /usb_backup/backup/rsnapshot/daily.0/

du -sh /usb_backup/backup/rsnapshot/daily.1
203G    /usb_backup/backup/rsnapshot/daily.1

du -sh /usb_backup/backup/rsnapshot/daily.2
203G    /usb_backup/backup/rsnapshot/daily.2

du -sh /usb_backup/backup/rsnapshot/daily.3
203G    /usb_backup/backup/rsnapshot/daily.3/

du -sh /usb_backup/backup/rsnapshot/daily.4
203G    /usb_backup/backup/rsnapshot/daily.4

du -sh /usb_backup/backup/rsnapshot/daily.5
203G    /usb_backup/backup/rsnapshot/daily.5

du -sh /usb_backup/backup/rsnapshot/daily.6
203G    /usb_backup/backup/rsnapshot/daily.6

I'm really starting to think, that rsyncing the rsnapshot somehow always copies all the files instead of just the links.

[markbl]

I'm really starting to think, that rsyncing the rsnapshot somehow always copies all the files instead of just the links.

Remember that rsnapshot works by using hardlinks to save space. Note that rsync does not preserve hardlinks by default with "-a". You must also specify "-H". See man rsync.