Linux rsync common usage

rsync is a command line file synchronisation application, which is available for

• Natively for almost all Linux distributions
• As a statically linked binary for ESXi server
• Most versions of Windows by way of a cut down cygwin environment

The rsync man page does not list the multitude of options in alphabetical order, which can make finding particular options long winded.  Some options are mutually exclusive and other options cancel other options.

Some common options and combinations are as follows.

Basic options suitable for local copy

options: -a archive, -v verbose, -h human readable, - P progress report

rsync -avhP /path/source/ /destination/

Copies the content of /path/source/. to /destination/.  and leaves intact any pre-existing files in /destination/ which are not in /path/source/

rsync -avhP /path/source /destination/

Creates a parent folder /destination/source/ and copies the contents of /path/source/. to it.

rsync -avhP --delete /path/source /destination/

Create a mirror of /path/source in /destination/ Delete any pre existing files in /destination/. which are not in source/.

Rsync MacOS Time Machine Backup Archive

MacOS includes rsync V2.69 accessible from the command line.

Rsync can be used to copy the Time Machine Backups.backupdb folder to and from volumes configured for use with time machine.  Hardlinks in the Backup.backupdb sub tree must be preserved using the -H parameter

rsync -avhHp /Volumes/TimeMachine/Backups.backupdb /Volumes/New/.

Optimising Throughput

Typical rsync operations carried out over fast LANs, often become bottlenecked at well below 10MBps (100Mbps).  Careful selection of the command options can improve throughput significantly.  Testing on a GBE LAN realised an improvement, from ~5MBps (50Mbps) to ~30MBps (300Mbps) for a 2GB file. 

The primary bottlenecks are caused by crc and compression algorithms within rsync, and the encryption algorithm within SSH.  Compression is disabled by default but often enabled without thought.  CRC calculations are not necessary during the initial replication, as there are no files in the destination to check against.  SSH encryption can be speeded up by selecting a less secure algorithm, such as blowfish, by passing switches to SSH through the rsync -e environment option.

• rsync compression is enabled and disabled with the -z switch.
• rsync crc checking may be disabled during initial replication with the [upper case] -W (--whole-file) switch.
• ssh blowfish encryption is selected with -e '-c blowfish'

rsync -e 'ssh -c blowfish' -avhPW user@host.local:~/work ~/backup/lanhost/

Options Suitable for Remote copy

Option: -z compress in transit

rsync -avhPz user@10.20.30.40:~/source /destination/

Connect to the host 10.20.30.40 using SSH as user and copy the content of /home/user/source to the local folder /destination/source/. Compress files before transmission and decompress before writing.

Option -e environment

rsync -e "-p 9022" -avhPz /path/source user@fqdn.net:/backup/

Pass the parameters -p 9022 to SSH, to connect to the custom SSH port 9022 on a host with the domain name fqdn.net,  as user.  Copy the contents from the local folder /path/source/. to the folder /backup/ on he remote host.

For MacOS use

rsync -e 'ssh -p 8022' -avhPz /path/source user@fqdn.net:/backup

Incremental Backup Options

Option -b backup, --backup= use archive folder

rsync - avhPzb --delete --backup-dir=/backup/archive/ /path/source /backup/today/

Mirrors the /path/source folder to /backup/today/.  Pre-existing files in /backup/today/ which do not appear in /path/source/. are moved to the /backup/archive/ folder.  Using the -b and --backup-dir options in a script, combined with date commands, allows an incremental backup to be constructed.

Windows Usage

rsync use under windows is essentially the same as linux.  Path references may not use the colon ':' character however.  Drives are referenced relative to the /cygdrive/ virual path.

rsync -avhP /cygdrive/c/documents\ and\ settings/ /cygdrive/r/backuppath/

Transfer Speeds

Data transfer between remote drives mapped with samba and wfs are much faster then using rsync's native ssh.  i.e. 10x to 100x

Background Operation

By default an rsync prcess will terminate when the shell is closed as happens when an ssh session is disconnected or a bash session ended.  To prevent rsync termination

1. Start the rsync process with nohup in the background. stdout is written to the nohup file in the current working directory and can be monitored with tail -f. However, there is no easy way to reconnect a terminal to the background process.

nohup rsync -avhp /source /dest &

2. Alternatively, the screen utility can be used to disconnect from a shell and reconnect later or from an alternative location.  After starting a local or ssh shell

screen

rsync -avhp /source /dest

type ctrl + A + D

To reconnect

screen -r