rss logo

Using Rsync on GNU/Linux: A Comprehensive Guide with Practical Examples

rsync logo

rsync (remote synchronization), is a powerful command-line tool for copying/synchronizing files and folders on local or remote hosts.

It is ideal for copying files and backing up a file server, for example. However, it is not suitable for files in use, such as databases or virtual machines, as in this case you will get corrupted (unusable) file copies.

Install rsync

rsync is not usually pre-installed, so the first thing to do is to install it in your Linux distribution.

  • Debian/Ubuntu:
user@host:~$ sudo apt install rsync
  • Arch Linux:
[root@host ~]# pacman -S rsync

Examples of rsync use

Here are a few examples of how rsync can be used.

Copying a file to another folder

Let's start by using rsync to copy a file from our /home/std/ folder to /tmp/.

Illustration of using the rsync command on Linux to transfer a file from one directory to another with the command rsync -av. Includes an example of transferring myfile_source to myfile_dest.
  • Command line:
user@host:~$ rsync -av /home/std/myfile_source /tmp/myfile_dest
  • Options:
    • -a or --archive which include:
      • -r: recurse into directories
      • -l: copy symlinks as symlinks
      • -p: preserve permissions
      • -t: preserve modification times
      • -g: preserve group
      • -o: preserve owner (super-user only)
      • -D (--devices)
    • -v: This option increases the amount of information you are given during the transfer.

Copy an entire folder

We can also use rsync to copy a folder recursively. Here, we'll copy our entire /home/std folder to /tmp/.

Illustration of rsync transferring a directory from /home/std to /tmp/folder_dest, preserving file and folder structure.
  • Command line:
user@host:~$ rsync -av /home/std/ /tmp/folder_dest
  • Options:
    • -a or --archive which include:
      • -r: recurse into directories
      • -l: copy symlinks as symlinks
      • -p: preserve permissions
      • -t: preserve modification times
      • -g: preserve group
      • -o: preserve owner (super-user only)
      • -D (--devices)
    • -v: This option increases the amount of information you are given during the transfer.

Copying over the network

Even more interestingly, we can use rsync to copy files or folders over the network. For this to work, both rsync and SSH must be installed on the source and destination servers.

Illustration of rsync transferring files and directories from a PC to a remote NAS server using SSH, demonstrating secure and efficient data synchronization.
  • Command line:
user@host:~$ rsync -av -e 'ssh -p 22' /home/std user@192.168.1.100:/data/
  • Options:
    • -a or --archive which include:
      • -r: recurse into directories
      • -l: copy symlinks as symlinks
      • -p: preserve permissions
      • -t: preserve modification times
      • -g: preserve group
      • -o: preserve owner (super-user only)
      • -D (--devices)
    • -v: This option increases the amount of information you are given during the transfer.
    • -e: Specify the remote shell to use, allows here to specify the ssh port (useful if our remote ssh service is listening on a different port)

Misc

  • Copy files from remote host to local host:
user@host:~$ rsync -av -e 'ssh -p 22' user@192.168.1.100:/data/std /home/

Backups

As explained above, rsync is particulary well suited to backing up file servers. We'll look at how to back up to USB drive, to a remote host and how to perform incremental backups to save disk space.

Backup to external hard drives

Here, we're going to perform a full backup of our GNU/Linux operating system. We will exclude all special or unnecessary folders (/dev, /mnt, /proc, /sys and /tmp). The destination folder on the external hard drive will be automatically named with the current date in back-YYYY-MM-dd format.

Illustration of using rsync to perform a system backup to a USB device, excluding specific directories like /tmp, /proc, and /dev, while preserving file structure and including timestamps for each backup.
  • Command line (the source is / (to save the entire system) and destination is /mnt/usb/):
root@host:~# rsync -av --delete --delete-excluded --exclude=/tmp --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/mnt / /mnt/usb/back-$(date "+%Y-%m-%d")
  • Options:
    • -a or --archive which include:
      • -r: recurse into directories
      • -l: copy symlinks as symlinks
      • -p: preserve permissions
      • -t: preserve modification times
      • -g: preserve group
      • -o: preserve owner (super-user only)
      • -D (--devices)
    • -v : This option increases the amount of information you are given during the transfer.
    • --exclude=PATTERN : exclude files matching PATTERN.
    • --delete (optional here) : delete extraneous files from dest dirs.
    • --delete-excluded (optional here) : also delete excluded files from dest dirs.

Incrementals backups

One of rsync's strengths is its ability to perform incremental backups. To do this, it uses hardlinks. For each file to be backed up, rsync compares it with the last backup (the current folder on the external hard drive, which is a symbolic link to the folder of the last backup): if the file is identical, it creates a hard link and consumes no disk space, otherwise (if the file is new or different) it creates a new file.

Of course, the current symbolic link must be created manually and recreated after each backup so that it points to the latest backup.

As you can see from the diagram, the first backup will occupy the most space (10 GB here), then the other backups will occupy only the difference (1 GB or 2 GB in this example).

One last thing to know: in the event we want to reclaim space on our external hard drive, we'll need to delete the backups from the oldest to the most recent (to avoid breaking hardlinks).

  • The three phases of incremental rsync backups:
    • PRE-BACKUP: the external hard drive is mounted on /mnt/usb/
    • BACKUP: rsync backup is performed, using the current symbolic link pointing to the last backup, in this case back-2022-08-14.
    • POST-BACKUP: once the rsync backup is complete, we need to update the current link so that it points to the most recent backup, which is now back-2022-08-15.
Diagram showing a three-phase backup process using rsync: full backup, incremental backups, and symlink management to update the current pointer for backups on a USB device.

Backup to USB drive with incrementals

This is detailed application of what was seen above.

Illustration of a three-phase process for incremental backups using rsync. Includes the initial full backup, incremental backups, and updating the current backup pointer on a USB device with symlink management.
  • Options:
    • -a or --archive which include:
      • -r: recurse into directories
      • -l: copy symlinks as symlinks
      • -p: preserve permissions
      • -t: preserve modification times
      • -g: preserve group
      • -o: preserve owner (super-user only)
      • -D (--devices)
    • -v: This option increases the amount of information you are given during the transfer.
    • --exclude=PATTERN: exclude files matching PATTERN.
    • --link-dest=/mnt/usb/current: hardlink to files in current when unchanged.
    • --delete: delete extraneous files from dest dirs.
    • --delete-excluded: also delete excluded files from dest dirs.
  • Command line to backup (the source is / (to save the entire system) and destination is /mnt/usb/):
user@host:~$ rsync -av --delete --delete-excluded --exclude=/tmp --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/mnt --exclude=/media --link-dest=/mnt/USB/current / /mnt/usb/back-$(date "+%Y-%m-%d")
  • Update the current symbolic link:
    • Remove the symbolic link named current:
user@host:~$ rm -f /mnt/usb/current
  • Recreate the current symbolic link:
user@host:~$ ln -s /mnt/usb/back-$(date "+%Y-%m-%d") /mnt/usb/current
  • Check the space used by incremental backups:
user@host:~$ du -sh /mnt/usb/back-*
10G	back-2022-08-11
1G	back-2022-08-12
2G	back-2022-08-13
1G	back-2022-08-14

Backup to a Network Server with incrementals

Same as above, except that this time the destination is a GNU/Linux server and the backup will take place over the network.

Diagram showcasing incremental backups using rsync over SSH to a remote server. Includes examples of full and incremental backups, file exclusion, and the use of a symlink for current backup management.
  • Options:
    • -a or --archive which include:
      • -r: recurse into directories
      • -l: copy symlinks as symlinks
      • -p: preserve permissions
      • -t: preserve modification times
      • -g: preserve group
      • -o: preserve owner (super-user only)
      • -D (--devices)
    • -v: This option increases the amount of information you are given during the transfer.
    • --exclude=PATTERN: exclude files matching PATTERN.
    • --link-dest=/mnt/usb/current: hardlink to files in current when unchanged.
    • --delete: delete extraneous files from dest dirs.
    • --delete-excluded: also delete excluded files from dest dirs.
  • Command line for backup (the source is / (to save the entire system) and the destination is /backup/ on the server 192.168.1.100):
user@host:~$ rsync -av --delete --delete-excluded --exclude=/tmp --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/mnt --exclude=/media --link-dest=/backup/current / user@192.168.1.100:/backup/back-$(date "+%Y-%m-%d")
  • Update the current symbolic link:
    • Remove the symbolic link named current:
user@host:~$ ssh -p 22 user@192.168.1.100 "rm /backup/current"
  • Recreate the current symbolic link:
user@host:~$ ssh -p 22 user@192.168.1.100 "ln -s /backup/back-$(date "+%Y-%m-%d") /backup/current"

Misc

I've listed everything I found useful here.

  • Show progress during transfer:
user@host:~$ rsync --progress -av /home/std/folder_source/ remoteuser@192.168.1.100:/tmp/folder_dest
  • Display statistics at the end of the transfer:
user@host:~$ rsync --stats -av /home/std/folder_source/ remoteuser@192.168.1.100:/tmp/folder_dest
  • Limit the bandwidth used to a defined threshold:
user@host:~$ rsync --progress -av --bwlimit=100k /home/std/folder_source/ remoteuser@192.168.1.100:/tmp/folder_dest
  • Transfer only files whose modification date does not exceed 3 days to the /tmp/ directory:

Note: requires local or ssh transfer without password to work.

user@host:~$ rsync --progress -av remoteuser@192.168.1.100:/ --no-relative --files-from=<(ssh remoteuser@192.168.1.100 find /data/* -mtime -3 -type f) /tmp/
  • Mirror copy (--delete: delete extraneous files from dest dirs):
user@host:~$ rsync --progress /home/std/ /tmp/folder_dest

How to use rsync with root privileges

  • On the remote host, add the following sudo entry (replace user with the username used by the ssh connection):
root@host:~# visudo
user       ALL=NOPASSWD:/usr/bin/rsync
  • rsync with sudo if we need root rights on the destination machine:
user@host:~$ rsync -av -e 'ssh -p 22' --rsync-path='sudo rsync' /home/std user@192.168.1.100:/data/