Backup FreeNAS Files Remotely Using FreeBSD and rsync

FreeNAS is an open source storage platform based on FreeBSD that supports file sharing across Windows, Apple, and Unix/Linux systems. rsync is an open source file copying utility for Linux/Unix systems. It’s famous for its “delta-transfer” algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. It offers a large number of options, including the ability to copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon; and, the ability to preserve symbolic links, hard links, file ownership, permissions, devices and times.

In this post I will describe how to configure FreeBSD, FreeNAS and rsync to securely backup files located on FreeNAS to a FreeBSD machine located at a remote location. All steps involved assume you have a running implementation of FreeBSD and FreeNAS. The software versions used in this post were as follows:

• FreeBSD 9.0-RELEASE (x64)
• FreeNAS v0.7.1 Shere (revision 5127)
• rsync v3.0.9

Configure FreeNAS

There are two ways for the FreeBSD machine to contact FreeNAS using rsync: using a remote-shell program as the transport (such as ssh or rsh) or contacting an rsync daemon directly via TCP. We’re going to use ssh as our transport method. We’ll also create a new user account in FreeNAS named “rsync,” which we will enable access to via ssh but limit its use solely for the purpose of backing up files using rsync.

Begin by logging into the FreeNAS web interface, navigate to Access->Users and Groups->Users and select the “+” icon to create a new user account. The following parameters will need to be configured:

Name – This is the login name of the new user. You’re welcome to use any user name you’d like here. For purposes of our example though let’s enter rsync.

Full Name – This field should match the Name field, so enter rsync again here.

Password – Our plans include generating and using ssh keys to authenticate the FreeBSD machine to the FreeNAS machine and not passwords, so leave these fields blank.

User ID – Unless you have a specific reason to change the User ID you should retain whatever FreeNAS has chosen for you here, typically UID 1001 if this is your first new user account on FreeNAS.

Shell – This parameter specifies the login shell for our new user rsync. The nologin option should not be selected as it will not permit access to the account. For purposes of our example, we’ll select the good old sh shell.

Primary group – This parameter sets the primary group that the user rsync will belong to. Unless you have a specific reason to change the group, the default group Guest should be retained here.

Additional group – No other group memberships need be selected.

Home directory – Here’s where we’ll enter the path to the home directory for the rsync user, which for purposes of our example should be set to /usr/home/rsync.

When complete, the configuration should resemble Figure 1. Now select “Add” to accept the changes and add rsync as a new user.

Now let’s configure and activate the rsync service in FreeNAS. Navigate to Services->Rsync->Server->Settings and check the “Enable” box, then click on the drop down list under “Map to user” and select rsync. The remaining fields can remain at their default settings. When complete, the configuration should resemble Figure 2. Now select “Save and Restart” to accept the changes and start the rsync server.

Next, we need to ensure the ssh service is running in FreeNAS. Navigate to Services->SSH and check the “Enable” box. To help improve security, we’ll avoid using the default TCP port 22 and, for purposes of our example, use port 13725 instead. Check the “Permit root login” box and ensure that “Password authentication” box is checked. When complete, the configuration should resemble Figure 3. Now select “Save and Restart” to accept the changes and start the ssh server.

Finally, connect via ssh to the the root account of the FreeNAS machine and create the directory /usr/home/rsync/.ssh, which we’ll use to hold the public RSA key that we’ll eventually generate and copy from the FreeBSD machine. Note: Alternatively, you can create this directory using FreeNAS’s built-in file management tools by navigating to Advanced->File Manager. Make sure to provide the correct local IP address or host name of the FreeNAS machine. If this machine is located on another network, then make sure to provide the correct IP address or URL of the network where the FreeNAS machine is located, as well as ensure that any NAT gateway is configured to pass TCP port 13725 traffic to the correct host:

Okay, we’re finished with thr FreeNAS configuration for now. Let’s move on and configure the FreeBSD machine.

Configure FreeBSD

The following steps assume you have the FreeBSD Ports Collection installed. If not, you can install it by using the following commands:

If the Ports Collection is already installed, make sure to update it:

Navigate to the rsync port and build it, accepting the default configuration options:

Next, we’ll choose a location that will be used to backup the FreeNAS files. For purposes of our example, we will create the subdirectory backup within the primary user’s home directory on the FreeBSD machine:

Then make sure the user owns this directory, as well as any files and subdirectories within it:

Recall that we shunned conventional password authentication when creating the rsync user account on FreeNAS. Instead, we’re going to generate an RSA public/private key pair on the FreeBSD machine, and copy the public key to the FreeNAS machine. rsync will be able to login and authenticate itself to the FreeNAS machine without the need for a password because the FreeBSD machine has the corresponding private key. rsync can now run autonomously because it will not be prompted for a password each time it attempts to login:

By default the RSA keys are 2048 bits. If you’re really paranoid you can increase this to 4096 bits if desired with the -b flag:

You’ll be asked by the ssh-keygen script where it should store the keys (~/.ssh is the default), and then asked to enter and confirm a password for the private key that will be stored in this machine. In our particular case, we don’t want to be prompted for a password so simply press enter. The private key (id_rsa) and the public key (id_rsa.pub) are generated and then stored in ~/.ssh (if you accepted the default directory). Now that our keys are generated, let’s copy the public key to the FreeNAS machine using the venerable secure copy command:

Note that in the case of the scp command we must designate the ssh port number using the upper-case -P option, and pass to it the same ssh port number that was designated when we configured FreeNAS. Here again, make sure to provide the correct local IP address or host name of the FreeNAS machine or, if this machine is located on another network, the correct IP address or URL of the network where the FreeNAS machine is located. When you successfully connect you’ll likely receive a message concerning the authenticity of the FreeNAS machine, along with a fingerprint of its public RSA key, and asked if you’re sure you want to continue connecting. Accept by typing yes and you’ll be presented with a password prompt for the root user on the FreeNAS machine. Enter the password and the public key will be copied.

Now let’s connect again via ssh to the root account of the FreeNAS machine and copy the public key to /home/rsync/.ssh/authorized_keys so that rsync can securely connect to FreeNAS without the need for a password. We’ll also take this opportunity to remove the public key from the root account:

Then lock down permissions and make sure the user rsync owns of all files and subdirectories within its home directory:

Finally, let’s make sure we can connect to the rsync account on FreeNAS using only public/private key authentication:

When you successfully connect you’ll once again receive a message concerning the authenticity of the FreeNAS machine. Accept by typing yes and you’ll be securely connected to the rsync user’s home directory (/home/rsync/) on FreeNAS without the need to enter a password.

Okay, let’s move on and configure rsync.

Configure rsync

For purposes of example, let’s assume that on the FreeNAS machines there is directory /mnt/files/foo containing a number of files along with two subdirectories – /mnt/files/foo/bar1 and /mnt/files/foo/bar2 – that we wish to backup on a routine basis to the FreeBSD machine. You could issue the following command to the FreeBSD machine accomplish that:

Long command, right? Let’s walk through it. First, we invoke the rsync command and pass to it the following options:

-a – Specifies that the files are transferred in “archive” mode, which ensures that any symbolic links, devices, attributes, permissions, ownerships etc are preserved.

-v – Increases the verbosity of the command so that you know in a little more detail what rsync is/is not doing.

-z – Specifies that rsync should use compression to reduce the size of data portions of its file transfers.

–delete – Specifies that rsync should delete files in the destination that no longer exist on the source side. This helps to keep the file systems specified in the source and destination synchronized.

Then we tell rsync that we’d like to use ssh and port 13725 by using the “-e ssh -p 13725” option. Finally, we tell rsync what the source and destination file systems are.

Another rsync option that I find handy is –exclude. Let’s say that for whatever reason you’d like to exclude the subdirectory /mnt/files/foo/bar2 and all the files therein. You’d use the following command to accomplish that:

Please consult the rsync man pages for further information on what rsync options are available for your specific needs.

Once you get the rsync command configured the way you’d like it, then it’s time to add it to cron on the FreeBSD machine so that it will perform backups on a routine basis. cron is a *nix utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon. These tasks are typically referred to as “cron jobs.” A “crontab” is a file which contains one or more cron job entries to be run at specified times. You can create a new crontab (or edit an exiting one) using the system’s default command line editor by using the command crontab -e under your user account. Here’s some example cron job entries using our rsync command. You could add one or many rsync cron job entries to your crontab depending on your needs, then simply uncomment the one you want to use. Also, cron will attempt to email to the user the output of the commands it runs. To silence this, we’ll redirect the command output to /dev/null:

While placing the full rsync command in the crontab works just fine, it can become a bit unwieldy, particularly if the command is lengthy. A more elegant approach is to capture the command and associated options in a shell script and invoke the script in a crontab instead of the full command. Here’s what a basic shell script might look like, again utilizing our example rsync command:

You’ll notice that in addition to running the rsync command, the script will also create the log file /home/iceflatline/cronlog and append the date/time and a brief summary of what transpired to it every time script runs. To use this script, modify its values to suit your particular need, including the correct IP address or URL for the FreeNAS machine, and give it a name, say for example rsync-freenas. Then create a bin/ subdirectory in the home directory on the FreeBSD machine, copy the file to that location, and make it executable:

Now then, a crontab entry invoking this script would look like this; i.e., a whole lot less complex:

Final Steps

Now that we have everything configured, login again to the FreeNAS web interface. Navigate to Services->SSH and uncheck both the “Permit root login” and “Password authentication” boxes. When complete, select “Save and Restart” to accept the changes and restart the ssh server.

Tips

Here’s a few of tips that may help improve your experience when using rsync in the manner described in this post

There’s no requirement that the machine you’re backing up FreeNAS files to be based on FreeBSD. You could just as easily use a machine based on a Linux distribution for example. Simply download and install rsync via the distribution’s package manager or compile it from source. The remaining steps should work as described.

Regardless of which operating system you use, backing up to a separate partition or, better yet, a separate hard disk(s) is recommended. This partition or disk can simply be mounted at a location in your file system, for example the /home/iceflatline/backup we used in our example, then, if something should require you to reinstall the OS, your files would still be intact. I prefer using a separate hard disk. That way, if/when I need to restore some or all of the files to FreeNAS, I need only unmount the drive from the FreeBSD machine, mount it in FreeNAS and have access again to the files.

Depending on the quantity and size of the files you want to backup, you may wish to consider initially locating the FreeBSD machine on the same network as the FreeNAS machine. The initial backup will likely go much faster on a 100 Mbps or 1000 Mbps local area network than it would over a wide area network. Once the initial backup is complete, then the FreeBSD machine can be moved to a remote location to start performing incremental backups.

A VPN (“Virtual Private Network”) can be setup between the FreeNAS and FreeBSD machine, thus perhaps eliminating the need to use a non-standard ssh port. Another benefit of using a VPN to connect the two machines is the ability to easily connect to the FreeBSD machine from within the FreeNAS machine’s network; handy if/when you want to quickly retrieve a file or perform routine maintenance on the FreeBSD machine.

Occasionally, you may add large and/or numerous files to FreeNAS where the time it takes to incrementally back them up to FreeBSD exceeds the time specified between cron jobs in the crontab. For example, if the cron job runs the rsync script or command every 60 minutes, but it takes 75 minutes to incrementally backup the new files added to FreeNAS, then errors will occur when the cron job triggers rsync to run again. To avoid this problem, you can edit the crontab to lengthen the time between cron jobs, or stop the cron job from running by commenting it out in the crontab and run rsync manually instead. Once rsync has fully backed up you can use the crontab again.

Conclusion

This concludes the post on how to remotely and securely backup and synchronize files located on a FreeNAS machine to a FreeBSD-based machine, using rsync, a fast and versatile file copying tool.

References

http://ss64.com/bash/cron.html

http://www.marksanborn.net/linux/learning-cron-by-example/

http://troy.jdmz.net/rsync/index.html