Tag: server

BSD

A Simple Script for Creating and Deleting Rolling ZFS Snapshots in FreeBSD

Here’s a simple little Bourne shell-compatible script I wrote to create and delete rolling ZFS snapshots on a small home server running the FreeBSD operating system. Sure there’s ports/packages available in FreeBSD to accomplish this, but there’s something to be said for keeping it simple and having one less piece of software to update and maintain.

As currently configured, the script recursively creates a daily snapshot of /pool_0/dataset_0. The name of each snapshot consists of the word “snap,” followed by the date and time the snapshot was taken (e.g., snap-202002102300). Snapshots are retained in pool_0/dataset_0/.zfs/snapshots. The script will then destroy any snapshot older that 90 days and log what its done to the file cronlog located in my home directory. A typical ~/cronlog entry looks like the following:

zfssnap.sh
Mon Feb 10 23:00:00 EST 2020

Taking today's snapshot: pool_0/dataset_0@snap-202002102300

Attempting to destroy old snapshots...
Destroying the following old snapshots:
pool_0/dataset_0@snap-202001312300
**********

zfssnap.sh

Mon Feb 10 23:00:00 EST 2020

Taking today's snapshot: pool_0/dataset_0@snap-202002102300

Attempting to destroy old snapshots...

Destroying the following old snapshots:

pool_0/dataset_0@snap-202001312300

**********

Using the script

To use the script, I save it as zfssnap.sh in ~/bin, where I keep most of my helper scripts, and make it executable:

chmod +x ~/bin/zfssnap.sh

1	chmod +x ~/bin/zfssnap.sh

Then I delegate some ZFS permissions to user iceflatline so snapshots can be created and destroyed without becoming the root user. However, this command is issued as the user root:

zfs allow -u iceflatline create,destroy,hold,mount,receive,send,snapshot pool_0

1	zfs allow -u iceflatline create,destroy,hold,mount,receive,send,snapshot pool_0

Note that I’ve set permissions at the zpool level, which means that all datasets under pool_0 will inherit these settings. Alternatively I could have applied the permissions just to dataset_0.

Then I make the snapshots retained in pool_0/dataset_0/.zfs/snapshots. Here too, the command must be issued as the user root:

zfs set snapdir=visible pool_0

1	zfs set snapdir=visible pool_0

Here again, I’ve applied applied this setting at the zpool level so when additional datasets are created in pool_0 they will inherit this setting. And here too, I could have applied this setting just to dataset_0.

Finally I configure a cronjob in the crontab for user iceflatline so that the script runs daily at 23:00:

### DAILY

# Run backup scripts every day at 2300
0 23 * * * /home/iceflatline/bin/zfssnap.sh

### DAILY

# Run backup scripts every day at 2300

0 23 * * * /home/iceflatline/bin/zfssnap.sh

If you’d like to use the script as is then simply modify the variable src_0 to reflect the name of your zpool and dataset. Else, modify the script to suite your needs.

Prefer to run the script more frequently than daily? Modify your cronjob. Here are couple of examples for running the script more frequently:

### HOURLY

# Run snapshot script every 15 minutes

#*/18 * * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script every 30 minutes

#*/33 * * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script at the top of every hour

#3 * * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script every three hours at the top of each hour

#3 */3 * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script every six hours at the top of each hour

#3 */6 * * * /home/iceflatine/bin/zfssnap.sh

### HOURLY

# Run snapshot script every 15 minutes

#*/18 * * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script every 30 minutes

#*/33 * * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script at the top of every hour

#3 * * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script every three hours at the top of each hour

#3 */3 * * * /home/iceflatine/bin/zfssnap.sh

# Run snapshot script every six hours at the top of each hour

#3 */6 * * * /home/iceflatine/bin/zfssnap.sh

Don’t like the snapshot naming convention. No problem, just modify the variable snap_prefix.

Prefer to have more or less than 90 snapshots? Modify the retention variable.

Want to add other zpools and/or datasets? Add them to another variable (e.g., src_1), then modify the remainder of the script to recognize and take action on them.

Prefer to have the script log its output to a different location? Modify the log variable.

You get the idea.

#!/bin/sh
### BEGIN INFO
# PROVIDE: 
# REQUIRE: 
# KEYWORD: 
# Description:	
# This script is used to create zfs snapshots and remove older snapshots. 
# The number of snapshots to retain is defined in the variable retention.
# Author: iceflatline <iceflatline@gmail.com>
#
# OPTIONS:
# -v: Be verbose
### END INFO

### START OF SCRIPT 
# These variables are named first because they are nested in other variables. 
snap_prefix=snap 
retention=90 

# Full paths to these utilities are needed when running the script from cron.
date=/bin/date
grep=/usr/bin/grep
sed=/usr/bin/sed
sort=/usr/bin/sort
xargs=/usr/bin/xargs
zfs=/sbin/zfs

src_0="pool_0/dataset_0"
today="$snap_prefix-`date +%Y%m%d%H%M`"
snap_today="$src_0@$today"
snap_old=`$zfs list -t snapshot -o name | $grep "$src_0@$snap_prefix*" | $sort -r | $sed 1,${retention}d | $xargs -n 1`
log=/home/iceflatline/cronlog

# Create a blank line between the previous log entry and this one.
echo >> $log

# Print the name of the script.
echo "zfssnap.sh" >> $log

# Print the current date/time.
$date >> $log

echo >> $log

# Look for today's snapshot and, if not found, create it.  
if $zfs list -H -o name -t snapshot | $grep "$snap_today" > /dev/null
then
	echo "Today's snapshot '$snap_today' already exists." >> $log
	# Uncomment if you want the script to exit when it does not create today's snapshot:
	#exit 1 
else
	echo "Taking today's snapshot: $snap_today" >> $log
	$zfs snapshot -r $snap_today >> $log 2>&1
fi

echo >> $log

# Remove snapshot(s) older than the value assigned to $retention.
echo "Attempting to destroy old snapshots..." >> $log

if [ -n "$snap_old" ]
then
	echo "Destroying the following old snapshots:" >> $log
	echo "$snap_old" >> $log
	$zfs list -t snapshot -o name | $grep "$src_0@$snap_prefix*" | $sort -r | $sed 1,${retention}d | $xargs -n 1 $zfs destroy -r >> $log 2>&1
else
    echo "Could not find any snapshots to destroy."	>> $log
fi

# Mark the end of the script with a delimiter.
echo "**********" >> $log
# END OF SCRIPT

#!/bin/sh

### BEGIN INFO

# PROVIDE:

# REQUIRE:

# KEYWORD:

# Description:

# This script is used to create zfs snapshots and remove older snapshots.

# The number of snapshots to retain is defined in the variable retention.

# Author: iceflatline <iceflatline@gmail.com>

# OPTIONS:

# -v: Be verbose

### END INFO

### START OF SCRIPT

# These variables are named first because they are nested in other variables.

snap_prefix=snap

retention=90

# Full paths to these utilities are needed when running the script from cron.

date=/bin/date

grep=/usr/bin/grep

sed=/usr/bin/sed

sort=/usr/bin/sort

xargs=/usr/bin/xargs

zfs=/sbin/zfs

src_0="pool_0/dataset_0"

today="$snap_prefix-`date +%Y%m%d%H%M`"

snap_today="$src_0@$today"

snap_old=`$zfs list -t snapshot -o name | $grep "$src_0@$snap_prefix*" | $sort -r | $sed 1,${retention}d | $xargs -n 1`

log=/home/iceflatline/cronlog

# Create a blank line between the previous log entry and this one.

echo >> $log

# Print the name of the script.

echo "zfssnap.sh" >> $log

# Print the current date/time.

$date >> $log

echo >> $log

# Look for today's snapshot and, if not found, create it.

if $zfs list -H -o name -t snapshot | $grep "$snap_today" > /dev/null

then

echo "Today's snapshot '$snap_today' already exists." >> $log

# Uncomment if you want the script to exit when it does not create today's snapshot:

#exit 1

else

echo "Taking today's snapshot: $snap_today" >> $log

$zfs snapshot -r $snap_today >> $log 2>&1

echo >> $log

# Remove snapshot(s) older than the value assigned to $retention.

echo "Attempting to destroy old snapshots..." >> $log

if [ -n "$snap_old" ]

then

echo "Destroying the following old snapshots:" >> $log

echo "$snap_old" >> $log

$zfs list -t snapshot -o name | $grep "$src_0@$snap_prefix*" | $sort -r | $sed 1,${retention}d | $xargs -n 1 $zfs destroy -r >> $log 2>&1

else

echo "Could not find any snapshots to destroy." >> $log

# Mark the end of the script with a delimiter.

echo "**********" >> $log

# END OF SCRIPT

Conclusion

There you have it. A simple way to create and destroy ZFS snapshots on a rolling basis under FreeBSD.

Linux

Install and Configure Folding@home on Ubuntu Server

Folding@home is a distributed computing project started by Stanford University to help understand protein folding, misfolding, and related diseases. The project uses the idle processing resources of thousands of personal computers belonging to people that have installed the Folding@home software on their systems. In this post I will describe how to install and configure the Folding@home client software on Ubuntu server. All steps assume that the Apache http server on Ubuntu server is installed and operating correctly.

Software versions used in this post were as follows:

Ubuntu server 16.04.3 LTS
fahclient_7.4.4_amd64.deb

So let’s get started…

Download and Install

First, update your package list:

sudo apt-get update

1	sudo apt-get update

Then download and install the latest 64-bit version of the Debian-based Folding@home client to a location of your choice. In this example we’ll use the user’s home directory:

cd ~
wget --no-check-certificate https://fah.stanford.edu/file-releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4_amd64.deb

1 2	cd ~ wget --no-check-certificate https://fah.stanford.edu/file-releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4_amd64.deb

Note that fahclient_7.4.4_amd64.deb was the most current version of the client at the time this post was published. You may need to update the file name in this command in order to fetch the most recent version of the software.

Now lets install the client:

cd ~
sudo dpkg -i --force-depends fahclient_7.4.4_amd64.deb

1 2	cd ~ sudo dpkg -i --force-depends fahclient_7.4.4_amd64.deb

The installer will request you to enter a Folding@home donor name, team number, passkey (optional), whether the client should be automatically started at system boot, and how much of your system resources should be used (See Figures 1-5, repectively).

If you have an existing Folding@home donor name enter that name here, else enter a new donor name:

Figure 1

If you’re joining an existing Folding@home team, enter that team number here, else simply enter 0 for no team:

Figure 2

The Folding@home passkey is a unique identifier that ties your Folding@home contributions directly to you (not just those with your username). The passkey uniquely identifies you as an individual donor and is associated with the results that you have completed. If you have an Folding@home passkey and would like to use it enter it here. If you don’t have a passkey and would like one, you can request one from Folding@home. Else simply select “Ok” here to move to the next screen:

Figure 3

Choose whether or not you want the Folding@home client to startup automatically when the Ubuntu server boots:

Figure 4

Finally, you can select how aggressively the client uses your system’s CPU resources:

Figure 5

The installer will finish and automatically start the Folding@home client. You can verify that the client is running using the following command:

sudo /etc/init.d/FAHClient status
fahclient is running with PID 30562

1 2	sudo /etc/init.d/FAHClient status fahclient is running with PID 30562

You can now delete the installation Folding@home package fahclient_7.4.4_amd64.deb from your home directory, if desired.

Configure

Now that the Folding@home client has been installed it’s time to configure Web Control, Folding@home’s graphical interface. Web Control is the default control program for monitoring your Folding@home client via an easy to use web page.

First, stop your running Folding@home client:

sudo /etc/init.d/FAHClient stop

1	sudo /etc/init.d/FAHClient stop

Then use sudo and your favorite editor to append the following lines to /etc/fahclient/config.xml, which will grant access to the Web Control dashboard to a specific IP address. Unfortunately Web Control can only grant access to a single IP address. In this example we’ve chosen to grant access to the host with the IP address 192.168.10.100:

<!-- Grant remote web access to the following IP -->
<allow>192.168.10.100</allow>
<web-allow>192.168.10.100</web-allow>

<web-allow>192.168.10.100</web-allow>

Finally, restart the Folding@home client:

sudo /etc/init.d/FAHClient start

1	sudo /etc/init.d/FAHClient start

You should now be able to access the Folding@home Web Control dash board at http://your-ubuntu-server-IP-address:7396/ (See Figure 6)

Figure 6

Conclusion
There you have it. A few minutes of your time and you can easily have Folding@home up and running on your Ubuntu server. I think Folding@Home is a great cause and have created my own Folding@Home team to contribute some of my spare CPU cycles. How about you? If you have a few cycles to spare how about dedicating some to Folding@home’s efforts? You’re also welcome to join join my team. To join, simply enter the team number 78746 when you install folding@home.

References
https://en.wikipedia.org/wiki/Folding@home
https://foldingathome.org/support/

Windows

Run a Windows Application as a Service with srvany

Occasionally the need arises for running an application in Windows as a service. This allows the application to run at boot time without the need for a user to be logged into the system in order for the application to start and operate. While built-in mechanisms such as Task Scheduler exist to help facilitate this, the ability to run an application as a service has certain advantages, notably the ability to prescribe certain actions be taken should the service fail. Take for example the the case where an application is listening for incoming IP connections. Being able to automatically restart the service without user intervention is desirable in order to avoid loss of service.

This post will describe how to use the Microsoft utilities instrsrv.exe and srvany.exe to install a Windows application as a service. instrsrv.exe is used to install the service while srvany.exe acts as a wrapper around the application and handles the service events. Both utilities are available as part of Microsoft’s Windows Server 2003 Resource Kit Tools. I have successful tested instrsrv.exe and srvany.exe on Windows 7/8.1; I have not tested them on Windows 10.

Let’s get started…

For purposes of example, we’ll assume we have a Windows application called “foobar” that is normally started by using the binary c:\foo\foobar.exe.

Start by downloading Microsoft’s Windows Server 2003 Resource Kit Tools and install it to a folder of your choice. Create a folder to contain the files instrsrv.exe and srvany.exe. We’ll use c:\srvany for our example. Now copy both files from the where you installed the Resource Kit tools to c:\srvany. After the files have been copied the Windows Server 2003 Resource Kit Tools may be uninstalled.

Now open a command prompt and install foobar as a system service using the following command. You may use any name you’d like for the service:

c:\srvany\instsrv.exe "foobar" c:\srvany\srvany.exe

1	c:\srvany\instsrv.exe "foobar" c:\srvany\srvany.exe

You should receive a response indicating that the service was successfully added.

Next, open the Windows registry editor and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\foobar. Use the mouse or keyboard to highlight foobar and select Edit->New->Key to create a new key named Parameters. Highlight Paramters and select Edit->New->String Value to create a new string value named Application. Double-click on Application and enter the full path to foobar.exe (e.g., c:\foo\foobar.exe). That’s it. You can now close the registry editor.

By default, the newly created service foobar is configured to run automatically when the system is restarted. To change this setting to Manual, run the Services applet by entering services.msc from a command prompt or by using WIN+R. Locate foobar from among the list of services, right-click on it and select Properties and change its Startup type to Manual. A service set to Manual can be started from within the Services applet, or by entering the following from a command prompt:

net start "foobar"

1	net start "foobar"

Selecting the Recovery tab from within properties will allow you to chose what the system should do in the event if the service fails.

You can delete the service by entering the following from a command prompt:

sc delete "foobar"

1	sc delete "foobar"

There you have it. A nice simple way to register and run a non-service application as a windows service.

BSD

Using Tarsnap in FreeBSD to Improve My Offsite Backups

In a recent post I described how I improved the reliability of my file system backups by using the data replication capabilities inherent in the FreeBSD Zettabyte File System (ZFS). In this post I will explore Tarsnap, another tool I recently started to use to perform secure offsite backups of my most important files.

The versions for the software used in this post are as follows:

FreeBSD 11.0-RELEASE
tarsnap 1.0.37

The steps discussed assume that the FreeBSD Ports Collection is installed. If not, you can install it using the following command:

portsnap fetch extract

1	portsnap fetch extract

If the Ports Collection is already installed, make sure to update it:

portsnap fetch update

1	portsnap fetch update

Okay, let’s get started. Okay, let’s get started. All commands are issued as the user root. While building the various ports you should accept all default configuration options unless otherwise instructed.

Create a Tarsnap account

Before installing Tarsnap I visited the Tarsnap registration page and created an account. Tarsnap operates on a prepaid basis, so you have to add some money to your account before you can start using it. The minimum amount is $5.00. Money will be deducted from your pre-paid amount based on the actual number of bytes stored and bandwidth used (after compression and data deduplication). Tarsnap prices are currently $.25 per Gigabyte per month for storage, and $.25 per byte for bandwidth.

Install Tarsnap

After creating an account it was time to install Tarsnap. First I made sure the Ports Collection was up to date:

portsnap fetch update

1	portsnap fetch update

Then proceeded with the install, accepting all default configuration options:

cd /usr/ports/sysutils/tarsnap 
make config-recursive install distclean

1 2	cd /usr/ports/sysutils/tarsnap make config-recursive install distclean

Next, I ran tarsnap-keygen, a utility which registers my machine with the Tarsnap server and generates a key that is used to encrypt and sign the archives that I create. I needed to have the e-mail address and password I used to create my Tarsnap account handy when running this command. In following example I’ve registered a machine with the host name tarsnap-test:

tarsnap-keygen --keyfile /root/tarsnap-test.key --user iceflatline@example.com --machine tarsnap-test

1	tarsnap-keygen --keyfile /root/tarsnap-test.key --user iceflatline@example.com --machine tarsnap-test

Note that if I had multiple machines containing files I wished to backup to Tarsnap, I would want create a separate key file for each machine.

By default tarsnap-keygen will create the key file /root/tarsnap.key. This can be changed by adding the option keyfile to specify a different location and/or key name. In the example above I’ve changed the name of my key file to tarsnap-test.key to help disambiguate keys in case I add additional machines to my Tarsnap account in the future.

Tarsnap creates the file /usr/local/etc/tarsnap.conf when installed. This config file is read by the tarsnap utility and specifies a number of default options, all of which will be ignored if the options in question are specified at the command line. Since I change the name of default key file, I revised the value for the option keyfile in /usr/local/etc/tarsnap.conf:

keyfile /root/tarsnap-test.key

1	keyfile /root/tarsnap-test.key

Note that you should store a copy of this key someplace safe. If you lose your Tarsnap key file(s), you will not be able to create new archives or access your archived data.

Using Tarsnap

After installing Tarsnap I was ready to create and backup my first archive. Tarsnap commands follow a syntax similar to the venerable tar utility. The -c option creates a new archive containing the specified files. The -f option specifies which file to write the archive to:

tarsnap -cf backup-20150729 pool_0/dataset_0/backup/

1	tarsnap -cf backup-20150729 pool_0/dataset_0/backup/

Performing subsequent backups of these files will go faster since Tarsnap’s deduplication feature will avoid sending data which was previously stored.

If I want to list all archives stored with Tarsnap I can use the following command:

tarsnap --list-archives | sort

1	tarsnap --list-archives \| sort

Adding one or more instance of the -v option to this command will make the output more verbose. For example, if -v is specified one or more times, the creation time of each archive is printed; if it is specified two or more times, the command line with which Tarsnap was invoked to create each archive is also printed.

If I want to list the files contained within a single archive I can use the following command:

tarsnap -tvf backup-20150729 | more

1	tarsnap -tvf backup-20150729 \| more

The -t option is used to print to stdout the files stored in the specified archive; the -v option of course makes the output a little more verbose.

If I wanted to delete one or more archives I can use the -d option:

tarsnap -df backup-20150729

1	tarsnap -df backup-20150729

When the time comes to restore one or more files from Tarsnap I have a couple of options. For example, I can recover all files contained in a particular archive using the following command. In this example, I’ve extracted all files contained in the archive backup-20150729 to /tmp where I can recover one of more files:

cd /tmp
tarsnap -xf backup-20150729

1 2	cd /tmp tarsnap -xf backup-20150729

Or I can extract just one of the directories in this archive:

cd /tmp
tarsnap -xf backup-20150729 pool_0/dataset_0/backup/foo-directory/

1 2	cd /tmp tarsnap -xf backup-20150729 pool_0/dataset_0/backup/foo-directory/

Note here that you must exclude the leading / from the directory you’re restoring from. So in this case, instead of /pool_0/dataset_0/some-directory/, it should be pool_0/dataset_0/backup/some-directory.

Or regress even further into the archive to recover a single file if desired:

cd /tmp
tarsnap -xf backup-20150729 pool_0/dataset_0/backup/foo-directory/foo-file

1 2	cd /tmp tarsnap -xf backup-20150729 pool_0/dataset_0/backup/foo-directory/foo-file

Finally, if for whatever reason I no longer wish to use Tarsnap on this machine I can invoke the nuke option, which will delete all of the archives stored:

tarsnap --nuke

1	tarsnap --nuke

To make sure you’re really serious, Tarsnap will ask you to type the text “No Tomorrow” when using this command.

Okay, after getting comfortable with the Tarsnap commands and backing up files manually for a couple of days, I created this ugly little script that creates a daily archive of a specified directory; looks for any archives older than 30 days and deletes them; and, logs its output to the file /home/iceflatline/cronlog:

#!/bin/sh
### BEGIN INFO
# PROVIDE: 
# REQUIRE: 
# KEYWORD: 
# Description:	
# This script is used to perform backups using the tarsnap utility. The number of backups to retain is defined in the variable retention.
# Author: iceflatline <iceflatline@gmail.com>
#
# OPTIONS:
# -c: Create an archive containing the specified files 
# -d: Delete the specified archive
# -f: Archive name
### END INFO

### START OF SCRIPT 
backup_prefix=backup 
retention=30

# Full paths to the following utilities are needed when running the script from cron.
date=/bin/date
grep=/usr/bin/grep
sed=/usr/bin/sed
sort=/usr/bin/sort
tarsnap=/usr/local/bin/tarsnap
xargs=/usr/bin/xargs

backup_old=`$tarsnap --list-archive | $grep "$backup_prefix" | $sort -r | $sed 1,${retention}d | $sort | $xargs -n 1`
backup_today="$backup_prefix-`date +%Y%m%d`"
dir_0=/pool_0/dataset_0/backup/
log=/home/iceflatline/cronlog

# Create a blank line between the previous log entry and this one.
echo >> $log

# Print the name of the script.
echo "tarsnap.sh" >> $log

# Print the current date/time.
$date >> $log

echo >> $log

# Look for today's backup and, if not found, create it.  
if $tarsnap --list-archives | $sort | $grep "$backup_today" > /dev/null
then
	echo "Today's backup '$backup_today' already exists." >> $log
else
	echo "Taking today's backup: $backup_today" >> $log
	$tarsnap -cf $backup_today $dir_0 >> $log 2>&1
fi

# echo >> $log

# Remove backup(s) older than the value assigned to $retention.
echo "Attempting to destroy old backups..." >> $log

if [ -n "$backup_old" ]
then
	echo "Destroying the following old backup:" >> $log
	echo "$backup_old" >> $log
	$tarsnap --list-archives | $grep "$backup_prefix*" | $sort -r | $sed 1,${retention}d | $sort | $xargs -n 1 $tarsnap -d -f >> $log 2>&1
else
    echo "Could not find any backups to destroy."	>> $log
fi

# Mark the end of the script with a delimiter.
echo "**********" >> $log
# END OF SCRIPT

#!/bin/sh

### BEGIN INFO

# PROVIDE:

# REQUIRE:

# KEYWORD:

# Description:

# This script is used to perform backups using the tarsnap utility. The number of backups to retain is defined in the variable retention.

# Author: iceflatline <iceflatline@gmail.com>

# OPTIONS:

# -c: Create an archive containing the specified files

# -d: Delete the specified archive

# -f: Archive name

### END INFO

### START OF SCRIPT

backup_prefix=backup

retention=30

# Full paths to the following utilities are needed when running the script from cron.

date=/bin/date

grep=/usr/bin/grep

sed=/usr/bin/sed

sort=/usr/bin/sort

tarsnap=/usr/local/bin/tarsnap

xargs=/usr/bin/xargs

backup_old=`$tarsnap --list-archive | $grep "$backup_prefix" | $sort -r | $sed 1,${retention}d | $sort | $xargs -n 1`

backup_today="$backup_prefix-`date +%Y%m%d`"

dir_0=/pool_0/dataset_0/backup/

log=/home/iceflatline/cronlog

# Create a blank line between the previous log entry and this one.

echo >> $log

# Print the name of the script.

echo "tarsnap.sh" >> $log

# Print the current date/time.

$date >> $log

echo >> $log

# Look for today's backup and, if not found, create it.

if $tarsnap --list-archives | $sort | $grep "$backup_today" > /dev/null

then

echo "Today's backup '$backup_today' already exists." >> $log

else

echo "Taking today's backup: $backup_today" >> $log

$tarsnap -cf $backup_today $dir_0 >> $log 2>&1

# echo >> $log

# Remove backup(s) older than the value assigned to $retention.

echo "Attempting to destroy old backups..." >> $log

if [ -n "$backup_old" ]

then

echo "Destroying the following old backup:" >> $log

echo "$backup_old" >> $log

$tarsnap --list-archives | $grep "$backup_prefix*" | $sort -r | $sed 1,${retention}d | $sort | $xargs -n 1 $tarsnap -d -f >> $log 2>&1

else

echo "Could not find any backups to destroy." >> $log

# Mark the end of the script with a delimiter.

echo "**********" >> $log

# END OF SCRIPT

I wrote the script to /home/iceflatline/bin/tarsnap.sh where I maintain some other scipts and made it executable:

chmod +x /home/iceflatline/tarsnap.sh

1	chmod +x /home/iceflatline/tarsnap.sh

Then added the following cron job to the crontab under user root. The script runs every day at 0800 local time:

# Run tarsnap backup script everyday at 0800
0 8 * * * /home/iceflatline/bin/tarsnap.sh

1 2	# Run tarsnap backup script everyday at 0800 0 8 * * * /home/iceflatline/bin/tarsnap.sh

Conclusion
Well, that’s it. A short post describing my experiences using Tarsnap, an easy, secure and inexpensive solution for performing offsite backups of my most important files.

References
https://www.tarsnap.com/gettingstarted.html
https://www.tarsnap.com/man-tarsnap.1.html

BSD

Replacing MySQL with MariaDB in FreeBSD

(20170113 — The steps in this post were amended to address changes in recent versions of software. Minor editorial corrections were also made — iceflatline)

In my post on how to install and configure Apache, MySQL, PHP and phpMyAdmin on FreeBSD for basic local web development activities, one of the components is the MySQL database server. But what if you prefer to use MariaDB? MariaDB is an open source alternative to MySQL, and available under the terms of the GNU GPL v2 license. It is developed by the MariaDB community with oversight by the MariaDB Foundation.

This post will describe how to install and configure the MariaDB 10.1 server, as well as how to configure it as a replacement for a MySQL 5.7 server. I strongly encourage you to test these steps first before using them on your development or production environment.

The versions of software discussed in this post are as follows:

FreeBSD 11.0-RELEASE

mysql57-server-5.7.17

mariadb101-server-10.1.20_1

The following steps discussed in this post assume you have the FreeBSD Ports Collection installed. If not, you can install it using the following command:

portsnap fetch extract

1	portsnap fetch extract

If the Ports Collection is already installed, make sure to update it:

portsnap fetch update

1	portsnap fetch update

Okay, let’s get started. All commands are issued as the user root. While building the various ports you should accept all default configuration options unless otherwise instructed.

Install the MariaDB server

If you’re installing the MariaDB server for the first time on a FreeBSD system that does not already contain a version of MySQL server use the following steps.

Navigate to the MariaDB server port and build it:

cd /usr/ports/databases/mariadb101-server
make config-recursive install distclean

1 2	cd /usr/ports/databases/mariadb101-server make config-recursive install distclean

Then use the sysrc command to add the following line to /etc/rc.conf:

sysrc mysql_enable="YES"

1	sysrc mysql_enable="YES"

Start the MariaDB server:

service mysql-server start

1	service mysql-server start

And create a password for the MariaDB server root user:

/usr/local/bin/mysqladmin -u root password 'your-password'

1	/usr/local/bin/mysqladmin -u root password 'your-password'

That’s it. Now you should be able to use the MariaDB server in the same way you would a MySQL server.

Replacing MySQL server with MariaDB server

If you’ve previously installed a MySQL server then you can replace it with a MariaDB server. First, make sure to backup any existing database(s). This is critical. MariaDB 10.1 is not a drop-in replacement for MySQL 5.7. Installing MariaDB requires you to destroy your existing databases and restore them after MariaDB is installed.

Stop the MySQL server:

service mysql-server stop

1	service mysql-server stop

Uninstall the MySQL server and client:

cd /usr/ports/databases/mysql57-server
make deinstall 

cd /usr/ports/databases/mysql57-client
make deinstall

cd /usr/ports/databases/mysql57-server

make deinstall

cd /usr/ports/databases/mysql57-client

make deinstall

Delete everything in the MySQL server data directory:

rm -r /var/db/mysql/*

1	rm -r /var/db/mysql/*

Then navigate to the MariaDB server port and build it:

cd /usr/ports/databases/mariadb101-server
make conig-recursive install distclean

1 2	cd /usr/ports/databases/mariadb101-server make conig-recursive install distclean

Start the MariaDB server:

service mysql-server start

1	service mysql-server start

Create a password for the MariaDB server root user:

mysqladmin -u root password 'your-password'

1	mysqladmin -u root password 'your-password'

Recreate your database(s) in the MariaDB server and restore their files from your backups. Then run the command mysql_upgrade. This command does two things: it ensures that your mysql privilege and event tables are updated with the new fields MariaDB uses; and it performs a check of all tables and marks them as compatible with MariaDB server. In most cases this should be a fast operation (depending on the number of database tables):

mysql_upgrade -u root -p

1	mysql_upgrade -u root -p

Conclusion

That’s it. A few minutes of your time with the FreeBSD Ports Collection and you can quickly install a MariaDB server from scratch or replace an existing MySQL server with it.

References

https://mariadb.com/kb/en/
https://mariadb.com/kb/en/mariadb/what-is-mariadb-101/