Backup/Restore Scripts - updated with restore script

[vlaurenz]

A note about that last posting of the dump_users functiuon. You need to define PROCESS_BLOCK_SIZE for it to work.

You are correct. I have it defined up top. Sorry, I should have mentioned that.

[kluss0]

I figured that it was defined at the top along with the rest of the variables. I just wanted to point it out in case someone reading the thread tried to cut and paster ver batum.

C-Ya,
Kenny

[cdonovan]

I'm quite new to Scalix but VERY impressed... I built a server at home, tested and then built a new one at the office... I use the original backup script from this post to create my backup files... I moved the backup files to my NEW install and run the scalix_restore.sh...

Everything seems to run fine, but when I go back to the Admin console, NO USERS are there... I can logon the the web client and ALL of my mail is there! Very confusing to this newbie anyways... Oh yes, I ran onexeccda and I an running version 10...

Thanks...

[florian]

please check the forum for hostname changes (assume you've done that) and also check the manpage for the sxmodfqdn command.

-- f.

[cdonovan]

Thanks for the quick response and YES, I've done both... Unless I'm missing something, everyone seems to change their server name back to the original server name to solve the problem...

My problem is, I can't do that! The name of the test server is a pointer to my home IP address and the new server is now at our office. I can't just change the domain name in another location... Poor planning on MY part... I could have built the server inside using it's real name!!

Chris

[florian]

basically all the directory entries are tagged with their "home server" name and sxmodfqdn fixes that.

In addition, there are a couple of config files in /etc/opt/scalix and /var/opt/scalix that have the server name.

finally, there is /opt/scalix/global/config, in which it is also contained.

helpful to see if there is any problem in the directory is the following command:

omsearch -s -m S/G/HOST-FQDN

hope this helps,
Florian.

[cdonovan]

Thanks again for the quick response... I thought since I was just testing, I would rebuidl Scalix 10 with the same name and do the restore and THEN switch to the new name... In doing so, I wanted to reboot first (I'm use to working with Windows) and the server did'nt come back up... Cooked until Monday now... Didn't want to work the weekend anyway!

[hkphooey]

Great thread! OK, so I've adapted these scripts for my own use, so in the spirit of this thead I thought I'd post my adaptations. The reason why I adapted them was that I wanted to rotate the users mailbox backups from the last 7 days. Every week I take a further backup of the mailboxes to an external server.

Here are the modificaitons. First I created a new function called rotate_backups which removes the oldest backup and shifts all the directories along one, creating a new backup in current.

Code: Select all

function rotate_backups
{
   # For a 7 day rotation
   working_dir="$BACKUP_DIR/users/$MAILNODE"
   echo_and_log "Beginning Backup Rotation"   
        rm -rf $working_dir/d-7
        mv $working_dir/d-6 $working_dir/d-7
        mv $working_dir/d-5 $working_dir/d-6
        mv $working_dir/d-4 $working_dir/d-5
        mv $working_dir/d-3 $working_dir/d-4
        mv $working_dir/d-2 $working_dir/d-3
        mv $working_dir/d-1 $working_dir/d-2
        mv $working_dir/current $working_dir/d-1
        mv $working_dir/working $working_dir/current

       if [ "$?" != "0" ]
       then
           exit_with_error "Backup Rotation encountered some problems"
       else
           echo_and_log "Completed Backup Rotation"
       fi
}

The backup itself is created in a temporary directory called working before the rotation, so we have to also modify the dump_users function:

Code: Select all

backup_file="$BACKUP_DIR/users/$MAILNODE/working/${nospaces}-${DATE}-mail.gz"   


Now we just have to add our working directory to the function that checks if directories exist, and we're good to go.

Code: Select all

    for dir in $BACKUP_DIR $BACKUP_DIR/users $BACKUP_DIR/users/$MAILNODE $BACKUP_DIR/users/$MAILNODE/working


I choose to call the rotate_backups function after we've restarted Scalix for minumum downtime, although its really pretty quick.

Then once a week I backup the "current" directory to my external server. There are probably more elegant solutions, but all seems good so far.

[vlaurenz]

Your error checking in the function rotate_backups is only checking if the following command completes successfully:

Code: Select all

mv $working_dir/working $working_dir/current

I would do something like this instead:

Code: Select all

function rotate_backups
{
   # For a 7 day rotation
   working_dir="$BACKUP_DIR/users/$MAILNODE"
   echo_and_log "Beginning Backup Rotation"
        ecode=0
        rm -rf $working_dir/d-7
        let ecode=$ecode+$?
        mv $working_dir/d-6 $working_dir/d-7
        let ecode=$ecode+$?
        mv $working_dir/d-5 $working_dir/d-6
        let ecode=$ecode+$?
        mv $working_dir/d-4 $working_dir/d-5
        let ecode=$ecode+$?
        mv $working_dir/d-3 $working_dir/d-4
        let ecode=$ecode+$?
        mv $working_dir/d-2 $working_dir/d-3
        let ecode=$ecode+$?
        mv $working_dir/d-1 $working_dir/d-2
        let ecode=$ecode+$?
        mv $working_dir/current $working_dir/d-1
        let ecode=$ecode+$?
        mv $working_dir/working $working_dir/current
        let ecode=$ecode+$?
 
       if [ "$ecode" != "0" ]
       then
           exit_with_error "Backup Rotation encountered some problems"
       else
           echo_and_log "Completed Backup Rotation"
       fi
}

[nesx]

Nice rotation technique,
but note ALL of the directories must exist prior to execution or the backup will fall over.

[Rahz]

Can someone post the current version of the script with all the modifications and fixes ? Or at least update the main sccript in the very first post? Would be good to keep that up to date so that others could use it without having to read through all 5 pages. Might help get more feedback also.

Thanks for the work!

[vlaurenz]

Can someone post the current version of the script with all the modifications and fixes ? Or at least update the main sccript in the very first post? Would be good to keep that up to date so that others could use it without having to read through all 5 pages. Might help get more feedback also.

Thanks for the work!

I don't think that there is one 'mater' version of the script any more. People have modified it to suit their own needs.

[ianare]

Here is my version of it. I made my own backup system based on day of week rather than numbers, that was easier for us to implement. Been using it for a couple days now, but please let me know if you get any problems.

Code: Select all

#!/bin/sh
###############################################################################
# ombackup:
#   a backup script for scalix mail servers
#
#   This script is used to backup Scalix mail servers; it exports each
#   user to a gzip compressed file using the 'omcpoutu' command, then
#   duplicates the scalix data directory using rsync.
#
#   Before using this program you should set the values of the variables
#   below to match your server/preferences.
#
#   For detailed descriptions of the available command line switches,
#   execute the program with the -h flag.
#
#
#   Copyright (C) 2006 Jon Allie <jon@jonallie.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
#
#
#
# ------------------------------ Modifications ----------------------------
#
# Ianaré Sévi <ianare@gmail.com> on 2006-10-10:
#
#   Combined various incarnations and patches of the original script
#   found here:  http://www.scalix.com/community/viewtopic.php?t=1922
#   into a fully working script. Huge thanks to all the original
#   contributors - I am but a shadow in their footsteps.
#
#   Added -r option and associated programming to allow rotation of backups
#   based on the day of the week (7 working backups)
#
###############################################################################



### main variables

MAILNODE="mailnode"
ROOT_BACKUP_DIR=/backup
SCALIX_DIR=/var/opt/scalix
SCALIX_BIN=/opt/scalix/bin
LOGFILE=/tmp/ombackup.log
USERFILE=/tmp/userfile.$$
DATE=`date +%Y%m%d`
PROCESS_BLOCK_SIZE=5
ROTATE_BACKUP=Y


### function declarations

function usage
{
    printf $"
Usage: ombackup [-h] [-b backup dir] [-d scalix data dir] [-s scalix bin dir]
                [-l logfile] [-u user file] [-m mailnode] [-r rotate backups (Y|N)]
                           
  ombackup comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
  are welcome to redistribute it under certain conditions.  See the GNU
  General Public Licence for details.
               
ombackup is a shell script to perform both user level and system level backups
of a Scalix mail server. User mailboxes are backed up via the 'omcpoutu' utility
and are stored in a configurable backup directory in a subdirectory named the
same as the mailnode being backed up. Systems level backups are performed by
copying the whole Scalix data dir (usually /var/opt/scalix) to a backup directory
using rysnc.

Most options can be configured by setting the values of the variables in the script
or can be passed to the script at runtime

Options:
    -h                  : print this message and exit
   
    -m <mailnode>       : mailnode to dump users from
   
    -b <backup dir>     : backup directory. This directory will store both the user and
                        system level backups. User backups are stored in a subdirectory
                        under this directory users/<mailnode>/<userfile>.
               
    -d <scalix dir>     : scalix data dir. Defaults to /var/opt/scalix
   
    -s <bin dir>        : scalix bin dir. Contains scalix utility binaries. Defaults to
                        /opt/scalix/bin
   
    -l <logfile>        : path to a logfile for logging backup actions.
   
    -u <userfile>       : userfile. This file is created during the user mailbox
                        backup. Defaults to /tmp/userfile.[pid]
                       
    -r <Y|N>          : wether or not to rotate backups on 7 day schedule.
                           
                           
Copyright (C) 2006 by Jon Allie <jon@jonallie.com>\n\n"

exit ${1:-0}
}


function badInput
{
    echo "Use -h for more information."
    echo
    exit 1
}


function log_it
{
    echo "[ `date` ]: $*" >>$LOGFILE

}

function echo_and_log
{
    echo $*
    log_it $*
}

function clean_up
{
    echo_and_log "Cleaning up temporary files"
    [ -f $USERFILE ] && rm -f $USERFILE
}


function exit_with_error
{
    echo_and_log "Error: $*"
    clean_up
    exit 1
}

function pre_check
{

    # look for scalix directories
    for dir in $SCALIX_BIN $SCALIX_DIR
    do
        [ -d $dir ] || exit_with_error "A required Scalix directory $dir doesn't exist."
    done


    # make sure that the $BACKUP_DIR structure exists, try to create it if not.   
    for dir in $BACKUP_DIR $BACKUP_DIR/users $BACKUP_DIR/users/$MAILNODE
    do
        if [ ! -d $dir ]
        then
            echo_and_log "$dir doesn't exist: creating it"
            mkdir $dir || exit_with_error "Unable to create required directory $dir"
        fi
    done
   
    # clear out user backup files
    rm -rf $BACKUP_DIR/users/$MAILNODE/*

}

function dump_users
{
    #index for processing block
    let i=1
    let index=1

    # Build userfile
    $SCALIX_BIN/omshowu -m $MAILNODE|cut -f1 -d'/' >$USERFILE
    [ "$?" != "0" ] && exit_with_error "Unable to build userfile $USERFILE from mailnode $MAILNODE"

    # Loop over userfile and create backups. Use 'while read' instead of 'for' because of spaces in names
    while read sc_username
    do
        # Create a version of the username without spaces and other crappy characters
        nospaces=`echo $sc_username|sed -e "s/[ \.;=*'?_!]//g"`

        BACKUP_FILE="$BACKUP_DIR/users/$MAILNODE/${nospaces}-mail.gz"

        if [ $i -le $PROCESS_BLOCK_SIZE ]
        then
            echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE -- User: $sc_username"
            ## BACKGROUND PROCESS
            $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - -F | gzip | cat > $BACKUP_FILE || echo_and_log "Error: Unable to complete backup operation for $sc_username" &
            PIDs[$index]=$!
            let i+=1
            let index=$i
        else
            echo "Process block is full."
            echo "Waiting for first complete process..."
            while [ $i -gt $PROCESS_BLOCK_SIZE ]
            do
                for p in `seq 1 $PROCESS_BLOCK_SIZE`
                do
                        ps ${PIDs[$p]} > /dev/null
                        if [ "$?" != "0" ]
                        then
                                echo "Process number $p of $PROCESS_BLOCK_SIZE has completed. -- User: $sc_username"
                                unset PIDs[$p]
                                let index=$p
                                echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE -- User: $sc_username"
                                ## BACKGROUND PROCESS
                                $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - -F | gzip | cat > $BACKUP_FILE || echo_and_log "Error: Unable to complete backup operation for $sc_username" &
                                PIDs[$index]=$!
                                break 2
                        fi
                done
            done
        fi
    done < $USERFILE

echo "All processes have been added."
echo "Waiting for those still running..."
wait
echo "All done!"
}

function sync_files
{
    echo_and_log "Beginning rsync of $SCALIX_DIR to $BACKUP_DIR"
    rsync -av --delete $SCALIX_DIR $BACKUP_DIR/ >>$LOGFILE
   
    if [ "$?" != "0" ]
    then
        exit_with_error "Rsync operation of $SCALIX_DIR to $BACKUP_DIR did not complete successfully"
    else
        echo_and_log "Completed rsync of $SCALIX_DIR to $BACKUP_DIR"
    fi
}

# process command line arguments
# -h            : show help
# -b <dir>      : backup directory
# -l <file>     : log file
# -u <userfile> : userfile
# -m <mailnode> : main mailnode
# -d <dir>      : location of the scalix data dir
# -s <dir>      : location of the scalix bin dir
# -r <Y|N>      : rotate backups or not

while getopts hb:l:u:m:s:r: opt
do
    case "$opt" in
        h) usage ;;
        b) BACKUP_DIR=$OPTARG ;;
        l) LOGFILE=$OPTARG ;;
        u) USERFILE=$OPTARG ;;
        m) MAILNODE=$OPTARG ;;
        d) SCALIX_DIR=$OPTARG ;;
        s) SCALIX_BIN=$OPTARG ;;
        r) ROTATE_BACKUP=$OPTARG ;;
        \?) badInput ;;
    esac
done

# validate that all required options are set
for x in "$LOGFILE" "$ROOT_BACKUP_DIR" "$MAILNODE" "$SCALIX_DIR" "$SCALIX_BIN" "$USERFILE" "$ROTATE_BACKUP"
do
    if [ -z "$x" ]
    then
        echo "A required parameter is missing, please check your command arguments."
        badInput
    fi
done

# rotate backups or not
if [ "$ROTATE_BACKUP" = "Y" ]
    then
        DAYNUMBER=`date +%A`
        BACKUP_DIR=$ROOT_BACKUP_DIR/$DAYNUMBER
    else
        BACKUP_DIR=$ROOT_BACKUP_DIR
fi

# initialize the logfile
>$LOGFILE

# call pre_check function to verify backup directory structure
pre_check

# shutoff remote client connections
echo_and_log "Shutting down remote client interface"
$SCALIX_BIN/omoff -d 0 rci
[ "$?" != "0" ] && exit_with_error "Unable to halt remote client interface"

# call dump_users function to make backups of user mailboxes
dump_users

#echo_and_log "Stopping scalix services"
/etc/init.d/scalix stop
[ "$?" != "0" ] && exit_with_error "Unable to halt scalix services"

# call sync_files function to make a backup of the $SCALIX_DIR
sync_files

# restart scalix services
echo_and_log "Starting Scalix services"
/etc/init.d/scalix start
[ "$?" != "0" ] && exit_with_error "Error restarting scalix services"

# explicily call the clean_up function to erase leftover files
clean_up

# exit successfully
echo_and_log "All operations complete"
exit 0


[ianare]

edit:
go to the wiki to get the script. Sorry for all the posts. Can a moderator my post on page 6?

http://www.scalix.com/wiki/index.php?ti ... ckupScript

[dkelly]

Can someone add this to the Scalix wiki and maintain it from there ?

See http://www.scalix.com/wiki/index.php?title=How-Tos

Cheers

Dave