Backup/Restore Scripts - updated with restore script

[BaldBoy]

Based upon previous examples we've been writing down our own version of the export/backup script.
What we wanted to achieve was:

• Better management of parallel processes: in previous versions of the script each process pool was awaited for it's complete end, first for users and then for folders. In "our" version the process pool is "global" so if export of users is almost done and leaves a queue or more free then the script begins exporting public folders.
• Checking of free process slot was tough intense and required in internal array for checking. In our version checking is more "gentle" and can be set up a wait time between checks
• Export rotation was based on daynames. In our version directory tree is slightly different (mailnode comes first) and is based upon dates in the form YYYYMMDD. Number of rotated exports is a count back in days: older exports previous than <current_date> minus <number of exports to keep> are deleted. Of course we use this to backup on a regular daily basis.
• We felt the need of an email report

I post it here in the hope might help someone.
Feel free to suggest improvements or bug fixes.

Code: Select all

#!/bin/bash
##############################################################################
## IMPORTANT: This script is not supported by Scalix. Use at your own risk. ##
##############################################################################
#
# NAME:      sxstoreexp - Perform a full/partial export of a Scalix Server
#           message store
#
# SYNOPSIS:  sxstoreexp
#
# DESCRIPTION:
#  Through the meaning of this script you'll be able to perform a full or
#  partial export of the contents of your Scalix Message Store.
#  This script is useful for disk backups of Scalix's Storage area and is
#  intended to minimize disk space occupation.
#
#  When job is done a detailed report is sent to system administrator
#  or any other email address you might find useful.
#
#  Derived from:
#  http://www.scalix.com/wiki/index.php?title=HowTos/AutoBackupScript
#  http://www.scalix.com/community/viewtopic.php?t=1922
#
#  Give credit to the following contributors:
#  (please do not remove - add your name if you contribute)
#
#  * Jon Allie <jon@jonallie.com>
#  * Ianaré Sévi <ianare@gmail.com>
#  * Dirk Ahrnke <da@it25.de>
#  * Andrea Lanfranchi (BaldBoy) <andrea.lanfranchi@isiweb.it>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
#
# Those who do not know how to use snail mail: The GPL is here:
# http://www.gnu.org/licenses/gpl.html
#
###############################################################################


#
#   Variables:
#   Following variables declaration can be fulfilled with constants. In this
#   case you will be able to invoke the command without specifying related
#   mandatory parameters. In either case specifying parameters from command
#   line bring variable constants to be overwritten (command line arguments
#   are MORE IMPORTANT than values specified here).
#

#   Target directory where to place exported items
#   (eg. /tmp/sxstorebackup)
#   Note that many subdirs will be created below the target you specify here
#   Related command argument switch -t | --target
#
    EXP_TARGET=/home/Isiweb/ScalixBackup

#   Do you want to export users' mailboxes ? By setting following value to Y
#   your request will be satisfied. :)
#
    EXP_USERS=Y    # Y | N

#   Do you also want to export Public Folder's content ? If so then set
#   the following to Y
#
    EXP_PUBLIC=Y   # Y | N

#   If you plan to use this script for backup purpouses you might find
#   useful to rotate backup plans. The following setting indicates how
#   many exports you want to keep on disk. Every export will be placed
#   in a subdirectory of EXP_TARGET named in the form YYYYMMDD. When a
#   directory date is older than <current-date> minus number of days
#   set in EXP_ROTATE parameter then that directory will be deleted.
#   Keep in mind that every export may consist of a huge amount
#   of data so ... take care of your disk space. YOU'RE WARNED!!
#   A value of 0 (zero) means no rotation.

    EXP_ROTATE=7

#   Do you want a mail report of the job to be delivered to someone ?
#   Say that here. Please note that multiple email address may be specified,
#   as separated by a comma. An empty value means no report is sent.
#
    EXP_MAIL_REPORT_FROM="webmaster@isiweb.it"
    EXP_MAIL_REPORT_TO="webmaster@isiweb.it"
    EXP_MAIL_REPORT_SUBJECT="Scalix Store Export Report"

#   This task can perform multiple parallel exports. This value sets how
#   many export processes can be run at the same time.
#   Note : increasing this number do not necessarily mean an equivalent
#   increase of speed of the export tasks. Use with care, only if your
#   server is well equipped and if you're exporting on target which is
#   located on hardware volume(s) different from the one(s) hosting
#   Scalix Storage Area.
#
    EXP_PROCESSES=3

#   Scalix's bin directory. This is where many of the Scalix utilities and
#   commands live. Unless you're not on a standard Scalix installation
#   do NOT change this value
#
    SCALIX_BIN=/opt/scalix/bin

#
#   Logging Facilities pointers
#   You should not need to change these
#
    STDLOG=/tmp/sxstoreexp.$$
    STDERR=/tmp/sxstoreexp.$$.stderr
    STDOUT=/tmp/sxstoreexp.$$.stdout

#  --------------------------------------------------------------------------
#  Here the program begins! DO NOT CHANGE ANYTHING BELOW THIS LINE UNLESS YOU
#  KNOW WHAT YOU'RE DOING. Obviously you can look at the code and learn how
#  much fun there's inside Linux
#  --------------------------------------------------------------------------

#  --------------------------------------------------------------------------
#  Begin Functions
#  --------------------------------------------------------------------------
function show_usage ()
{
   echo "
Help not vailable yet
"
   exit 1
}

function show_params ()
{

   trace_std "
------------------------------------------------------------------------------
   Running Params
   - Scalix Directory ..... : $SCALIX_BIN
   - Target Directory ..... : $EXP_TARGET
   - Export User's Mboxes . : $EXP_USERS
   - Export Public Folders  : $EXP_PUBLIC
   - Paralell Exp Processes : $EXP_PROCESSES
   - Rotate Exports         : $EXP_ROTATE
------------------------------------------------------------------------------
"
}

function pre_check ()
{

    # Look for mandatory directories
    for DIR in $SCALIX_BIN $EXP_TARGET
    do
   [ -d $DIR ] || trace_err "Required directory missing : $DIR"
    done

}

function init()
{
    # Extract default mailnode to extract from
    SCALIX_MN=`$SCALIX_BIN/omshowmn | grep '**' | awk '{ print $2 }'`
    eval set -- "$SCALIX_MN"

    # Append to EXP_TARGET current MailNode
    EXP_TARGET_FULL=$EXP_TARGET/$SCALIX_MN
    if [ ! -d $EXP_TARGET_FULL ]
    then
   mkdir $EXP_TARGET_FULL
    fi

    # Append to EXP_TARGET_FULL current Date
    DAYNUMBER=`date +%Y%m%d`
    EXP_TARGET_FULL=$EXP_TARGET_FULL/$DAYNUMBER
    if [ ! -d $EXP_TARGET_FULL ]
    then
   mkdir $EXP_TARGET_FULL
    else
        trace_std "Cleaning up $EXP_TARGET_FULL"
   rm -dfr $EXP_TARGET_FULL
   mkdir $EXP_TARGET_FULL
   wait
    fi

    # Create Subfolders for objects storage
    mkdir $EXP_TARGET_FULL/Users
    mkdir $EXP_TARGET_FULL/Public_Folders
    wait

   # If we have to rotate exports than let's calculate
   # which is the limit date for which we have to keep
   # exports on disk

    if [ "$EXP_ROTATE" != "0" ]
    then
   DATEFROM=`date -d "$EXP_ROTATE days ago" +%Y%m%d`
   ls $EXP_TARGET/$SCALIX_MN/ | grep '[0-9]' | awk '{ if (int($1)<=int('$DATEFROM')) { print $1 }}' | while read OLDDIR
   do
      # Loop through each directory which is older than $DATEFROM date
      trace_std "Removing old export directory : $OLDDIR"
      rm -dfr $EXP_TARGET/$SCALIX_MN/$OLDDIR
      wait
   done
    fi
}

function trace_err ()
{
    echo "$1"
    echo "`date +%x\ %X` $1" >> $STDERR
}

function trace_std ()
{
    echo "$1"
    echo "`date +%x\ %X` $1" >> $STDLOG
}

function exp_users()
{

    # setup directory for output
    DIR=$EXP_TARGET_FULL/Users

    # Build users's list and loop through their names
    $SCALIX_BIN/omshowu -m $SCALIX_MN|cut -f1 -d'/' | while read sc_username
    do

        # Create a version of the username without spaces and other crappy characters
        nospaces=`echo $sc_username|sed -e "s/[ \.;=*'?_!]//g"`

   # Build file name for user's mailbox
        BACKUP_FILE="$DIR/${nospaces}.mbox.gz"

   # Try to figure out how many processes slot we have free
   # Be kind ... do not hammer
   sleep 2
   PIDS=$(pgrep sxmboxexp | grep '[0-9]' -c)

   # Hmmm ... we have to wait until number of running
   # processes gets below $EXP_PROCESSES watermark.
   while [ $PIDS -ge $EXP_PROCESSES ]
   do
      # Be kind ... do not hammer
      sleep 2
      PIDS=$(pgrep sxmboxexp | grep '[0-9]' -c)
   done

   # If we have a process slot free then submit new queue
   # Export user's mailbox using sxmboxexp
   trace_std "Mailbox $SCALIX_MN/$sc_username : export queued ..."
   $SCALIX_BIN/sxmboxexp -u "$sc_username/$SCALIX_MN" -a - -F | gzip -c > $BACKUP_FILE || trace_std "Error: mailbox $SCALIX_MN/$sc_username" &

    done

    # All queues submitted
    trace_std "All mailbox export queues submitted ... "

}

function exp_folders()
{

    # create directory for output
    DIR=$EXP_TARGET_FULL/Public_Folders
    IDX=$EXP_TARGET_FULL/Public_Folders/index.txt

    touch $IDX

    # Build PF list and loop through their names
    $SCALIX_BIN/sxmboxlist --public $SCALIX_MN | grep '^F-' | sort | while read pfid pfname
    do

   # Build file name for user's mailbox
        BACKUP_FILE="$DIR/$pfid.pflr.gz"

   # Try to figure out how many processes slot we have free
   # Be kind ... do not hammer
   sleep 2
   PIDS=$(pgrep sxmboxexp | grep '[0-9]' -c)

   # Hmmm ... we have to wait until number of running
   # processes gets below $EXP_PROCESSES watermark.
   while [ $PIDS -ge $EXP_PROCESSES ]
   do
      # Be kind ... do not hammer
      sleep 2
      PIDS=$(pgrep sxmboxexp | grep '[0-9]' -c)
   done

   # If we have a process slot free then submit new queue
   # Export user's mailbox using sxmboxexp
   trace_std "Folder $SCALIX_MN/$pfname : export queued ..."
        echo "$pfid --> $pfname" >> $IDX
   $SCALIX_BIN/sxmboxexp -p -f $pfid -a - -F | gzip -c > $BACKUP_FILE || trace_std "Error: folder $SCALIX_MN/$pfname" &

    done

    # All queues submitted
    trace_std "All folders export queues submitted ... "

}

#  --------------------------------------------------------------------------
#  End Functions
#  --------------------------------------------------------------------------


if [ "$#" -eq 0 ]
then   # No given options/arguments - running using hardcoded values

  trace_std "No given params. Using hardcoded ones."

else

  #  process command line arguments
  #  -h | --help         : show help
  #  -t | --target <dir>   : target export
  #  -u | --users <Y|N>      : export users
  #  -p | --public <Y|N>   : export public folders

  TEMP=`getopt -o ht:u::p: --long 'help',target:,users::,public: -n "$0" -- "$@"`
  if [ $? != 0 ] ; then show_usage ; exit 1 ; fi
  eval set -- "$TEMP"

  while true ; do
   case "$1" in
      -h|--help) show_usage ; shift ;;
      -t|--target) EXP_TARGET=$2; shift 2;;
      -u|--users)
         case "$2" in
            "") EXP_USERS=Y; shift 2;;
            "Y"|"N") EXP_USERS=$2; shift 2;;
            *) show_usage;;
         esac;;
      -p|--public)
         case "$2" in
            "") EXP_PUBLIC=Y; shift 2;;
            "Y"|"N") EXP_PUBLIC=$2; shift 2;;
            *) show_usage;;
         esac;;
      --) shift ; break ;;
      *) echo "Internal error!" ; exit 1 ;;
   esac
  done

fi

# Show user which params will be used to perform this export
show_params

# Initialize logging facilities
rm -f $STDLOG $STDERR $STDOUT
touch $STDLOG $STDERR $STDOUT

# Check if everything is ok
pre_check

# Initialize
init

# If we have any error then exit


# We can now begin to export users' mailboxes
if [ "$EXP_USERS" = "Y" ]
then
   exp_users
fi

# Do the same with public folders if required
if [ "$EXP_PUBLIC" = "Y" ]
then
   exp_folders
fi

# Loop through active processes to monitor completion
PIDS=$(pgrep sxmboxexp | grep '[0-9]' -c)
while [ $PIDS != 0 ]
do
# Be kind ... do not hammer
sleep 2
PIDS=$(pgrep sxmboxexp | grep '[0-9]' -c)
done

# Job done !
trace_std "All queues completed!"

# Report Usage on Disk for this export
echo "
Report of disk usage after export operation
" >> $STDLOG
du $EXP_TARGET_FULL -h -s >> $STDLOG

# Report Usage on Disk for target directory
echo "
Report of disk usage for export target directory
" >> $STDLOG
du $EXP_TARGET -h -s >> $STDLOG

# Send Report to Admin(s)
if [ -s $MAINTLOG ]
then
   mail -r $EXP_MAIL_REPORT_FROM -s "$($SCALIX_BIN/omcheckgc -h) - $EXP_MAIL_REPORT_SUBJECT" $EXP_MAIL_REPORT_TO < $STDLOG
fi

# Clean up logging facilities
rm -f $STDLOG $STDERR $STDOUT


[Fution_]

Hi

I ran the backup script and this went through gr8. Tryed to do the restore script and now sitting with the following problem.
The restore for the single user goes through but whe you look in the Scalix admin module the user that was restored is not there. Please help.
PS how do i resore the whole backup that was created?

Regards

[Mon1018]

Hi, This post is very informative, however I would like some specific information. If someone can help me then please send me a private message. Best Regards,

[BaldBoy]

Post your issues here. Maybe others have already solved your problems.

[Hicham]

Hi all,

I want to know how to restore the public folder ?

Thanks ,

Hicham

[kennyguo]

on the commercial side of things and GUI support for backup, we obviously can't beat Exchange; nevertheless, the eco-system how we like to call it, is building. For backup, you might want to check out german vendor SEP (www.sep.de, also selling in the US through RICIS who also happen to be one of our distributors) and their product Sesam that have a Scalix agent. V1 of this, which is currently on the market, is still somewhat rough, but we're working together with them (being based out of Germany, I'm actually managing the relatioonship on a technical side) to create a solution that allows for a fully GUI-driven single item (i..e single Mail message) restore.

[charon]

The Script back-ups the primary mailnode users and all public folders
how do i get the other mailnodes user data stored?

[TCWardrobe]

I had an issue with users not getting backed up because there were some accounts with the very same name. This is because I have some users who need different identities but with the same name, so I changed BaldBoys script at two points to take the common name rather than the username as an argument to smboxexp. Here are the changes:

Code: Select all

    # Build users's list and loop through their names
    $SCALIX_BIN/omshowu -m $SCALIX_MN|cut -f3 -d'/' | while read sc_username

and

Code: Select all

   # Export user's mailbox using sxmboxexp
   trace_std "Mailbox $SCALIX_MN/$sc_username : export queued ..."
   $SCALIX_BIN/sxmboxexp -u "$sc_username" -a - -F | gzip -c > $BACKUP_FILE || trace_std "Error: mailbox $SCALIX_MN/$sc_username" &

It is rather a dirty hack but it works. This may break other areas of the script that I am not aware of (I think handling different mailnodes is one thing), so I would appreciate comments on this.

Btw. it it possible to write a ncurses like restore programm whcih would allow to restore wanted data in for example a public folder? Or how do you folks restore with these scripts just parts of a users data?

greetings
Michael

[charon]

Nobody has an idea how to backup secondary mailnodes?

Here is the Original Code:

Code: Select all

# Extract default mailnode to extract from
    SCALIX_MN=`$SCALIX_BIN/omshowmn | grep '**' | awk '{ print $2 }'`
    eval set -- "$SCALIX_MN"

[ITCons]

In regard to the activity on this thread and the very good script written for backups of the Scalix server, I had the task to do a migration from one Scalix version to another.

So I used the backup script, since we are running version 11 of Scalix on all installations. After that, I needed to import all mailboxes into the new Server.

This is where I didnt really find a suitable solution, which is where I wrote a small script that takes the exported mailboxes and imports them into the new server version, adding users and mails. The only thing there is to do after the import is to change the mailbox password at the first login. So here it goes...

Code: Select all

#!/bin/bash
################################################################################
# omrestore:
#   a utility to restore a whole Mailserver after a version upgrade
#   use at own risk!
#
#   This script can be run to import the previously exported mailboxes with the
#   ombackup.sh script
#
#   Before using this program you should set the values of the variables
#   below to match your server/preferences.
#
#   Copyright (C) 2006 Jon Allie <jon@jonallie.com>
#   Copyright (C) 2008 Adrian Oliver Schmid <me@ascsoftware.ch>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
#
#
################################################################################

### useful variables
BACKUP_DIR=/backup/location/Users
SCALIX_DIR=/var/opt/scalix
SCALIX_BIN=/opt/scalix/bin
RESTORE_MN="scalix"
RESTORE_DOMAIN="domain.com"
LOGFILE=$BACKUP_DIR/omrestore.log
SECRET="password"
DATE=`date +%Y%m%d`

### function declarations

function log_it
{
    echo "[ `date` ]: $*" >>$LOGFILE

}

function echo_and_log
{
    echo $*
    log_it $*
}

function clean_up
{
    [ -f $USERFILE ] && rm -f $USERFILE
}


function exit_with_error
{
    echo_and_log "Error: $*"
    clean_up
    exit 1
}

function validate_mailnode
{
    ## check to see if the restore node exists. If not, create it

    $SCALIX_BIN/omshowmn -m $RESTORE_MN >/dev/null 2>&1

    if [ "$?" != "0" ]
    then
        echo_and_log "Restore mailnode $RESTORE_MN doesn't exist"
        echo_and_log "Creating restore mailnode $RESTORE_MN"

        $SCALIX_BIN/omaddmn -m $RESTORE_MN >/dev/null 2>&1

        if [ "$?" != "0" ]
        then
            exit_with_error "Unable to create restore mailnode $RESTORE_MN"
        else
            echo_and_log "Successfully created restore mailnode $RESTORE_MN"
        fi
    else
        echo_and_log "Restore mailnode $RESTORE_MN exists"
    fi
}


function ask
{
          #
          # ask a question with a yes/no answer;
          # Usage: ask [-y|-n] "question"
          # Set default: -y = yes; -n = no; otherwise no default;
          # Returns: 0 (true) = yes; 1 (false) = no;
          # Note: changing the default does not effect the return value;
          #
          ASK_DFLT=
          # process options/args
          for ASK_OPT do
                case "$ASK_OPT" in
                  -[yY]*) ASK_DFLT='y' ;;
                  -[nN]*) ASK_DFLT='n' ;;
                  --) shift; break ;;
                  -*) ;;
                  *) break
                  esac
                shift
            done
          ASK_PROMPT="$*"

          # get the response
         while : ; do
                echo "$ASK_PROMPT (y/n)?${ASK_DFLT:+ [$ASK_DFLT]} " >&2
                read ASK_ANSWER ASK_JUNK

                : ${ASK_ANSWER:=$ASK_DFLT}

                case "$ASK_ANSWER" in
                  [yY]*)
                        return 0
                        ;;
                  [nN]*)
                        return 1
                        ;;
                  *)
                        echo "ASK" >&2
                  esac
            done

          return
        }


##### main program loop begins

# initialize the logfie
>$LOGFILE

# check for the scalix bin directory
if [ -d $SCALIX_BIN ]
then
    [ -f $SCALIX_BIN/omcpinu ] || exit_with_error "Unable to locate scalix binaries in $SCALIX_BIN"
else
    exit_with_error "Scalix binary directory $SCALIX_BIN doesn't exist."
fi

# call the validate_mailnode function to verify and/or create the restore mailnode
validate_mailnode

for file in $BACKUP_DIR/*; do
  if [ ${file##*.} == "gz" -o ${file##*.} == "mbox" ]; then
    echo $file; # or whatever you want to do

    echo "About to import User mailbox:" $file
    if ask -y "Do you want to import this Mailaccount?"
    then
      echo "Please enter the Users Full Name (with upper/lower case):"
      read UserFullName
      echo "Please enter First name (lower case only):"
      read FirstName
      echo "Please enter Last name (lower case only):"
      read LastName

      # Add the user via command line
      $SCALIX_BIN/omdelu -n "$UserFullName/$RESTORE_MN"
      $SCALIX_BIN/omaddu -n "$UserFullName/$RESTORE_MN"  -p $SECRET -e
      $SCALIX_BIN/ommodu -o "$UserFullName/$RESTORE_MN" -n "$UserFullName/$RESTORE_MN/IA=\"$UserFullName\" \
          <$FirstName.$LastName@$RESTORE_DOMAIN>=\"$UserFullName\" <$FirstName@$RESTORE_DOMAIN>"

      # Now unzip the file and import the account
      if [ ${file##*.} == "gz" ]; then
        gzip -d "$file" | omcpinu -f -
        # We need to cut the last 3 characters off from $file!!
        fileextracted=${file%.*}
        $SCALIX_BIN/sxmboximp -u "$UserFullName" -a "$fileextracted" -s --listlevel folder
      else
        if [ ${file##*.} == "mbox" ]; then
         $SCALIX_BIN/sxmboximp -u "$UserFullName" -a "$file" -s --listlevel folder
        fi
      fi
    fi
  fi
done

# exit successful
echo_and_log "All tasks completed successfully."
exit 0




Feel free to comment or add functionality

***************************
Adrian Oliver Schmid
Linux System Administrator LPIC I
http://www.lpi.org/

Friends dont let friends do Windows.