Backup/Restore Scripts - updated with restore script

Discuss the Scalix Server software

Moderators: ScalixSupport, admin

ja30278

Backup/Restore Scripts - updated with restore script

Postby ja30278 » Wed Mar 29, 2006 4:53 pm

Hi All,

I'm in the process of implementing Scalix and (like lots of people here) I was disappointed with the backup and restore options, especially for single users. I've had enough problems with LVM snapshots that I don't trust them for my production backups, and the technote for ISS and SUR just made my head hurt. With that in mind I've spent the past day or so working on backup/restore scripts for my scalix install and I thought I'd post them here in the hope that someone else would be able to use them, or, better yet, improve on them. I haven't quite finished the restore script, but the backup script is in a workable state. Ive tested this multiple times on my semi-production Scalix 10.0 server, but I make no guarantees on its use.

Before using it, you'll need to edit the variable section at the top of the file to match your setup (at minimum, you'll need to specify your mailnode) . Running the script with the '-h' option should give instructions on passing command arguments.

This script will create the following folder structure for storing backups

Code: Select all

$BACKUP_DIR/
        |----------------users/
        |                           |----------------[mailnode]
        |                                                         |---------userfile.gz
        |                                                         |---------userfile2.gz
        |
        |-----------------[rsync of scalix data]





Code: Select all

#!/bin/sh
###############################################################################
# ombackup:
#   a backup script for scalix mail servers
#
#   This script is used to backup Scalix mail servers; it exports each
#   user to a gzip compressed file using the 'omcpoutu' command, then
#   duplicates the scalix data directory using rsync.
#
#   Before using this program you should set the values of the variables
#   below to match your server/preferences.
#
#   For detailed descriptions of the available command line switches,
#   execute the program with the -h flag.
#
#
#   Copyright (C) 2006 Jon Allie <jon@jonallie.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
#
#
###############################################################################


### useful variables
MAILNODE=""
BACKUP_DIR=/backup
SCALIX_DIR=/var/opt/scalix
SCALIX_BIN=/opt/scalix/bin
LOGFILE=/tmp/ombackup.log
USERFILE=/tmp/userfile.$$
DATE=`date +%Y%m%d`

### function declarations

function usage
{
    printf $"
Usage: ombackup [-h] [-b backup dir] [-d scalix data dir] [-s scalix bin dir]
                [-l logfile] [-u user file] [-m mailnode]
                           
  ombackup comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
  are welcome to redistribute it under certain conditions.  See the GNU
  General Public Licence for details.
               
ombackup is a shell script to perform both user level and system level backups
of a Scalix mail server. User mailboxes are backed up via the 'omcpoutu' utility
and are stored in a configurable backup directory in a subdirectory named the
same as the mailnode being backed up. Systems level backups are performed by
copying the whole Scalix data dir (usually /var/opt/scalix) to a backup directory
using rysnc.

Most options can be configured by setting the values of the variables in the script
or can be passed to the script at runtime

Options:
    -h                  : print this message and exit
   
    -m <mailnode>       : mailnode to dump users from
   
    -b <backup dir>     : backup directory. This directory will store both the user and
                        system level backups. User backups are stored in a subdirectory
                        under this directory users/<mailnode>/<userfile>.
               
    -d <scalix dir>     : scalix data dir. Defaults to /var/opt/scalix
   
    -s <bin dir>        : scalix bin dir. Contains scalix utility binaries. Defaults to
                        /opt/scalix/bin
   
    -l <logfile>        : path to a logfile for logging backup actions.
   
    -u <userfile>       : userfile. This file is created during the user mailbox
                        backup. Defaults to /tmp/userfile.[pid]
                           
                           
Copyright (C) 2006 by Jon Allie <jon@jonallie.com>\n\n"



exit ${1:-0}

}


function log_it
{
    echo "[ `date` ]: $*" >>$LOGFILE

}

function echo_and_log
{
    echo $*
    log_it $*
}

function clean_up
{
    echo_and_log "Cleaning up temporary files"
    [ -f $USERFILE ] && rm -f $USERFILE
}


function exit_with_error
{
    echo_and_log "Error: $*"
    clean_up
    exit 1
}

function pre_check
{

    # look for scalix directories
    for dir in $SCALIX_BIN $SCALIX_DIR
    do
        [ -d $dir ] || exit_with_error "A required Scalix directory $dir doesn't exist."
    done


    # make sure that the $BACKUP_DIR structure exists, try to create it if not.
    for dir in $BACKUP_DIR $BACKUP_DIR/users $BACKUP_DIR/users/$MAILNODE
    do
        if [ ! -d $dir ]
        then
            echo_and_log "$dir doesn't exist: creating it"
            mkdir $dir || exit_with_error "Unable to create required directory $dir"
        fi
    done

}

function dump_users
{

    # build userfile
    $SCALIX_BIN/omshowu -m $MAILNODE|cut -f1 -d'/' >$USERFILE
    [ "$?" != "0" ] && exit_with_error "Unable to build userfile $USERFILE from mailnode $MAILNODE"


    # loop over userfile and create backups. Use 'while read' instead of 'for' because of spaces in names

    while read sc_username
    do

        nospaces=`echo $sc_username|sed -e 's/[ \.]//g'` # create a version of the username with spaces removed to use in the filename
        backup_file="$BACKUP_DIR/users/$MAILNODE/${nospaces}-${DATE}-mail.gz"   
           
        $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - |gzip >$backup_file
        if [ "$?" != "0" ]
        then
            echo_and_log "Error: unable to complete backup operation for $sc_username ot $backup_file"
        else
            echo_and_log "Success: backed up $sc_username to $backup_file"
        fi
   
    done < $USERFILE
   
}

function sync_files
{
    echo_and_log "Beginning rsync of $SCALIX_DIR to $BACKUP_DIR"
    rsync -av --delete $SCALIX_DIR $BACKUP_DIR/ >>$LOGFILE
   
    if [ "$?" != "0" ]
    then
        exit_with_error "Rsync operation of $SCALIX_DIR to $BACKUP_DIR did not complete successfully"
    else
        echo_and_log "Completed rsync of $SCALIX_DIR to $BACKUP_DIR"
    fi
}



# process command line arguments
# -h            : show help
# -b <dir>      : backup directory
# -l <file>     : log file
# -u <userfile> : userfile
# -m <mailnode> : main mailnode
# -d <dir>      : location of the scalix data dir
# -s <dir>      : location of the scalix bin dir

while getopts hb:l:u:m:s: opt
do
    case "$opt" in
        h) usage ;;
        b) BACKUP_DIR=$OPTARG ;;
        l) LOGFILE=$OPTARG ;;
        u) USERFILE=$OPTARG ;;
        m) MAILNODE=$OPTARG ;;
        d) SCALIX_DIR=$OPTARG ;;
        s) SCALIX_BIN=$OPTARG ;;
        \?) usage ;;
    esac
done

# validate that all required options are set
for x in "$LOGFILE" "$BACKUP_DIR" "$MAILNODE" "$SCALIX_DIR" "$SCALIX_BIN" "$USERFILE"
do
    if [ -z "$x" ]
    then
        echo "A required parameter is missing: please check your command arguments"
        usage 1
    fi
done

# initialize the logfile
>$LOGFILE

# call pre_check function to verify backup directory structure
pre_check

# shutoff remote client connections
echo_and_log "Shutting down remote client interface"
$SCALIX_BIN/omoff -d 0 rci
[ "$?" != "0" ] && exit_with_error "Unable to halt remote client interface"

# call dump_users function to make backups of user mailboxes
dump_users

echo_and_log "Stopping scalix services"
/etc/init.d/scalix stop
[ "$?" != "0" ] && exit_with_error "Unable to halt scalix services"

# call sync_files function to make a backup of the $SCALIX_DIR
sync_files

# restart scalix services
echo_and_log "Starting Scalix services"
/etc/init.d/scalix start
[ "$?" != "0" ] && exit_with_error "Error restarting scalix services"

# explicily call the clean_up function to erase leftover files
clean_up

# exit successfully
echo_and_log "All operations complete"
exit 0

Last edited by ja30278 on Thu Mar 30, 2006 6:56 pm, edited 1 time in total.

cdclark
Posts: 73
Joined: Tue Mar 07, 2006 2:20 pm

Postby cdclark » Wed Mar 29, 2006 5:46 pm

Well I for one very much appreciate your efforts. I've been trying to figure out how I would backup Scalix on a production server for a couple of days. Naturally I went to the Scalix 10 Adminsitration guide, which suggests using mirrordir from sourceforge. Well, apparently that project no longer exists at sf.net, and I don't really want to chase down a version that might not be the right one. So next I consulted the KB, which suggests using LVM snapshots, which are totally new to me. That's not a huge issue, but I don't really have time right now to learn all there is to know about LVM.

Can you clarify one thing about your script? It looks to me like its doing a full rsync snapshot of the /var/opt/scalix directory, AND THEN making separate per-user archives? I presume the latter is for easier single-user restore?

ja30278

Postby ja30278 » Wed Mar 29, 2006 6:06 pm

cdclark wrote:Can you clarify one thing about your script? It looks to me like its doing a full rsync snapshot of the /var/opt/scalix directory, AND THEN making separate per-user archives? I presume the latter is for easier single-user restore?



Yep. It actually makes the single user dumps first (since scalix actually has to be running during this process). Then it shuts down the scalix services and rysncs the entire data directory.

In my experience, almost all restores are of the "I accidentally deleted this message" variety rather than a total bare metal restore. I'm not sure if you've read the technote on SIngle User restore using the Item Structure server, but the process for restoring a single user from a complete backup of the scalix directory is......daunting. My intention is to use the mailbox dumps for most restores and the full rsyncs for disaster recovery. Obviously I'll be backing up the /backup to tape for long term storage.

After running this script, you end up with a /backup directory that contains a 'scalix' dir that is an exact mirror of /var/opt/scalix and a 'users' dir that has a subdirectory named the same as your mailnode that contains your mailbox backups, one per user, with the filename <user>-<date>-mail-gz.

There are instructions elsewhere on the forum for restoring from a single user backup, but basically it involves creating a 'restore' mailnode, creating restore user in that mailnode with the same CN as the user you're restoring, and using 'omcpinu' to load the mail into that mailbox.

The restore script I'm working on will basically allow you to run 'omrestore -u "User Name" -f <mail file>' to load a users mail into a termporary restore mailbox (auto creating the restore mail node and user as needed). From there you can copy individual messages as to the existing account, or give users access to retrieve messages themselves. I'd also like to include the capability to wholesale overwrite a users mailbox with the backup, but it seems to me that retrieving individual deleted messages will be far more common (at least for me).

cdclark
Posts: 73
Joined: Tue Mar 07, 2006 2:20 pm

Postby cdclark » Wed Mar 29, 2006 6:19 pm

Yeah I realized the true order of operations after I posted. I configured and ran the script and it appears to do what its supposed to.

I look forward to your restore script as well.

axsom1
Posts: 69
Joined: Tue Aug 17, 2004 12:31 pm

Postby axsom1 » Wed Mar 29, 2006 6:28 pm

Do you not run the ISS service then? I would presume you don't.

I also thank you for the efforts you've put forth. I re-read the ISS and SUR technote every few months and it still continues to confuse the heck out of me (we've been live for over 18 months).

I also tried LVM snapshots but quickly learned there is/was a bug with snapshots and RHEL4 (I haven't updated to update 3 yet to see if snapshots are working).

I look forward to testing your script out and having a look at your restore solution as well.

Regards,
John

ja30278

Postby ja30278 » Wed Mar 29, 2006 6:58 pm

axsom1 wrote:Do you not run the ISS service then? I would presume you don't.

I also thank you for the efforts you've put forth. I re-read the ISS and SUR technote every few months and it still continues to confuse the heck out of me (we've been live for over 18 months).

I also tried LVM snapshots but quickly learned there is/was a bug with snapshots and RHEL4 (I haven't updated to update 3 yet to see if snapshots are working).

I look forward to testing your script out and having a look at your restore solution as well.

Regards,
John



Nope. I'm not running the ISS service. I felt guilty for awhile since the technote makes it seem like you _ought_ to be running it, but after the 10th time reading the ISS/SUR technote I realized it wouldn't do me any good to run it if I couldn't understand how to restore from it. It also seemed difficult to get to an individual message using ISS.

I had the same issues with LVM (CentOS) which is what started me down this path.

Can anyone shed some light on omsuspend? do you usually background it? It just seems to 'block' on the termininal for whatever the '-s' argument is. Plus, if the suspend is limited to the time given by the -s argument, why would you need to resume with -r? I had orginally intended to suspend during the backup rather than shutdown the service, but omsuspend seemed a little lacking.

axsom1
Posts: 69
Joined: Tue Aug 17, 2004 12:31 pm

Postby axsom1 » Wed Mar 29, 2006 7:27 pm

I think omsuspend is more (based on the examples) for syncing disks when using a software RAID or for creating a consistent snapshot.

My guess is if you don't want to stop the services then snapshots would need to be utilized.

John

ja30278

Backup/Restore Scripts - Updated wih restore script

Postby ja30278 » Thu Mar 30, 2006 6:54 pm

OK,

Here's my restore script. It isn't perfect (please feel free to improve) but it is certainly functional. I've tested this my server but I totally disclaim any unpleasant results you may experience. Note that this script will happily delete users WITHOUT prompting, so use with care.

Basically this is designed to automate the process of creating a restore mailnode, creating the restore user, and loading the data using omcpinu. You can also use this to do a full restore of a user's mailbox by setting the mailnode to your real 'production' mailnode. Note that this will delete and recreate the user, so make certain that the values for auth ID and internet address are correct if using this functionality.

As with the backup script you'll need to populate the variables section to match your values before running. $RESTORE_NAME and $RESTORE_FILE would typically always be passed on the command line.

I've tried to make it fairly 'self-documenting'; let me know if something isn't clear. I'm not a programmer by trade so suggestions and improvements are welcome. Personally I'd like to see a shared set of tools based on omcpoutu and omcpinu rather than ISS.

Code: Select all

#!/bin/bash
################################################################################
# omrestore:
#   a utility for single user restore on Scalix mail servers
#
#
#   Before using this program you should set the values of the variables
#   below to match your server/preferences.
#
#   For detailed descriptions of the available command line switches,
#   execute the program with the -h flag.
#
#
#   Copyright (C) 2006 Jon Allie <jon@jonallie.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
#
#
################################################################################

### useful variables
BACKUP_DIR=/backup
SCALIX_DIR=/var/opt/scalix
SCALIX_BIN=/opt/scalix/bin
RESTORE_ID="restore-user@example.com"
RESTORE_MN="restore"
RESTORE_PASS="restore"
RESTORE_FILE=""
RESTORE_USER=""
RESTORE_IAM="restore-user@example.com"
LOGFILE=/tmp/omrestore.log
DATE=`date +%Y%m%d`

### function declarations

function usage
{
    printf $"
Usage: omrestore -u \"User Name\" -f mailfile [-m mailnode] [-i auth id]
                 [-a internet address] [-l logfile] [-h]

     omrestore comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
     are welcome to redistribute it under certain conditions.  See the GNU
     General Public Licence for details.

omrestore is a shell script for restoring a single user's mailbox from a data file
(in the format produced by the 'omcpoutu' utiltiy) to a Scalix mail server. The restore
is performed using the 'omcpinu' utility provided by Scalix. Note that this utility WILL
completely delete the account with the name given WITHOUT PROMPTING. You should only use
this utility against your normal mailnode if this is the desired behaviour.

Most options can be configured by setting the values of the variables in the script;
however, The name of the user to be restored and the mail file to restore from will
usually be passed at runtime.

Options:

    -h              : print this message and exit
   
    -u <\"user\">     : Name of the user you're restoring. This name MUST match the
                    name that the backup file was created from. This user will be
                    created if it does not exist on the specified mailnode. If the
                    name contains spaces, it should be enclosed in quotes
                   
    -f <mail file>  : Mail file to read mailbox data from.
   
    -m <mailnode>   : Mailnode to restore to. Note that this should NOT be your
                    normal mailnode unless you want to completely overwrite the
                    users mailbox with the contents of the restore file. If this
                    mailnode doesn't exist, it will be created automatically.
               
    -i <auth id>    : Authentication ID of the restore user. ATTN: if this user
                    exists, it will be deleted.
                   
    -a <address>    : Internet address for the restore user
   
    -l <logfile>    : Path to a logfile for logging restore actions.             


Copyright (C) 2006 by Jon Allie <jon@jonallie.com>\n\n
"

exit ${1:-0}

}


function log_it
{
    echo "[ `date` ]: $*" >>$LOGFILE

}

function echo_and_log
{
    echo $*
    log_it $*
}

function clean_up
{
    [ -f $USERFILE ] && rm -f $USERFILE
}


function exit_with_error
{
    echo_and_log "Error: $*"
    clean_up
    exit 1
}

function validate_mailnode
{
    ## check to see if the restore node exists. If not, create it

    $SCALIX_BIN/omshowmn -m $RESTORE_MN >/dev/null 2>&1

    if [ "$?" != "0" ]
    then
        echo_and_log "Restore mailnode $RESTORE_MN doesn't exist"
        echo_and_log "Creating restore mailnode $RESTORE_MN"
   
        $SCALIX_BIN/omaddmn -m $RESTORE_MN >/dev/null 2>&1
   
        if [ "$?" != "0" ]
        then
            exit_with_error "Unable to create restore mailnode $RESTORE_MN"
        else
            echo_and_log "Successfully created restore mailnode $RESTORE_MN"
        fi
    else
        echo_and_log "Restore mailnode $RESTORE_MN exists"
    fi
}

function validate_user
{
    # destroys the restore user account (if it exists) and recreates it.
   
    $SCALIX_BIN/omshowu -n $RESTORE_ID >/dev/null 2>&1
    if [ "$?" = "0" ]
    then
        echo_and_log "Found existing user with ID $RESTORE_ID , deleting"
        $SCALIX_BIN/omdelu $RESTORE_ID
        [ "$?" != "0" ] && exit_with_error "Unable to delete existing user with ID $RESTORE_ID"
    fi

    $SCALIX_BIN/omshowu -n "$RESTORE_NAME/$RESTORE_MN" >/dev/null 2>&1
    if [ "$?" = "0" ]
    then
        echo_and_log "Found existing user with name $RESTORE_NAME on $RESTORE_MN , deleting"
        $SCALIX_BIN/omdelu -n "$RESTORE_NAME/$RESTORE_MN"
        [ "$?" != ")" ] && exit_with_error "Unable to delete existing user $RESTORE_USER"
    fi

    echo_and_log "Creating user: $RESTORE_USER/$RESTORE_MN IA=${RESTORE_IAM} AUTH ID=${RESTORE_ID} PASS=${RESTORE_PASS}"
    $SCALIX_BIN/omaddu -x -n "$RESTORE_USER/$RESTORE_MN/IA=${RESTORE_IAM}" $RESTORE_ID -p $RESTORE_PASS
    if [ "$?" != "0" ]
    then
        exit_with_error "Failed to create user $RESTORE_USER/$RESTORE_MN"
    else
        echo_and_log "Successfully created user $RESTORE_USER/$RESTORE_MN"
    fi

}


##### main program loop begins

# process command line arguments
# -h            : help
# -m <mailnode> : restore mailnode
# -i <id>       : restore user authid.
# -a <addr>     : restore user internet mail address.
# -p <pass>     : restore user password
# -l <file>     : logfile
# -f <file>     : file to restore
# -u <user>     : username to restore

while getopts hm:u:p:i:a:l:f: opt
do
    case "$opt" in
        h) usage ;;
        m) RESTORE_MN=$OPTARG ;;
        u) RESTORE_USER=$OPTARG ;;
        p) RESTORE_PASS=$OPTARG ;;
        i) RESTORE_ID=$OPTARG ;;
        a) RESTORE_IAM=$OPTARG ;;
        l) LOGFILE=$OPTARG ;;
        f) RESTORE_FILE=$OPTARG ;;
        \?) usage ;;
    esac
done

# validate that all required options are set
for x in "$LOGFILE" "$RESTORE_MN" "$RESTORE_ID" "$RESTORE_PASS" "$RESTORE_FILE" "$RESTORE_USER" "$RESTORE_IAM"
do
    if [ -z "$x" ]
    then
        echo "A required parameter is missing. Please check your command arguments"
        usage 1
    fi
done

# initialize the logfie
>$LOGFILE

# check to make cetain the restore file exists
[ -f $RESTORE_FILE ] || exit_with_error "The specified restore file $RESTORE_FILE doesn't exist"

# a 'ghetto' email address validator for the internet address
echo $RESTORE_IAM|grep -E '\w+\@\w+\.???' >/dev/null
[ "$?" != "0" ] && exit_with_error "Invalid internet address $RESTORE_IAM , does not appear to be a valid email address"


# check for the scalix bin directory
if [ -d $SCALIX_BIN ]
then
    [ -f $SCALIX_BIN/omcpinu ] || exit_with_error "Unable to locate scalix binaries in $SCALIX_BIN"
else
    exit_with_error "Scalix binary directory $SCALIX_BIN doesn't exist."
fi

# call the validate_mailnode function to verify and/or create the restore mailnode
validate_mailnode


# call the validate_user function to destroy (if necessary) and create the restore user
validate_user


# load the mail data with omcpinu
echo_and_log "Beginning restore of mail from $RESTORE_FILE to $RESTORE_USER/$RESTORE_MN"
$SCALIX_BIN/omcpinu -f $RESTORE_FILE -m $RESTORE_MN >>$LOGFILE
if [ "$?" != "0" ]
then
    exit_with_error "Failed to load mailbox data from $RESTORE_FILE. There may be more information available in the logfile at $LOGFILE"
else
    echo_and_log "Finished loading mailbox data from $RESTORE_FILE"
fi

# exit successful
echo_and_log "All tasks completed successfully."
exit 0

florian
Scalix
Scalix
Posts: 3852
Joined: Fri Dec 24, 2004 8:16 am
Location: Frankfurt, Germany
Contact:

Postby florian » Thu Mar 30, 2006 8:51 pm

Wow, this is an amazing thread!

First, let me say thank you for all the good work and useful information. Thanks go to Jon in particular for posting the results of his scripting work.

Backup and restore is an important topic. Naturally and realistically. Historically (talk about the old days), Scalix has kinda treated it as "admins know how to do that and the application provides all interfaces to actually implement it". Now, we realize in the meantime that this is not enough.

So let me add a few short remarks to get going here...

1) We actually do recommend what is being implemented here, a two-tiered backup/restore strategy based on export/import for user-level restore and a full backup for more desaster-recovery like scenarios. I've only looked over jon's script quickly, but it seems to do "The Right Thing" (TM).

2) if you want to do the full backup in online mode, you will need some snapshot mechanism to have a consistent, steady image of your store. and yes, omsuspend is to be used DURING the actual snapshot operation. and yes, it blocks the terminal and is supposed to be used in the background. Snapshots can be provided in software by LVM or EVMS and also in hardware by disk arrays that have the capabiility. You will find examples of it's use in the omsuspend man page and also in our sxbackup script that can be found in the admin_resource_kit directory of the tarball.

3) We understand that this is an area of great concern, nevertheless. Therefore, the next release of Scalix, coming sometime late summer (in the northern hemisphere!!!) will feature enhancements to (a) prevent having to go to backup to recover accidentally deleted messages (according to admins using our product, this is the number one reason to go for restore; the message store is highly stable, so there are few crashes) (b) allow for more flexibility for the export/import mode of backup (e.g. think public folders and not requiring "shadow users" anymore) and many more nice features around that. I believe you will all appreciate (or actually *love*) this.

4) I'm one of the few people who think they understand how ISDB, ISS and SUR works. I have to admit that the procedure make some heads and eyes turn even inside the company. We know that we'll need to work on the docs front and we will be putting a mechanism in place next week and make some announcements that will help this effort.

Having said that, let me contribute something I've written up with/for a customer that has the most important steps should anybody want to give this a try anyway. This does not come with a full description of what all this means, but it certainly covers the major steps.

Code: Select all

1. Make sure ISS is running

   omon iss
   omsetsvc -a iss Y

2. Create initial Database "0"

   omscan -aA -l /tmp/scanlog
   omupdtis -I -l /tmp/scanlog

   omscan -t

   Last omscan tool run on 10.10.05 at 09:29:12; duration 21 minute(s).

   omupdtis -r -f 10.10.05 -F 09:29:12


3. Create Backup

   Do Snapshot, record time of snapshot, e.g. 10.10.05 10:30:20
   Do Backup from Snapshot using Backup tool

4. If needed, perform Single User Restore

   a) Create copy of Database "0"

      cd /var/opt/scalix/is

      cp -a 0 1

      Note: replace "1" with any unused number

   b) Apply log records to copy until time of backup run

      omupdtis -d 1 -t 10.10.05 -T 10:30:20

      Note: -d must be the same as number of copy of database from Step a)
            -t and -T must have exact date and time of snapshot

   c) Generate list of needed files

      omprepsur -n "Florian Scalix/nice,easy" -d /tmp/workdir -f /tmp/filelist.txt
        -i 1 -t 10.10.05 -T 10:30:20

      Note: This is user specific, so name is name of user to be restored
            Again, -i must specify database number assigned from step 1
            Again, -t and -T must have exact date and time of snapshot
            -d must be a non-existing or empty directory
            -f must be a non-existing new filename

   d) Retrieve all files listed in /tmp/filelist.txt and restore them under /tmp/workdir

      Note: How to do this is specific to the backup tool used.

   e) Verify all files have been retrieved

      omprepsur -n "Florian Scalix/nice,easy" -d /tmp/workdir -f /tmp/filelist2.txt
        -i 1 -t 10.10.05 -T 10:30:20

      Note: -f has a new filelist filename

      Check in filelist2.txt that all entries are either state O or state P, no longer
      state N or state ?

   f) Create an import file from directory tree

      omdosur -n "Florian Scalix/nice,easy" -d /tmp/workdir -f /tmp/florian.cpinu
        -i 1 -t 10.10.05 -T 10:30:20

   g) /tmp/florian.cpinu will contain a omcpoutu/omcpinu-type file that contains the
      mailbox of the user at the time of the backup. this can be imported into a shadow
      account as for the import part of the omcpinu/omcpoutu cookbook.


This references a cpoutu/cpinu cookbook - that's this one:

Code: Select all

1. Exporting a user

   omcpoutu -n "Florian Scalix/nice,easy" -f mi.cpout       

   in Scalix 9.4, user can be logged on and file can be compressed:

   omcpoutu -n "Florian Scalix/nice,easy" -f - -F | gzip -c -9 >mi.cpout.gz
                                                ^ ^
                                                | - Force export of logged on user
                                                |
                                                --- Write to pipe to that gzip works


2. Importing to a Shadow Account:

   a) Check out Naming attributes

      omshowu -n "Florian Scalix/nice,easy"

      ...
      User Name : Florian Scalix /CN=Florian Scalix
      MailNode : nice,easy
      ...

   b) Create temporary restore mailnode and user

      omaddmn -m restore
      omaddu -n "Florian Scalix/restore/CN=Florian Scalix/IA=xxx@xxx.com" -x -p scalix

      Notes:
      (1) First and Last Name MUST be the same
      (2) Mailnode MUST NOT be the same
      (3) CN and CN-TX MUST be the same
      (4) IA MUST NOT be the same and does not matter if it's a valid address
      (5) -x means to prevent user from being added to addressbook, -p sets temporary password

   c) Import dump file into temporary account

      omcpinu -f mi.cpout -m restore

      Note: DON'T forget to specify temporary mailnode using -m, otherwise data will
      be imported into original mailbox, resulting in duplicate messages !!!!

      In 9.4 and with a compressed file:

      gzip -d <mi.cpout.gz | omcpinu -f - -m restore

3. Create a profile for Shadow Account in Outlook or Thunderbird and copy over needed messages.

4. Delete temporary shadow account and mailnode

   omdelu -n "Florian Scalix/restore"
   omdelmn -m restore



Again, thanks for all the input. Most valuable and appreciated.

Florian.
Florian von Kurnatowski, Die Harder!

axsom1
Posts: 69
Joined: Tue Aug 17, 2004 12:31 pm

Postby axsom1 » Thu Mar 30, 2006 9:08 pm

Thanks for the info Florian.

I really look forward to seeing what you all have up your sleeve in regards to comment 3a above.

Currently this is the most frustrating thing about Scalix for me and I'm glad that you all are continually striving to improve both your products and the lives of admins who support your products!

Cheers!
John (NOT the same Jon that provided these wonderful scripts)

ja30278

Postby ja30278 » Thu Mar 30, 2006 9:26 pm

Florian,

Thanks for getting in on this discussion! I have to say that the admin presence on these forums is amazing; You guys really seem commited to making this a useful place to find help.

I think part of the disconnect is that Exchage, despite its many faults, has enough marketshare that most commercial backup solutions have plugins to make message-level restore brain-dead simple. Scalix, as you said, is more 'unix-y' in it's approach to backup and restore.


I feel like I'm being a little dense when it comes to omsuspend, but I still don't feel like I get it. If I were to use it in a script, and lauched it in the background, how would I know whether the suspend were successful? Also, I'm still not quite clear, even after reading the man page, about the duration of the suspend. If the '-s' value controls the total length of the suspend, e.g. omsuspend -s 120 will suspend for two minutes, then resume, then what is the use-case for the '-r' options? Is it used solely to prematurely end a suspend that's running in the background? If that's true then it's use would truly be limited to LVM snapshots or something similar. Not a big deal, but I am curious.


Thanks for posting the samples; It's nice to know I'm doing the Right Thing (TM). I may try to take another look at ISS at some point. You can never have too many backups

Jon

ja30278

Postby ja30278 » Thu Mar 30, 2006 9:45 pm

OK..last post for the night, I'm missing out on my game time ;)

I thought it might be easer to download these in a tarball rather than copying/pasting from a web page.

http://jonallie.com/files/scalix_backup.tar.gz

florian
Scalix
Scalix
Posts: 3852
Joined: Fri Dec 24, 2004 8:16 am
Location: Frankfurt, Germany
Contact:

Postby florian » Fri Mar 31, 2006 3:03 am

No problem...

omsuspend - revisited...

Yes, the idea is to do something like

omsuspend -s 120 &
sleep 2
sync; sync; sync
<create_snapshot>
omsuspend -r

omsuspend by itself is a very minimal operation, so it will take effect quickly. There is no need to sync too extensively because all critical writes are synchronous anyway.

it has it's maximum length because other processes (including user processes servicing outlook) will block on it - and those hangs because of their timeouts, have a maximum useful time.

on the commercial side of things and GUI support for backup, we obviously can't beat Exchange; nevertheless, the eco-system how we like to call it, is building. For backup, you might want to check out german vendor SEP (www.sep.de, also selling in the US through RICIS who also happen to be one of our distributors) and their product Sesam that have a Scalix agent. V1 of this, which is currently on the market, is still somewhat rough, but we're working together with them (being based out of Germany, I'm actually managing the relatioonship on a technical side) to create a solution that allows for a fully GUI-driven single item (i..e single Mail message) restore.

Hope this helps some more, enjoy,
Florian.
Florian von Kurnatowski, Die Harder!

nader
Posts: 27
Joined: Thu Dec 22, 2005 6:17 pm

Postby nader » Fri Mar 31, 2006 6:45 pm

Hi,

I am new to the whole Linux/Scalix environment. When I ran the script, I keep getting the follwoing errors (and there's a whole bunch of them):

Code: Select all

rsync: chown "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.getUsers.tcl.vDgDnU" failed: Operation not permitted (1)
rsync: rename "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.getUsers.tcl.vDgDnU" -> "scalix/access/apps/admin/getUsers.tcl": Permission denied (13)
rsync: chown "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.home.action.NPVMQl" failed: Operation not permitted (1)
rsync: chown "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/data/000000t" failed: Operation not permitted (1)
rsync: rename "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.home.action.NPVMQl" -> "scalix/access/apps/admin/home.action": Permission denied (13)
rsync: chown "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.home.page.hgLZlN" failed: Operation not permitted (1)
rsync: rename "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.home.page.hgLZlN" -> "scalix/access/apps/admin/home.page": Permission denied (13)
rsync: chown "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/data/000000u" failed: Operation not permitted (1)
rsync: chown "/root/DCD_TREE/DCD_NW5/SCALIX/scalix_backup/scalix/access/apps/admin/.initApp.xXGhTe" failed: Operation not permitted (1)


My own rsync based little scriplet generates similar errors. Are these errors something that can be ignored? What is rsync trying to tell me?

Any help would be greatly appreciated.

Kind Regards,

Nader.

ja30278

Postby ja30278 » Sat Apr 01, 2006 4:15 pm

silly question: are you running the scripts as root?


Return to “Scalix Server”



Who is online

Users browsing this forum: No registered users and 13 guests

cron