Backup/Restore Scripts - updated with restore script

[vlaurenz]

I'm made some changes to dump_users that I think may be helpful to some of you. With this version, there is no need to shutdown/start RCI, or run omsuspend before and after dump_users is called. Each user is kicked off before they are backed up. Also, it allows you to send the backups to a remote host.

As you can see, I have not made the changes to run the omcpoutu processes in the background. Still waiting on word from Florian.

Code: Select all

function dump_users
{
    # Build userfile
    $SCALIX_BIN/omshowu -m $MAILNODE|cut -f1 -d'/' >$USERFILE
    [ "$?" != "0" ] && exit_with_error "Unable to build userfile $USERFILE from mailnode $MAILNODE"

    # Loop over userfile and create backups. Use 'while read' instead of 'for' because of spaces in names
    while read sc_username
    do
        echo "Processing $sc_username ..."

        # Create a version of the username without spaces and other crappy characters
        nospaces=`echo $sc_username|sed -e "s/[ \.;=*'?_!]//g"`
        REMOTE_BACKUP_FILE="$REMOTE_BACKUP_DIR/$MAILNODE/${nospaces}-${DATE}-mail.gz"

        # Lock the user's account so they can't reconnect when you log them off
        $SCALIX_BIN/ommodu "$sc_username" -K || echo_and_log "Error: Unable to LOCK the account of $sc_username"

        # Disconnect them from the system
        kill -9 `$SCALIX_BIN/omstat -u rci -c |grep "$sc_username" |awk '{ print $6 }'`

        # Compress and send the backup mailbox file to its remote home
        $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - | gzip | ssh $REMOTE_BACKUP_HOST "cat > $REMOTE_BACKUP_FILE"
        if [ "$?" != "0" ]
        then
            echo_and_log "Error: Unable to complete backup operation for $sc_username to $REMOTE_BACKUP_HOST:$REMOTE_BACKUP_FILE"
        else
            echo "Success: Backed up to $REMOTE_BACKUP_HOST:$REMOTE_BACKUP_FILE"
        fi

        # Unlock the user's account when we are done with the backup
        $SCALIX_BIN/ommodu "$sc_username" -k || echo_and_log "Error: Unable to UNLOCK the account of $sc_username"
    done < $USERFILE
}


[vlaurenz]

An even "cleaner" version:

Code: Select all

function dump_users
{
    # Build userfile
    $SCALIX_BIN/omshowu -m $MAILNODE|cut -f1 -d'/' >$USERFILE
    [ "$?" != "0" ] && exit_with_error "Unable to build userfile $USERFILE from mailnode $MAILNODE"

    # Loop over userfile and create backups. Use 'while read' instead of 'for' because of spaces in names
    while read sc_username
    do
        echo "Processing $sc_username ..."

        # Create a version of the username without spaces and other crappy characters
        nospaces=`echo $sc_username|sed -e "s/[ \.;=*'?_!]//g"`
        REMOTE_BACKUP_FILE="$REMOTE_BACKUP_DIR/$MAILNODE/${nospaces}-${DATE}-mail.gz"

        # Check if the user is logged in and get the PID of their RCI process
        uPID=`$SCALIX_BIN/omstat -u rci -c |grep "$sc_username" |grep "$MAILNODE" |awk '{ print $6 }'`
        if [ "$uPID" != "" ]
        then
                # If they are connected...
                # Lock the user's account so they can't reconnect when you log them off
                $SCALIX_BIN/ommodu "$sc_username/$MAILNODE" -K || echo_and_log "Error: Unable to LOCK the account of $sc_username"
                # Disconnect them from the system
                kill -9 "$uPID"
        fi

        # Compress and send the backup mailbox file to its remote home
        $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - | gzip | ssh $REMOTE_BACKUP_HOST "cat > $REMOTE_BACKUP_FILE"
        if [ "$?" != "0" ]
        then
            echo_and_log "Error: Unable to complete backup operation for $sc_username to $REMOTE_BACKUP_HOST:$REMOTE_BACKUP_FILE"
        else
            echo "Success: Backed up to $REMOTE_BACKUP_HOST:$REMOTE_BACKUP_FILE"
        fi

        if [ "$uPID" != "" ]
        then
                # If we locked the account before...
                # Unlock the user's account when we are done with the backup
                $SCALIX_BIN/ommodu "$sc_username/$MAILNODE" -k || echo_and_log "Error: Unable to UNLOCK the account of $sc_username
"
        fi
    done < $USERFILE
}

[florian]

Florian, would you recommend running, say, 350 omcpoutu processes in parallel on a production Scalix server? I'm asking, because at the moment, my single user backups using omcpoutu are taking far too long. I'd love to be able to fire them off in parallel if that would speed things up some.

Honest answer is - can't say.

The processes will be fairly independent, however obviously running that many of them in parallel will put a certain load on the machine.

Better would be a script that tracks how many are running and let (n) run a the same time, and whenever one is finished starts the next user out of this "thread pool" (poor man's that is... ).

You can then play around with the (n) parameter and see how your system (and logged-on users) react to the addtional load.

Cheers,
-- F.

P.S. was off for a few days, long weekend.

[btisdall]

Another approach might be:

Code: Select all

while [ mailboxes remaining to backup ]; do
if [ load avg < n ]; then
start omcpoutu process &
sleep a bit #allow things to stabilise
else
sleep a bit #wait to see if a process has finished
fi
done

[vlaurenz]

Bear with my ugly bash scripting, but here's an algorithm I've been working on. It works, and I'm wondering what you all think of it.

Code: Select all

#!/bin/bash

let i=1
let index=1
PROCESS_BLOCK_SIZE=5

for time in 10 15 20 1 11 3 6 2 4 7 12 9 3
do
do
        if [ $i -le $PROCESS_BLOCK_SIZE ]
        then
            echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE"
            sleep $time || echo "ERROR" &
            PIDs[$index]=$!
            let i+=1
            let index=$i
        else
            echo "Process block is full."
            echo "Waiting for first complete process..."
            while [ $i -gt $PROCESS_BLOCK_SIZE ]
            do
                for p in `seq 1 $PROCESS_BLOCK_SIZE`
                do
                    ps ${PIDs[$p]} > /dev/null
                    if [ "$?" != "0" ]
                    then
                            echo "Process number $p of $PROCESS_BLOCK_SIZE has completed."
                            unset PIDs[$p]
                            let index=$p
                            echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE"
                            sleep $time || echo "ERROR" &
                            PIDs[$index]=$!
                            break 2
                    fi
                done
            done
        fi
done

echo "All processes have been added."
echo "Waiting for those still running..."
wait
echo "All done!"


[vlaurenz]

Tested and working beautifully. Although, it definitely puts a toll on the load level.

It should probably get cleaned up though. Anyone?

Code: Select all

function dump_users
{
    #index for processing block
    let i=1
    let index=1

    # Build userfile
    $SCALIX_BIN/omshowu -m $MAILNODE|cut -f1 -d'/' >$USERFILE
    [ "$?" != "0" ] && exit_with_error "Unable to build userfile $USERFILE from mailnode $MAILNODE"

    # Loop over userfile and create backups. Use 'while read' instead of 'for' because of spaces in names
    while read sc_username
    do
        # Create a version of the username without spaces and other crappy characters
        nospaces=`echo $sc_username|sed -e "s/[ \.;=*'?_!]//g"`
        REMOTE_BACKUP_FILE="$REMOTE_BACKUP_DIR/$MAILNODE/${nospaces}-${DATE}-mail.gz"

        # Check if the user is logged in and get the PID(s) of their RCI process(es)
        $SCALIX_BIN/omstat -u rci -c |grep "$sc_username" |grep "$MAILNODE" > $CONNECTIONS
        while read connection
        do
            uPID=`echo $connection |awk '{ print $6 }'`
            if [ "$uPID" != "" ]
            then
                # If they are connected...
                # Lock the user's account so they can't reconnect when you log them off
                $SCALIX_BIN/ommodu "$sc_username/$MAILNODE" -K || echo_and_log "Error: Unable to LOCK the account of $sc_usernam
e"
                # Disconnect them from the system
                kill -9 "$uPID"
            fi
        done < $CONNECTIONS

        if [ $i -le $PROCESS_BLOCK_SIZE ]
        then
            echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE -- User: $sc_username"
            ## BACKGROUND PROCESS
            $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - | gzip | ssh $REMOTE_BACKUP_HOST "cat > $REMOTE_BACKUP_FILE" |
| echo_and_log "Error: Unable to complete backup operation for $sc_username" &
            PIDs[$index]=$!
            uName[$index]="$sc_username"
            let i+=1
            let index=$i
        else
            echo "Process block is full."
            echo "Waiting for first complete process..."
            while [ $i -gt $PROCESS_BLOCK_SIZE ]
            do
                for p in `seq 1 $PROCESS_BLOCK_SIZE`
                do
                        ps ${PIDs[$p]} > /dev/null
                        if [ "$?" != "0" ]
                        then
                                echo "Process number $p of $PROCESS_BLOCK_SIZE has completed. -- User: ${uName[$p]}"
                                unset PIDs[$p]
                                let index=$p
                                # We might have locked the account before...
                                # Unlock the user's account when we are done with the backup
                                $SCALIX_BIN/ommodu "${uName[$index]}/$MAILNODE" -k || echo_and_log "Error: Unable to UNLOCK the
account of ${uName[$index]}"
                                echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE -- User: $sc_username"
                                ## BACKGROUND PROCESS
                                $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - | gzip | ssh $REMOTE_BACKUP_HOST "cat > $R
EMOTE_BACKUP_FILE" || echo_and_log "Error: Unable to complete backup operation for $sc_username" &
                                PIDs[$index]=$!
                                uName[$index]="$sc_username"
                                break 2
                        fi
                done
            done
        fi
    done < $USERFILE

echo "All processes have been added."
echo "Waiting for those still running..."
wait

for name in ${uName[*]}
do
        # Unlock the remaining accounts
        echo "Unlocking $name/$MAILNODE"
        $SCALIX_BIN/ommodu "$name/$MAILNODE" -k || echo_and_log "Error: Unable to UNLOCK the account of $name"
done
echo "All done!"
}

[btisdall]

Will pore over it when I get chance, just wanted to give you some appreciation right now for all the work you've put in - I'll defintely give this a spin.

Best,

Ben.

[florian]

this is an extremely long and useful thread now - anybody willing to capture all experiences and move this over into a wiki-based howto at www.scalix.com/wiki?



Florian.

[vlaurenz]

I don't know if you all are still interested, but I've realized that you don't need to log users off to omcpoutu. I've been going under that assumption since I first read this thread. The '-F' flag tells omcpoutu "to force the store to continue even if the user is currently logged on".

I've been told by support that this will not cause any corruption of data. So...

Code: Select all

function dump_users
{
    #index for processing block
    let i=1
    let index=1

    # Build userfile
    $SCALIX_BIN/omshowu -m $MAILNODE|cut -f1 -d'/' >$USERFILE
    [ "$?" != "0" ] && exit_with_error "Unable to build userfile $USERFILE from mailnode $MAILNODE"

    # Loop over userfile and create backups. Use 'while read' instead of 'for' because of spaces in names
    while read sc_username
    do
        # Create a version of the username without spaces and other crappy characters
        nospaces=`echo $sc_username|sed -e "s/[ \.;=*'?_!]//g"`
        REMOTE_BACKUP_FILE="$REMOTE_BACKUP_DIR/$MAILNODE/${nospaces}-${DATE}-mail.gz"

        if [ $i -le $PROCESS_BLOCK_SIZE ]
        then
            echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE -- User: $sc_username"
            ## BACKGROUND PROCESS
            $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - -F | gzip | ssh $REMOTE_BACKUP_HOST "cat > $REMOTE_BACKUP_FILE" |
| echo_and_log "Error: Unable to complete backup operation for $sc_username" &
            PIDs[$index]=$!
            let i+=1
            let index=$i
        else
            echo "Process block is full."
            echo "Waiting for first complete process..."
            while [ $i -gt $PROCESS_BLOCK_SIZE ]
            do
                for p in `seq 1 $PROCESS_BLOCK_SIZE`
                do
                        ps ${PIDs[$p]} > /dev/null
                        if [ "$?" != "0" ]
                        then
                                echo "Process number $p of $PROCESS_BLOCK_SIZE has completed. -- User: ${uName[$p]}"
                                unset PIDs[$p]
                                let index=$p
                                echo "Adding Process: Number $index of $PROCESS_BLOCK_SIZE -- User: $sc_username"
                                ## BACKGROUND PROCESS
                                $SCALIX_BIN/omcpoutu -n "$sc_username/$MAILNODE" -f - -F | gzip | ssh $REMOTE_BACKUP_HOST "cat > $R
EMOTE_BACKUP_FILE" || echo_and_log "Error: Unable to complete backup operation for $sc_username" &
                                PIDs[$index]=$!
                                break 2
                        fi
                done
            done
        fi
    done < $USERFILE

echo "All processes have been added."
echo "Waiting for those still running..."
wait
echo "All done!"
}

[kluss0]

Hi All,

Thanks to everyone that has put effort into this thread. This is really good stuff. I have been doing backups in a similar way for a while, and apparently it has been the root cause of some of my problems.

Several months back, I started to get corrupted mailboxes on a regular basis. The logs would show an error of "Content record has not been upgraded to the current container format" (http://www.scalix.com/community/viewtop ... highlight=) . I have worked with support quite regularly on the issue, and until recently, no one has been able to figure out what the cause of the corruption was. A few days ago, I was told that Scalix Engineering believes that my backups may be introducing the problem.

From an e-mail from support:

Right now, the way you're backing up your server is to stop rci, imap, smtp and maybe pop. Engineering believes this may actually introduce a problem in that when the last service that is talking to a mailbox is being shutdown, it's supposed to clean up all of the pending deletes. However, if that process aborts or is killed prematurely, it will cause the item hasn't been upgraded error. The reason you may be seeing it more than other sites is that most sites use LVM and perform snapshots rather than stopping those services every night.

The script that I use is:

Code: Select all

#!/bin/bash

#Shut down services that would touch a mailbox during the backup process

 /opt/scalix/bin/omoff -w -d 0 rci imap smtpd

#Create complete backups using rsync

rsync --delete -a /var/opt/scalix/ /opt/backups/full/var/opt/scalix
rsync --delete -a /opt/scalix/ /opt/backups/full/opt/scalix

# Delete the oldest backup, and rotate the directories, then create a new
# directory for the current backup

rm -rf /opt/backups/7old
mv /opt/backups/6old /opt/backups/7old
mv /opt/backups/5old /opt/backups/6old
mv /opt/backups/4old /opt/backups/5old
mv /opt/backups/3old /opt/backups/4old
mv /opt/backups/2old /opt/backups/3old
mv /opt/backups/1old /opt/backups/2old
mv /opt/backups/current /opt/backups/1old

mkdir /opt/backups/current

# Create a user list of all users on the mailnode and dump it into a file

/opt/scalix/bin/omshowu -m postal,aylus > /opt/backups/current/userlist

# Parse the user file to create an omcpoutu command for each user.
 
sed -e 's/\ \/postal\,aylus\/CN=/\"\ -f\ \ - >"\/opt\/backups\/current\//g' -e 's/^/\/opt\/scalix\/bin\/omcpoutu\ -n\ \"/g' -e 's/$/\.out\"\ /g' /opt/backups/current/userlist > /opt/backups/current/mailbackupscript

# Execute the series of omcpoutu commands and log issues.
/opt/backups/current/mailbackupscript > /opt/backups/current/logfile.txt

# Move the audit log into the backup directory (not really needed anymore)

mv /var/opt/scalix/logs/audit /opt/backups/current/

# Re-start the services
/opt/scalix/bin/omon -w rci imap smtpd



It's a simple script, and not nearly as eligent as what has been posted on this thread. I am going to give this script a try, and see if it handles things better then mine.

Has anyone else experienced any similar problems that could be attributed to the non-LVM backups? I specifically chose to not use LVM because of past bad experiences with it, so I am extremely reluctant to implement it at this point. However, if doing backups this way is what is causing the corruptions to occur, I may have no choice.

On a related note, once Scalix 11 comes out, we are all going to have to re-vamp our scripts to use the new archive commands, as I believe omcpoutu/omcpinu will be going away.

Thanks,
Kenny

[Derek]

Does anyone know what daemons and services are required for omcpoutu?

I'm doing the straight-foward rsync to a remote box method. I'm also doing single user backups with omcpoutu, but I'm executing that on the remote machine. So, rather than completely starting up Scalix, I'd rather just start up what's necessary for omcpoutu.

[florian]

On a related note, once Scalix 11 comes out, we are all going to have to re-vamp our scripts to use the new archive commands, as I believe omcpoutu/omcpinu will be going away.

Friends... what do you think of us? Obviously, the old commands _WILL_ remain available for downward compatibility reasons.

However, you will certainly _WANT_ to revamp the scripts to make use of the new features included in the updated version, such as...
- import/export of public folder data
- import/export of selected folders - or items
- import "into" different folder of same mailbox (no shadows needed for single-item restore)
- import "into" same or different folder of any mailbox (name does not have to be the same, etc.)
- import/export of some user configuration data like delegation settings, rules, etc.
... and some more!

Most of this, btw., is already working in the Scalix 11 Community Preview currently available, so if you want to try or modify the scripts to be ready when Scalix 11 hits production quality.... give it a go!

Cheers,
Florian.

[florian]

Derek,

you'll need the base daemons - no actual services are needed. The easiest way to get there when your Scalix is down is to run

Code: Select all

omrc -n

- this will only start the required processes.

Cheers,
Florian.

[vlaurenz]

Kenny and Derek,

I suggest you use the -F flag with omcpoutu and avoid keeping any daemons or services off while the omcpoutu processes complete. It's working fine for me this way.

I am currently doing LVM snapshots for full backups and my per-user-backups run with a version of ombackup script in this thread.

[kluss0]

Florian,

I'm glad to hear that backward compatability will be maintained. I am going to start working with the community edition to prepair for the enterprise upgrade in November. In the mean time, I need to get my backups and the corruption problem under control.

A note about that last posting of the dump_users functiuon. You need to define PROCESS_BLOCK_SIZE for it to work.

Thanks,
Kenny