Backup/restore of large mailstore (over 100gb)

[adhodgson]

Hi all,

We have a mailstore which is just going over the 100gb mark, and we are having issues with our current backup process. We currently take a snapshot of the LVM with /var/opt/scalix, after suspending Scalix for a few minutes, then tar/gz the lot to another LVM on the machine, for backing up onto tape. Thia has worked fairly well so far, but is now taking just over 7-8 hours to complete, and sometimes the snapshot goes seriously wrong, meaning I have to reboot the server, and juggle round with LVM devices in /dev/mapper to get it back up again.

I am just wondering if anyone else is doing similar, or anything else with similar volumes of data?

Thanks.
Andrew.

[jaime.pinto]

Not too long ago we were using RHE4 and the snapshot there definitely didn't work very well. Recently we upgraded to RHE5, and we're getting inconclusive results: with smaller raids (and hardware raid controllers) snapshot seems to work and still keep the system up even if we run out of space on the snapshot partition (80% use + 20% snapshot). If the raid is larger (2TB>)it locks up at times.

The preliminary conclusion is that we should not activate more than 50% of the total volume for "normal" use, and assign the other half to snapshot. This way we are guarantied never to run out of space on the snapshot, regardless of how long it is on.That is probably OK for 300-500GB storage, since HDs are so cheap these days (the route we're taking now)

In your case you're doing the backup in 2 stages, plus the tar...
What about this setup?

1) primary storage 500GB raid1 (minimum 2 disks), 50% use + 50% snapshot.
That would give you about 220GB of usable storage, which is more than double of your current use.

2) turn snapshot on and backup the primary storage directly to tape (no tar)

3) If you want to work with a secondary storage, have just a 3rd HD of 250GB.
* Turn snapshot on
* use rsync (not tar) to replicate the primary storage on this disk (much faster)
* Turn snapshot off
* backup the secondary storage to tape

On a side note, I'm particularly interested on some feedback from other people using snapshots on redhat in general.

[adhodgson]

Hi.

We have 200GB for Snapshot storage. I have 1TB, 400GB Scalix store, 400GB backup space (in case I decided to use rsync to the second store) and 200GB left for snapshot space. Actually the snapshot has been fairly reliable on this RHEL4 system, it has only gone wrong like this once, but I have just seen some logs today where it couldn't unmount the snapshot for some reason, so I suspect it may have done this again, and so will have to reboot early Monday morning, but this is only the second time in just over 6 months.

We use BackupExec and the BE agent on the server to back up the Tar file onto tape. I am wondering whether there is a way I could script this on BES so it goes direct to the BES server/tape and not on the Scalix system? My major concern is the time this is taking - over 6 hours to complete the tar operation - is this typical? Adding another tape into the mix is possibly going to cause more issues, since we have to find some way of putting another tape drive in the rack, as the server is full of disks for the RAID10 setup. We do have a tape library, but this is mapped to the BES server through a Qlogic switch using a zone.

Andrew.

[kanderson]

Any chance you could use the snapshot to rsync to a different device (or to the other 400 Gigs)?

Then you could end release the LVM lock, and do the backup while the system is running normally.

If you rsync to a different box completely (an old cheap desktop with a big SATA drive), you could back it up during the day with no impact on the production system.

Kev.

[adhodgson]

Hi,

This is possible. One issue with Rsync and BackupExec is that I am not sure how the permissions would fair, and also, restoring one tar file from the tape is easier than restoring a whole Scalix mailstore with each individual file. However, I want to try and get the snapshot off the volume quicker than it is being released now. . . I spoke to the backup guys yesterday, and found that they were doing a full backup on the system on some nights, which was slowing down the backup I was doing. I will be speaking to them again today to see if they have any ideas.

Andrew.

[djtremors]

Hey guys,

Hope this is a good idea for my first post.

Why not instead of using rsync and/or LVM to snapshot a live system, why not just get another server with the same capacity and fast enough to mirror and use DRBD to mirror the live server in realtime and then disconnect when you are ready to backup and set the device to primary mode, backup and then set it to secondary and reconnect to the primary ?
There is no need to handle the primary and the secondary could even act as a backup DR of the live one?

It's not hard to set up but very hard once you've had it running.

[kanderson]

Two reasons.

1) There is a very real possibility that the server would not start for you. This could be due to corrupt data in the message store, but more likely, it would be due to the fact that the rsync will copy over lock files which show services running. The second is quickly and easily fixed with omreset, and the first would likely be fixed with omscan. But the timeline involved could still be significant, especially if especially if you have a large mailstore.

2) It's still going to be out of date somewhat. Many servers I look after take hours for an rsync to run, and that means that the best I can do is have a server that's a few hours out of date.

Is it better than nothing? Sure. Is it a good DR solution for a company that actually needs one? Not really.

Note too that if you rsync over a WAN, it'll take a VERY long time to copy data over from a semi-large server. Also note that during the sync, there will be a noticable performance hit.


Kev.

[djtremors]

@kanderson : was this related to my post? I wasn't talking about rsync but drbd.

[Valerion]

I am not a DRDB expert, but wouldn't it affect access times? If all data is synched immediately on access (no other way to keep it 100% in sync), I can see disk IO worsening. Scalix is VERY disk-IO intensive, so this may translate into a slower server.

Personally I would go for a FC SAN solution, though it is costly and not everyone can do it.

[djtremors]

I use DRBD for my HA mirror and i saw it sync 40MB/s over gigabit crossover cable. Go fiber if you want.

DRBD has 3 modes (A,B and C). C is the most guarenteed sync as it waits for the sync to be written to the other nodes disk. B waits for it to be written to the cache of the other node whereas A waits for it to be in the transmit buffer ( or something, check site for accuracey).

Don't forget it's mirroring changes so only writes would be more effected but reads would be as fast as your disks can go. You are now talking about using FC SAN but here we're trying to keep costs down but have an easy way to mirror, so having a next door neighbour mirroring I find it to be perfect for me on what I'm doing. Possibly a huge enterprise would do it another way as huge amount of access by 200+ people probably would get effected.

[PeterR]

HI djtremors !

DRBD is a nice idea, it could work if underlaying hardware + interconnection between DRBD nodes is realy fast. I'm concerned about I/O Issues, since, like Kanderson and Valerion already mention, Scalix is pretty damm I/O intensive.

DBRD adds another abstraction layer between the hardware ( HDD / RAID ) and the computer filesystem ( ext3 / xfs / var/opt/scalix) . Each layer between hardware and scalix database consumes some CPU + IO -time, not much but a littel.
The main bottleneck is imho the conenction between two DRBD Servers. Like ISCSI you use ethernet or fiber, which is definetly slower and hase more overhead than direct attached storage ( like HDD in the same computer where scalix runs. )
Not in bandwidth but in latency ...

Or in short:

If everthing runs fast with no IO-wait on direct attached storage, DBRB will slow down but it also will work and may be worth a try .

I use DRBD for my HA mirror and i saw it sync 40MB/s over gigabit crossover cable. Go fiber if you want.
.

Hmm I have here upto 65MB/s but using scalix leads to a load between 2 and 10 with iowait >90 %

7 Users , 30 GB total Mail Storage CE 11.2 Version on SATA Raid1 ext3 System runnung CentOS 5.1, Thunderbird & Kmail no Outlook MUA.

Greeting PeteR

[seany]

This sounds like an appropriate place to put my backup question

We're about to install our hardware to run Scalix. The storage is a FC box with 16x500gb SATA drives. Our initial mail store (imported from Notes) is likely to be around 700gb. The storage is overspecced really, but this is to allow possible expansion later.

Our backup requirements are twofold. Firstly, an offsite backup of the daily snapshot, and secondly the ability to retrieve deleted emails for a two month period.

The current idea is to:

> Partition the FC array in two - one partition for the live environment, and one for backup staging.

> Copy the live data (LVM snapshot) to the staging area overnight. We will then use rsync to update a remote system (in our other datacentre). This system will run Scalix and we will use it to restore in the event of a disaster.

To retrieve deleted emails, the idea is to just make the users recovery bin retain data for two months.

My question is: does this sound feasible, given the amount of data? We have a fast link to the other datacentre (10gbit), will this be a problem with rsync? And better ideas?

Thanks,

Sean

[mikevl]

Hi

This sounds feesable in theory. I would consider two things

Will the SATA be SATAII?
If you have a mail store that large it would indicate you have a lot of users > 500. Would those users all be using Outlook? SATA may not provide the performance you need. If you are intent on using SATA there are a few things that you can do to speed up the performance of your system. These may include

Provide seperate spindle(s) for the /var/opt/scalix/xx/s/temp folder may help
I think there is another directory you may need to put on a seperate spindle but I can't remember waht it is

Mike

[seany]

Thanks Mike.

Yes these are SATA II drives, 16 of them in a FC array (an Eonstor). There are only 310 users initially, but the idea is to have space to grow later.

[mikevl]

Hi

The Eonstor is a great machine..

You will need to do all you can to maximise the speed of the array as suggested above. You are on the limit of SATA performance. I was looking for an earlier post where a company had 700 users on SATA and that was NOT working for them. Very high I/O wait states.

I guess you would have an internal disk for the OS?

I would be interested in following how your installation goes?

Mike