Extremely slow backups to tape

[kevinm128]

Running 11.4.5 on RHEL 4. Over the course of a couple of weeks the time needed to back up 140GB of RAID 10 data increased from 11 hours to 22 hours. Regular email performance seems unchanged. Hardware checks out OK and backing an AIX box up to the same tape drive through the same network switch is still just as fast as it always has been. So far Red Hat hasn't come up with anything. Has anyone seen this behavior before?

Thanks,

Kevin

[PrisonMind]

hi,

Please check the SCSI ... like SCSI Speed setting/cables/terminator etc.
also check the drive for hradware compression.
If it is switched on, pls do not use software compress.
Next point is, if you have many small files (like scalix has), it is difficult for the drive to go to streaming mode. The stopping and starting points to this possibility.
Have you tried fbackup in commandline mode ? That might be an alternative.
Is the tapedrive the only device on that that controler ? If you have daisychained any disks or anything else to it, this is another chance for the problem.
Pls keep us updated.

regards

[SidebandSamurai]

What kind of tape backup hardware are you using? What software are you using?

Sideband Samurai

[kevinm128]

We are backing up to an IBM 3580 LTO2 SCSI tape drive on an IBM p630. The backup administration software is Storix. The Scalix box is connected via gigabit Ethernet through a gigabit switch. I've tried backing up Volume Group 00 instead of the snapshot, which is /mnt/sxbackup on Volume Group 01, and the backup of VG00 runs at least twice as fast as the Scalix snapshot. That indicates to me that the problem is not hardware related, or maybe it is, VG01 is on its own RAID 10 array but we haven't noticed any performance problems with Scalix. I'm stumped on this one, I've tried both Red Hat and Storix support and they haven't been able to help.

Kevin

[SidebandSamurai]

Good Hardware!

I know yo have observed the hardware in action, while backing up, is the tape pausing or "Shoe Shining" instead of running from beginning to end in one run? It should not be "Shoe Shining". That would indicate you are not streaming data fast enough to the tape drive an the tape drive is waiting for the buffer to fill up. If allowed to continue "Shoe Shining" it can wear the tape faster than normal, and dirty the heads quicker.

Other things to check. Check your job and make sure "Compression" is not enabled for the job. The tape drive has hardware compression built in and is enabled by default. This hardware compression is far superior than any software compression you can enable. Besides if you compress the data with software, then compress it with hardware you end up with a slightly (by about 10%) larger backup. I know that this feels weird but I have seen strange problems like this when the file system client is compressing the data and sending it out the network port for backup. Because the server is really busy it takes cycles from the compression routine from the file system client and slows the backup down.

I also noticed you are backing up VG01, I hope VG01 is not the actual mail store. That would definitely give you corrupted backups. You should take a snapshot of the mail store and backup the snapshot.

Are you backing up via a separate VLan? In other words for your Scalix server you should have two LAN cards. One for regular traffic that servers all your users. One is then dedicated for backup. Also make sure 1000MB is enabled on the server by doing an ifconfig.

I find it quite odd that backups would all of a sudden double. there has to be an explanation to this. how about other backup jobs? are they taking longer right around that time? maybe some kind of maintenance routine is kicking off with Storix that is slowing the backup down.

i would give my suggestions a try and report back.

Good Luck

Sideband Samurai