Huge backup for few mails

[mouseman]

I added /var/opt/scalix/sx/... on the scalix server to our daily differential backup.
This backup is eating up the tape space !
Even for few mails on the weekend the backup grows up to 100-150 MByte
When I take a look at the log I see lots of files in .../s/data and below written to tape.
Can anyone tell me what is going here - is there some strange kind of nightly re-indexing which is confusing the differential calculation ?
Should I change my backup strategy or directory?
Thanks for any hints !
Thomas

[kanderson]

You are probably getting that much new email.

Assuming you have 25 users, that would work out to about 5 messages per user, each with a 1 meg attachment. That's not at all unreasonable. Alternately, it would work out to 50 messages per person each with a 100K message size. Again, not unreasonable by any stretch.

Kev.

[mouseman]

nice idea - I should have been more precise. Sorry, my fault.
Our old mailserver (cyrus imap) is getting the same mail data.
The differential backup has a size of 15 MB, scalix backup is
180 MB large.

[kanderson]

You COULD just grab /var/opt/scalix/??/s, as this would skip PostgreSQL's cache, (Which can easily be rebuild, so it's expendable). But you'd still have the mime headers in the mailstore. I'm told these can result in a 30% increase in disk use, but you're looking at much more than that. What are the results of omshowlvl and omshowaud?

Not sure there's a good way to prevent getting everything like that.

You could use sxmboxexp to dump each user and all the Public Folders, and then just archive the backups. That's probably easier to restore a single user from, but it would be harder to restore the entire mailstore from it. It would also result in duplication, as each user would have a copy of each message.

Kev.

[florian]

I just checked the data on two of our internal production systems. They are not huge, but we are heavy users of email, so it should provide some indication.

System 1: 1573649 files in data directory, 44GB Message Store
Files modified in the last 24h: 6943 (0,44%)
Files modified in the last 3d: 26434 (1,67%)
Files modified in the last 1w: 46288 (2,94%)
Files modified in the last 30d:153393 (9,74%)

System 2: 808829 files in data directory,24GB Message Store
Files modified in the last 24h: 4522 (0,55%)
Files modified in the last 3d: 10998 (1,35%)
Files modified in the last 1w: 21365 (2,64%)
Files modified in the last 30d: 121213 (14,98%)

These numbers make good sense to me. Yours seem to be a bit on the high side.

One important question is - has this system recently been upgraded from Scalix 10 to Scalix 11? In this case, what you see might be normal as all data files will be rewritten at least once to upgrade some internal data structures. This happens on first access (read or write) of the respective file.

Also, are you sure that you exclude the s/temp and s/tmp directories from your backups?

Florian.

[dkelly]

The reason why the incremental is so big is that you are backing up a directory called ~/dits which is a table that maps an index value to a pointer into the message store. This is pretty intrinsic to the operation of Scalix and is used by all our clients. As it's updated for every message that's added/deleted in the message store, the file will be present on every incremental.

It also means that you cannot restore that directory for any other backup other than the most recent.

Cheers

Dave